[Workaround for] OOM / memory "502 Bad Gateway" crashes w/ large videos' playback or download-to-client [SEE ALSO #37 & #53] by deldesir · Pull Request #51 · iiab/calibre-web

deldesir · 2023-11-23T06:17:49Z

Calibre-Web tends to load the entire video file to RAM before playing, making it challenging when only large videos are available due to oom-killer restrictions on excessive memory usage. This PR will fix this.

Use native HTML5 video player

Play video directly using native HTML5 player

Co-authored-by: A Holt <holta@users.noreply.github.com>

holta · 2023-11-28T17:46:16Z

Original problem:

Some videos download as overweight (giant) files that (1) are not appropriate for schools, and (2) crash Calibre-Web with "502 Bad Gateway" #37

Side Question: Does this allow everyone access to all books, regardless whether they have a Calibre-Web account/access/authorization?

(As an intentional or unintentional side effect?)

Question copied to its own ticket:

Give everyone access to all books, intentionally ignoring Calibre-Web ACL accounts/authorization? #53

deldesir · 2023-11-29T00:23:51Z

Downloads is now optimized with this PR. The previously 4GB video is redownloaded and it's now 210.8MB. Tested on Ubuntu 22.04.3.

Should work with most YouTube videos assuming a video+audio stream, with 720p resolution and vp9 codec exists for the URL.

deldesir · 2023-12-04T16:14:15Z

Smoke-tested together with #57 on Ubuntu 24.04 (Noble Numbat)

Improve video path and align with nginx

deldesir · 2023-12-06T00:33:54Z

+                    if file.endswith(".webm") or file.endswith(".mp4"):
+                        video_file = os.path.join(video_path, file)
+                        # align with nginx path
+                        video_file = video_file.replace(config.config_calibre_dir, "/books-direct")


Aligned with iiab/iiab#3680

holta · 2023-12-06T00:45:53Z

@deldesir this looks great!

Is this well-tested with the latest PR Restrict NGINX (public?) access to "large" files only (e.g. .mp3, .mp4, .webm) in /library/calibre-web — for PR #51 OOM workaround #57 ?
Can (A) long audiobooks and (B) long audiocasts and (C) long musical performances increasingly be Guaranteed Playable & Guaranteed Downloadable on 512 MB Rapberry Pi Zero W's and similar — without OOM ("502 Bad Gateway" out-of-memory failures) — thanks to a follow-up PR building on this one?

holta · 2023-12-06T07:10:10Z

+                for file in os.listdir(video_path):
+                    if file.endswith(".webm") or file.endswith(".mp4"):
+                        video_file = os.path.join(video_path, file)
+                        # align with nginx path


Suggested change

# align with nginx path

# align with nginx path (e.g. "books-direct") specified on Lines ~24 and ~29 of:

# https://github.com/iiab/calibre-web/blob/master/scripts/calibre-web-nginx.conf

@deldesir can you take the above suggestion seriously — or if not explain why?

In lieu of soft-coding (or posting constants on top of files, as is traditional to make software more maintainable...) please at least consider clear documentation explaining how to maintain de facto global constants.

I get it. Definitely useful adding context in comments.

Co-authored-by: A Holt <holta@users.noreply.github.com>

holta · 2023-12-09T21:08:51Z

QUESTION: I'd love to understand the Root Cause better (#37 and corollary #53) — before we merge this PR and its companion PR (#57) — which together risk papering over a deeper problem?

Hopefully @deldesir can explain better the Root Cause in coming days, or at least what's understood about it so far!

BACKGROUND:

A "crutch" (temporary workaround) is sometimes necessary of course, to heal any injury!
Conversely some other "crutches" create unhealthy dependencies and long-term maintenance burdens we should avoid — e.g. if the "root cause" injury can be somewhat better understood by everybody, or at least circumscribed and documented — on the road towards fixing that properly in 2024!?

🙏 Either way let's articulate & clarify the likeliest road forward, as precisely as we can anyway. 🙏

deldesir added 3 commits November 23, 2023 01:02

Update omg.yml

7708d3f

Create watchvideo.html

d2fe87d

Use native HTML5 video player

Update web.py

6f1363e

Play video directly using native HTML5 player

deldesir added bug Something isn't working enhancement New feature or request labels Nov 23, 2023

deldesir requested a review from holta November 23, 2023 06:17

deldesir self-assigned this Nov 23, 2023

This was linked to issues Nov 23, 2023

Some videos download as overweight (giant) files that (1) are not appropriate for schools, and (2) crash Calibre-Web with "502 Bad Gateway" #37

Open

Serve videos [using Video.js ?] #46

Open

This was referenced Nov 23, 2023

Some videos download as overweight (giant) files that (1) are not appropriate for schools, and (2) crash Calibre-Web with "502 Bad Gateway" #37

Open

Add Video.js #49

Closed

holta mentioned this pull request Nov 23, 2023

Create calibre-web-nginx.conf #50

Merged

holta reviewed Nov 23, 2023

View reviewed changes

Comment thread scripts/omg.yml Outdated

Update scripts/omg.yml

bd235e6

Co-authored-by: A Holt <holta@users.noreply.github.com>

holta reviewed Nov 23, 2023

View reviewed changes

Comment thread scripts/omg.yml Outdated

deldesir added 2 commits November 24, 2023 06:12

Update omg.yml

07ba82a

Symlink library

a0b909c

holta reviewed Nov 27, 2023

View reviewed changes

Comment thread scripts/omg.yml Outdated

deldesir mentioned this pull request Nov 28, 2023

Give everyone access to all books, intentionally ignoring Calibre-Web ACL accounts/authorization? #53

Open

Update scripts/omg.yml

8a3a90a

Co-authored-by: A Holt <holta@users.noreply.github.com>

holta reviewed Nov 28, 2023

View reviewed changes

Comment thread scripts/omg.yml

Remove "&"

fa70213

holta mentioned this pull request Nov 30, 2023

Time to put scripts/omg.yml on a diet! #55

Merged

holta changed the title ~~Fix OOM issue - affecting large videos playback~~ Fix OOM issue - affecting large videos' playback Nov 30, 2023

This was referenced Dec 2, 2023

Discarded Updates (PR polluted) #56

Closed

Restrict NGINX (public?) access to "large" files only (e.g. .mp3, .mp4, .webm) in /library/calibre-web — for PR #51 OOM workaround #57

Closed

deldesir added 2 commits December 4, 2023 09:36

Delete scripts/omg.yml

52cf159

Create omg.yml

6b7cbd4

Merge branch 'iiab:master' into deldesir-oom-fix

3342ab3

holta mentioned this pull request Dec 4, 2023

Introduce var 'calibreweb_venv_wipe: False' for TDD (test-driven development) of Calibre-Web iiab/iiab#3678

Merged

holta changed the title ~~Fix OOM issue - affecting large videos' playback~~ [Workaround for] OOM / memory "502 Bad Gateway" crashes w/ large videos' playback, or download to client [SEE ALSO #37 & #53] Dec 4, 2023

holta changed the title ~~[Workaround for] OOM / memory "502 Bad Gateway" crashes w/ large videos' playback, or download to client [SEE ALSO #37 & #53]~~ [Workaround for] OOM / memory "502 Bad Gateway" crashes w/ large videos' playback or download-to-client [SEE ALSO #37 & #53] Dec 4, 2023

holta mentioned this pull request Dec 5, 2023

enable-or-disable.yml: Don't create symlink /library/www/html/calibre-web -> /library/calibre-web (NGINX can go directly if/as nec!) iiab/iiab#3680

Merged

deldesir added 2 commits December 5, 2023 19:27

Merge branch 'iiab:master' into deldesir-oom-fix

d48d1ae

Update web.py

3cd8a97

Improve video path and align with nginx

deldesir commented Dec 6, 2023

View reviewed changes

holta reviewed Dec 6, 2023

View reviewed changes

Comment thread cps/web.py Outdated

holta reviewed Dec 6, 2023

View reviewed changes

deldesir and others added 2 commits December 6, 2023 05:42

Update cps/web.py

1ac69ad

Co-authored-by: A Holt <holta@users.noreply.github.com>

Update cps/web.py

013ee9d

Co-authored-by: A Holt <holta@users.noreply.github.com>

EMG70 mentioned this pull request Dec 8, 2023

scripts/lb-wrapper: Try "--format-sort='tbr~1000'" [i.e. target 1000 kbit/s codecs] #62

Merged

holta mentioned this pull request Dec 8, 2023

Create scripts/lb-wrapper.greedy à la 'yt-dlp dl --format-sort size' [and refactor /usr/local/bin/lb-wrapper] #64

Merged

holta mentioned this pull request Dec 10, 2023

Install /usr/local/bin/lb-wrapper.greedy for Calibre-Web OOM (out of memory) testing iiab/iiab#3681

Merged

Merge branch 'iiab:master' into deldesir-oom-fix

fdea24c

holta mentioned this pull request Dec 24, 2023

Clicking "Watch in Browser - MP4" works w/ 8GB RAM but crashes calibre-web.service w/ 2GB RAM / memory (MP4 is 1.0GB; users see "502 Bad Gateway" due to OOM crash) #79

Open

Merge branch 'iiab:master' into deldesir-oom-fix

bf1996b

deldesir closed this Jul 1, 2024

deldesir deleted the deldesir-oom-fix branch July 1, 2024 21:42

deldesir restored the deldesir-oom-fix branch July 1, 2024 22:03

deldesir reopened this Jul 1, 2024

Merge branch 'iiab:master' into deldesir-oom-fix

7d29b79

deldesir closed this May 17, 2026

deldesir deleted the deldesir-oom-fix branch May 17, 2026 19:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Workaround for] OOM / memory "502 Bad Gateway" crashes w/ large videos' playback or download-to-client [SEE ALSO #37 & #53]#51

[Workaround for] OOM / memory "502 Bad Gateway" crashes w/ large videos' playback or download-to-client [SEE ALSO #37 & #53]#51
deldesir wants to merge 18 commits into
iiab:masterfrom
deldesir:deldesir-oom-fix

deldesir commented Nov 23, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

holta commented Nov 28, 2023 •

edited

Loading

Uh oh!

Uh oh!

deldesir commented Nov 29, 2023 •

edited

Loading

Uh oh!

deldesir commented Dec 4, 2023 •

edited

Loading

Uh oh!

deldesir Dec 6, 2023

Uh oh!

holta commented Dec 6, 2023 •

edited

Loading

Uh oh!

Uh oh!

holta Dec 6, 2023 •

edited

Loading

Uh oh!

holta Dec 6, 2023

Uh oh!

deldesir Dec 6, 2023

Uh oh!

holta commented Dec 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	# align with nginx path
	# align with nginx path (e.g. "books-direct") specified on Lines ~24 and ~29 of:
	# https://github.com/iiab/calibre-web/blob/master/scripts/calibre-web-nginx.conf

Conversation

deldesir commented Nov 23, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

holta commented Nov 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

deldesir commented Nov 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deldesir commented Dec 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deldesir Dec 6, 2023

Choose a reason for hiding this comment

Uh oh!

holta commented Dec 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

holta Dec 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

holta Dec 6, 2023

Choose a reason for hiding this comment

Uh oh!

deldesir Dec 6, 2023

Choose a reason for hiding this comment

Uh oh!

holta commented Dec 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

holta commented Nov 28, 2023 •

edited

Loading

deldesir commented Nov 29, 2023 •

edited

Loading

deldesir commented Dec 4, 2023 •

edited

Loading

holta commented Dec 6, 2023 •

edited

Loading

holta Dec 6, 2023 •

edited

Loading