Build your own "YouTube/Vimeo[*] Learning Library" using Internet-in-a-Box !

[*] Downloading videos from Vimeo.com and similar sites needs additional testing, Huge Thanks if you can help!!

We aim to help educators download the videos and audio recordings they truly need, from almost 2000 websites.

News

1. Uploading of local videos should be smoother — let us know how it's working for you: #214
2. "Tasks" view errors continue to be much more clearly explained (work in progress!)
3. Tons of incremental improvements from the upstream version of Calibre-Web.
4. Log files are cleaner, thanks to improvements in xklb: #230
5. Upgrading IIAB Calibre-Web is now easy, just run: sudo iiab-update -f
6. A $58 Internet-in-a-Box is now for sale at the Wikipedia Store.🌹

On Deck

These serious issues will be fixed as soon as we can!

• Memory crash with large/long videos: #79
• Two kinds of SQL errors with some playlists: #186
• Upcoming and truly live YouTube video URLs will soon be handled gracefully: PR #212

Install Instructions

https://github.com/iiab/calibre-web/wiki#wrench-installation

What's Changed

• Tighten messaging of formerly live video's HH:MM:SS duration, in "Tasks" view (as it downloads) by @holta in #216
• REVISED sync from upstream (30 commits, 65 files changed) by @holta in #219
• Sync from upstream (29 commits, 66 files changed) by @holta in #218
• Import shutil.move to recover from upstream change (janeczku/calibre-web@bc1c097) to cps/editbooks.py by @deldesir in #221
• Remove [FFmpeg] scene detection filter [to make thumbnail/cover/poster creation more reliable] by @deldesir in #225
• [Update & clarify FFmpeg -vf (video filters) explanation, re: extracting a thumbnail/cover/poster] by @deldesir in #226
• Increase fragment stuck timeout to 60 seconds [was 30s] by @deldesir in #227
• Ensure error and live_status columns are created from the start by @deldesir in #191
• Increase timeout to 120s [fragment_stuck_timeout = 120 # seconds] by @deldesir in #229
• Update & tighten up omg.yml (cloud-init rapid install script) by @holta in #231

Full Changelog: 2024.07.05...2024.07.26

Build your own "YouTube/Vimeo[*] Learning Library" using Internet-in-a-Box !

[*] Downloading videos from Vimeo.com and similar sites needs additional testing, Huge Thanks if you can help!!

We aim to help educators download the videos and audio recordings they truly need, from almost 2000 websites.

News

1. Uploading of local videos is working better. Stay tuned as work progresses to more resiliently handle punctuation (etc!) in filenames: #214
2. "Tasks" view errors are starting to be much more clearly explained (work in progress!)
3. Many small improvements were included from the upstream version of Calibre-Web.

On Deck

These serious issues will be fixed as soon as we can!

• Memory crash with large/long videos: #79
• Two kinds of SQL errors with some playlists: #186
• Upcoming and truly live YouTube video URLs will soon be handled gracefully: PR #212

Install Instructions

https://github.com/iiab/calibre-web/wiki#wrench-installation

What's Changed

• Make image and videos uploads use the same data structure by @deldesir in #207
• Fix video upload failure [e.g. filenames that contain square brackets] by @deldesir in #208
• Fix regression due to ValueError in file uploading mechanism by @deldesir in #210
• Adjust message for formerly "live" videos by @deldesir in #211
• Ensure progress is 100% on task success by @deldesir in #213

Full Changelog: 2024.06.28...2024.07.05

Build your own "YouTube/Vimeo[*] Learning Library" using Internet-in-a-Box !

[*] Downloading videos from Vimeo.com and similar sites needs additional testing, Huge Thanks if you can help!!

We aim to help educators download the videos and audio recordings they truly need, from almost 2000 websites.

News

1. IIAB Calibre-Web now works on Raspberry Pi!

2. YouTube mislabels many video URLs as "live" — when in reality these video-URLs-labeled-as-live can be:

   1. not yet live — video doesn't exist yet 🛑
   2. actually live — complete video doesn't exist yet ⚠️
   3. formerly live — FIXED: these videos are now downloadable thanks to PR chapmanjacobd/library#41 ✅

   So our IIAB Calibre-Web's "Tasks" view should (very soon in coming days!) help educators/parents/librarians sort out what's really happening in all 3 cases: #202

3. Uploading of your own video isn't quite yet working perfectly; please persist and let us know your experience(s) here: #203

On Deck

These serious issues will be fixed as soon as we can!

• Memory crash with large/long videos: #79
• Two kinds of SQL errors with some playlists: #186

Install Instructions

https://github.com/iiab/calibre-web/wiki#wrench-installation

What's Changed

• Update omg.yml [make last modif date more visible, in raw text view] by @avni in #198
• Clear input field right after URL is submitted by @deldesir in #200
• Escape quotes to fix SyntaxError for RPI [Concatenate instead of using f-string, for Raspberry Pi] by @deldesir in #201
• Add --live flag to support downloading of "live" videos by @deldesir in #199

New Contributors

• @avni made their first contribution in #198

Full Changelog: 2024.06.21...2024.06.28

Build your own "YouTube/Vimeo[*] Learning Library" using Internet-in-a-Box !

[*] Downloading videos from Vimeo.com and similar sites needs additional testing, Huge Thanks if you can help!!

We aim to help educators download the videos and audio recordings they truly need, from almost 2000 websites.

News

1. "Tasks" view explanations continue to improve, e.g. when the "Download to IIAB" button (for video downloads) partially or completely fails.
2. A video thumbnail should now be auto-generated, when you upload your own local/community video.
3. Code safety is improving — with a focus on defensive programming — to help us handle the humongous number of yt-dlp failure modes! 😉

Install Instructions

https://github.com/iiab/calibre-web/wiki#wrench-installation

What's Changed

• Return the real shelf title by @deldesir in #192
• Adjust message to warn about problematic videos [e.g. "live" videos, that might not really be live anymore, but are not currently downloading] by @deldesir in #194
• Safer lb-wrapper: Wrap [most] lb parameters (bash vars) in single quotes by @holta in #195
• Prevent stuck videos from carrying over to next download session ["unavailable fragments" ?] by @deldesir in #193
• Generate cover [a.k.a. thumbnail/poster] for video manually uploaded by @deldesir in #196

Full Changelog: 2024.06.14...2024.06.21

Build your own "YouTube/Vimeo[*] Learning Library" using Internet-in-a-Box !

[*] Downloading videos from Vimeo.com and similar sites needs additional testing, Huge Thanks if you can help!!

We aim to help educators download the videos and audio recordings they truly need, from almost 2000 websites.

NEWS: The "Tasks" view is dramatically improved this week, (1) showing useful details of a YouTube playlist / channel (as it's being downloaded to become an offline bookshelf), and (2) making many error messages far more understandable.

WARNING: Uploading of OFFline video is still a bit rough (thumbnail is missing for now!) but at least the video does upload: #107 (comment)

Install Instructions

https://github.com/iiab/calibre-web/wiki#wrench-installation

What's Changed

• Read error from xklb-metadata.db [leveraging yt-dlp STDERR] by @deldesir in #175
• Set outtmpl externally [to avoid missing thumbnails; NOTE: archival-quality book & video titles are preserved in 2 SQLite db's] by @deldesir in #177
• Handle restricted/unavailable videos [and list them in "Tasks" view, when downloading (stale!) playlists] by @deldesir in #179
• Build list of URLs simpler [refactor PR #179 for readability / maintainability] by @deldesir in #180
• Clear input field when download modal is shown by @deldesir in #184
• Prepend the video description with the source URL by @deldesir in #183
• Display shelf title [name & link in "Tasks" view, when downloading a playlist or channel to IIAB] by @deldesir in #182
• uploader.py: Put "Original Internet URL" under description by @holta in #185

Full Changelog: 2024.06.07...2024.06.14

Build your own "YouTube/Vimeo[*] Learning Library" using Internet-in-a-Box !

[*] Downloading videos from Vimeo.com and similar sites needs additional testing, Huge Thanks if you can help!!

We aim to help educators download the videos and audio recordings they truly need, from almost 2000 websites.

Install Instructions

https://github.com/iiab/calibre-web/wiki#wrench-installation

What's Changed

• Increase "fragment stuck" timeout from 10s to 30s [try to avoid erroneously labeling slow video downloads as "stuck"] by @deldesir in #169
• Enhance timeout check to avoid false positives during processing [of video download to IIAB] by @deldesir in #170
• Match playlist pattern [try to improve video downloads from ~1000 sites, not just YouTube and Vimeo — thanks to yt-dlp] by @deldesir in #174
• Simplify playlists identification [for non-YouTube playlists e.g. Vimeo showcases, Bilibili.tv / Bilibili.com anthologies, etc] by @deldesir in #173
• Make error status messages user-friendly by @deldesir in #172

Full Changelog: 2024.05.31...2024.06.07

Build your own "YouTube/Vimeo[*] Learning Library" using Internet-in-a-Box !

[*] Vimeo is not quite yet working, but we aim to help educators download the videos and audio recordings they truly need, from almost 2000 websites.

Install Instructions

https://github.com/iiab/calibre-web/wiki#wrench-installation

What's Changed

• Add additional media support by @deldesir in #1
• Make covers square by @deldesir in #2
• Online media download support by @deldesir in #6
• Add download task by @deldesir in #10
• Remove unsupported formats by @deldesir in #16
• Handle request by @deldesir in #17
• Improve task by @deldesir in #18
• Revert prefix by @deldesir in #19
• Remove addToBookshelf by @deldesir in #21
• Fix unresponsive event by @deldesir in #23
• Remove addToBookshelf by @deldesir in #27
• Handle failed request by @deldesir in #26
• Fix modal by @deldesir in #32
• Add lb wrapper by @deldesir in #34
• Stabilize dynamic [URL href for API] endpoints resolution by @deldesir in #33
• Update lb-wrapper by @deldesir in #35
• Improvements on bookshelf automation by @deldesir in #36
• Add omg.yml by @deldesir in #38
• Update lb-wrapper by @deldesir in #39
• Update media db path [sets SURVEY_DB_FILE to /library/downloads/calibre-web/survey.db] by @deldesir in #40
• Fix path [SURVEY_DB_FILE in lb-wrapper] by @deldesir in #42
• Tighten metadata [auto-extract author and publisher from 'path' field of /library/downloads/calibre-web/survey.db] by @deldesir in #41
• Update lb-wrapper by @deldesir in #43
• Clarify cps/constants.py and scripts/lb-wrapper [& refine URL validation, to accept http:// legacy URLs w/o SSL/TLS] by @holta in #44
• Pin max format [Set "best" <= 720p with VP9 codec for now: later consider 30FPS instead of 60FPS and WebM to tighten up further?] by @deldesir in #48
• Create calibre-web-nginx.conf by @deldesir in #50
• Time to put scripts/omg.yml on a diet! by @holta in #55
• Use correct option [lb --write-thumbnail, and also fix logging explanation to match recent changes] by @deldesir in #54
• Fix missing cover [for MP4; not just WebM] by @deldesir in #59
• scripts/lb-wrapper: Try yt-dlp with "--format best --format-sort 'tbr~1000'" [testing shows need for URL validation, clearly communicated, for users who accidentally request the impossible after clicking "Download to IIAB" > "Start"] by @holta in #63
• scripts/lb-wrapper: Try "--format-sort='tbr~1000'" [i.e. target 1000 kbit/s codecs] by @holta in #62
• Create scripts/lb-wrapper.greedy à la 'yt-dlp dl --format-sort size' [and refactor /usr/local/bin/lb-wrapper] by @holta in #64
• Trim outer whitespace from URL ["Download to IIAB" user input] by @deldesir in #66
• Find thumbnail [a.k.a. cover/poster. For MP4; not just WebM] by @deldesir in #65
• Find webp file [thumbnail/cover/poster for MP4; not just WebM ?] by @deldesir in #69
• Pull full description text from 'captions' table by @deldesir in #71
• Support PNG, WEBP, GIF, JPG covers/thumbnails by @deldesir in #72
• Increase Buffersize to 10GB by @deldesir in #75
• Unstretch covers/posters [CSS crops each rectangle to its central square "thumbnail"] by @deldesir in #80
• Remove advanced settings from "Download" modal dialog by @deldesir in #81
• cps/editbooks.py: move instead of copyfile (for #45) by @holta in #82
• Suggested improvements for scripts/lb-wrapper [refactor logging & subprocess substitution, etc] by @holta in #87
• Refactor pattern matching [cps/tasks/download.py: parse yt-dlp output for download progress percentage] by @deldesir in #88
• 3 regex's (STDERR, STDIN, Python) to parse yt-dlp download progress, for live display in "Tasks" view by @holta in #93
• Update tasks messages based on the current status by @deldesir in #90
• Get lb-wrapper command directly : cps/tasks/download.py by @deldesir in #94
• Improve error handling for failed downloads by @deldesir in #95
• Enhance database operations for better debugging by @deldesir in #96
• Revert to upload task by @deldesir in #102
• Fix shelf regression by @deldesir in #103
• Tighten lb-wrapper: XKLB already writes version# to xklb.log by @holta in #108
• Rename and persist xklb database by @deldesir in #109
• Refactor lb-wrapper for command separation by @deldesir in #110
• Decouple the metadata fetching with TaskMetadataExtract by @deldesir in #111
• Create cps/tasks/metadata_extract.py by @deldesir in #112
• Call download task from TaskMetadataExtract by @deldesir in #113
• Retrieve relevant description by @deldesir in #116
• Make URLs clickable [in "Tasks" view] by @deldesir in #115
• Update the video path upon successful download by @deldesir in #118
• Ignore failed URLs in new downloads by @deldesir in #121
• Column error does not exist until lb dl is run by @deldesir in #122
• Ensure fail-safe downloads when an invalid URL was submitted in a previous session by @deldesir in #123
• Download a single video instead of all videos in a playlist by @deldesir in #124
• Make url clickable in metadata task as well by @deldesir in #126
• Add upstream commits by @deldesir in #128
• Quick Testing of 75 upstream commits (89 files changes) w/o bureaucratizing by @holta in #127
• UPDATE paths instead of REPLACE + FOOTNOTE re: Archivists-vs-YouTube having conflicting world views, on how to define "Duplicate Videos" by @deldesir in #125
• Among failed URLs recorded in xklb-metadata db, select the requested failed one instead of the first one encountered by @deldesir in #130
• Increase task cleanup trigger to 101 by @deldesir in #133
• Fix "405 METHOD NOT ALLOWED" by (1) Changing /media to /meta (2) Keeping URL query parameters by @deldesir in #131
• Handle complex URL [e.g. single video AND playlist in the same YouTube URL; e.g. YouTube URLs containing @channelname] by @deldesir in #134
• Handle hanging due to unavailable fragments by @deldesir in #137
• Refactor for better maintainability by @deldesir in #142
• Add Max Download limits by @deldesir in #143
• Update to retrieve key metadata by @deldesir in #144
• Calculate each video's views-per-day [very rough approximation for popularity / importance / quality] by @deldesir in #145
• Remove redundant log message by @deldesir in #148
• Remove redundant log message by @deldesir in #149
• Ignore failed/residual/stuck video downloads: download only newly requested one(s) by @deldesir in #147
• Log yt-dlp version by @deldesir in #159
• Sync with upstream by @deldesir in #160
• Fix missing cover by @deldesir in #162
• Add lb search support by @deldesir in https://github.com/iiab/calibre-...