Skip to content

Conversation

deldesir
Copy link
Collaborator

@deldesir deldesir commented Jan 26, 2024

🚀 Pull Request Overview:

This pull request refactors the download task to be called from within the media metadata one (TaskMediaMeta). This ensures every single video is downloaded in its own task.

📋 Checklist:

  • Tested the changes thoroughly.

🔗 Related Issue(s):
Issue #104

📌 Testing scenarios:
See Issue #97

cc @EMG70

PS This PR builds upon #109, #110, #111

@deldesir deldesir requested a review from holta January 26, 2024 13:08
@deldesir deldesir self-assigned this Jan 26, 2024
@deldesir deldesir changed the title Call download task from TaskMediaMeta Call download task from TaskMetadataExtract Jan 26, 2024
@holta holta added the enhancement New feature or request label Jan 26, 2024
@EMG70
Copy link

EMG70 commented Jan 26, 2024

iiab-diagnostics http://sprunge.us/hrfrUF?en

1.Valid Download: https://www.youtube.com/watch?v=AE1aNFeV_2s✅
2. Invalid URL: https://www.youtube.com/watch?v=AE1aNFeV_5s ( url intentionally amended to differ on second from last strings (_5s instead of _2s) from Test 1 above) ✅
error observed in Tasks
Screenshot from 2024-01-26 23-20-03
3.No Requested Files in the Database:
Ensure the SQLite database contains no requested files by providing a YouTube live video URL
Task shows 'Non type:object is not subscriptable ' ✅

Run the download task and verify that it logs "No requested files found in the database." ✅
''' [2024-01-26 21:12:10,799] INFO {cps:168} *** "greenlet" version does not meet the requirements. Should: <2.1.0, Found: 3.0.3, please consider installing required version ***
[2024-01-26 21:12:10,799] INFO {cps:177} Starting Calibre Web...
[2024-01-26 21:12:10,833] DEBUG {cps.services:26} Cannot import goodreads, showing authors-metadata will not work: No module named 'betterreads'
[2024-01-26 21:12:10,835] DEBUG {cps.services:34} Cannot import simpleldap, logging in with ldap will not work: No module named 'flask_simpleldap'
[2024-01-26 21:12:10,890] DEBUG {cps.services:49} Cannot import gmail, sending books via Gmail Oauth2 Verification will not work: No module named 'google_auth_oauthlib'
[2024-01-26 21:12:10,915] WARN {py.warnings:109} /usr/local/calibre-web-py3/lib/python3.11/site-packages/flask_limiter/extension.py:336: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
warnings.warn(
[2024-01-26 21:12:11,111] DEBUG {cps.gdriveutils:81} Cannot import pydrive, httplib2, using gdrive will not work: No module named 'pydrive'
[2024-01-26 21:12:11,310] INFO {apscheduler.scheduler:181} Scheduler started
[2024-01-26 21:12:11,311] INFO {apscheduler.scheduler:895} Added job "end scheduled task" to job store "default"
[2024-01-26 21:12:11,539] DEBUG {cps.comic:46} Cannot import comicapi, extracting comic metadata will not work: No module named 'comicapi'
[2024-01-26 21:12:11,541] DEBUG {cps.comic:53} Cannot import rarfile, extracting cover files from rar files will not work: No module named 'rarfile'
[2024-01-26 21:12:11,541] DEBUG {cps.comic:59} Cannot import py7zr, extracting cover files from CB7 files will not work: No module named 'py7zr'
[2024-01-26 21:12:11,663] DEBUG {cps.admin:81} Cannot import Flask-Dance, login with Oauth will not work: No module named 'flask_dance'
[2024-01-26 21:12:11,671] DEBUG {cps.gdrive:44} Cannot import googleapiclient, using GDrive will not work: No module named 'googleapiclient'
[2024-01-26 21:12:11,746] DEBUG {cps.search_metadata:62} Import error for metadata source: douban - No module named 'html2text'
[2024-01-26 21:12:11,764] DEBUG {cps.search_metadata:62} Import error for metadata source: lubimyczytac - No module named 'dateutil'
[2024-01-26 21:12:11,766] DEBUG {cps.search_metadata:62} Import error for metadata source: scholar - No module named 'scholarly'
[2024-01-26 21:12:11,769] DEBUG {cps.search_metadata:62} Import error for metadata source: amazon - No module named 'bs4'
[2024-01-26 21:12:11,914] INFO {cps.server:268} Starting Tornado server on :8083
[2024-01-26 21:24:16,792] INFO {cps.server:320} webserver stop (restart=False)
[2024-01-26 21:24:16,792] INFO {apscheduler.scheduler:212} Scheduler has been shut down
[2024-01-26 21:24:16,793] INFO {cps.server:298} Performing shutdown of Calibre-Web
[2024-01-26 21:24:18,289] INFO {cps:168} *** "greenlet" version does not meet the requirements. Should: <2.1.0, Found: 3.0.3, please consider installing required version ***
[2024-01-26 21:24:18,289] INFO {cps:177} Starting Calibre Web...
[2024-01-26 21:24:18,298] DEBUG {cps.services:26} Cannot import goodreads, showing authors-metadata will not work: No module named 'betterreads'
[2024-01-26 21:24:18,299] DEBUG {cps.services:34} Cannot import simpleldap, logging in with ldap will not work: No module named 'flask_simpleldap'
[2024-01-26 21:24:18,356] DEBUG {cps.services:49} Cannot import gmail, sending books via Gmail Oauth2 Verification will not work: No module named 'google_auth_oauthlib'
[2024-01-26 21:24:18,388] WARN {py.warnings:109} /usr/local/calibre-web-py3/lib/python3.11/site-packages/flask_limiter/extension.py:336: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
warnings.warn(
[2024-01-26 21:24:18,563] DEBUG {cps.gdriveutils:81} Cannot import pydrive, httplib2, using gdrive will not work: No module named 'pydrive'
[2024-01-26 21:24:18,700] INFO {apscheduler.scheduler:181} Scheduler started
[2024-01-26 21:24:18,701] INFO {apscheduler.scheduler:895} Added job "end scheduled task" to job store "default"
[2024-01-26 21:24:18,829] DEBUG {cps.comic:46} Cannot import comicapi, extracting comic metadata will not work: No module named 'comicapi'
[2024-01-26 21:24:18,831] DEBUG {cps.comic:53} Cannot import rarfile, extracting cover files from rar files will not work: No module named 'rarfile'
[2024-01-26 21:24:18,831] DEBUG {cps.comic:59} Cannot import py7zr, extracting cover files from CB7 files will not work: No module named 'py7zr'
[2024-01-26 21:24:18,977] DEBUG {cps.admin:81} Cannot import Flask-Dance, login with Oauth will not work: No module named 'flask_dance'
[2024-01-26 21:24:18,982] DEBUG {cps.gdrive:44} Cannot import googleapiclient, using GDrive will not work: No module named 'googleapiclient'
[2024-01-26 21:24:19,023] DEBUG {cps.search_metadata:62} Import error for metadata source: douban - No module named 'html2text'
[2024-01-26 21:24:19,033] DEBUG {cps.search_metadata:62} Import error for metadata source: lubimyczytac - No module named 'dateutil'
[2024-01-26 21:24:19,034] DEBUG {cps.search_metadata:62} Import error for metadata source: scholar - No module named 'scholarly'
[2024-01-26 21:24:19,034] DEBUG {cps.search_metadata:62} Import error for metadata source: amazon - No module named 'bs4'
[2024-01-26 21:24:19,146] INFO {cps.server:268} Starting Tornado server on :8083
[2024-01-26 21:38:36,379] INFO {cps.server:320} webserver stop (restart=False)
[2024-01-26 21:38:36,380] INFO {apscheduler.scheduler:212} Scheduler has been shut down
[2024-01-26 21:38:36,381] INFO {cps.server:298} Performing shutdown of Calibre-Web
[2024-01-26 21:38:38,036] INFO {cps:168} *** "greenlet" version does not meet the requirements. Should: <2.1.0, Found: 3.0.3, please consider installing required version ***
[2024-01-26 21:38:38,036] INFO {cps:177} Starting Calibre Web...
[2024-01-26 21:38:38,050] DEBUG {cps.services:26} Cannot import goodreads, showing authors-metadata will not work: No module named 'betterreads'
[2024-01-26 21:38:38,051] DEBUG {cps.services:34} Cannot import simpleldap, logging in with ldap will not work: No module named 'flask_simpleldap'
[2024-01-26 21:38:38,113] DEBUG {cps.services:49} Cannot import gmail, sending books via Gmail Oauth2 Verification will not work: No module named 'google_auth_oauthlib'
[2024-01-26 21:38:38,146] WARN {py.warnings:109} /usr/local/calibre-web-py3/lib/python3.11/site-packages/flask_limiter/extension.py:336: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
warnings.warn(
[2024-01-26 21:38:38,326] DEBUG {cps.gdriveutils:81} Cannot import pydrive, httplib2, using gdrive will not work: No module named 'pydrive'
[2024-01-26 21:38:38,463] INFO {apscheduler.scheduler:181} Scheduler started
[2024-01-26 21:38:38,464] INFO {apscheduler.scheduler:895} Added job "end scheduled task" to job store "default"
[2024-01-26 21:38:38,611] DEBUG {cps.comic:46} Cannot import comicapi, extracting comic metadata will not work: No module named 'comicapi'
[2024-01-26 21:38:38,614] DEBUG {cps.comic:53} Cannot import rarfile, extracting cover files from rar files will not work: No module named 'rarfile'
[2024-01-26 21:38:38,614] DEBUG {cps.comic:59} Cannot import py7zr, extracting cover files from CB7 files will not work: No module named 'py7zr'
[2024-01-26 21:38:38,763] DEBUG {cps.admin:81} Cannot import Flask-Dance, login with Oauth will not work: No module named 'flask_dance'
[2024-01-26 21:38:38,771] DEBUG {cps.gdrive:44} Cannot import googleapiclient, using GDrive will not work: No module named 'googleapiclient'
[2024-01-26 21:55:29,706] INFO {cps.server:320} webserver stop (restart=False)
[2024-01-26 21:55:29,707] INFO {apscheduler.scheduler:212} Scheduler has been shut down
[2024-01-26 21:55:31,516] INFO {cps:168} *** "greenlet" version does not meet the requirements. Should: <2.1.0, Found: 3.0.3, please consider installing required version ***
[2024-01-26 21:55:31,516] INFO {cps:177} Starting Calibre Web...
[2024-01-26 21:55:31,527] DEBUG {cps.services:26} Cannot import goodreads, showing authors-metadata will not work: No module named 'betterreads'
[2024-01-26 21:55:31,528] DEBUG {cps.services:34} Cannot import simpleldap, logging in with ldap will not work: No module named 'flask_simpleldap'
[2024-01-26 21:55:31,588] DEBUG {cps.services:49} Cannot import gmail, sending books via Gmail Oauth2 Verification will not work: No module named 'google_auth_oauthlib'
[2024-01-26 21:55:31,622] WARN {py.warnings:109} /usr/local/calibre-web-py3/lib/python3.11/site-packages/flask_limiter/extension.py:336: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
warnings.warn(
[2024-01-26 21:55:31,850] DEBUG {cps.gdriveutils:81} Cannot import pydrive, httplib2, using gdrive will not work: No module named 'pydrive'
[2024-01-26 21:55:32,051] INFO {apscheduler.scheduler:181} Scheduler started
[2024-01-26 21:55:32,052] INFO {apscheduler.scheduler:895} Added job "end scheduled task" to job store "default"
[2024-01-26 21:55:32,260] DEBUG {cps.comic:46} Cannot import comicapi, extracting comic metadata will not work: No module named 'comicapi'
[2024-01-26 21:55:32,263] DEBUG {cps.comic:53} Cannot import rarfile, extracting cover files from rar files will not work: No module named 'rarfile'
[2024-01-26 21:55:32,263] DEBUG {cps.comic:59} Cannot import py7zr, extracting cover files from CB7 files will not work: No module named 'py7zr'
[2024-01-26 21:55:32,426] DEBUG {cps.admin:81} Cannot import Flask-Dance, login with Oauth will not work: No module named 'flask_dance'
[2024-01-26 21:55:32,435] DEBUG {cps.gdrive:44} Cannot import googleapiclient, using GDrive will not work: No module named 'googleapiclient'
[2024-01-26 21:55:32,511] DEBUG {cps.search_metadata:62} Import error for metadata source: douban - No module named 'html2text'
[2024-01-26 21:55:32,527] DEBUG {cps.search_metadata:62} Import error for metadata source: lubimyczytac - No module named 'dateutil'
[2024-01-26 21:55:32,529] DEBUG {cps.search_metadata:62} Import error for metadata source: scholar - No module named 'scholarly'
[2024-01-26 21:55:32,530] DEBUG {cps.search_metadata:62} Import error for metadata source: amazon - No module named 'bs4'
[2024-01-26 21:55:32,652] INFO {cps.server:268} Starting Tornado server on :8083
[2024-01-26 22:13:43,929] INFO {cps.server:320} webserver stop (restart=False)
[2024-01-26 22:13:43,930] INFO {apscheduler.scheduler:212} Scheduler has been shut down
[2024-01-26 22:13:43,931] INFO {cps.server:298} Performing shutdown of Calibre-Web
[2024-01-26 22:13:45,385] INFO {cps:168} *** "greenlet" version does not meet the requirements. Should: <2.1.0, Found: 3.0.3, please consider installing required version ***
[2024-01-26 22:13:45,385] INFO {cps:177} Starting Calibre Web...
[2024-01-26 22:13:45,395] DEBUG {cps.services:26} Cannot import goodreads, showing authors-metadata will not work: No module named 'betterreads'
[2024-01-26 22:13:45,396] DEBUG {cps.services:34} Cannot import simpleldap, logging in with ldap will not work: No module named 'flask_simpleldap'
[2024-01-26 22:13:45,450] DEBUG {cps.services:49} Cannot import gmail, sending books via Gmail Oauth2 Verification will not work: No module named 'google_auth_oauthlib'
[2024-01-26 22:13:45,482] WARN {py.warnings:109} /usr/local/calibre-web-py3/lib/python3.11/site-packages/flask_limiter/extension.py:336: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
warnings.warn(
[2024-01-26 22:13:45,738] DEBUG {cps.gdriveutils:81} Cannot import pydrive, httplib2, using gdrive will not work: No module named 'pydrive'
[2024-01-26 22:13:45,998] INFO {apscheduler.scheduler:181} Scheduler started
[2024-01-26 22:13:45,999] INFO {apscheduler.scheduler:895} Added job "end scheduled task" to job store "default"
[2024-01-26 22:13:46,122] DEBUG {cps.comic:46} Cannot import comicapi, extracting comic metadata will not work: No module named 'comicapi'
[2024-01-26 22:13:46,125] DEBUG {cps.comic:53} Cannot import rarfile, extracting cover files from rar files will not work: No module named 'rarfile'
[2024-01-26 22:13:46,125] DEBUG {cps.comic:59} Cannot import py7zr, extracting cover files from CB7 files will not work: No module named 'py7zr'
[2024-01-26 22:13:46,254] DEBUG {cps.admin:81} Cannot import Flask-Dance, login with Oauth will not work: No module named 'flask_dance'
[2024-01-26 22:13:46,260] DEBUG {cps.gdrive:44} Cannot import googleapiclient, using GDrive will not work: No module named 'googleapiclient'
[2024-01-26 22:13:46,324] DEBUG {cps.search_metadata:62} Import error for metadata source: douban - No module named 'html2text'
[2024-01-26 22:13:46,340] DEBUG {cps.search_metadata:62} Import error for metadata source: lubimyczytac - No module named 'dateutil'
[2024-01-26 22:13:46,340] DEBUG {cps.search_metadata:62} Import error for metadata source: scholar - No module named 'scholarly'
[2024-01-26 22:13:46,341] DEBUG {cps.search_metadata:62} Import error for metadata source: amazon - No module named 'bs4'
[2024-01-26 22:13:46,471] INFO {cps.server:268} Starting Tornado server on :8083
[2024-01-26 22:17:43,046] INFO {cps.server:320} webserver stop (restart=False)
[2024-01-26 22:17:43,047] INFO {apscheduler.scheduler:212} Scheduler has been shut down
[2024-01-26 22:17:43,049] INFO {cps.server:298} Performing shutdown of Calibre-Web
[2024-01-26 22:17:44,577] INFO {cps:168} *** "greenlet" version does not meet the requirements. Should: <2.1.0, Found: 3.0.3, please consider installing required version ***
[2024-01-26 22:17:44,577] INFO {cps:177} Starting Calibre Web...
[2024-01-26 22:17:44,589] DEBUG {cps.services:26} Cannot import goodreads, showing authors-metadata will not work: No module named 'betterreads'
[2024-01-26 22:17:44,590] DEBUG {cps.services:34} Cannot import simpleldap, logging in with ldap will not work: No module named 'flask_simpleldap'
[2024-01-26 22:17:44,641] DEBUG {cps.services:49} Cannot import gmail, sending books via Gmail Oauth2 Verification will not work: No module named 'google_auth_oauthlib'
[2024-01-26 22:17:44,677] WARN {py.warnings:109} /usr/local/calibre-web-py3/lib/python3.11/site-packages/flask_limiter/extension.py:336: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
warnings.warn(
[2024-01-26 22:17:44,854] DEBUG {cps.gdriveutils:81} Cannot import pydrive, httplib2, using gdrive will not work: No module named 'pydrive'
[2024-01-26 22:17:45,007] INFO {apscheduler.scheduler:181} Scheduler started
[2024-01-26 22:17:45,008] INFO {apscheduler.scheduler:895} Added job "end scheduled task" to job store "default"
[2024-01-26 22:17:45,158] DEBUG {cps.comic:46} Cannot import comicapi, extracting comic metadata will not work: No module named 'comicapi'
[2024-01-26 22:17:45,160] DEBUG {cps.comic:53} Cannot import rarfile, extracting cover files from rar files will not work: No module named 'rarfile'
[2024-01-26 22:17:45,160] DEBUG {cps.comic:59} Cannot import py7zr, extracting cover files from CB7 files will not work: No module named 'py7zr'
[2024-01-26 22:17:45,289] DEBUG {cps.admin:81} Cannot import Flask-Dance, login with Oauth will not work: No module named 'flask_dance'
[2024-01-26 22:17:45,295] DEBUG {cps.gdrive:44} Cannot import googleapiclient, using GDrive will not work: No module named 'googleapiclient'
[2024-01-26 22:17:45,355] DEBUG {cps.search_metadata:62} Import error for metadata source: douban - No module named 'html2text'
[2024-01-26 22:17:45,368] DEBUG {cps.search_metadata:62} Import error for metadata source: lubimyczytac - No module named 'dateutil'
[2024-01-26 22:17:45,369] DEBUG {cps.search_metadata:62} Import error for metadata source: scholar - No module named 'scholarly'
[2024-01-26 22:17:45,370] DEBUG {cps.search_metadata:62} Import error for metadata source: amazon - No module named 'bs4'
[2024-01-26 22:17:45,505] INFO {cps.server:268} Starting Tornado server on :8083
[2024-01-26 22:30:15,053] DEBUG {cps.web:1410} You are now logged in as: 'Admin'
[2024-01-26 22:30:15,067] DEBUG {cps.ub:82} Login and store session : dc6e31a14246e6e1c7dcec411f0c290f265be3131f2ea8069eeb74e0391ffb948b36fc805be9601c632fc777cc4a4e966ec238e9c9f9a12b877e8dbc37ab2733
[2024-01-26 22:30:15,069] DEBUG {cps.ub:84} Found stored session: dc6e31a14246e6e1c7dcec411f0c290f265be3131f2ea8069eeb74e0391ffb948b36fc805be9601c632fc777cc4a4e966ec238e9c9f9a12b877e8dbc37ab2733
[2024-01-26 22:30:32,470] DEBUG {cps.services.worker:91} Add Task for user: Admin - Metadata fetch task for https://www.youtube.com/watch?v=AE1aNFeV_2s
[2024-01-26 22:30:32,471] INFO {cps.tasks.metadata_extract:29} Starting to fetch metadata for URL: https://www.youtube.com/watch?v=AE1aNFeV_2s
[2024-01-26 22:30:32,472] INFO {cps.tasks.metadata_extract:38} Subprocess args: ['lb-wrapper', 'tubeadd', 'https://www.youtube.com/watch?v=AE1aNFeV_2s']
[2024-01-26 22:30:35,062] INFO {cps.tasks.metadata_extract:73} No playlists table found in the database
[2024-01-26 22:30:35,062] INFO {cps.tasks.metadata_extract:79} Shelf title: None
[2024-01-26 22:30:35,063] DEBUG {cps.services.worker:91} Add Task for user: Admin - Download task for https://www.youtube.com/watch?v=AE1aNFeV_2s
[2024-01-26 22:30:35,063] INFO {cps.tasks.download:30} Starting download task for URL: https://www.youtube.com/watch?v=AE1aNFeV_2s
[2024-01-26 22:30:35,064] INFO {cps.tasks.download:39} Subprocess args: ['lb-wrapper', 'dl', 'https://www.youtube.com/watch?v=AE1aNFeV_2s']
[2024-01-26 22:32:11,075] INFO {cps.editbooks:384} Requested file: /library/downloads/calibre-web/Youtube/AIM_Agriculture_Farm/From_Being_a_Nurse_to_Chicken_Princess_on_the_Smallest_Space_144.60k_[AE1aNFeV_2s].mp4
[2024-01-26 22:32:11,076] INFO {cps.editbooks:389} Processing file: <io.BufferedReader name='/library/downloads/calibre-web/Youtube/AIM_Agriculture_Farm/From_Being_a_Nurse_to_Chicken_Princess_on_the_Smallest_Space_144.60k[AE1aNFeV_2s].mp4'>
[2024-01-26 22:32:11,077] DEBUG {cps.uploader:371} Temporary file: /tmp/calibre_web/2861118703d5eb3d07943c3d96e34df3
[2024-01-26 22:32:11,119] DEBUG {cps.helper:543} Moving title: /tmp/calibre_web/2861118703d5eb3d07943c3d96e34df3 to /library/calibre-web/AIM Agriculture Farm/From Being a Nurse to Chicken Princess on the Smallest Space (1)/From Being a Nurse to Chicken Princess on the Smallest Space - AIM Agriculture Farm
[2024-01-26 22:32:11,146] INFO {cps.tasks.download:92} Successfully sent the requested file to http://10.201.17.167/books/meta
[2024-01-26 22:33:31,240] DEBUG {cps.services.worker:91} Add Task for user: Admin - Metadata fetch task for https://www.youtube.com/watch?v=AE1aNFeV_5s
[2024-01-26 22:33:31,241] INFO {cps.tasks.metadata_extract:29} Starting to fetch metadata for URL: https://www.youtube.com/watch?v=AE1aNFeV_5s
[2024-01-26 22:33:31,241] INFO {cps.tasks.metadata_extract:38} Subprocess args: ['lb-wrapper', 'tubeadd', 'https://www.youtube.com/watch?v=AE1aNFeV_5s']
[2024-01-26 22:33:32,883] INFO {cps.tasks.metadata_extract:73} No playlists table found in the database
[2024-01-26 22:33:32,883] INFO {cps.tasks.metadata_extract:79} Shelf title: None
[2024-01-26 22:33:32,883] DEBUG {cps.services.worker:91} Add Task for user: Admin - Download task for https://www.youtube.com/watch?v=AE1aNFeV_5s
[2024-01-26 22:33:32,883] INFO {cps.tasks.download:30} Starting download task for URL: https://www.youtube.com/watch?v=AE1aNFeV_5s
[2024-01-26 22:33:32,883] INFO {cps.tasks.download:39} Subprocess args: ['lb-wrapper', 'dl', 'https://www.youtube.com/watch?v=AE1aNFeV_5s']
[2024-01-26 22:33:37,015] ERROR {cps.tasks.download:100} An error occurred during the subprocess execution: 'NoneType' object is not subscriptable
[2024-01-26 22:35:25,046] DEBUG {cps.services.worker:91} Add Task for user: Admin - Metadata fetch task for https://www.youtube.com/watch?v=BJ3Yv572V1A
[2024-01-26 22:35:25,048] INFO {cps.tasks.metadata_extract:29} Starting to fetch metadata for URL: https://www.youtube.com/watch?v=BJ3Yv572V1A
[2024-01-26 22:35:25,048] INFO {cps.tasks.metadata_extract:38} Subprocess args: ['lb-wrapper', 'tubeadd', 'https://www.youtube.com/watch?v=BJ3Yv572V1A']
[2024-01-26 22:35:27,217] INFO {cps.tasks.metadata_extract:73} No playlists table found in the database
[2024-01-26 22:35:27,217] INFO {cps.tasks.metadata_extract:79} Shelf title: None
[2024-01-26 22:35:27,218] DEBUG {cps.services.worker:91} Add Task for user: Admin - Download task for https://www.youtube.com/watch?v=AE1aNFeV_5s
[2024-01-26 22:35:27,218] DEBUG {cps.services.worker:91} Add Task for user: Admin - Download task for https://www.youtube.com/watch?v=BJ3Yv572V1A
[2024-01-26 22:35:27,218] INFO {cps.tasks.download:30} Starting download task for URL: https://www.youtube.com/watch?v=AE1aNFeV_5s
[2024-01-26 22:35:27,218] INFO {cps.tasks.download:39} Subprocess args: ['lb-wrapper', 'dl', 'https://www.youtube.com/watch?v=AE1aNFeV_5s']
[2024-01-26 22:35:29,234] ERROR {cps.tasks.download:100} An error occurred during the subprocess execution: 'NoneType' object is not subscriptable
[2024-01-26 22:35:29,234] INFO {cps.tasks.download:30} Starting download task for URL: https://www.youtube.com/watch?v=BJ3Yv572V1A
[2024-01-26 22:35:29,234] INFO {cps.tasks.download:39} Subprocess args: ['lb-wrapper', 'dl', 'https://www.youtube.com/watch?v=BJ3Yv572V1A']
[2024-01-26 22:35:30,036] ERROR {cps.tasks.download:100} An error occurred during the subprocess execution: 'NoneType' object is not subscriptable
[2024-01-26 23:24:41,642] DEBUG {cps.updater:329} Stable version: {'version': '0.6.22 Beta'}
[2024-01-26 23:24:54,213] DEBUG {cps.updater:329} Stable version: {'version': '0.6.22 Beta'}
[2024-01-26 23:25:42,258] DEBUG {cps.updater:329} Stable version: {'version': '0.6.22 Beta'}
'''

Check that the progress is set to 0 and an error is displayed in Tasks. ✅

@holta
Copy link
Member

holta commented Jan 27, 2024

@deldesir Which ticket should address 'NoneType' object is not subscriptable (3 identical errors) in @EMG70's screenshot below?

IMG-20240126-WA0010~4

@deldesir
Copy link
Collaborator Author

@deldesir Which ticket should address 'NoneType' object is not subscriptable (3 identical errors) in @EMG70's screenshot below?

IMG-20240126-WA0010~4

Let's keep track of it in #114.

To allow all videos from a single session to be added in the same bookshelf (when a playlist URL is provided)
Introduce a slow down while ensuring we catch up with progress cycles
@holta
Copy link
Member

holta commented Jan 29, 2024

@deldesir

Awesome if the race-condition is being solved!

@EMG70 would love to test all 4 PR's together (#110, #111, #112, #113) when it's time!

@deldesir
Copy link
Collaborator Author

It's time. Waiting for his confirmation.

@EMG70
Copy link

EMG70 commented Jan 30, 2024

Test after applying PR 110,111,112 AND 113
iiab-dagnostics http://sprunge.us/gxJl3I?en
Playlist https://www.youtube.com/playlist?list=PLT98CRl2KxKEUHie1m24-wkyHpEsa4Y70 was downloaded.The first video showed progress until 74% and stayed there until all videos were downloaded successfully in the background.Playlist wasd auto populated in shelf and home page OK
I was hoping to see each video showing download progress one after the other.
The second test was with playlist intentionally amended to give a wrong url ,results showed (failed:nontype,object is not subscriptable.See screenshot,last row.
Screenshot from 2024-01-30 18-14-23
Screenshot from 2024-01-30 18-21-07

@deldesir
Copy link
Collaborator Author

Thanks @EMG70 for your feedback. Improvements will be added after they are merged.

@holta
Copy link
Member

holta commented Jan 31, 2024

A small "description" regression turns out to be unrelated to these 4 PR's (#110, #111, #112, #113) in the end!

So that issue will be addressed separately here:

@holta
Copy link
Member

holta commented Jan 31, 2024

Thank you @EMG70 and @deldesir for careful TDD (focused testing) over the past week.

Looking to the future now with these PRs are others:

@holta holta merged commit 4c3cad6 into iiab:master Jan 31, 2024
@deldesir deldesir deleted the deldesir-task-download branch July 1, 2024 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants