Merges upload and download tasks messages to avoid too many tasks rows

1. I don't quite understand what is being proposed. Please explain more, Thanks! (Roughly Speaking: Who wants this? What should we be aiming for and why?)

2. An opposite viewpoint (might) be that cramming everything onto 1 line (of "Tasks" view) can potentially / accidentally make tangible understanding + debugging + copy/pasting of "disappearing messages" more frustrating and unfriendly❓

Imagine downloading a playlist of a dozen videos! It's better to have the list of all videos downloaded in the relevant row IMHO. This keeps the table tidy and helps identify failed videos in a playlist.

• Introduces a list files_uploaded to store links of successfully uploaded files.

I and others are trying to understand this PR:

• What does files_uploaded really mean?
• What does uploaded files really mean (presumably the same as above?)

Users should never see the word "uploaded" if they just clicked "Download to IIAB" — the word "uploaded" appears to be extremely confusing language making this PR very difficult to understand (?) And if so...

• Should the word "uploaded" be completely hidden from users if it's leading to mass confusion?

Imagine downloading a playlist of a dozen videos! It's better to have the list of all videos downloaded in the relevant row IMHO. This keeps the table tidy and helps identify failed videos in a playlist.

@deldesir Please deepen explanation(s) of what you are proposing!

• Are you saying that all playlist info (e.g. when downloading from a YouTube playlist or channel) should be squeezed into one row within "Tasks" view?
• What happens when a playlist has 1000 videos?
  • How would one see 100 video download failures and 900 video download successes?
• Are some rows (or table entries) effectively becoming quite dynamic, i.e. almost animations?
  • How would copy/pasting work — e.g. if the contents of an increasing number of table entries regularly change or disappear?

PS @EMG70 asks how PRs #96 and #99 should be tested separately and/or together...

• Introduces a list files_uploaded to store links of successfully uploaded files.

I and others are trying to understand this PR:

• What does files_uploaded really mean?
• What does uploaded files really mean (presumably the same as above?)

They're the same. files_uploaded is the variable that holds the links of all the videos downloaded in a single run/call.

Users should never see the word "uploaded" if they just clicked "Download to IIAB" — the word "uploaded" appears to be extremely confusing language making this PR very difficult to understand (?) And if so...

I agree. The subsequent upload tasks that kick in right after videos' been downloaded do not fit well. They are there just because I forgot to remove them initially when I came with download task.

• Should the word "uploaded" be completely hidden from users if it's leading to mass confusion?

It should because we already have this reserved for direct files uploads/imports using the upload button.

• Are you saying that all playlist info (e.g. when downloading from a YouTube playlist or channel) should be squeezed into one row within "Tasks" view?

Yes, and single downloads too. The upload task along with a download one was a pure accident and wasn't designed. I was learning about the upload task and made the download one after it.

• What happens when a playlist has 1000 videos?

  • How would one see 100 video download failures and 900 video download successes?

I can't be sure I know exactly what the answer is unless we try this scenario ourselves. I presume links of those videos would not be shown because.

• Are some rows (or table entries) effectively becoming quite dynamic, i.e. almost animations?

Prior to #99, there was already some dynamic updating at play for almost all columns. This PR builds on it.

• How would copy/pasting work — e.g. if the contents of an increasing number of table entries regularly changes or disappears?

This video that keeps failing https://www.youtube.com/watch?v=QXQrvT23rPw (due to xklb "media check failure") would benefit from a dynamic sort of healing/recovering, but we're not there yet. Copy pasting the table was never possible to me. I was planning of implementing a mechanism to dump it.

PS @EMG70 asks how PRs #96 and #99 should be tested separately and/or together...

Please test them together using the same testing scenarios from #97. If this turns out to be non practical or non user-friendly, I'll do necessary changes.

This video that keeps failing https://www.youtube.com/watch?v=QXQrvT23rPw (due to xklb "media check failure")

@deldesir What ticket number is tracking the above issue?

would benefit from a dynamic sort of healing/recovering, but we're not there yet. Copy pasting the table was never possible to me. I was planning of implementing a mechanism to dump it.

• What happens when a playlist has 1000 videos?

  • How would one see 100 video download failures and 900 video download successes?

I can't be sure I know exactly what the answer is unless we try this scenario ourselves. I presume links of those videos would not be shown because.

1. @deldesir Please open a ticket asking the key questions here:

   Playlists/Channels containing ~1000 videos need to work and work well!

2. RELATED: Sometimes soon Download to IIAB > Start should automatically identify playlists/channels then immediately prompt the IIAB implementer — which of those ~1000 videos do they need?

   • Latest ~100 videos
   • "Best" ~100 videos
   • Earliest ~100 videos
   • ALL ~1000 videos

This video that keeps failing https://www.youtube.com/watch?v=QXQrvT23rPw (due to xklb "media check failure")

@deldesir What ticket number is tracking the above issue?

@deldesir seems to have posted this to:

• "Media check failed" reported for specific videos #101

I have used same videos from PR97 for consistency
iiab-diagnostics http://sprunge.us/UxvHwB?en

1.Valid Download- still downloads ok but no thumbnail ❌
https://www.youtube.com/watch?v=QXQrvT23rPw The sahara

2.Invalid URL ✅ fails as as expected in tasks
https://www.youtube.com/watch?v=1RQVkzbU2jo&list=PLg2tfDG3Ww4uBdOZ-j-znfOarR7TKGs-w TLSMaths E7 Module

3.No Requested Files in the Database: Live video feed fails as expected ✅
https://www.youtube.com/watch?v=BJ3Yv572V1A Nat Geographic live video

4.Shelf Title Retrieval:
https://www.youtube.com/playlist?list=PL7yh-TELLS1FuqLSjl5bgiQIEH25VEmIc Python Advanced Tutorials playlist
Hung up and all videos in waiting mode thereafter.I had to restart calibre under Admin in order to resume downloading process .❌
Could it be a broken playlist?

Playlist downloads ok as one row showing all videos ✅
https://www.youtube.com/playlist?list=PLj8W7XIvO93rJHSYzkk7CgfiLQRUEC2Sq Playlist

5.Download Cancellation:
cancellation of video once started does not stop it from proceeding ❌
https://www.youtube.com/watch?v=CWG17Nj6iT0

I think viewing all downloaded videos in one row is a good idea IMHO as they look all compacted to allow librarian/admin to immediately see output than having them as individual rows.

I think viewing all downloaded videos in one row is a good idea IMHO as they look all compacted to allow librarian/admin to immediately see output than having them as individual rows.

I strongly disagree as cramming 100 videos (that's our target for most playlists/channels) into a giant paragraph of smallprint becomes... a disgusting / overwhelming mess (and I'm being very polite!)

That's fair enough.The admin /librarian can still scroll and see all downloads if viewed as one video per row.👍

admin /librarian can still scroll and see all downloads

Yes:

And the "Tasks" view will likely show basic/vital details for each of the playlist/channel's 100 videos, for example each row might show...

(1) HH:MM:SS = duration of each video (2) date-of-publication of the video (3) "views-per-year" (or views-per-day) i.e. relative popularity of each video, e.g. as reported by YouTube

RECAP: "Tasks" view is not just a bug-reporting / diagnostic tool.

It's also a critical Dashboard of Curatorial Insights, that might profoundly help educators grasp the structure of their own/evolving content choices! 🎨

1.Valid Download- still downloads ok but no thumbnail ❌
https://www.youtube.com/watch?v=QXQrvT23rPw The sahara

4.Shelf Title Retrieval:
https://www.youtube.com/playlist?list=PL7yh-TELLS1FuqLSjl5bgiQIEH25VEmIc Python Advanced Tutorials playlist
Hung up and all videos in waiting mode thereafter.I had to restart calibre under Admin in order to resume downloading process .❌
Could it be a broken playlist?

@deldesir please respond to both above, thanks to @EMG70's hard work above, Thank you!

5.Download Cancellation:
cancellation of video once started does not stop it from proceeding ❌
https://www.youtube.com/watch?v=CWG17Nj6iT0

PS I strongly suggest we not worry about 5. just yet. As all PR's MUST be as COMPACT as humanly possible. With CLEARLY-STATED PURPOSE. Any PR "afflicted by inflation" needs to broken apart into smaller PR's (each very valuable, communicating as many of the critical 5 W's as possible for each change!)

• @deldesir Please open a ticket asking the key questions here:
  Playlists/Channels containing ~1000 videos need to work and work well!

• RELATED: Sometimes soon Download to IIAB > Start should automatically identify playlists/channels then immediately prompt the IIAB implementer — which of those ~1000 videos do they need?

  • Latest ~100 videos
  • "Best" ~100 videos
  • Earliest ~100 videos
  • ALL ~1000 videos

CLARIFICATION: We will focus exclusively on playlists/channels' "TOP 100" videos (based on views-per-day, as reported by YouTube) going forward.

Featuritis a.k.a. Scope Creep beyond agreed upon Requirements a.k.a. Second System Effect is the death of all projects — and the straight truth is we have to put an end to the endless wish list of bells & whistles right now! 💯

Noted. I will revert to "upload" task for every video downloaded.

I will revert to "upload" task for every video downloaded.

Thank You!

And if 100 rows (depicting a playlist/channel's 100 videos) are extremely ugly in particular ways — we will assess as that time how to make that evolving UX progressively more purposeful — for the "curriculum designers" who will depend on it! 👣

Somewhat Related:

• Enhance download task messages for improved information when downloading a video #89 (comment)

I am perhaps late to the party, but I have a couple of comments.

1. I support the one video per row model. A playlist is just a way of getting the names of several videos of interest. I would treat the videos separately.
2. This means the download, any processing, and upload into Calibre should be done for each video as it completes.
3. If one or more videos fails it can be reported and the others can continue.
4. It also means that the algorithm that decides on video and audio format does not have to require all videos to be the same.
5. I have the impression that worker threads are used for parallel processing. Are there caps on the number of simultaneous videos that can be processed.

Of course, my views are based on what I did in Admin Console where lists of content can be selected at the same time, but each member of the list is processed separately in a queue. This video download facility would have been a nice addition to CMDSRV.

@tim-moody's insights are right on. One other point I and @deldesir should have mentioned... we're also looking for a "slightly gamified" (and definitely compelling!) view of each video as it's chosen/downloaded/ETC — to make "content gardening" much more fun!

RECAP: "Tasks" view is not just a bug-reporting / diagnostic tool.

It's also a critical Dashboard of Curatorial Insights, that might profoundly help educators grasp the structure of their own/evolving content choices! 🎨

(We don't yet have a great design: but hopefully in about a week we'll have an early prototype or visual mock-up at minimum, to kick the tires to start exploring this!)

Somewhat off-topic but then again maybe not (!) is this odd NYT article from earlier this week 😁

https://nytimes.com/2024/01/14/business/media/netflix-streaming-movies-ratings.html

@tim-moody wrote:

5. I have the impression that worker threads are used for parallel processing. Are there caps on the number of simultaneous videos that can be processed.

Great Question! I don't yet understand it and it's definitely a work in progress. @deldesir is sick this week but texted me...

"It works as a queue currently. All subsequent tasks are in waiting mode until the running task is done."

Hopefully he can explain more in later weeks how this works and how it might be evolving, when he's feeling better 😃

This PR is no more relevant. @holta can we close it since we have retained one video per row approach?

This PR is no more relevant. @holta can we close it since we have retained one video per row approach?

Ok if this PR is no longer relevant!