Repository navigation
YouTube videos with lots of comments seem impossible to download, keep failing while downloading comments #380
Description
Activity
Do you have any details of the error that's occurring? 404/503? If it's one that you see retrying before failure you could try bumping up the
--extractor-retriesvalue (default 3)It's probably YouTube doing some sort of rate limiting.
Do you have any details of the error that's occurring? 404/503? If it's one that you see retrying before failure you could try bumping up the
--extractor-retriesvalue (default 3)It's probably YouTube doing some sort of rate limiting.
Thanks for your reply. Pasted below is the error I receive:
It appears to be an error 1059? Another download right this very second got to 9890/23026 and has unfortunately errored as well. It doesn't seem to retry, it just hangs on "Download comment reply thread" anywhere between 10 seconds to 10 minutes before spitting out the above error and moving on to the next video. Edit: This one particular one was stuck for 7 minutes.
As for the possibility of being rate limited, is it possible to make the scraping of comments resumable, so in the case that it errors out, I can retry again and not have to start from scratch? It will consistently error out at a few thousand comments for the great majority of videos, so being able to resume would at least ensure progress is being made. Is that what --continue would do? I just noticed I'm using --no-continue, wondering if this is causing the comments to restart on errors.
As well, to ease troubleshooting: I have Python 3.7.4, and yt-dlp 2021.06.01. I'm using the following commands:
--verbose --force-ipv4 --sleep-interval 5 --max-sleep-interval 30 --ignore-errors --no-continue --no-overwrites --download-archive archive.log --add-metadata --write-description --write-info-json --write-annotations --write-thumbnail --embed-thumbnail --all-subs --sub-format "srt" --embed-subs --get-comments --match-filter "!is_live & !live" --output "%(uploader)s/%(upload_date)s - %(title)s/%(title).40s [%(id)s].%(ext)s" --merge-output-format "mkv" --batch-file "Source - Channels.txt" --cookies cookies.txt
It doesn't seem to matter whether--sleep-requests1is used.That's usually a connection issue between you and YouTube. I've seen that error sometimes when running over proxies/vpns. However I might look into allowing the retry behavior cover more cases such as this, since it is a bit annoying and not uncommon.
I think we've discussed the idea of recovering before (for playlists etc.) and figured it wasn't really feasible. As with YouTube comments/playlists, etc. you cannot request a specific page number. It might be technically possible to pass the last used continuation token, but I'm not sure how we could implement that in yt-dlp (and there are many issues with that approach too).
It might be technically possible to pass the last used continuation token, but I'm not sure how we could implement that in yt-dlp (and there are many issues with that approach too).
Do continuation tokens expire or otherwise become invalid after some time? If not, it is possible by treating comment extraction as a "fragment download" and writing each new "fragment" along with a
.ytdlfile with continuation token. But that would need a complete rewrite of the current comment extraction codeDo continuation tokens expire or otherwise become invalid after some time?
Not sure, we'd need to experiment with this.
If not, it is possible by treating comment extraction as a "fragment download" and writing each new "fragment" along with a .ytdl file with continuation token. But that would need a complete rewrite of the current comment extraction code
Hmm maybe. Also - if we were to implement something like that then we might want to make it work for other places where continuations are used (e.g. playlists) too.
That's usually a connection issue between you and YouTube. I've seen that error sometimes when running over proxies/vpns. However I might look into allowing the retry behavior cover more cases such as this, since it is a bit annoying and not uncommon.
I think we've discussed the idea of recovering before (for playlists etc.) and figured it wasn't really feasible. As with YouTube comments/playlists, etc. you cannot request a specific page number. It might be technically possible to pass the last used continuation token, but I'm not sure how we could implement that in yt-dlp (and there are many issues with that approach too).
Ah, I see. I don't use a proxy or vpn myself FWIW.
I've read the follow ups posted by pukkandan and yourself. I really hope it's possible to add the fragment download functionality to comments!
As well, just wondering: is it possible to make the archiving of comments a separate thing, using a separate archive.log (archive-comments.log), so that we could first download videos without comments, and then go back and download the comments to those videos at a later date?
I'm not exactly sure how it would work: if it just works on the urls in the videos' archive.log, it will download comments for videos that may have otherwise been manually deleted after the download.EDIT: I just had an idea that it could use the archive.log, but not have permission to create directories, meaning it will only be able to write the comment.json files if the directory already exists, otherwise it will skip. Is that in the realm of possibilities?
If it's possible to use the output naming convention (in my case
"%(uploader)s/%(upload_date)s - %(title)s/%(title).40s [%(id)s].%(ext)s") to look through the directories and get the urls from the%(ext)spart of the filenames, perhaps that would be a way of doing it. Just spitballing here, no idea if that's possible.The reason I ask is because I'm currently redownloading my entire youtube-dl archive using yt-dlp in order to get comments. I just took the urls from the archive.log and find+replace'd them to make standard video urls which I then put in the batch.txt. It's just annoying having to delete the videos that I don't want again (for some reason, some channel urls also downloaded those channels' personal playlists such as their favourites, which I don't want).
If it was possible to just get the comments without redownloading the videos, I could have done that and not have to use 2x the storage to fit everything before finally comparing the old with the new and deleting the old.there is another youtube-dl fork that supports downloading youtube comments - https://www.reddit.com/r/DataHoarder/comments/ha9uwv/i_made_a_youtubedl_fork_that_supports_downloading/
there is another youtube-dl fork that supports downloading youtube comments - https://www.reddit.com/r/DataHoarder/comments/ha9uwv/i_made_a_youtubedl_fork_that_supports_downloading/
Interesting, do you know if this works without erroring on videos with thousands of comments?
there is another youtube-dl fork that supports downloading youtube comments - https://www.reddit.com/r/DataHoarder/comments/ha9uwv/i_made_a_youtubedl_fork_that_supports_downloading/
Interesting, do you know if this works without erroring on videos with thousands of comments?
don't know, have not tried it yet.
there is another youtube-dl fork that supports downloading youtube comments -
https://www.reddit.com/r/DataHoarder/comments/ha9uwv/i_made_a_youtubedl_fork_that_supports_downloading/Interesting, do you know if this works without erroring on videos with thousands of comments?
yt-dlp's comment download was adapted from this fork in #31 and later improved in #95, #167, #221
I've recently been able to download several videos with over 30 000 comments using the default delays, so I don't think that YouTube is imposing strict rate limiting here. At least when it comes to SSL timeouts like the one you received, it is probably dependent on your connection and the path it takes to the YouTube servers, as others have mentioned.
I do occasionally seem to receive an error page in response (showing up as a JSON parse error) as well, but this seems to happen less often for me now than it did a few months ago. Still, it'd certainly be preferable if it was possible to retry these requests instead of having them be fatal to the download process.
I've recently been able to download several videos with over 30 000 comments using the default delays, so I don't think that YouTube is imposing strict rate limiting here. At least when it comes to SSL timeouts like the one you received, it is probably dependent on your connection and the path it takes to the YouTube servers, as others have mentioned.
I do occasionally seem to receive an error page in response (showing up as a JSON parse error) as well, but this seems to happen less often for me now than it did a few months ago. Still, it'd certainly be preferable if it was possible to retry these requests instead of having them be fatal to the download process.
What is this path you mention? Wouldn't a connection without proxy nor VPN be taking the most direct path, or is it preferable to use a proxy or VPN in this case?
I have wired internet with 1ms ping, so I feel my connection is as stable as I can ask for.As a side, over the past 3 days I've been having random videos throttle at 50KB/s which I could either wait them out, or restart the script, but restarting takes about half an hour to retry the videos with thousands of comments that it consistently errors on before it finally gets to the throttled video, only to then throttle maybe 3 videos later. If it weren't for the videos with thousands of comments always failing, retrying the downloads would be very trivial and take no time at all.
I've recently been able to download several videos with over 30 000 comments using the default delays, so I don't think that YouTube is imposing strict rate limiting here. At least when it comes to SSL timeouts like the one you received, it is probably dependent on your connection and the path it takes to the YouTube servers, as others have mentioned.
I do occasionally seem to receive an error page in response (showing up as a JSON parse error) as well, but this seems to happen less often for me now than it did a few months ago. Still, it'd certainly be preferable if it was possible to retry these requests instead of having them be fatal to the download process.What is this path you mention? Wouldn't a connection without proxy nor VPN be taking the most direct path, or is it preferable to use a proxy or VPN in this case?
I have wired internet with 1ms ping, so I feel my connection is as stable as I can ask for.As a side, over the past 3 days I've been having random videos throttle at 50KB/s which I could either wait them out, or restart the script, but restarting takes about half an hour to retry the videos with thousands of comments that it consistently errors on before it finally gets to the throttled video, only to then throttle maybe 3 videos later. If it weren't for the videos with thousands of comments always failing, retrying the downloads would be very trivial and take no time at all.
does your ISP throttle your internet connection ??
I've recently been able to download several videos with over 30 000 comments using the default delays, so I don't think that YouTube is imposing strict rate limiting here. At least when it comes to SSL timeouts like the one you received, it is probably dependent on your connection and the path it takes to the YouTube servers, as others have mentioned.
I do occasionally seem to receive an error page in response (showing up as a JSON parse error) as well, but this seems to happen less often for me now than it did a few months ago. Still, it'd certainly be preferable if it was possible to retry these requests instead of having them be fatal to the download process.What is this path you mention? Wouldn't a connection without proxy nor VPN be taking the most direct path, or is it preferable to use a proxy or VPN in this case?
I have wired internet with 1ms ping, so I feel my connection is as stable as I can ask for.
As a side, over the past 3 days I've been having random videos throttle at 50KB/s which I could either wait them out, or restart the script, but restarting takes about half an hour to retry the videos with thousands of comments that it consistently errors on before it finally gets to the throttled video, only to then throttle maybe 3 videos later. If it weren't for the videos with thousands of comments always failing, retrying the downloads would be very trivial and take no time at all.does your ISP throttle your internet connection ??
Absolutely not. The throttling is being done by YouTube, not my ISP. I can actually boot up my mini PC and run the script (with different video urls) on it and get full speed whenever my main PC gets throttled while it's still running (ie. 2 instances of the script on 2 separate video lists, specifically 2 halves of the Game Grumps channel's videos), so that's why I'm not sure if it's throttling me based on my ISP or on my machine or what.
Is there a soft limit to how many videos you can download at full speed within a time frame before youtube starts throttling you? I'm wondering because when I was getting throttled earlier, I'd restart the script, and depending on how long I waited it would download a different amount of videos at full speed. For example, it would download 1-2 videos before throttling if I waited for ~10 minutes, and if I waited for about an hour, it would download closer to 10 videos before throttling. I downloaded exactly 5664 videos since May 29th (9 days ago), 4164 on my main PC and 1500 on my mini PC.
Another issue I've noticed is that even when comments successfully get fully downloaded, if for whatever reason the video download fails or the script is stopped during the download, in order to redownload the video, the entire comments must be redownloaded as well.
It would be great if it can detect the presence of a fully downloaded comments section, which I assume is made apparent with the presence of the info.json (since that file will not exist until all comments are successfully downloaded) and skip straight to the redownloading of the video in such a case. Redownloading the comments would just make it likely to fail again.Okay, this is weird: my friend granted me access to his laptop via AnyDesk while he sleeps to allow me to run yt-dlp on it, and it works flawlessly. Not a single video has failed to download the comments.
It runs so well in fact that it has no issues running 2 simultaneous instances.
He's running inferior specs compared to my main pc, but superior specs compared to my mini pc: i5-7300HQ, 8GB RAM, on wi-fi. My main PC has a ryzen 1800x and 16GB RAM with ethernet, and I tried using my mini pc (A10-8700P, 8GB RAM) with wi-fi and also with ethernet. No matter what, both of my machines often fail at downloading large quantities of comments, and simply will not download more than 10k comments without failing (100+ attempts on a few vids with no luck), while his laptop hasn't failed a single time. I'm actually downloading videos using the
worstformat quality and deleting the vids so that I can transfer the comment info.json's to my pc, where I then download the vid without comments in the best format quality.I have no idea what's going on here. Could it be simply because he's in the states while I'm in Canada? Is that even a thing that can matter? He has no VPN, nor do I. I installed the same Python 3.7.4 (32-bit) on his system as I did on both of mine, installed ffmpeg using chocolatey the same way as I did on my machines, and installed the latest yt-dlp 2021-06-09 that my machines also use. Nothing's different from these 3 things.
The reason I ask is because I'm currently redownloading my entire youtube-dl archive using yt-dlp in order to get comments. I just took the urls from the archive.log and find+replace'd them to make standard video urls which I then put in the batch.txt. It's just annoying having to delete the videos that I don't want again
If it was possible to just get the comments without redownloading the videos, I could have done that and not have to use 2x the storage to fit everything before finally comparing the old with the new and deleting the old.
I'm actually downloading videos using the worst format quality and deleting the vids so that I can transfer the comment info.json's to my pc, where I then download the vid without comments in the best format quality.
Use
--skip-downloadis it possible to make the archiving of comments a separate thing, using a separate archive.log (archive-comments.log), so that we could first download videos without comments, and then go back and download the comments to those videos at a later date?
Just run these seperately:
yt-dlp --write-comments --skip-download --download-archive archive-comments.log --force-write-archive -- URL yt-dlp --download-archive archive.log -- URL(for some reason, some channel urls also downloaded those channels' personal playlists such as their favourites, which I don't want).
youtube-dl nowadays downloads whatever youtube page the link points to. So if you point it to the channels home page, it will download all the videos in the home page and not the channel's uploaded videos. The workaround is to give link to the videos page like this:
https://www.youtube.com/c/3blue1brown/videos. yt-dlp on the other hand, automatically redirects the home page to the videos page since that is the expected behaviour by most users.As a side, over the past 3 days I've been having random videos throttle at 50KB/s which I could either wait them out, or restart the script, but restarting takes about half an hour to retry the videos with thousands of comments that it consistently errors on before it finally gets to the throttled video, only to then throttle maybe 3 videos later.
This is a recent thing. There is already an issue open in youtube-dl. Currently the only known workaround is to use a multi-connection downloader like
aria2cHe's running inferior specs compared to my main pc, but superior specs compared to my mini pc: i5-7300HQ, 8GB RAM, on wi-fi.
Specs don't matter. The only thing that matters is the internet's stability
I am closing this since I believe all the questions have been answered
Reacted by Rings4The reason I ask is because I'm currently redownloading my entire youtube-dl archive using yt-dlp in order to get comments. I just took the urls from the archive.log and find+replace'd them to make standard video urls which I then put in the batch.txt. It's just annoying having to delete the videos that I don't want again
If it was possible to just get the comments without redownloading the videos, I could have done that and not have to use 2x the storage to fit everything before finally comparing the old with the new and deleting the old.
I'm actually downloading videos using the worst format quality and deleting the vids so that I can transfer the comment info.json's to my pc, where I then download the vid without comments in the best format quality.
Use
--skip-downloadis it possible to make the archiving of comments a separate thing, using a separate archive.log (archive-comments.log), so that we could first download videos without comments, and then go back and download the comments to those videos at a later date?
Just run these seperately:
yt-dlp --write-comments --skip-download --download-archive archive-comments.log --force-write-archive -- URL yt-dlp --download-archive archive.log -- URL(for some reason, some channel urls also downloaded those channels' personal playlists such as their favourites, which I don't want).
youtube-dl nowadays downloads whatever youtube page the link points to. So if you point it to the channels home page, it will download all the videos in the home page and not the channel's uploaded videos. The workaround is to give link to the videos page like this:
https://www.youtube.com/c/3blue1brown/videos. yt-dlp on the other hand, automatically redirects the home page to the videos page since that is the expected behaviour by most users.As a side, over the past 3 days I've been having random videos throttle at 50KB/s which I could either wait them out, or restart the script, but restarting takes about half an hour to retry the videos with thousands of comments that it consistently errors on before it finally gets to the throttled video, only to then throttle maybe 3 videos later.
This is a recent thing. There is already an issue open in youtube-dl. Currently the only known workaround is to use a multi-connection downloader like
aria2cHe's running inferior specs compared to my main pc, but superior specs compared to my mini pc: i5-7300HQ, 8GB RAM, on wi-fi.
Specs don't matter. The only thing that matters is the internet's stability
I am closing this since I believe all the questions have been answered
Thank you very much, your answers were very helpful! Thanks for telling me about aria2c, I installed it and am using the following commands
--external-downloader aria2c --external-downloader-args '-j 16 -x 16 -s 16 -k 1M'and it's running like a dream. So much faster than before.Everything you addressed has been super helpful, and I appreciate it very much.
Just the last thing you mentioned about internet stability: my internet is very stable. I've never had a stability issue before. Therefore I don't know why comments only fail for me on my 1ms ping wired internet when it doesn't fail for my friend's 12ms ping wireless internet. Are there other way of measuring internet stability that relate more closely to what yt-dlp expects when downloading comments, perhaps?
I see the latest version of yt-dlp (2021.07.24) now has the ability to resume downloading comments upon failure! This is huge! Thanks so much for implementing this!
@Rings4 It retries (
--extractor-retries), but cannot resume if the retries are exhaustedI see. In my case only a single retry was necessary each time for it to go back on track. In one video it had to retry once at 6k comments and a second time at 7k, and it managed to finish. What are the max retries, and does it only exhaust itself of retries when they're done in a row?
3 by default and can be changed with
--extractor-retriesand does it only exhaust itself of retries when they're done in a row?
Yes, any success resets the counter
That's great to hear, thanks for the info! You're doing god's work and your responsiveness is to be commended.
Checklist
Question
Trying to download videos with comments is very failure prone: why is that? I thought adding
--cookies cookies.txtand especially--sleep-requests 1would prevent them from erroring out but it still happens. Thus, videos with 20k+ comments seem impossible to download. Especially annoying with--sleep-requests 1because it will take over 10 minutes for it to download the first 5k or so comments, only to fail anyway.It doesn't even matter how many comments a vid has, even some vids with 800 comments will fail at like 200 or 600. What can I do to prevent these from erroring out?