Skip to content

[youtube] Add options to skip webpage and js player for video extraction#860

Merged
pukkandan merged 12 commits intoyt-dlp:masterfrom
coletdjnz:player_js_skip
Sep 6, 2021
Merged

[youtube] Add options to skip webpage and js player for video extraction#860
pukkandan merged 12 commits intoyt-dlp:masterfrom
coletdjnz:player_js_skip

Conversation

@coletdjnz
Copy link
Copy Markdown
Member

@coletdjnz coletdjnz commented Sep 2, 2021 •
edited
Loading

Please follow the guide below

  • You will be asked some questions, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your pull request (like that [x])
  • Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check one of the following options:

  • I am the original author of this code and I am willing to release it under Unlicense
  • I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

  • (Future?) Bug fix
  • Improvement
  • New extractor
  • New feature

Description of your pull request and other information

Not sure what to call this.

This code was originally from #682.

This PR adds:

  • fallback player url extraction when it fails to be extracted from the webpage
  • don't download js player unnecessarily for clients that don't require it
  • try extract js player url from any additional client configs
  • ability to skip the js player usage/download (inc iframe fallback).
    • This overrides the clients' config setting.
    • This can be enabled with --extractor-args youtube:player_skip=js
    • Formats that require signature decryption will be skipped automatically (since no player url)
  • ability to skip the webpage download
    • This can be enabled with --extractor-args youtube:player_skip=webpage

KNOWN ISSUES:

  • authentication for multi-channel accounts and multi-account cookies may not work correctly if the webpage (and any additional client configs) are not downloaded.

TODO:

  • documentation
  • player_skip=js or something else?

@coletdjnz
Copy link
Copy Markdown
Member Author

coletdjnz commented Sep 2, 2021 •
edited
Loading

Some example usage

  1. --extractor-args youtube:player_skip=webpage,configs,js;player_client=android,web
    • This makes extraction solely rely on the API - so no webpage downloads. Might help with avoiding 429s in some situations.
    • Since the web client usually needs the player js, formats from the web client requiring signature decryption will be skipped.
    • 3 initial API requests (android, web, initial data)

2. --extractor-args youtube:player_skip=webpage,configs,js;player_client=android
- Same as above, but without the web client (some minor metadata may be missed - be sure to check)
- 2 initial API requests (android, initial data) Do not use - missing metadata

3. --extractor-args youtube:player_skip=webpage;player_client=android
- since the android client doesn't require the player js, this is the same as 2. Do not use - missing metadata

  1. --extractor-args youtube:player_skip=webpage

    • basically forces fallbacks as if the webpage failed to download.
    • player url should be downloaded using the iframe API fallback (once, only if required)

  2. --extractor-args youtube:player_skip=webpage;player_client=web_music

    • config webpage for web_music is downloaded (thus player js details), so web formats should work correctly.

Not recommended:

  1. --extractor-args youtube:player_skip=webpage,configs,js;player_client=web
    • In some cases you'll get a no formats error (formats from the web client requiring signature decryption will be skipped).
  2. --extractor-args youtube:player_skip=js;player_client=web

@coletdjnz coletdjnz mentioned this pull request Sep 2, 2021
Closed
3 tasks
pukkandan
pukkandan reviewed Sep 2, 2021
View reviewed changes
Comment thread yt_dlp/extractor/youtube.py Outdated
Comment thread yt_dlp/extractor/youtube.py Outdated
Comment thread yt_dlp/extractor/youtube.py Outdated
@coletdjnz coletdjnz added the pending-fixes PR has had changes requested label Sep 3, 2021
coletdjnz and others added 5 commits September 5, 2021 06:32
@coletdjnz @pukkandan
remove unnecessary pop param
c8ac420 
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
@coletdjnz @pukkandan
improve player js url extraction
37f110e 
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
@coletdjnz @pukkandan
Update yt_dlp/extractor/youtube.py
1b7a20e 
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
@coletdjnz
cleanup
1caa46e
@coletdjnz
Add documentation
a404506
@coletdjnz coletdjnz removed the pending-fixes PR has had changes requested label Sep 5, 2021
@coletdjnz coletdjnz requested a review from pukkandan September 5, 2021 06:57
@coletdjnz coletdjnz changed the title [youtube] Formalise API fallbacks for YouTubeIE [youtube] Add options to skip webpage and js player for video extraction Sep 5, 2021
pukkandan
pukkandan reviewed Sep 5, 2021
View reviewed changes
Comment thread yt_dlp/extractor/youtube.py Outdated
Comment thread yt_dlp/extractor/youtube.py
Comment thread yt_dlp/extractor/youtube.py Outdated
pukkandan
pukkandan reviewed Sep 6, 2021
View reviewed changes
Comment thread README.md Outdated
@pukkandan pukkandan merged commit b6de707 into yt-dlp:master Sep 6, 2021
@MinePlayersPE MinePlayersPE mentioned this pull request Sep 8, 2021
Closed
5 tasks
nixxo pushed a commit to nixxo/yt-dlp that referenced this pull request Nov 22, 2021
@coletdjnz
[youtube] Improvements to JS player extraction (See desc) (yt-dlp#860)
025b111 
* fallback player url extraction when it fails to be extracted from the webpage
* don't download js player unnecessarily for clients that don't require it
* try to extract js player url from any additional client configs
* ability to skip the js player usage/download using `player_skip=js`
* ability to skip the initial webpage download using `player_skip=webpage`

known issue:
* authentication for multi-channel accounts and multi-account cookies may not work correctly if the webpage or client configs are skipped
*  formats from the web client requiring signature decryption will be skipped if player js extraction is skipped

Authored by: coletdjnz
@coletdjnz coletdjnz deleted the player_js_skip branch March 4, 2022 06:04
@dasl- dasl- mentioned this pull request Apr 11, 2022
Closed
@pukkandan pukkandan mentioned this pull request Aug 4, 2022
Closed
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pukkandan pukkandan pukkandan left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@coletdjnz @pukkandan