diff options
88 files changed, 1547 insertions, 6426 deletions
diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml deleted file mode 100644 index 04de08708..000000000 --- a/.github/FUNDING.yml +++ /dev/null @@ -1,13 +0,0 @@ -# These are supported funding model platforms - -github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2] -patreon: # Replace with a single Patreon username -open_collective: # Replace with a single Open Collective username -ko_fi: # Replace with a single Ko-fi username -tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel -community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry -liberapay: # Replace with a single Liberapay username -issuehunt: # Replace with a single IssueHunt username -otechie: # Replace with a single Otechie username - -custom: ['https://github.com/yt-dlp/yt-dlp/blob/master/Collaborators.md#collaborators'] diff --git a/.github/ISSUE_TEMPLATE/1_broken_site.yml b/.github/ISSUE_TEMPLATE/1_broken_site.yml deleted file mode 100644 index 397f92824..000000000 --- a/.github/ISSUE_TEMPLATE/1_broken_site.yml +++ /dev/null @@ -1,63 +0,0 @@ -name: Broken site -description: Report broken or misfunctioning site -labels: [triage, site-bug] -body: - - type: checkboxes - id: checklist - attributes: - label: Checklist - description: | - Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: - options: - - label: I'm reporting a broken site - required: true - - label: I've verified that I'm running yt-dlp version **2022.02.04**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) - required: true - - label: I've checked that all provided URLs are alive and playable in a browser - required: true - - label: I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command) - required: true - - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates - required: true - - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue) - required: true - - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required - - type: input - id: region - attributes: - label: Region - description: "Enter the region the site is accessible from" - placeholder: "India" - - type: textarea - id: description - attributes: - label: Description - description: | - Provide an explanation of your issue in an arbitrary form. - Provide any additional information, any suggested solutions, and as much context and examples as possible - placeholder: WRITE DESCRIPTION HERE - validations: - required: true - - type: textarea - id: log - attributes: - label: Verbose log - description: | - Provide the complete verbose output of yt-dlp **that clearly demonstrates the problem**. - Add the `-vU` flag to your command line you run yt-dlp with (`yt-dlp -vU <your command line>`), copy the WHOLE output and insert it below. - It should look similar to this: - placeholder: | - [debug] Command-line config: ['-vU', 'http://www.youtube.com/watch?v=BaW_jenozKc'] - [debug] Portable config file: yt-dlp.conf - [debug] Portable config: ['-i'] - [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252 - [debug] yt-dlp version 2022.02.04 (exe) - [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0 - [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1 - [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets - [debug] Proxy map: {} - yt-dlp is up to date (2022.02.04) - <more lines> - render: shell - validations: - required: true diff --git a/.github/ISSUE_TEMPLATE/2_site_support_request.yml b/.github/ISSUE_TEMPLATE/2_site_support_request.yml deleted file mode 100644 index 9b02f6f72..000000000 --- a/.github/ISSUE_TEMPLATE/2_site_support_request.yml +++ /dev/null @@ -1,74 +0,0 @@ -name: Site support request -description: Request support for a new site -labels: [triage, site-request] -body: - - type: checkboxes - id: checklist - attributes: - label: Checklist - description: | - Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: - options: - - label: I'm reporting a new site support request - required: true - - label: I've verified that I'm running yt-dlp version **2022.02.04**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) - required: true - - label: I've checked that all provided URLs are alive and playable in a browser - required: true - - label: I've checked that none of provided URLs [violate any copyrights](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge - required: true - - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates - required: true - - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue) - required: true - - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and am willing to share it if required - - type: input - id: region - attributes: - label: Region - description: "Enter the region the site is accessible from" - placeholder: "India" - - type: textarea - id: example-urls - attributes: - label: Example URLs - description: | - Provide all kinds of example URLs for which support should be added - placeholder: | - - Single video: https://www.youtube.com/watch?v=BaW_jenozKc - - Single video: https://youtu.be/BaW_jenozKc - - Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc - validations: - required: true - - type: textarea - id: description - attributes: - label: Description - description: | - Provide any additional information - placeholder: WRITE DESCRIPTION HERE - validations: - required: true - - type: textarea - id: log - attributes: - label: Verbose log - description: | - Provide the complete verbose output **using one of the example URLs provided above**. - Add the `-vU` flag to your command line you run yt-dlp with (`yt-dlp -vU <your command line>`), copy the WHOLE output and insert it below. - It should look similar to this: - placeholder: | - [debug] Command-line config: ['-vU', 'http://www.youtube.com/watch?v=BaW_jenozKc'] - [debug] Portable config file: yt-dlp.conf - [debug] Portable config: ['-i'] - [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252 - [debug] yt-dlp version 2022.02.04 (exe) - [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0 - [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1 - [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets - [debug] Proxy map: {} - yt-dlp is up to date (2022.02.04) - <more lines> - render: shell - validations: - required: true diff --git a/.github/ISSUE_TEMPLATE/3_site_feature_request.yml b/.github/ISSUE_TEMPLATE/3_site_feature_request.yml deleted file mode 100644 index 3ad71edc7..000000000 --- a/.github/ISSUE_TEMPLATE/3_site_feature_request.yml +++ /dev/null @@ -1,72 +0,0 @@ -name: Site feature request -description: Request a new functionality for a supported site -labels: [triage, site-enhancement] -body: - - type: checkboxes - id: checklist - attributes: - label: Checklist - description: | - Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: - options: - - label: I'm reporting a site feature request - required: true - - label: I've verified that I'm running yt-dlp version **2022.02.04**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) - required: true - - label: I've checked that all provided URLs are alive and playable in a browser - required: true - - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates - required: true - - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue) - required: true - - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required - - type: input - id: region - attributes: - label: Region - description: "Enter the region the site is accessible from" - placeholder: "India" - - type: textarea - id: example-urls - attributes: - label: Example URLs - description: | - Example URLs that can be used to demonstrate the requested feature - placeholder: | - https://www.youtube.com/watch?v=BaW_jenozKc - validations: - required: true - - type: textarea - id: description - attributes: - label: Description - description: | - Provide an explanation of your site feature request in an arbitrary form. - Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient). - Provide any additional information, any suggested solutions, and as much context and examples as possible - placeholder: WRITE DESCRIPTION HERE - validations: - required: true - - type: textarea - id: log - attributes: - label: Verbose log - description: | - Provide the complete verbose output of yt-dlp that demonstrates the need for the enhancement. - Add the `-vU` flag to your command line you run yt-dlp with (`yt-dlp -vU <your command line>`), copy the WHOLE output and insert it below. - It should look similar to this: - placeholder: | - [debug] Command-line config: ['-vU', 'http://www.youtube.com/watch?v=BaW_jenozKc'] - [debug] Portable config file: yt-dlp.conf - [debug] Portable config: ['-i'] - [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252 - [debug] yt-dlp version 2022.02.04 (exe) - [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0 - [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1 - [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets - [debug] Proxy map: {} - yt-dlp is up to date (2022.02.04) - <more lines> - render: shell - validations: - required: true diff --git a/.github/ISSUE_TEMPLATE/4_bug_report.yml b/.github/ISSUE_TEMPLATE/4_bug_report.yml deleted file mode 100644 index 791a7ee14..000000000 --- a/.github/ISSUE_TEMPLATE/4_bug_report.yml +++ /dev/null @@ -1,57 +0,0 @@ -name: Bug report -description: Report a bug unrelated to any particular site or extractor -labels: [triage, bug] -body: - - type: checkboxes - id: checklist - attributes: - label: Checklist - description: | - Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: - options: - - label: I'm reporting a bug unrelated to a specific site - required: true - - label: I've verified that I'm running yt-dlp version **2022.02.04**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) - required: true - - label: I've checked that all provided URLs are alive and playable in a browser - required: true - - label: I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command) - required: true - - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates - required: true - - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue) - required: true - - type: textarea - id: description - attributes: - label: Description - description: | - Provide an explanation of your issue in an arbitrary form. - Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient). - Provide any additional information, any suggested solutions, and as much context and examples as possible - placeholder: WRITE DESCRIPTION HERE - validations: - required: true - - type: textarea - id: log - attributes: - label: Verbose log - description: | - Provide the complete verbose output of yt-dlp **that clearly demonstrates the problem**. - Add the `-vU` flag to **your** command line you run yt-dlp with (`yt-dlp -vU <your command line>`), copy the WHOLE output and insert it below. - It should look similar to this: - placeholder: | - [debug] Command-line config: ['-vU', 'http://www.youtube.com/watch?v=BaW_jenozKc'] - [debug] Portable config file: yt-dlp.conf - [debug] Portable config: ['-i'] - [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252 - [debug] yt-dlp version 2022.02.04 (exe) - [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0 - [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1 - [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets - [debug] Proxy map: {} - yt-dlp is up to date (2022.02.04) - <more lines> - render: shell - validations: - required: true diff --git a/.github/ISSUE_TEMPLATE/5_feature_request.yml b/.github/ISSUE_TEMPLATE/5_feature_request.yml deleted file mode 100644 index a3a824f52..000000000 --- a/.github/ISSUE_TEMPLATE/5_feature_request.yml +++ /dev/null @@ -1,32 +0,0 @@ -name: Feature request -description: Request a new functionality unrelated to any particular site or extractor -labels: [triage, enhancement] -body: - - type: checkboxes - id: checklist - attributes: - label: Checklist - description: | - Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: - options: - - label: I'm reporting a feature request - required: true - - label: I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme) - required: true - - label: I've verified that I'm running yt-dlp version **2022.02.04**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) - required: true - - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates - required: true - - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue) - required: true - - type: textarea - id: description - attributes: - label: Description - description: | - Provide an explanation of your site feature request in an arbitrary form. - Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient). - Provide any additional information, any suggested solutions, and as much context and examples as possible - placeholder: WRITE DESCRIPTION HERE - validations: - required: true diff --git a/.github/ISSUE_TEMPLATE/6_question.yml b/.github/ISSUE_TEMPLATE/6_question.yml deleted file mode 100644 index 061158ed3..000000000 --- a/.github/ISSUE_TEMPLATE/6_question.yml +++ /dev/null @@ -1,53 +0,0 @@ -name: Ask question -description: Ask yt-dlp related question -labels: [question] -body: - - type: checkboxes - id: checklist - attributes: - label: Checklist - description: | - Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: - options: - - label: I'm asking a question and **not** reporting a bug/feature request - required: true - - label: I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme) - required: true - - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue) - required: true - - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions including closed ones - required: true - - type: textarea - id: question - attributes: - label: Question - description: | - Ask your question in an arbitrary form. - Please make sure it's worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient). - Provide any additional information and as much context and examples as possible. - If your question contains "isn't working" or "can you add", this is most likely the wrong template. - If you are in doubt if this is the right template, use another template! - placeholder: WRITE QUESTION HERE - validations: - required: true - - type: textarea - id: log - attributes: - label: Verbose log - description: | - If your question involes a yt-dlp command, provide the complete verbose output of that command. - Add the `-vU` flag to **your** command line you run yt-dlp with (`yt-dlp -vU <your command line>`), copy the WHOLE output and insert it below. - It should look similar to this: - placeholder: | - [debug] Command-line config: ['-vU', 'http://www.youtube.com/watch?v=BaW_jenozKc'] - [debug] Portable config file: yt-dlp.conf - [debug] Portable config: ['-i'] - [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252 - [debug] yt-dlp version 2021.12.01 (exe) - [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0 - [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1 - [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets - [debug] Proxy map: {} - yt-dlp is up to date (2021.12.01) - <more lines> - render: shell diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml deleted file mode 100644 index 9cdffa4b1..000000000 --- a/.github/ISSUE_TEMPLATE/config.yml +++ /dev/null @@ -1,8 +0,0 @@ -blank_issues_enabled: false -contact_links: - - name: Get help from the community on Discord - url: https://discord.gg/H5MNcFW63r - about: Join the yt-dlp Discord for community-powered support! - - name: Matrix Bridge to the Discord server - url: https://matrix.to/#/#yt-dlp:matrix.org - about: For those who do not want to use Discord diff --git a/.github/ISSUE_TEMPLATE_tmpl/1_broken_site.yml b/.github/ISSUE_TEMPLATE_tmpl/1_broken_site.yml deleted file mode 100644 index fd6435ba6..000000000 --- a/.github/ISSUE_TEMPLATE_tmpl/1_broken_site.yml +++ /dev/null @@ -1,63 +0,0 @@ -name: Broken site -description: Report broken or misfunctioning site -labels: [triage, site-bug] -body: - - type: checkboxes - id: checklist - attributes: - label: Checklist - description: | - Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: - options: - - label: I'm reporting a broken site - required: true - - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) - required: true - - label: I've checked that all provided URLs are alive and playable in a browser - required: true - - label: I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command) - required: true - - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates - required: true - - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue) - required: true - - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required - - type: input - id: region - attributes: - label: Region - description: "Enter the region the site is accessible from" - placeholder: "India" - - type: textarea - id: description - attributes: - label: Description - description: | - Provide an explanation of your issue in an arbitrary form. - Provide any additional information, any suggested solutions, and as much context and examples as possible - placeholder: WRITE DESCRIPTION HERE - validations: - required: true - - type: textarea - id: log - attributes: - label: Verbose log - description: | - Provide the complete verbose output of yt-dlp **that clearly demonstrates the problem**. - Add the `-vU` flag to your command line you run yt-dlp with (`yt-dlp -vU <your command line>`), copy the WHOLE output and insert it below. - It should look similar to this: - placeholder: | - [debug] Command-line config: ['-vU', 'http://www.youtube.com/watch?v=BaW_jenozKc'] - [debug] Portable config file: yt-dlp.conf - [debug] Portable config: ['-i'] - [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252 - [debug] yt-dlp version %(version)s (exe) - [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0 - [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1 - [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets - [debug] Proxy map: {} - yt-dlp is up to date (%(version)s) - <more lines> - render: shell - validations: - required: true diff --git a/.github/ISSUE_TEMPLATE_tmpl/2_site_support_request.yml b/.github/ISSUE_TEMPLATE_tmpl/2_site_support_request.yml deleted file mode 100644 index f380c1331..000000000 --- a/.github/ISSUE_TEMPLATE_tmpl/2_site_support_request.yml +++ /dev/null @@ -1,74 +0,0 @@ -name: Site support request -description: Request support for a new site -labels: [triage, site-request] -body: - - type: checkboxes - id: checklist - attributes: - label: Checklist - description: | - Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: - options: - - label: I'm reporting a new site support request - required: true - - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) - required: true - - label: I've checked that all provided URLs are alive and playable in a browser - required: true - - label: I've checked that none of provided URLs [violate any copyrights](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge - required: true - - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates - required: true - - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue) - required: true - - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and am willing to share it if required - - type: input - id: region - attributes: - label: Region - description: "Enter the region the site is accessible from" - placeholder: "India" - - type: textarea - id: example-urls - attributes: - label: Example URLs - description: | - Provide all kinds of example URLs for which support should be added - placeholder: | - - Single video: https://www.youtube.com/watch?v=BaW_jenozKc - - Single video: https://youtu.be/BaW_jenozKc - - Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc - validations: - required: true - - type: textarea - id: description - attributes: - label: Description - description: | - Provide any additional information - placeholder: WRITE DESCRIPTION HERE - validations: - required: true - - type: textarea - id: log - attributes: - label: Verbose log - description: | - Provide the complete verbose output **using one of the example URLs provided above**. - Add the `-vU` flag to your command line you run yt-dlp with (`yt-dlp -vU <your command line>`), copy the WHOLE output and insert it below. - It should look similar to this: - placeholder: | - [debug] Command-line config: ['-vU', 'http://www.youtube.com/watch?v=BaW_jenozKc'] - [debug] Portable config file: yt-dlp.conf - [debug] Portable config: ['-i'] - [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252 - [debug] yt-dlp version %(version)s (exe) - [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0 - [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1 - [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets - [debug] Proxy map: {} - yt-dlp is up to date (%(version)s) - <more lines> - render: shell - validations: - required: true diff --git a/.github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.yml b/.github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.yml deleted file mode 100644 index 88b1f1217..000000000 --- a/.github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.yml +++ /dev/null @@ -1,72 +0,0 @@ -name: Site feature request -description: Request a new functionality for a supported site -labels: [triage, site-enhancement] -body: - - type: checkboxes - id: checklist - attributes: - label: Checklist - description: | - Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: - options: - - label: I'm reporting a site feature request - required: true - - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) - required: true - - label: I've checked that all provided URLs are alive and playable in a browser - required: true - - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates - required: true - - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue) - required: true - - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required - - type: input - id: region - attributes: - label: Region - description: "Enter the region the site is accessible from" - placeholder: "India" - - type: textarea - id: example-urls - attributes: - label: Example URLs - description: | - Example URLs that can be used to demonstrate the requested feature - placeholder: | - https://www.youtube.com/watch?v=BaW_jenozKc - validations: - required: true - - type: textarea - id: description - attributes: - label: Description - description: | - Provide an explanation of your site feature request in an arbitrary form. - Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient). - Provide any additional information, any suggested solutions, and as much context and examples as possible - placeholder: WRITE DESCRIPTION HERE - validations: - required: true - - type: textarea - id: log - attributes: - label: Verbose log - description: | - Provide the complete verbose output of yt-dlp that demonstrates the need for the enhancement. - Add the `-vU` flag to your command line you run yt-dlp with (`yt-dlp -vU <your command line>`), copy the WHOLE output and insert it below. - It should look similar to this: - placeholder: | - [debug] Command-line config: ['-vU', 'http://www.youtube.com/watch?v=BaW_jenozKc'] - [debug] Portable config file: yt-dlp.conf - [debug] Portable config: ['-i'] - [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252 - [debug] yt-dlp version %(version)s (exe) - [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0 - [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1 - [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets - [debug] Proxy map: {} - yt-dlp is up to date (%(version)s) - <more lines> - render: shell - validations: - required: true diff --git a/.github/ISSUE_TEMPLATE_tmpl/4_bug_report.yml b/.github/ISSUE_TEMPLATE_tmpl/4_bug_report.yml deleted file mode 100644 index 03a6ba551..000000000 --- a/.github/ISSUE_TEMPLATE_tmpl/4_bug_report.yml +++ /dev/null @@ -1,57 +0,0 @@ -name: Bug report -description: Report a bug unrelated to any particular site or extractor -labels: [triage, bug] -body: - - type: checkboxes - id: checklist - attributes: - label: Checklist - description: | - Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: - options: - - label: I'm reporting a bug unrelated to a specific site - required: true - - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) - required: true - - label: I've checked that all provided URLs are alive and playable in a browser - required: true - - label: I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command) - required: true - - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates - required: true - - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue) - required: true - - type: textarea - id: description - attributes: - label: Description - description: | - Provide an explanation of your issue in an arbitrary form. - Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient). - Provide any additional information, any suggested solutions, and as much context and examples as possible - placeholder: WRITE DESCRIPTION HERE - validations: - required: true - - type: textarea - id: log - attributes: - label: Verbose log - description: | - Provide the complete verbose output of yt-dlp **that clearly demonstrates the problem**. - Add the `-vU` flag to **your** command line you run yt-dlp with (`yt-dlp -vU <your command line>`), copy the WHOLE output and insert it below. - It should look similar to this: - placeholder: | - [debug] Command-line config: ['-vU', 'http://www.youtube.com/watch?v=BaW_jenozKc'] - [debug] Portable config file: yt-dlp.conf - [debug] Portable config: ['-i'] - [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252 - [debug] yt-dlp version %(version)s (exe) - [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0 - [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1 - [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets - [debug] Proxy map: {} - yt-dlp is up to date (%(version)s) - <more lines> - render: shell - validations: - required: true diff --git a/.github/ISSUE_TEMPLATE_tmpl/5_feature_request.yml b/.github/ISSUE_TEMPLATE_tmpl/5_feature_request.yml deleted file mode 100644 index eb5d3d634..000000000 --- a/.github/ISSUE_TEMPLATE_tmpl/5_feature_request.yml +++ /dev/null @@ -1,32 +0,0 @@ -name: Feature request -description: Request a new functionality unrelated to any particular site or extractor -labels: [triage, enhancement] -body: - - type: checkboxes - id: checklist - attributes: - label: Checklist - description: | - Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: - options: - - label: I'm reporting a feature request - required: true - - label: I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme) - required: true - - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) - required: true - - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates - required: true - - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue) - required: true - - type: textarea - id: description - attributes: - label: Description - description: | - Provide an explanation of your site feature request in an arbitrary form. - Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient). - Provide any additional information, any suggested solutions, and as much context and examples as possible - placeholder: WRITE DESCRIPTION HERE - validations: - required: true diff --git a/.github/ISSUE_TEMPLATE_tmpl/6_question.yml b/.github/ISSUE_TEMPLATE_tmpl/6_question.yml deleted file mode 100644 index 061158ed3..000000000 --- a/.github/ISSUE_TEMPLATE_tmpl/6_question.yml +++ /dev/null @@ -1,53 +0,0 @@ -name: Ask question -description: Ask yt-dlp related question -labels: [question] -body: - - type: checkboxes - id: checklist - attributes: - label: Checklist - description: | - Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: - options: - - label: I'm asking a question and **not** reporting a bug/feature request - required: true - - label: I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme) - required: true - - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue) - required: true - - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions including closed ones - required: true - - type: textarea - id: question - attributes: - label: Question - description: | - Ask your question in an arbitrary form. - Please make sure it's worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient). - Provide any additional information and as much context and examples as possible. - If your question contains "isn't working" or "can you add", this is most likely the wrong template. - If you are in doubt if this is the right template, use another template! - placeholder: WRITE QUESTION HERE - validations: - required: true - - type: textarea - id: log - attributes: - label: Verbose log - description: | - If your question involes a yt-dlp command, provide the complete verbose output of that command. - Add the `-vU` flag to **your** command line you run yt-dlp with (`yt-dlp -vU <your command line>`), copy the WHOLE output and insert it below. - It should look similar to this: - placeholder: | - [debug] Command-line config: ['-vU', 'http://www.youtube.com/watch?v=BaW_jenozKc'] - [debug] Portable config file: yt-dlp.conf - [debug] Portable config: ['-i'] - [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252 - [debug] yt-dlp version 2021.12.01 (exe) - [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0 - [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1 - [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets - [debug] Proxy map: {} - yt-dlp is up to date (2021.12.01) - <more lines> - render: shell diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md deleted file mode 100644 index 684bf59e9..000000000 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ /dev/null @@ -1,28 +0,0 @@ -## Please follow the guide below - -- You will be asked some questions, please read them **carefully** and answer honestly -- Put an `x` into all the boxes [ ] relevant to your *pull request* (like that [x]) -- Use *Preview* tab to see how your *pull request* will actually look like - ---- - -### Before submitting a *pull request* make sure you have: -- [ ] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions) -- [ ] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests -- [ ] Checked the code with [flake8](https://pypi.python.org/pypi/flake8) - -### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check one of the following options: -- [ ] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/) -- [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence) - -### What is the purpose of your *pull request*? -- [ ] Bug fix -- [ ] Improvement -- [ ] New extractor -- [ ] New feature - ---- - -### Description of your *pull request* and other information - -Explanation of your *pull request* in arbitrary form goes here. Please make sure the description explains the purpose and effect of your *pull request* and is worded well enough to be understood. Provide as much context and examples as possible. diff --git a/.github/banner.svg b/.github/banner.svg deleted file mode 100644 index 35dc93eae..000000000 --- a/.github/banner.svg +++ /dev/null @@ -1,31 +0,0 @@ -<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" preserveAspectRatio="xMidYMid" width="699.935" height="173.764" viewBox="0 0 717 178"> - <defs> - <style> - .cls-1, .cls-4 { - fill: red; - } - - .cls-1, .cls-2, .cls-3, .cls-4 { - fill-rule: evenodd; - } - - .cls-2 { - fill: #666; - } - - .cls-3 { - fill: #fff; - } - - .cls-3, .cls-4 { - stroke: #282828; - stroke-linejoin: round; - stroke-width: 1px; - } - </style> - </defs> - <path d="M89.846,166.601 L87.111,166.601 L87.111,172.000 L82.173,172.000 L82.173,153.812 L90.024,153.812 C94.064,153.812 96.773,156.370 96.773,160.242 C96.773,164.158 93.993,166.601 89.846,166.601 ZM88.851,157.755 L87.111,157.755 L87.111,162.764 L88.851,162.764 C90.583,162.764 91.622,161.796 91.622,160.242 C91.622,158.679 90.583,157.755 88.851,157.755 ZM67.898,153.812 L72.835,153.812 L72.835,168.021 L80.189,168.021 L80.189,172.000 L67.898,172.000 L67.898,153.812 ZM56.572,172.000 L49.574,172.000 L49.574,153.812 L56.501,153.812 C62.113,153.812 65.630,157.223 65.630,162.906 C65.630,168.590 62.113,172.000 56.572,172.000 ZM56.252,158.004 L54.511,158.004 L54.511,167.808 L56.394,167.808 C59.094,167.808 60.657,166.707 60.657,162.906 C60.657,159.105 59.094,158.004 56.252,158.004 ZM38.211,162.906 L46.736,162.906 L46.736,166.601 L38.211,166.601 L38.211,162.906 ZM31.253,172.000 L26.387,172.000 L26.387,157.791 L20.916,157.791 L20.916,153.812 L36.724,153.812 L36.724,157.791 L31.253,157.791 L31.253,172.000 ZM12.007,172.000 L7.104,172.000 L7.104,166.281 L0.426,153.812 L5.932,153.812 L9.484,161.201 L9.627,161.201 L13.179,153.812 L18.685,153.812 L12.007,166.281 L12.007,172.000 Z" class="cls-1"/> - <path d="M714.317,161.947 C714.104,160.988 713.536,159.993 711.689,159.993 C710.019,159.993 708.634,160.846 708.456,162.018 C708.278,163.048 708.918,163.617 710.445,164.007 L712.399,164.505 C714.743,165.109 715.738,166.281 715.418,168.199 C715.028,170.544 712.577,172.284 709.415,172.284 C706.609,172.284 704.904,171.041 704.797,168.732 L706.893,168.235 C707.000,169.691 707.959,170.437 709.664,170.437 C711.617,170.437 713.038,169.478 713.216,168.306 C713.394,167.347 712.861,166.707 711.511,166.387 L709.344,165.855 C706.928,165.251 706.005,164.007 706.325,162.125 C706.715,159.816 709.131,158.182 712.008,158.182 C714.708,158.182 715.951,159.461 716.306,161.414 L714.317,161.947 ZM702.671,165.890 L692.751,165.890 C692.245,169.229 693.648,170.401 696.276,170.401 C697.955,170.401 699.269,169.691 700.042,168.270 L701.960,168.838 C700.974,170.899 698.736,172.284 695.957,172.284 C692.023,172.284 690.069,169.478 690.770,165.286 C691.454,161.095 694.403,158.182 698.088,158.182 C700.939,158.182 703.674,159.922 702.813,165.002 L702.671,165.890 ZM697.768,160.064 C695.477,160.064 693.461,162.143 693.044,164.078 L700.823,164.078 C701.223,161.770 700.051,160.064 697.768,160.064 ZM687.862,172.000 L685.446,172.000 L683.066,166.707 L678.910,172.000 L676.494,172.000 L681.965,165.180 L678.768,158.359 L681.183,158.359 L683.528,163.936 L687.720,158.359 L690.135,158.359 L684.594,165.180 L687.862,172.000 ZM673.886,154.630 C673.886,153.848 674.560,153.209 675.377,153.209 C676.194,153.209 676.869,153.848 676.869,154.630 C676.869,155.411 676.194,156.050 675.377,156.050 C674.560,156.050 673.886,155.411 673.886,154.630 ZM673.513,172.000 L671.417,172.000 L673.690,158.359 L675.786,158.359 L673.513,172.000 ZM670.212,154.914 C668.826,154.914 668.151,155.624 667.903,156.974 L667.672,158.359 L670.745,158.359 L670.460,160.135 L667.379,160.135 L665.416,172.000 L663.320,172.000 L665.301,160.135 L663.107,160.135 L663.391,158.359 L665.603,158.359 L665.914,156.477 C666.269,154.132 668.365,152.960 670.318,152.960 C671.348,152.960 671.952,153.173 672.237,153.315 L671.348,155.127 C671.135,155.056 670.816,154.914 670.212,154.914 ZM649.225,172.000 L649.580,169.904 L649.332,169.904 C648.745,170.650 647.582,172.284 644.962,172.284 C641.543,172.284 639.616,169.549 640.327,165.215 C641.046,160.917 643.879,158.182 647.324,158.182 C649.989,158.182 650.539,159.816 650.877,160.526 L651.054,160.526 L652.173,153.812 L654.269,153.812 L651.250,172.000 L649.225,172.000 ZM647.182,160.064 C644.527,160.064 642.911,162.302 642.440,165.180 C641.952,168.093 642.849,170.401 645.477,170.401 C647.999,170.401 649.811,168.270 650.326,165.180 C650.832,162.125 649.749,160.064 647.182,160.064 ZM635.980,172.000 L633.884,172.000 L635.305,163.475 C635.660,161.343 634.701,160.064 632.747,160.064 C630.723,160.064 629.053,161.414 628.627,163.794 L627.277,172.000 L625.181,172.000 L627.454,158.359 L629.479,158.359 L629.124,160.491 L629.302,160.491 C630.154,159.105 631.611,158.182 633.671,158.182 C636.406,158.182 638.005,159.851 637.436,163.333 L635.980,172.000 ZM621.349,172.000 L619.253,172.000 L619.573,170.153 L619.466,170.153 C618.898,171.041 617.442,172.320 615.062,172.320 C612.468,172.320 610.657,170.792 611.083,168.128 C611.616,165.002 614.458,164.434 617.051,164.114 C619.573,163.794 620.603,163.865 620.781,162.871 L620.781,162.800 C621.065,161.059 620.354,160.029 618.436,160.029 C616.447,160.029 615.097,161.095 614.458,162.089 L612.611,161.379 C614.067,158.892 616.554,158.182 618.614,158.182 C620.354,158.182 623.551,158.679 622.841,163.013 L621.349,172.000 ZM616.660,165.926 C614.991,166.139 613.428,166.636 613.179,168.235 C612.930,169.691 613.996,170.437 615.665,170.437 C618.152,170.437 619.786,168.767 620.070,167.062 L620.390,165.144 C619.964,165.570 617.548,165.819 616.660,165.926 ZM597.804,159.993 C596.135,159.993 594.749,160.846 594.572,162.018 C594.394,163.048 595.033,163.617 596.561,164.007 L598.515,164.505 C600.859,165.109 601.854,166.281 601.534,168.199 C601.143,170.544 598.692,172.284 595.531,172.284 C592.724,172.284 591.019,171.041 590.913,168.732 L593.009,168.235 C593.115,169.691 594.074,170.437 595.779,170.437 C597.733,170.437 599.154,169.478 599.332,168.306 C599.509,167.347 598.976,166.707 597.627,166.387 L595.460,165.855 C593.044,165.251 592.121,164.007 592.440,162.125 C592.831,159.816 595.247,158.182 598.124,158.182 C600.824,158.182 602.067,159.461 602.422,161.414 L600.433,161.947 C600.220,160.988 599.651,159.993 597.804,159.993 ZM588.786,165.890 L578.866,165.890 C578.360,169.229 579.763,170.401 582.392,170.401 C584.071,170.401 585.385,169.691 586.157,168.270 L588.076,168.838 C587.090,170.899 584.852,172.284 582.072,172.284 C578.138,172.284 576.185,169.478 576.886,165.286 C577.570,161.095 580.518,158.182 584.204,158.182 C587.054,158.182 589.790,159.922 588.928,165.002 L588.786,165.890 ZM583.884,160.064 C581.593,160.064 579.577,162.143 579.160,164.078 L586.939,164.078 C587.339,161.770 586.166,160.064 583.884,160.064 ZM574.722,160.171 C572.733,160.171 571.046,161.530 570.744,163.368 L569.323,172.000 L567.227,172.000 L569.500,158.359 L571.525,158.359 L571.170,160.420 L571.312,160.420 C572.023,159.070 573.586,158.146 575.255,158.146 C576.001,158.146 576.534,158.324 576.889,158.644 L575.894,160.384 C575.646,160.242 575.255,160.171 574.722,160.171 ZM561.299,172.000 L561.690,169.691 L561.548,169.691 C560.695,171.076 559.132,172.178 557.072,172.178 C554.515,172.178 552.952,170.508 553.520,167.027 L554.976,158.359 L557.072,158.359 L555.651,166.885 C555.332,168.874 556.362,170.153 558.102,170.153 C559.665,170.153 561.797,168.981 562.223,166.423 L563.573,158.359 L565.669,158.359 L563.395,172.000 L561.299,172.000 ZM551.534,160.135 L548.594,160.135 L547.271,168.093 C546.987,169.869 547.839,170.153 548.763,170.153 C549.225,170.153 549.509,170.082 549.686,170.046 L549.829,171.929 C549.509,172.036 548.976,172.178 548.195,172.178 C546.418,172.178 544.713,171.041 545.104,168.661 L546.507,160.135 L544.465,160.135 L544.749,158.359 L546.800,158.359 L547.342,155.091 L549.438,155.091 L548.896,158.359 L551.818,158.359 L551.534,160.135 ZM539.780,172.000 L537.684,172.000 L538.004,170.153 L537.897,170.153 C537.329,171.041 535.873,172.320 533.493,172.320 C530.900,172.320 529.088,170.792 529.514,168.128 C530.047,165.002 532.889,164.434 535.482,164.114 C538.004,163.794 539.034,163.865 539.212,162.871 L539.212,162.800 C539.496,161.059 538.786,160.029 536.867,160.029 C534.878,160.029 533.528,161.095 532.889,162.089 L531.042,161.379 C532.498,158.892 534.985,158.182 537.045,158.182 C538.786,158.182 541.983,158.679 541.272,163.013 L539.780,172.000 ZM535.091,165.926 C533.422,166.139 531.859,166.636 531.610,168.235 C531.361,169.691 532.427,170.437 534.097,170.437 C536.583,170.437 538.217,168.767 538.501,167.062 L538.821,165.144 C538.395,165.570 535.979,165.819 535.091,165.926 ZM527.316,165.890 L517.397,165.890 C516.891,169.229 518.294,170.401 520.922,170.401 C522.601,170.401 523.915,169.691 524.688,168.270 L526.606,168.838 C525.620,170.899 523.382,172.284 520.603,172.284 C516.669,172.284 514.715,169.478 515.416,165.286 C516.100,161.095 519.049,158.182 522.734,158.182 C525.585,158.182 528.320,159.922 527.459,165.002 L527.316,165.890 ZM522.414,160.064 C520.123,160.064 518.107,162.143 517.690,164.078 L525.469,164.078 C525.869,161.770 524.697,160.064 522.414,160.064 ZM514.282,154.914 C512.897,154.914 512.222,155.624 511.973,156.974 L511.742,158.359 L514.815,158.359 L514.531,160.135 L511.449,160.135 L509.487,172.000 L507.391,172.000 L509.371,160.135 L507.178,160.135 L507.462,158.359 L509.673,158.359 L509.984,156.477 C510.339,154.132 512.435,152.960 514.389,152.960 C515.419,152.960 516.023,153.173 516.307,153.315 L515.419,155.127 C515.206,155.056 514.886,154.914 514.282,154.914 ZM493.506,172.000 L496.525,153.812 L498.621,153.812 L495.601,172.000 L493.506,172.000 ZM489.674,172.000 L487.578,172.000 L487.898,170.153 L487.791,170.153 C487.223,171.041 485.766,172.320 483.386,172.320 C480.793,172.320 478.981,170.792 479.408,168.128 C479.941,165.002 482.782,164.434 485.375,164.114 C487.898,163.794 488.928,163.865 489.105,162.871 L489.105,162.800 C489.390,161.059 488.679,160.029 486.761,160.029 C484.772,160.029 483.422,161.095 482.782,162.089 L480.935,161.379 C482.392,158.892 484.878,158.182 486.938,158.182 C488.679,158.182 491.876,158.679 491.166,163.013 L489.674,172.000 ZM484.985,165.926 C483.315,166.139 481.752,166.636 481.504,168.235 C481.255,169.691 482.321,170.437 483.990,170.437 C486.477,170.437 488.111,168.767 488.395,167.062 L488.715,165.144 C488.288,165.570 485.873,165.819 484.985,165.926 ZM475.576,172.000 L473.480,172.000 L474.901,163.475 C475.256,161.343 474.297,160.064 472.343,160.064 C470.319,160.064 468.649,161.414 468.223,163.794 L466.873,172.000 L464.777,172.000 L467.051,158.359 L469.075,158.359 L468.720,160.491 L468.898,160.491 C469.750,159.105 471.207,158.182 473.267,158.182 C476.002,158.182 477.601,159.851 477.032,163.333 L475.576,172.000 ZM455.511,172.284 C451.745,172.284 449.703,169.407 450.395,165.109 C451.070,160.917 453.948,158.182 457.571,158.182 C461.336,158.182 463.388,161.059 462.686,165.393 C462.011,169.549 459.134,172.284 455.511,172.284 ZM457.535,160.064 C454.658,160.064 452.873,162.587 452.420,165.393 C451.994,168.057 452.811,170.401 455.546,170.401 C458.423,170.401 460.208,167.924 460.661,165.109 C461.088,162.444 460.271,160.064 457.535,160.064 ZM446.401,154.630 C446.401,153.848 447.076,153.209 447.893,153.209 C448.710,153.209 449.385,153.848 449.385,154.630 C449.385,155.411 448.710,156.050 447.893,156.050 C447.076,156.050 446.401,155.411 446.401,154.630 ZM446.028,172.000 L443.932,172.000 L446.206,158.359 L448.301,158.359 L446.028,172.000 ZM442.763,160.135 L439.823,160.135 L438.500,168.093 C438.216,169.869 439.069,170.153 439.992,170.153 C440.454,170.153 440.738,170.082 440.916,170.046 L441.058,171.929 C440.738,172.036 440.205,172.178 439.424,172.178 C437.648,172.178 435.943,171.041 436.333,168.661 L437.736,160.135 L435.694,160.135 L435.978,158.359 L438.030,158.359 L438.571,155.091 L440.667,155.091 L440.125,158.359 L443.047,158.359 L442.763,160.135 ZM431.380,154.630 C431.380,153.848 432.055,153.209 432.872,153.209 C433.689,153.209 434.364,153.848 434.364,154.630 C434.364,155.411 433.689,156.050 432.872,156.050 C432.055,156.050 431.380,155.411 431.380,154.630 ZM431.007,172.000 L428.911,172.000 L431.184,158.359 L433.280,158.359 L431.007,172.000 ZM422.770,172.000 L423.126,169.904 L422.877,169.904 C422.291,170.650 421.128,172.284 418.508,172.284 C415.089,172.284 413.162,169.549 413.872,165.215 C414.591,160.917 417.424,158.182 420.870,158.182 C423.534,158.182 424.085,159.816 424.422,160.526 L424.600,160.526 L425.719,153.812 L427.815,153.812 L424.795,172.000 L422.770,172.000 ZM420.728,160.064 C418.073,160.064 416.456,162.302 415.986,165.180 C415.497,168.093 416.394,170.401 419.023,170.401 C421.545,170.401 423.357,168.270 423.872,165.180 C424.378,162.125 423.294,160.064 420.728,160.064 ZM407.216,172.000 L407.572,169.904 L407.323,169.904 C406.737,170.650 405.573,172.284 402.954,172.284 C399.535,172.284 397.608,169.549 398.318,165.215 C399.037,160.917 401.870,158.182 405.316,158.182 C407.980,158.182 408.531,159.816 408.868,160.526 L409.046,160.526 L410.165,153.812 L412.261,153.812 L409.241,172.000 L407.216,172.000 ZM405.174,160.064 C402.519,160.064 400.902,162.302 400.432,165.180 C399.943,168.093 400.840,170.401 403.469,170.401 C405.991,170.401 407.803,168.270 408.318,165.180 C408.824,162.125 407.740,160.064 405.174,160.064 ZM393.971,172.000 L391.875,172.000 L392.195,170.153 L392.088,170.153 C391.520,171.041 390.063,172.320 387.683,172.320 C385.090,172.320 383.279,170.792 383.705,168.128 C384.238,165.002 387.080,164.434 389.673,164.114 C392.195,163.794 393.225,163.865 393.403,162.871 L393.403,162.800 C393.687,161.059 392.976,160.029 391.058,160.029 C389.069,160.029 387.719,161.095 387.080,162.089 L385.232,161.379 C386.689,158.892 389.175,158.182 391.236,158.182 C392.976,158.182 396.173,158.679 395.463,163.013 L393.971,172.000 ZM389.282,165.926 C387.612,166.139 386.049,166.636 385.801,168.235 C385.552,169.691 386.618,170.437 388.287,170.437 C390.774,170.437 392.408,168.767 392.692,167.062 L393.012,165.144 C392.586,165.570 390.170,165.819 389.282,165.926 ZM372.842,172.000 L370.746,172.000 L372.167,163.475 C372.522,161.308 371.528,160.064 369.574,160.064 C367.513,160.064 365.773,161.414 365.347,163.794 L363.997,172.000 L361.901,172.000 L364.920,153.812 L367.016,153.812 L365.915,160.491 L366.093,160.491 C366.945,159.070 368.330,158.182 370.497,158.182 C373.268,158.182 374.867,159.816 374.298,163.333 L372.842,172.000 ZM360.448,160.135 L357.508,160.135 L356.185,168.093 C355.901,169.869 356.753,170.153 357.677,170.153 C358.139,170.153 358.423,170.082 358.601,170.046 L358.743,171.929 C358.423,172.036 357.890,172.178 357.109,172.178 C355.333,172.178 353.627,171.041 354.018,168.661 L355.421,160.135 L353.379,160.135 L353.663,158.359 L355.714,158.359 L356.256,155.091 L358.352,155.091 L357.810,158.359 L360.732,158.359 L360.448,160.135 ZM349.065,154.630 C349.065,153.848 349.740,153.209 350.557,153.209 C351.374,153.209 352.049,153.848 352.049,154.630 C352.049,155.411 351.374,156.050 350.557,156.050 C349.740,156.050 349.065,155.411 349.065,154.630 ZM348.692,172.000 L346.596,172.000 L348.869,158.359 L350.965,158.359 L348.692,172.000 ZM337.615,172.000 L336.372,161.521 L336.159,161.521 L331.434,172.000 L329.374,172.000 L327.491,158.359 L329.694,158.359 L330.901,168.803 L331.043,168.803 L335.697,158.359 L337.935,158.359 L339.072,168.767 L339.214,168.767 L343.903,158.359 L346.105,158.359 L339.675,172.000 L337.615,172.000 ZM316.983,172.000 L314.319,172.000 L310.296,165.526 L308.600,166.885 L307.747,172.000 L305.651,172.000 L308.671,153.812 L310.767,153.812 L308.999,164.434 L309.239,164.434 L316.237,158.359 L318.830,158.359 L312.090,164.203 L316.983,172.000 ZM303.559,160.171 C301.569,160.171 299.882,161.530 299.580,163.368 L298.159,172.000 L296.063,172.000 L298.337,158.359 L300.362,158.359 L300.006,160.420 L300.149,160.420 C300.859,159.070 302.422,158.146 304.091,158.146 C304.837,158.146 305.370,158.324 305.726,158.644 L304.731,160.384 C304.482,160.242 304.091,160.171 303.559,160.171 ZM286.797,172.284 C283.031,172.284 280.989,169.407 281.682,165.109 C282.356,160.917 285.234,158.182 288.857,158.182 C292.622,158.182 294.674,161.059 293.972,165.393 C293.297,169.549 290.420,172.284 286.797,172.284 ZM288.822,160.064 C285.944,160.064 284.159,162.587 283.706,165.393 C283.280,168.057 284.097,170.401 286.832,170.401 C289.710,170.401 291.495,167.924 291.948,165.109 C292.374,162.444 291.557,160.064 288.822,160.064 ZM280.512,154.914 C279.126,154.914 278.452,155.624 278.203,156.974 L277.972,158.359 L281.045,158.359 L280.760,160.135 L277.679,160.135 L275.716,172.000 L273.620,172.000 L275.601,160.135 L273.407,160.135 L273.691,158.359 L275.903,158.359 L276.214,156.477 C276.569,154.132 278.665,152.960 280.618,152.960 C281.649,152.960 282.252,153.173 282.537,153.315 L281.649,155.127 C281.435,155.056 281.116,154.914 280.512,154.914 ZM259.735,172.000 L262.755,153.812 L264.851,153.812 L261.831,172.000 L259.735,172.000 ZM253.595,172.000 L253.950,169.904 L253.701,169.904 C253.115,170.650 251.952,172.284 249.332,172.284 C245.913,172.284 243.986,169.549 244.696,165.215 C245.416,160.917 248.249,158.182 251.694,158.182 C254.358,158.182 254.909,159.816 255.246,160.526 L255.424,160.526 L256.543,153.812 L258.639,153.812 L255.619,172.000 L253.595,172.000 ZM251.552,160.064 C248.897,160.064 247.281,162.302 246.810,165.180 C246.321,168.093 247.218,170.401 249.847,170.401 C252.369,170.401 254.181,168.270 254.696,165.180 C255.202,162.125 254.119,160.064 251.552,160.064 ZM233.670,165.180 L233.990,163.226 L241.947,163.226 L241.627,165.180 L233.670,165.180 ZM230.478,165.890 L220.558,165.890 C220.052,169.229 221.455,170.401 224.084,170.401 C225.762,170.401 227.077,169.691 227.849,168.270 L229.768,168.838 C228.782,170.899 226.544,172.284 223.764,172.284 C219.830,172.284 217.876,169.478 218.578,165.286 C219.262,161.095 222.210,158.182 225.896,158.182 C228.746,158.182 231.482,159.922 230.620,165.002 L230.478,165.890 ZM225.576,160.064 C223.285,160.064 221.269,162.143 220.851,164.078 L228.631,164.078 C229.030,161.770 227.858,160.064 225.576,160.064 ZM209.063,172.284 C206.434,172.284 205.830,170.650 205.475,169.904 L205.226,169.904 L204.871,172.000 L202.846,172.000 L205.866,153.812 L207.962,153.812 L206.860,160.526 L207.038,160.526 C207.606,159.816 208.708,158.182 211.372,158.182 C214.817,158.182 216.736,160.917 216.025,165.215 C215.315,169.549 212.473,172.284 209.063,172.284 ZM210.803,160.064 C208.246,160.064 206.541,162.125 206.043,165.180 C205.546,168.270 206.576,170.401 209.098,170.401 C211.727,170.401 213.432,168.093 213.929,165.180 C214.391,162.302 213.468,160.064 210.803,160.064 ZM196.634,172.000 L197.025,169.691 L196.883,169.691 C196.031,171.076 194.468,172.178 192.407,172.178 C189.850,172.178 188.287,170.508 188.855,167.027 L190.311,158.359 L192.407,158.359 L190.986,166.885 C190.667,168.874 191.697,170.153 193.437,170.153 C195.000,170.153 197.132,168.981 197.558,166.423 L198.908,158.359 L201.004,158.359 L198.730,172.000 L196.634,172.000 ZM186.869,160.135 L183.929,160.135 L182.606,168.093 C182.322,169.869 183.174,170.153 184.098,170.153 C184.560,170.153 184.844,170.082 185.022,170.046 L185.164,171.929 C184.844,172.036 184.311,172.178 183.530,172.178 C181.754,172.178 180.048,171.041 180.439,168.661 L181.842,160.135 L179.800,160.135 L180.084,158.359 L182.135,158.359 L182.677,155.091 L184.773,155.091 L184.231,158.359 L187.153,158.359 L186.869,160.135 ZM173.020,172.000 L173.410,169.691 L173.268,169.691 C172.416,171.076 170.853,172.178 168.792,172.178 C166.235,172.178 164.672,170.508 165.240,167.027 L166.697,158.359 L168.792,158.359 L167.372,166.885 C167.052,168.874 168.082,170.153 169.823,170.153 C171.386,170.153 173.517,168.981 173.943,166.423 L175.293,158.359 L177.389,158.359 L175.115,172.000 L173.020,172.000 ZM155.157,172.284 C151.391,172.284 149.349,169.407 150.041,165.109 C150.716,160.917 153.594,158.182 157.217,158.182 C160.982,158.182 163.034,161.059 162.332,165.393 C161.657,169.549 158.780,172.284 155.157,172.284 ZM157.181,160.064 C154.304,160.064 152.519,162.587 152.066,165.393 C151.640,168.057 152.457,170.401 155.192,170.401 C158.069,170.401 159.854,167.924 160.307,165.109 C160.734,162.444 159.917,160.064 157.181,160.064 ZM136.511,177.293 C135.801,177.293 135.197,177.151 135.019,177.044 L135.836,175.197 C137.293,175.588 138.207,175.366 139.317,173.350 L140.063,172.000 L137.293,158.359 L139.495,158.359 L141.449,169.229 L141.591,169.229 L147.168,158.359 L149.512,158.359 L141.023,174.202 C139.886,176.298 138.429,177.293 136.511,177.293 ZM123.225,166.849 L115.871,166.849 L113.181,172.000 L110.872,172.000 L120.569,153.812 L122.843,153.812 L126.501,172.000 L124.192,172.000 L123.225,166.849 ZM121.315,156.690 L121.173,156.690 L116.893,164.895 L122.860,164.895 L121.315,156.690 Z" class="cls-2"/> - <path d="M252.245,116.350 L252.245,102.200 L309.303,102.200 L309.303,116.350 L252.245,116.350 ZM208.254,81.088 L245.342,59.291 L208.254,38.180 L216.242,25.227 L260.862,52.844 L260.862,65.739 L216.413,93.355 L208.254,81.088 Z" class="cls-3"/> - <path d="M508.108,52.635 C507.921,55.093 507.643,57.527 507.274,59.937 L504.214,78.017 C503.658,81.170 502.754,84.324 501.502,87.475 C500.250,90.628 498.464,93.479 496.147,96.028 C493.829,98.579 491.047,100.503 487.802,101.800 C484.556,103.097 481.311,103.747 478.067,103.747 C476.211,103.747 474.357,103.491 472.504,102.982 C470.648,102.474 469.072,101.615 467.775,100.409 C466.475,99.205 465.410,97.767 464.576,96.098 C463.741,94.429 463.092,92.714 462.629,90.952 L455.953,131.146 L436.482,131.146 L453.310,28.922 L472.921,28.922 L471.391,38.240 C472.504,36.665 473.777,35.180 475.216,33.790 C476.652,32.399 478.228,31.240 479.944,30.313 C481.659,29.387 483.467,28.737 485.369,28.365 C487.268,27.996 489.145,27.809 491.001,27.809 C493.411,27.809 495.706,28.226 497.886,29.061 C500.063,29.895 501.871,31.171 503.310,32.886 C504.746,34.602 505.835,36.549 506.578,38.727 C507.319,40.907 507.806,43.156 508.039,45.472 C508.269,47.791 508.293,50.179 508.108,52.635 ZM487.455,48.184 C486.851,46.841 485.877,45.798 484.534,45.055 C483.189,44.314 481.729,43.942 480.153,43.942 C478.762,43.942 477.393,44.151 476.050,44.568 C474.705,44.986 473.499,45.681 472.434,46.655 C471.367,47.628 470.556,48.765 470.000,50.062 C469.444,51.362 469.027,52.659 468.748,53.956 L465.828,72.037 C465.641,73.149 465.480,74.286 465.341,75.444 C465.202,76.605 465.178,77.717 465.271,78.782 C465.363,79.849 465.526,80.916 465.758,81.981 C465.988,83.048 466.384,84.022 466.940,84.902 C467.497,85.784 468.283,86.456 469.305,86.918 C470.324,87.383 471.391,87.614 472.504,87.614 C474.079,87.614 475.633,87.314 477.163,86.710 C478.693,86.108 480.036,85.204 481.196,83.998 C482.354,82.794 483.235,81.425 483.839,79.895 C484.441,78.365 484.882,76.859 485.160,75.375 L488.081,57.294 C488.359,55.719 488.474,54.143 488.428,52.565 C488.381,50.990 488.057,49.530 487.455,48.184 ZM433.422,86.501 L433.422,102.635 L384.744,102.635 L384.744,86.501 L401.433,86.501 L412.977,16.544 L400.460,16.544 L400.460,0.410 L435.230,0.410 L420.905,86.501 L433.422,86.501 ZM354.285,102.635 L355.815,93.177 C354.793,94.846 353.542,96.354 352.060,97.697 C350.575,99.042 348.976,100.179 347.261,101.105 C345.545,102.033 343.736,102.704 341.837,103.121 C339.936,103.539 338.058,103.747 336.204,103.747 C333.792,103.747 331.497,103.330 329.320,102.495 C327.140,101.661 325.332,100.388 323.896,98.671 C322.457,96.956 321.368,94.985 320.627,92.760 C319.884,90.535 319.397,88.264 319.167,85.945 C318.934,83.628 318.911,81.264 319.097,78.852 C319.282,76.442 319.606,74.032 320.071,71.620 L322.992,53.539 C323.548,50.295 324.452,47.096 325.704,43.942 C326.955,40.791 328.764,37.962 331.128,35.459 C333.492,32.955 336.274,31.056 339.473,29.756 C342.672,28.459 345.892,27.809 349.139,27.809 C351.086,27.809 352.964,28.066 354.772,28.574 C356.580,29.085 358.155,29.943 359.500,31.147 C360.843,32.353 361.934,33.790 362.769,35.459 C363.603,37.128 364.205,38.844 364.577,40.605 L371.253,0.410 L390.724,0.410 L373.895,102.635 L354.285,102.635 ZM362.004,52.705 C361.956,51.592 361.795,50.503 361.517,49.436 C361.239,48.371 360.822,47.419 360.265,46.585 C359.709,45.751 358.920,45.103 357.901,44.638 C356.880,44.175 355.815,43.942 354.702,43.942 C353.124,43.942 351.573,44.245 350.043,44.846 C348.513,45.451 347.168,46.355 346.010,47.559 C344.849,48.765 343.969,50.110 343.367,51.592 C342.763,53.076 342.370,54.606 342.185,56.182 L339.125,74.262 C338.847,75.840 338.730,77.415 338.777,78.991 C338.823,80.569 339.147,82.029 339.751,83.372 C340.353,84.717 341.326,85.760 342.672,86.501 C344.015,87.244 345.475,87.614 347.053,87.614 C348.443,87.614 349.810,87.405 351.156,86.988 C352.499,86.571 353.705,85.875 354.772,84.902 C355.836,83.928 356.671,82.794 357.275,81.494 C357.877,80.197 358.270,78.900 358.457,77.600 L361.517,59.520 C361.702,58.407 361.841,57.272 361.934,56.112 C362.026,54.954 362.049,53.817 362.004,52.705 Z" class="cls-4"/> -</svg> diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml deleted file mode 100644 index 1ca67034f..000000000 --- a/.github/workflows/build.yml +++ /dev/null @@ -1,413 +0,0 @@ -name: Build -on: workflow_dispatch - -jobs: - build_unix: - runs-on: ubuntu-latest - outputs: - version_suffix: ${{ steps.version_suffix.outputs.version_suffix }} - ytdlp_version: ${{ steps.bump_version.outputs.ytdlp_version }} - upload_url: ${{ steps.create_release.outputs.upload_url }} - sha256_bin: ${{ steps.sha256_bin.outputs.sha256_bin }} - sha512_bin: ${{ steps.sha512_bin.outputs.sha512_bin }} - sha256_tar: ${{ steps.sha256_tar.outputs.sha256_tar }} - sha512_tar: ${{ steps.sha512_tar.outputs.sha512_tar }} - - steps: - - uses: actions/checkout@v2 - with: - fetch-depth: 0 - - name: Set up Python - uses: actions/setup-python@v2 - with: - python-version: '3.8' - - name: Install packages - run: sudo apt-get -y install zip pandoc man - - name: Set version suffix - id: version_suffix - env: - PUSH_VERSION_COMMIT: ${{ secrets.PUSH_VERSION_COMMIT }} - if: "env.PUSH_VERSION_COMMIT == ''" - run: echo ::set-output name=version_suffix::$(date -u +"%H%M%S") - - name: Bump version - id: bump_version - run: | - python devscripts/update-version.py ${{ steps.version_suffix.outputs.version_suffix }} - make issuetemplates - - name: Push to release - id: push_release - run: | - git config --global user.name github-actions - git config --global user.email github-actions@example.com - git add -u - git commit -m "[version] update" -m "Created by: ${{ github.event.sender.login }}" -m ":ci skip all" - git push origin --force ${{ github.event.ref }}:release - echo ::set-output name=head_sha::$(git rev-parse HEAD) - - name: Update master - id: push_master - env: - PUSH_VERSION_COMMIT: ${{ secrets.PUSH_VERSION_COMMIT }} - if: "env.PUSH_VERSION_COMMIT != ''" - run: git push origin ${{ github.event.ref }} - - name: Get Changelog - id: get_changelog - run: | - changelog=$(cat Changelog.md | grep -oPz '(?s)(?<=### ${{ steps.bump_version.outputs.ytdlp_version }}\n{2}).+?(?=\n{2,3}###)') || true - echo "changelog<<EOF" >> $GITHUB_ENV - echo "$changelog" >> $GITHUB_ENV - echo "EOF" >> $GITHUB_ENV - - - name: Build lazy extractors - id: lazy_extractors - run: python devscripts/make_lazy_extractors.py - - name: Run Make - run: make all tar - - name: Get SHA2-256SUMS for yt-dlp - id: sha256_bin - run: echo "::set-output name=sha256_bin::$(sha256sum yt-dlp | awk '{print $1}')" - - name: Get SHA2-256SUMS for yt-dlp.tar.gz - id: sha256_tar - run: echo "::set-output name=sha256_tar::$(sha256sum yt-dlp.tar.gz | awk '{print $1}')" - - name: Get SHA2-512SUMS for yt-dlp - id: sha512_bin - run: echo "::set-output name=sha512_bin::$(sha512sum yt-dlp | awk '{print $1}')" - - name: Get SHA2-512SUMS for yt-dlp.tar.gz - id: sha512_tar - run: echo "::set-output name=sha512_tar::$(sha512sum yt-dlp.tar.gz | awk '{print $1}')" - - - name: Install dependencies for pypi - env: - PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }} - if: "env.PYPI_TOKEN != ''" - run: | - python -m pip install --upgrade pip - pip install setuptools wheel twine - - name: Build and publish on pypi - env: - TWINE_USERNAME: __token__ - TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} - if: "env.TWINE_PASSWORD != ''" - run: | - rm -rf dist/* - python setup.py sdist bdist_wheel - twine upload dist/* - - - name: Install SSH private key - env: - BREW_TOKEN: ${{ secrets.BREW_TOKEN }} - if: "env.BREW_TOKEN != ''" - uses: yt-dlp/ssh-agent@v0.5.3 - with: - ssh-private-key: ${{ env.BREW_TOKEN }} - - name: Update Homebrew Formulae - env: - BREW_TOKEN: ${{ secrets.BREW_TOKEN }} - if: "env.BREW_TOKEN != ''" - run: | - git clone git@github.com:yt-dlp/homebrew-taps taps/ - python3 devscripts/update-formulae.py taps/Formula/yt-dlp.rb "${{ steps.bump_version.outputs.ytdlp_version }}" - git -C taps/ config user.name github-actions - git -C taps/ config user.email github-actions@example.com - git -C taps/ commit -am 'yt-dlp: ${{ steps.bump_version.outputs.ytdlp_version }}' - git -C taps/ push - - - name: Create Release - id: create_release - uses: actions/create-release@v1 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - with: - tag_name: ${{ steps.bump_version.outputs.ytdlp_version }} - release_name: yt-dlp ${{ steps.bump_version.outputs.ytdlp_version }} - commitish: ${{ steps.push_release.outputs.head_sha }} - body: | - #### [A description of the various files]((https://github.com/yt-dlp/yt-dlp#release-files)) are in the README - - --- - - ### Changelog: - ${{ env.changelog }} - draft: false - prerelease: false - - name: Upload yt-dlp Unix binary - id: upload-release-asset - uses: actions/upload-release-asset@v1 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - with: - upload_url: ${{ steps.create_release.outputs.upload_url }} - asset_path: ./yt-dlp - asset_name: yt-dlp - asset_content_type: application/octet-stream - - name: Upload Source tar - uses: actions/upload-release-asset@v1 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - with: - upload_url: ${{ steps.create_release.outputs.upload_url }} - asset_path: ./yt-dlp.tar.gz - asset_name: yt-dlp.tar.gz - asset_content_type: application/gzip - - build_macos: - runs-on: macos-11 - needs: build_unix - outputs: - sha256_macos: ${{ steps.sha256_macos.outputs.sha256_macos }} - sha512_macos: ${{ steps.sha512_macos.outputs.sha512_macos }} - sha256_macos_zip: ${{ steps.sha256_macos_zip.outputs.sha256_macos_zip }} - sha512_macos_zip: ${{ steps.sha512_macos_zip.outputs.sha512_macos_zip }} - - steps: - - uses: actions/checkout@v2 - # In order to create a universal2 application, the version of python3 in /usr/bin has to be used - - name: Install Requirements - run: | - brew install coreutils - /usr/bin/python3 -m pip install -U --user pip Pyinstaller==4.9 -r requirements.txt - - name: Bump version - id: bump_version - run: /usr/bin/python3 devscripts/update-version.py - - name: Build lazy extractors - id: lazy_extractors - run: /usr/bin/python3 devscripts/make_lazy_extractors.py - - name: Run PyInstaller Script - run: /usr/bin/python3 pyinst.py --target-architecture universal2 --onefile - - name: Upload yt-dlp MacOS binary - id: upload-release-macos - uses: actions/upload-release-asset@v1 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - with: - upload_url: ${{ needs.build_unix.outputs.upload_url }} - asset_path: ./dist/yt-dlp_macos - asset_name: yt-dlp_macos - asset_content_type: application/octet-stream - - name: Get SHA2-256SUMS for yt-dlp_macos - id: sha256_macos - run: echo "::set-output name=sha256_macos::$(sha256sum dist/yt-dlp_macos | awk '{print $1}')" - - name: Get SHA2-512SUMS for yt-dlp_macos - id: sha512_macos - run: echo "::set-output name=sha512_macos::$(sha512sum dist/yt-dlp_macos | awk '{print $1}')" - - - name: Run PyInstaller Script with --onedir - run: | - /usr/bin/python3 pyinst.py --target-architecture universal2 --onedir - zip ./dist/yt-dlp_macos.zip ./dist/yt-dlp_macos - - name: Upload yt-dlp MacOS onedir - id: upload-release-macos-zip - uses: actions/upload-release-asset@v1 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - with: - upload_url: ${{ needs.build_unix.outputs.upload_url }} - asset_path: ./dist/yt-dlp_macos.zip - asset_name: yt-dlp_macos.zip - asset_content_type: application/zip - - name: Get SHA2-256SUMS for yt-dlp_macos.zip - id: sha256_macos_zip - run: echo "::set-output name=sha256_macos_zip::$(sha256sum dist/yt-dlp_macos.zip | awk '{print $1}')" - - name: Get SHA2-512SUMS for yt-dlp_macos.zip - id: sha512_macos_zip - run: echo "::set-output name=sha512_macos_zip::$(sha512sum dist/yt-dlp_macos.zip | awk '{print $1}')" - - build_windows: - runs-on: windows-latest - needs: build_unix - outputs: - sha256_win: ${{ steps.sha256_win.outputs.sha256_win }} - sha512_win: ${{ steps.sha512_win.outputs.sha512_win }} - sha256_py2exe: ${{ steps.sha256_py2exe.outputs.sha256_py2exe }} - sha512_py2exe: ${{ steps.sha512_py2exe.outputs.sha512_py2exe }} - sha256_win_zip: ${{ steps.sha256_win_zip.outputs.sha256_win_zip }} - sha512_win_zip: ${{ steps.sha512_win_zip.outputs.sha512_win_zip }} - - steps: - - uses: actions/checkout@v2 - # 3.8 is used for Win7 support - - name: Set up Python 3.8 - uses: actions/setup-python@v2 - with: - python-version: '3.8' - - name: Install Requirements - # Custom pyinstaller built with https://github.com/yt-dlp/pyinstaller-builds - run: | - python -m pip install --upgrade pip setuptools wheel py2exe - pip install "https://yt-dlp.github.io/Pyinstaller-Builds/x86_64/pyinstaller-4.9-py3-none-any.whl" -r requirements.txt - - name: Bump version - id: bump_version - env: - version_suffix: ${{ needs.build_unix.outputs.version_suffix }} - run: python devscripts/update-version.py ${{ env.version_suffix }} - - name: Build lazy extractors - id: lazy_extractors - run: python devscripts/make_lazy_extractors.py - - name: Run PyInstaller Script - run: python pyinst.py - - name: Upload yt-dlp.exe Windows binary - id: upload-release-windows - uses: actions/upload-release-asset@v1 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - with: - upload_url: ${{ needs.build_unix.outputs.upload_url }} - asset_path: ./dist/yt-dlp.exe - asset_name: yt-dlp.exe - asset_content_type: application/vnd.microsoft.portable-executable - - name: Get SHA2-256SUMS for yt-dlp.exe - id: sha256_win - run: echo "::set-output name=sha256_win::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA256).Hash.ToLower())" - - name: Get SHA2-512SUMS for yt-dlp.exe - id: sha512_win - run: echo "::set-output name=sha512_win::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA512).Hash.ToLower())" - - - name: Run PyInstaller Script with --onedir - run: | - python pyinst.py --onedir - Compress-Archive -LiteralPath ./dist/yt-dlp -DestinationPath ./dist/yt-dlp_win.zip - - name: Upload yt-dlp Windows onedir - id: upload-release-windows-zip - uses: actions/upload-release-asset@v1 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - with: - upload_url: ${{ needs.build_unix.outputs.upload_url }} - asset_path: ./dist/yt-dlp_win.zip - asset_name: yt-dlp_win.zip - asset_content_type: application/zip - - name: Get SHA2-256SUMS for yt-dlp_win.zip - id: sha256_win_zip - run: echo "::set-output name=sha256_win_zip::$((Get-FileHash dist\yt-dlp_win.zip -Algorithm SHA256).Hash.ToLower())" - - name: Get SHA2-512SUMS for yt-dlp_win.zip - id: sha512_win_zip - run: echo "::set-output name=sha512_win_zip::$((Get-FileHash dist\yt-dlp_win.zip -Algorithm SHA512).Hash.ToLower())" - - - name: Run py2exe Script - run: python setup.py py2exe - - name: Upload yt-dlp_min.exe Windows binary - id: upload-release-windows-py2exe - uses: actions/upload-release-asset@v1 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - with: - upload_url: ${{ needs.build_unix.outputs.upload_url }} - asset_path: ./dist/yt-dlp.exe - asset_name: yt-dlp_min.exe - asset_content_type: application/vnd.microsoft.portable-executable - - name: Get SHA2-256SUMS for yt-dlp_min.exe - id: sha256_py2exe - run: echo "::set-output name=sha256_py2exe::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA256).Hash.ToLower())" - - name: Get SHA2-512SUMS for yt-dlp_min.exe - id: sha512_py2exe - run: echo "::set-output name=sha512_py2exe::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA512).Hash.ToLower())" - - build_windows32: - runs-on: windows-latest - needs: build_unix - - outputs: - sha256_win32: ${{ steps.sha256_win32.outputs.sha256_win32 }} - sha512_win32: ${{ steps.sha512_win32.outputs.sha512_win32 }} - - steps: - - uses: actions/checkout@v2 - # 3.7 is used for Vista support. See https://github.com/yt-dlp/yt-dlp/issues/390 - - name: Set up Python 3.7 32-Bit - uses: actions/setup-python@v2 - with: - python-version: '3.7' - architecture: 'x86' - - name: Install Requirements - run: | - python -m pip install --upgrade pip setuptools wheel - pip install "https://yt-dlp.github.io/Pyinstaller-Builds/i686/pyinstaller-4.9-py3-none-any.whl" -r requirements.txt - - name: Bump version - id: bump_version - env: - version_suffix: ${{ needs.build_unix.outputs.version_suffix }} - run: python devscripts/update-version.py ${{ env.version_suffix }} - - name: Build lazy extractors - id: lazy_extractors - run: python devscripts/make_lazy_extractors.py - - name: Run PyInstaller Script for 32 Bit - run: python pyinst.py - - name: Upload Executable yt-dlp_x86.exe - id: upload-release-windows32 - uses: actions/upload-release-asset@v1 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - with: - upload_url: ${{ needs.build_unix.outputs.upload_url }} - asset_path: ./dist/yt-dlp_x86.exe - asset_name: yt-dlp_x86.exe - asset_content_type: application/vnd.microsoft.portable-executable - - name: Get SHA2-256SUMS for yt-dlp_x86.exe - id: sha256_win32 - run: echo "::set-output name=sha256_win32::$((Get-FileHash dist\yt-dlp_x86.exe -Algorithm SHA256).Hash.ToLower())" - - name: Get SHA2-512SUMS for yt-dlp_x86.exe - id: sha512_win32 - run: echo "::set-output name=sha512_win32::$((Get-FileHash dist\yt-dlp_x86.exe -Algorithm SHA512).Hash.ToLower())" - - finish: - runs-on: ubuntu-latest - needs: [build_unix, build_windows, build_windows32, build_macos] - - steps: - - name: Make SHA2-256SUMS file - env: - SHA256_BIN: ${{ needs.build_unix.outputs.sha256_bin }} - SHA256_TAR: ${{ needs.build_unix.outputs.sha256_tar }} - SHA256_WIN: ${{ needs.build_windows.outputs.sha256_win }} - SHA256_PY2EXE: ${{ needs.build_windows.outputs.sha256_py2exe }} - SHA256_WIN_ZIP: ${{ needs.build_windows.outputs.sha256_win_zip }} - SHA256_WIN32: ${{ needs.build_windows32.outputs.sha256_win32 }} - SHA256_MACOS: ${{ needs.build_macos.outputs.sha256_macos }} - SHA256_MACOS_ZIP: ${{ needs.build_macos.outputs.sha256_macos_zip }} - run: | - echo "${{ env.SHA256_BIN }} yt-dlp" >> SHA2-256SUMS - echo "${{ env.SHA256_TAR }} yt-dlp.tar.gz" >> SHA2-256SUMS - echo "${{ env.SHA256_WIN }} yt-dlp.exe" >> SHA2-256SUMS - echo "${{ env.SHA256_PY2EXE }} yt-dlp_min.exe" >> SHA2-256SUMS - echo "${{ env.SHA256_WIN32 }} yt-dlp_x86.exe" >> SHA2-256SUMS - echo "${{ env.SHA256_WIN_ZIP }} yt-dlp_win.zip" >> SHA2-256SUMS - echo "${{ env.SHA256_MACOS }} yt-dlp_macos" >> SHA2-256SUMS - echo "${{ env.SHA256_MACOS_ZIP }} yt-dlp_macos.zip" >> SHA2-256SUMS - - name: Upload 256SUMS file - id: upload-sums - uses: actions/upload-release-asset@v1 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - with: - upload_url: ${{ needs.build_unix.outputs.upload_url }} - asset_path: ./SHA2-256SUMS - asset_name: SHA2-256SUMS - asset_content_type: text/plain - - name: Make SHA2-512SUMS file - env: - SHA512_BIN: ${{ needs.build_unix.outputs.sha512_bin }} - SHA512_TAR: ${{ needs.build_unix.outputs.sha512_tar }} - SHA512_WIN: ${{ needs.build_windows.outputs.sha512_win }} - SHA512_PY2EXE: ${{ needs.build_windows.outputs.sha512_py2exe }} - SHA512_WIN_ZIP: ${{ needs.build_windows.outputs.sha512_win_zip }} - SHA512_WIN32: ${{ needs.build_windows32.outputs.sha512_win32 }} - SHA512_MACOS: ${{ needs.build_macos.outputs.sha512_macos }} - SHA512_MACOS_ZIP: ${{ needs.build_macos.outputs.sha512_macos_zip }} - run: | - echo "${{ env.SHA512_BIN }} yt-dlp" >> SHA2-512SUMS - echo "${{ env.SHA512_TAR }} yt-dlp.tar.gz" >> SHA2-512SUMS - echo "${{ env.SHA512_WIN }} yt-dlp.exe" >> SHA2-512SUMS - echo "${{ env.SHA512_WIN_ZIP }} yt-dlp_win.zip" >> SHA2-512SUMS - echo "${{ env.SHA512_PY2EXE }} yt-dlp_min.exe" >> SHA2-512SUMS - echo "${{ env.SHA512_WIN32 }} yt-dlp_x86.exe" >> SHA2-512SUMS - echo "${{ env.SHA512_MACOS }} yt-dlp_macos" >> SHA2-512SUMS - echo "${{ env.SHA512_MACOS_ZIP }} yt-dlp_macos.zip" >> SHA2-512SUMS - - name: Upload 512SUMS file - id: upload-512sums - uses: actions/upload-release-asset@v1 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - with: - upload_url: ${{ needs.build_unix.outputs.upload_url }} - asset_path: ./SHA2-512SUMS - asset_name: SHA2-512SUMS - asset_content_type: text/plain diff --git a/.github/workflows/core.yml b/.github/workflows/core.yml deleted file mode 100644 index 4fb65e0c1..000000000 --- a/.github/workflows/core.yml +++ /dev/null @@ -1,31 +0,0 @@ -name: Core Tests -on: [push, pull_request] -jobs: - tests: - name: Core Tests - if: "!contains(github.event.head_commit.message, 'ci skip')" - runs-on: ${{ matrix.os }} - strategy: - fail-fast: false - matrix: - os: [ubuntu-18.04] - # py3.9 is in quick-test - python-version: [3.7, 3.8, 3.10-dev, pypy-3.6, pypy-3.7] - run-tests-ext: [sh] - include: - # atleast one of the tests must be in windows - - os: windows-latest - python-version: 3.6 - run-tests-ext: bat - steps: - - uses: actions/checkout@v2 - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v2 - with: - python-version: ${{ matrix.python-version }} - - name: Install pytest - run: pip install pytest - - name: Run tests - continue-on-error: False - run: ./devscripts/run_tests.${{ matrix.run-tests-ext }} core - # Linter is in quick-test diff --git a/.github/workflows/download.yml b/.github/workflows/download.yml deleted file mode 100644 index dd242fa56..000000000 --- a/.github/workflows/download.yml +++ /dev/null @@ -1,28 +0,0 @@ -name: Download Tests -on: [push, pull_request] -jobs: - tests: - name: Download Tests - if: "contains(github.event.head_commit.message, 'ci run dl')" - runs-on: ${{ matrix.os }} - strategy: - fail-fast: true - matrix: - os: [ubuntu-18.04] - python-version: [3.7, 3.8, 3.9, 3.10-dev, pypy-3.6, pypy-3.7] - run-tests-ext: [sh] - include: - - os: windows-latest - python-version: 3.6 - run-tests-ext: bat - steps: - - uses: actions/checkout@v2 - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v2 - with: - python-version: ${{ matrix.python-version }} - - name: Install pytest - run: pip install pytest - - name: Run tests - continue-on-error: true - run: ./devscripts/run_tests.${{ matrix.run-tests-ext }} download diff --git a/.github/workflows/quick-test.yml b/.github/workflows/quick-test.yml deleted file mode 100644 index d8e14f470..000000000 --- a/.github/workflows/quick-test.yml +++ /dev/null @@ -1,33 +0,0 @@ -name: Quick Test -on: [push, pull_request] -jobs: - tests: - name: Core Test - if: "!contains(github.event.head_commit.message, 'ci skip all')" - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - name: Set up Python - uses: actions/setup-python@v2 - with: - python-version: 3.9 - - name: Install test requirements - run: pip install pytest pycryptodomex - - name: Run tests - run: ./devscripts/run_tests.sh core - flake8: - name: Linter - if: "!contains(github.event.head_commit.message, 'ci skip all')" - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - name: Set up Python - uses: actions/setup-python@v2 - with: - python-version: 3.9 - - name: Install flake8 - run: pip install flake8 - - name: Make lazy extractors - run: python devscripts/make_lazy_extractors.py - - name: Run flake8 - run: flake8 . diff --git a/.gitignore b/.gitignore index 5dc82ccbe..fb09c3d6d 100644 --- a/.gitignore +++ b/.gitignore @@ -14,10 +14,7 @@ cookies *.frag.urls *.info.json *.live_chat.json -*.meta *.part* -*.tmp -*.temp *.unknown_video *.ytdl .cache/ @@ -67,7 +64,6 @@ zip/ tmp/ venv/ completions/ - # Misc *~ *.DS_Store @@ -81,36 +77,126 @@ updates_key.pem *.egg-info .tox *.class - # Generated AUTHORS README.txt +README.md +CONTRIBUTING.md .mailmap *.1 *.bash-completion *.fish *.tar.gz *.zsh -*.spec -test/testdata/sigs/player-*.js - +test/testdata/player-*.js # Binary -/youtube-dl -/youtube-dlc -/yt-dlp +/ytdlp yt-dlp.zip -*.exe - # Text Editor / IDE .idea *.iml .vscode *.sublime-* - # Lazy extractors */extractor/lazy_extractors.py - # Plugins ytdlp_plugins/extractor/* !ytdlp_plugins/extractor/__init__.py !ytdlp_plugins/extractor/sample.py +# VS Code related files +# branding +.readthedocs.yml +hypervideo.1 +hypervideo.bash-completion +hypervideo.fish +hypervideo +hypervideo.exe +hypervideo.tar.gz +setup.py +tox.in +# Ignore nonfree JS or SWF bytecode +.github/ISSUE_TEMPLATE/1_broken_site.md +.github/ISSUE_TEMPLATE/2_site_support_request.md +.github/ISSUE_TEMPLATE/3_site_feature_request.md +.github/ISSUE_TEMPLATE/4_bug_report.md +.github/ISSUE_TEMPLATE/5_feature_request.md +.github/ISSUE_TEMPLATE/6_question.md +.github/ISSUE_TEMPLATE_tmpl/1_broken_site.md +.github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md +.github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md +.github/ISSUE_TEMPLATE_tmpl/4_bug_report.md +.github/ISSUE_TEMPLATE_tmpl/5_feature_request.md +.github/PULL_REQUEST_TEMPLATE.md +.github/ISSUE_TEMPLATE.md +.github/ISSUE_TEMPLATE_tmpl.md +.github/FUNDING.yml +.github/banner.svg +.github/workflows/build.yml +.github/workflows/core.yml +.github/workflows/download.yml +.github/workflows/quick-test.yml +devscripts/create-github-release.py +devscripts/gh-pages.unused/add-version.py +devscripts/gh-pages.unused/generate-download.py +devscripts/gh-pages.unused/sign-versions.py +devscripts/gh-pages.unused/update-copyright.py +devscripts/gh-pages.unused/update-feed.py +devscripts/gh-pages.unused/update-sites.py +devscripts/make_issue_template.py +devscripts/release.sh +devscripts/show-downloads-statistics.py +devscripts/wine-py2exe.sh +devscripts/make_readme.py +devscripts/update-formulae.py +devscripts/update-version.py +test/swftests/.gitignore +test/swftests/ArrayAccess.as +test/swftests/ClassCall.as +test/swftests/ClassConstruction.as +test/swftests/ConstArrayAccess.as +test/swftests/ConstantInt.as +test/swftests/DictCall.as +test/swftests/EqualsOperator.as +test/swftests/LocalVars.as +test/swftests/MemberAssignment.as +test/swftests/NeOperator.as +test/swftests/PrivateCall.as +test/swftests/PrivateVoidCall.as +test/swftests/StaticAssignment.as +test/swftests/StaticRetrieval.as +test/swftests/StringBasics.as +test/swftests/StringCharCodeAt.as +test/swftests/StringConversion.as +test/swftests.unused/.gitignore +test/swftests.unused/ArrayAccess.as +test/swftests.unused/ClassCall.as +test/swftests.unused/ClassConstruction.as +test/swftests.unused/ConstArrayAccess.as +test/swftests.unused/ConstantInt.as +test/swftests.unused/DictCall.as +test/swftests.unused/EqualsOperator.as +test/swftests.unused/LocalVars.as +test/swftests.unused/MemberAssignment.as +test/swftests.unused/NeOperator.as +test/swftests.unused/PrivateCall.as +test/swftests.unused/PrivateVoidCall.as +test/swftests.unused/StaticAssignment.as +test/swftests.unused/StaticRetrieval.as +test/swftests.unused/StringBasics.as +test/swftests.unused/StringCharCodeAt.as +test/swftests.unused/StringConversion.as +test/test_iqiyi_sdk_interpreter.py +test/test_swfinterp.py.disabled +test/test_unicode_literals.py.disabled +test/test_update.py.disabled +test/test_write_annotations.py.disabled +test/test_iqiyi_sdk_interpreter.py~upstream_master +test/test_swfinterp.py +test/test_update.py +test/versions.json +yt_dlp/swfinterp.py +yt_dlp/update.py +ytdlp_plugins/extractor/__init__.py +ytdlp_plugins/extractor/sample.py +ytdlp_plugins/postprocessor/__init__.py +ytdlp_plugins/postprocessor/sample.py diff --git a/.readthedocs.yml b/.readthedocs.yml deleted file mode 100644 index 052f7bfca..000000000 --- a/.readthedocs.yml +++ /dev/null @@ -1,22 +0,0 @@ -# .readthedocs.yaml -# Read the Docs configuration file -# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details - -# Required -version: 2 - -# Build documentation in the docs/ directory with Sphinx -sphinx: - configuration: docs/conf.py - -# Optionally build your docs in additional formats such as PDF -formats: - - epub - - pdf - - htmlzip - -# Optionally set the version of Python and requirements required to build your docs -python: - version: 3 - install: - - requirements: docs/requirements.txt diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bfa8ae410..6f27a10f7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,61 +1,26 @@ -# CONTRIBUTING TO YT-DLP - -- [OPENING AN ISSUE](#opening-an-issue) - - [Is the description of the issue itself sufficient?](#is-the-description-of-the-issue-itself-sufficient) - - [Are you using the latest version?](#are-you-using-the-latest-version) - - [Is the issue already documented?](#is-the-issue-already-documented) - - [Why are existing options not enough?](#why-are-existing-options-not-enough) - - [Have you read and understood the changes, between youtube-dl and yt-dlp](#have-you-read-and-understood-the-changes-between-youtube-dl-and-yt-dlp) - - [Is there enough context in your bug report?](#is-there-enough-context-in-your-bug-report) - - [Does the issue involve one problem, and one problem only?](#does-the-issue-involve-one-problem-and-one-problem-only) - - [Is anyone going to need the feature?](#is-anyone-going-to-need-the-feature) - - [Is your question about yt-dlp?](#is-your-question-about-yt-dlp) - - [Are you willing to share account details if needed?](#are-you-willing-to-share-account-details-if-needed) -- [DEVELOPER INSTRUCTIONS](#developer-instructions) - - [Adding new feature or making overarching changes](#adding-new-feature-or-making-overarching-changes) - - [Adding support for a new site](#adding-support-for-a-new-site) - - [yt-dlp coding conventions](#yt-dlp-coding-conventions) - - [Mandatory and optional metafields](#mandatory-and-optional-metafields) - - [Provide fallbacks](#provide-fallbacks) - - [Regular expressions](#regular-expressions) - - [Long lines policy](#long-lines-policy) - - [Quotes](#quotes) - - [Inline values](#inline-values) - - [Collapse fallbacks](#collapse-fallbacks) - - [Trailing parentheses](#trailing-parentheses) - - [Use convenience conversion and parsing functions](#use-convenience-conversion-and-parsing-functions) -- [EMBEDDING YT-DLP](README.md#embedding-yt-dlp) - - - -# OPENING AN ISSUE - -Bugs and suggestions should be reported at: [yt-dlp/yt-dlp/issues](https://github.com/yt-dlp/yt-dlp/issues). Unless you were prompted to or there is another pertinent reason (e.g. GitHub fails to accept the bug report), please do not send bug reports via personal email. For discussions, join us in our [discord server](https://discord.gg/H5MNcFW63r). - -**Please include the full output of yt-dlp when run with `-vU`**, i.e. **add** `-vU` flag to **your command line**, copy the **whole** output and post it in the issue body wrapped in \`\`\` for better formatting. It should look similar to this: -``` -$ yt-dlp -vU <your command line> -[debug] Command-line config: ['-v', 'demo.com'] -[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8 -[debug] yt-dlp version 2021.09.25 (zip) -[debug] Python version 3.8.10 (CPython 64bit) - Linux-5.4.0-74-generic-x86_64-with-glibc2.29 -[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4 +**Please include the full output of hypervideo when run with `-v`**, i.e. **add** `-v` flag to **your command line**, copy the **whole** output and post it in the issue body wrapped in \`\`\` for better formatting. It should look similar to this: +``` +$ hypervideo -v <your command line> +[debug] System config: [] +[debug] User config: [] +[debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=BaW_jenozKcj'] +[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251 +[debug] hypervideo version 1.1.11 +[debug] Git HEAD: 135392e +[debug] Python version 2.6.6 - Windows-2003Server-5.2.3790-SP2 +[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4 [debug] Proxy map: {} -Current Build Hash 25cc412d1d3c0725a1f2f5b7e4682f6fb40e6d15f7024e96f7afd572e9919535 -yt-dlp is up to date (2021.09.25) ... ``` **Do not post screenshots of verbose logs; only plain text is acceptable.** -The output (including the first lines) contains important debugging information. Issues without the full output are often not reproducible and therefore will be closed as `incomplete`. - -The templates provided for the Issues, should be completed and **not removed**, this helps aide the resolution of the issue. +The output (including the first lines) contains important debugging information. Issues without the full output are often not reproducible and therefore do not get solved in short order, if ever. Please re-read your issue once again to avoid a couple of common mistakes (you can and should use this as a checklist): ### Is the description of the issue itself sufficient? -We often get issue reports that we cannot really decipher. While in most cases we eventually get the required information after asking back multiple times, this poses an unnecessary drain on our resources. +We often get issue reports that we cannot really decipher. While in most cases we eventually get the required information after asking back multiple times, this poses an unnecessary drain on our resources. Many contributors, including myself, are also not native speakers, so we may misread some parts. So please elaborate on what feature you are requesting, or what bug you want to be fixed. Make sure that it's obvious @@ -63,31 +28,25 @@ So please elaborate on what feature you are requesting, or what bug you want to - How it could be fixed - How your proposed solution would look like -If your report is shorter than two lines, it is almost certainly missing some of these, which makes it hard for us to respond to it. We're often too polite to close the issue outright, but the missing info makes misinterpretation likely. We often get frustrated by these issues, since the only possible way for us to move forward on them is to ask for clarification over and over. +If your report is shorter than two lines, it is almost certainly missing some of these, which makes it hard for us to respond to it. We're often too polite to close the issue outright, but the missing info makes misinterpretation likely. As a committer myself, I often get frustrated by these issues, since the only possible way for me to move forward on them is to ask for clarification over and over. -For bug reports, this means that your report should contain the **complete** output of yt-dlp when called with the `-vU` flag. The error message you get for (most) bugs even says so, but you would not believe how many of our bug reports do not contain this information. +For bug reports, this means that your report should contain the *complete* output of hypervideo when called with the `-v` flag. The error message you get for (most) bugs even says so, but you would not believe how many of our bug reports do not contain this information. -If the error is `ERROR: Unable to extract ...` and you cannot reproduce it from multiple countries, add `--write-pages` and upload the `.dump` files you get [somewhere](https://gist.github.com). +If your server has multiple IPs or you suspect censorship, adding `--call-home` may be a good idea to get more diagnostics. If the error is `ERROR: Unable to extract ...` and you cannot reproduce it from multiple countries, add `--dump-pages` (warning: this will yield a rather large output, redirect it to the file `log.txt` by adding `>log.txt 2>&1` to your command-line) or upload the `.dump` files you get when you add `--write-pages` [somewhere](https://gist.github.com/). **Site support requests must contain an example URL**. An example URL is a URL you might want to download, like `https://www.youtube.com/watch?v=BaW_jenozKc`. There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. `https://www.youtube.com/`) is *not* an example URL. ### Are you using the latest version? -Before reporting any issue, type `yt-dlp -U`. This should report that you're up-to-date. This goes for feature requests as well. +Before reporting any issue, type `doas pacman -Sy hypervideo`. This should report that you're up-to-date. About 20% of the reports we receive are already fixed, but people are using outdated versions. This goes for feature requests as well. ### Is the issue already documented? -Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or browse the [GitHub Issues](https://github.com/yt-dlp/yt-dlp/search?type=Issues) of this repository. If there is an issue, feel free to write something along the lines of "This affects me as well, with version 2021.01.01. Here is some more information on the issue: ...". While some issues may be old, a new post into them often spurs rapid activity. - -Additionally, it is also helpful to see if the issue has already been documented in the [youtube-dl issue tracker](https://github.com/ytdl-org/youtube-dl/issues). If similar issues have already been reported in youtube-dl (but not in our issue tracker), links to them can be included in your issue report here. +Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or browse the [GitHub Issues](https://github.com/ytdl-org/youtube-dl/search?type=Issues) of this repository. If there is an issue, feel free to write something along the lines of "This affects me as well, with version 2015.01.01. Here is some more information on the issue: ...". While some issues may be old, a new post into them often spurs rapid activity. ### Why are existing options not enough? -Before requesting a new feature, please have a quick peek at [the list of supported options](README.md#usage-and-options). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem. - -### Have you read and understood the changes, between youtube-dl and yt-dlp - -There are many changes between youtube-dl and yt-dlp [(changes to default behavior)](README.md#differences-in-default-behavior), and some of the options available have a different behaviour in yt-dlp, or have been removed all together [(list of changes to options)](README.md#deprecated-options). Make sure you have read and understand the differences in the options and how this may impact your downloads before opening an issue. +Before requesting a new feature, please have a quick peek at [the list of supported options](https://github.com/ytdl-org/youtube-dl/blob/master/README.md#options). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem. ### Is there enough context in your bug report? @@ -99,86 +58,68 @@ We are then presented with a very complicated request when the original problem Some of our users seem to think there is a limit of issues they can or should open. There is no limit of issues they can or should open. While it may seem appealing to be able to dump all your issues into one ticket, that means that someone who solves one of your issues cannot mark the issue as closed. Typically, reporting a bunch of issues leads to the ticket lingering since nobody wants to attack that behemoth, until someone mercifully splits the issue into multiple ones. -In particular, every site support request issue should only pertain to services at one site (generally under a common domain, but always using the same backend technology). Do not request support for vimeo user videos, White house podcasts, and Google Plus pages in the same issue. Also, make sure that you don't post bug reports alongside feature requests. As a rule of thumb, a feature request does not include outputs of yt-dlp that are not immediately related to the feature at hand. Do not post reports of a network error alongside the request for a new video service. +In particular, every site support request issue should only pertain to services at one site (generally under a common domain, but always using the same backend technology). Do not request support for vimeo user videos, White house podcasts, and Google Plus pages in the same issue. Also, make sure that you don't post bug reports alongside feature requests. As a rule of thumb, a feature request does not include outputs of hypervideo that are not immediately related to the feature at hand. Do not post reports of a network error alongside the request for a new video service. ### Is anyone going to need the feature? Only post features that you (or an incapacitated friend you can personally talk to) require. Do not post features because they seem like a good idea. If they are really useful, they will be requested by someone who requires them. -### Is your question about yt-dlp? - -Some bug reports are completely unrelated to yt-dlp and relate to a different, or even the reporter's own, application. Please make sure that you are actually using yt-dlp. If you are using a UI for yt-dlp, report the bug to the maintainer of the actual application providing the UI. In general, if you are unable to provide the verbose log, you should not be opening the issue here. - -If the issue is with `youtube-dl` (the upstream fork of yt-dlp) and not with yt-dlp, the issue should be raised in the youtube-dl project. - -### Are you willing to share account details if needed? - -The maintainers and potential contributors of the project often do not have an account for the website you are asking support for. So any developer interested in solving your issue may ask you for account details. It is your personal discretion whether you are willing to share the account in order for the developer to try and solve your issue. However, if you are unwilling or unable to provide details, they obviously cannot work on the issue and it cannot be solved unless some developer who both has an account and is willing/able to contribute decides to solve it. - -By sharing an account with anyone, you agree to bear all risks associated with it. The maintainers and yt-dlp can't be held responsible for any misuse of the credentials. - -While these steps won't necessarily ensure that no misuse of the account takes place, these are still some good practices to follow. - -- Look for people with `Member` (maintainers of the project) or `Contributor` (people who have previously contributed code) tag on their messages. -- Change the password before sharing the account to something random (use [this](https://passwordsgenerator.net/) if you don't have a random password generator). -- Change the password after receiving the account back. - - +### Is your question about hypervideo? +It may sound strange, but some bug reports we receive are completely unrelated to hypervideo and relate to a different, or even the reporter's own, application. Please make sure that you are actually using hypervideo. If you are using a UI for hypervideo, report the bug to the maintainer of the actual application providing the UI. On the other hand, if your UI for hypervideo fails in some way you believe is related to hypervideo, by all means, go ahead and report the bug. # DEVELOPER INSTRUCTIONS -Most users do not need to build yt-dlp and can [download the builds](https://github.com/yt-dlp/yt-dlp/releases) or get them via [the other installation methods](README.md#installation). +Most users do not need to build hypervideo and can [download the builds](https://ytdl-org.github.io/youtube-dl/download.html) or get them from their distribution. -To run yt-dlp as a developer, you don't need to build anything either. Simply execute +To run hypervideo as a developer, you don't need to build anything either. Simply execute - python -m yt_dlp + python -m youtube_dl To run the test, simply invoke your favorite test runner, or execute a test file directly; any of the following work: python -m unittest discover python test/test_download.py nosetests - pytest See item 6 of [new extractor tutorial](#adding-support-for-a-new-site) for how to run extractor specific test cases. -If you want to create a build of yt-dlp yourself, you can follow the instructions [here](README.md#compile). - - -## Adding new feature or making overarching changes - -Before you start writing code for implementing a new feature, open an issue explaining your feature request and atleast one use case. This allows the maintainers to decide whether such a feature is desired for the project in the first place, and will provide an avenue to discuss some implementation details. If you open a pull request for a new feature without discussing with us first, do not be surprised when we ask for large changes to the code, or even reject it outright. +If you want to create a build of hypervideo yourself, you'll need -The same applies for changes to the documentation, code style, or overarching changes to the architecture +* python +* make (only GNU make is supported) +* pandoc +* zip +* nosetests +### Adding support for a new site -## Adding support for a new site - -If you want to add support for a new site, first of all **make sure** this site is **not dedicated to [copyright infringement](https://www.github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free)**. yt-dlp does **not support** such sites thus pull requests adding support for them **will be rejected**. +If you want to add support for a new site, first of all **make sure** this site is **not dedicated to [copyright infringement](README.md#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free)**. hypervideo does **not support** such sites thus pull requests adding support for them **will be rejected**. After you have ensured this site is distributing its content legally, you can follow this quick list (assuming your service is called `yourextractor`): -1. [Fork this repository](https://github.com/yt-dlp/yt-dlp/fork) -1. Check out the source code with: +1. [Fork this repository](https://git.conocimientoslibres.ga/software/hypervideo) +2. Check out the source code with: - git clone git@github.com:YOUR_GITHUB_USERNAME/yt-dlp.git + git clone https://git.conocimientoslibres.ga/software/hypervideo -1. Start a new git branch with +3. Start a new git branch with - cd yt-dlp + cd hypervideo git checkout -b yourextractor -1. Start with this simple template and save it to `yt_dlp/extractor/yourextractor.py`: +4. Start with this simple template and save it to `youtube_dl/extractor/yourextractor.py`: ```python # coding: utf-8 + from __future__ import unicode_literals + from .common import InfoExtractor - - + + class YourExtractorIE(InfoExtractor): _VALID_URL = r'https?://(?:www\.)?yourextractor\.com/watch/(?P<id>[0-9]+)' - _TESTS = [{ + _TEST = { 'url': 'https://yourextractor.com/watch/42', 'md5': 'TODO: md5 sum of the first 10241 bytes of the video file (use --test)', 'info_dict': { @@ -192,12 +133,12 @@ After you have ensured this site is distributing its content legally, you can fo # * A regular expression; start the string with re: # * Any Python type (for example int or float) } - }] + } def _real_extract(self, url): video_id = self._match_id(url) webpage = self._download_webpage(url, video_id) - + # TODO more code goes here, for example ... title = self._html_search_regex(r'<h1>(.+?)</h1>', webpage, 'title') @@ -206,59 +147,45 @@ After you have ensured this site is distributing its content legally, you can fo 'title': title, 'description': self._og_search_description(webpage), 'uploader': self._search_regex(r'<div[^>]+id="uploader"[^>]*>([^<]+)<', webpage, 'uploader', fatal=False), - # TODO more properties (see yt_dlp/extractor/common.py) + # TODO more properties (see youtube_dl/extractor/common.py) } ``` -1. Add an import in [`yt_dlp/extractor/extractors.py`](yt_dlp/extractor/extractors.py). -1. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, the tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc. Note that tests with `only_matching` key in test's dict are not counted in. You can also run all the tests in one go with `TestDownload.test_YourExtractor_all` -1. Make sure you have atleast one test for your extractor. Even if all videos covered by the extractor are expected to be inaccessible for automated testing, tests should still be added with a `skip` parameter indicating why the particular test is disabled from running. -1. Have a look at [`yt_dlp/extractor/common.py`](yt_dlp/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](yt_dlp/extractor/common.py#L91-L426). Add tests and code for as many as you want. -1. Make sure your code follows [yt-dlp coding conventions](#yt-dlp-coding-conventions) and check the code with [flake8](https://flake8.pycqa.org/en/latest/index.html#quickstart): +5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/extractors.py). +6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc. Note that tests with `only_matching` key in test's dict are not counted in. +7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/ytdl-org/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L94-L303). Add tests and code for as many as you want. +8. Make sure your code follows [hypervideo coding conventions](#hypervideo-coding-conventions) and check the code with [flake8](https://flake8.pycqa.org/en/latest/index.html#quickstart): - $ flake8 yt_dlp/extractor/yourextractor.py + $ flake8 youtube_dl/extractor/yourextractor.py -1. Make sure your code works under all [Python](https://www.python.org/) versions supported by yt-dlp, namely CPython and PyPy for Python 3.6 and above. Backward compatibility is not required for even older versions of Python. -1. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files, [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this: +9. Make sure your code works under all [Python](https://www.python.org/) versions claimed supported by hypervideo, namely 2.6, 2.7, and 3.2+. +10. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files and [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this: - $ git add yt_dlp/extractor/extractors.py - $ git add yt_dlp/extractor/yourextractor.py - $ git commit -m '[yourextractor] Add extractor' + $ git add youtube_dl/extractor/extractors.py + $ git add youtube_dl/extractor/yourextractor.py + $ git commit -m '[yourextractor] Add new extractor' $ git push origin yourextractor -1. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it. +11. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it. In any case, thank you very much for your contributions! -**Tip:** To test extractors that require login information, create a file `test/local_parameters.json` and add `"usenetrc": true` or your username and password in it: -```json -{ - "username": "your user name", - "password": "your password" -} -``` - -## yt-dlp coding conventions +## hypervideo coding conventions This section introduces a guide lines for writing idiomatic, robust and future-proof extractor code. -Extractors are very fragile by nature since they depend on the layout of the source data provided by 3rd party media hosters out of your control and this layout tends to change. As an extractor implementer your task is not only to write code that will extract media links and metadata correctly but also to minimize dependency on the source's layout and even to make the code foresee potential future changes and be ready for that. This is important because it will allow the extractor not to break on minor layout changes thus keeping old yt-dlp versions working. Even though this breakage issue may be easily fixed by a new version of yt-dlp, this could take some time, during which the the extractor will remain broken. - +Extractors are very fragile by nature since they depend on the layout of the source data provided by 3rd party media hosters out of your control and this layout tends to change. As an extractor implementer your task is not only to write code that will extract media links and metadata correctly but also to minimize dependency on the source's layout and even to make the code foresee potential future changes and be ready for that. This is important because it will allow the extractor not to break on minor layout changes thus keeping old hypervideo versions working. Even though this breakage issue is easily fixed by emitting a new version of hypervideo with a fix incorporated, all the previous versions become broken in all repositories and distros' packages that may not be so prompt in fetching the update from us. Needless to say, some non rolling release distros may never receive an update at all. ### Mandatory and optional metafields -For extraction to work yt-dlp relies on metadata your extractor extracts and provides to yt-dlp expressed by an [information dictionary](yt_dlp/extractor/common.py#L91-L426) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by yt-dlp: +For extraction to work hypervideo relies on metadata your extractor extracts and provides to hypervideo expressed by an [information dictionary](https://github.com/ytdl-org/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L94-L303) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by youtube-dl: - `id` (media identifier) - `title` (media title) - `url` (media download URL) or `formats` -The aforementioned metafields are the critical data that the extraction does not make any sense without and if any of them fail to be extracted then the extractor is considered completely broken. While all extractors must return a `title`, they must also allow it's extraction to be non-fatal. - -For pornographic sites, appropriate `age_limit` must also be returned. - -The extractor is allowed to return the info dict without url or formats in some special cases if it allows the user to extract usefull information with `--ignore-no-formats-error` - Eg: when the video is a live stream that has not started yet. +In fact only the last option is technically mandatory (i.e. if you can't figure out the download location of the media the extraction does not make any sense). But by convention hypervideo also treats `id` and `title` as mandatory. Thus the aforementioned metafields are the critical data that the extraction does not make any sense without and if any of them fail to be extracted then the extractor is considered completely broken. -[Any field](yt_dlp/extractor/common.py#219-L426) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerant** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields. +[Any field](https://github.com/ytdl-org/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L188-L303) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerant** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields. #### Example @@ -267,15 +194,13 @@ Say you have some source dictionary `meta` that you've fetched as JSON with HTTP ```python meta = self._download_json(url, video_id) ``` - + Assume at this point `meta`'s layout is: ```python { + ... "summary": "some fancy summary text", - "user": { - "name": "uploader name" - }, ... } ``` @@ -294,30 +219,6 @@ description = meta['summary'] # incorrect The latter will break extraction process with `KeyError` if `summary` disappears from `meta` at some later time but with the former approach extraction will just go ahead with `description` set to `None` which is perfectly fine (remember `None` is equivalent to the absence of data). - -If the data is nested, do not use `.get` chains, but instead make use of the utility functions `try_get` or `traverse_obj` - -Considering the above `meta` again, assume you want to extract `["user"]["name"]` and put it in the resulting info dict as `uploader` - -```python -uploader = try_get(meta, lambda x: x['user']['name']) # correct -``` -or -```python -uploader = traverse_obj(meta, ('user', 'name')) # correct -``` - -and not like: - -```python -uploader = meta['user']['name'] # incorrect -``` -or -```python -uploader = meta.get('user', {}).get('name') # incorrect -``` - - Similarly, you should pass `fatal=False` when extracting optional data from a webpage with `_search_regex`, `_html_search_regex` or similar methods, for instance: ```python @@ -338,35 +239,10 @@ description = self._search_regex( On failure this code will silently continue the extraction with `description` set to `None`. That is useful for metafields that may or may not be present. - -Another thing to remember is not to try to iterate over `None` - -Say you extracted a list of thumbnails into `thumbnail_data` using `try_get` and now want to iterate over them - -```python -thumbnail_data = try_get(...) -thumbnails = [{ - 'url': item['url'] -} for item in thumbnail_data or []] # correct -``` - -and not like: - -```python -thumbnail_data = try_get(...) -thumbnails = [{ - 'url': item['url'] -} for item in thumbnail_data] # incorrect -``` - -In the later case, `thumbnail_data` will be `None` if the field was not found and this will cause the loop `for item in thumbnail_data` to raise a fatal error. Using `for item in thumbnail_data or []` avoids this error and results in setting an empty list in `thumbnails` instead. - - ### Provide fallbacks When extracting metadata try to do so from multiple sources. For example if `title` is present in several places, try extracting from at least some of them. This makes it more future-proof in case some of the sources become unavailable. - #### Example Say `meta` from the previous example has a `title` and you are about to extract it. Since `title` is a mandatory meta field you should end up with something like: @@ -385,7 +261,6 @@ title = meta.get('title') or self._og_search_title(webpage) This code will try to extract from `meta` first and if it fails it will try extracting `og:title` from a `webpage`. - ### Regular expressions #### Don't capture groups you don't use @@ -407,6 +282,7 @@ Incorrect: r'(id|ID)=(?P<id>\d+)' ``` + #### Make regular expressions relaxed and flexible When using regular expressions try to write them fuzzy, relaxed and flexible, skipping insignificant parts that are more likely to change, allowing both single and double quotes for quoted values and so on. @@ -422,49 +298,34 @@ Say you need to extract `title` from the following HTML code: The code for that task should look similar to: ```python -title = self._search_regex( # correct +title = self._search_regex( r'<span[^>]+class="title"[^>]*>([^<]+)', webpage, 'title') ``` Or even better: ```python -title = self._search_regex( # correct +title = self._search_regex( r'<span[^>]+class=(["\'])title\1[^>]*>(?P<title>[^<]+)', webpage, 'title', group='title') ``` -Note how you tolerate potential changes in the `style` attribute's value or switch from using double quotes to single for `class` attribute: +Note how you tolerate potential changes in the `style` attribute's value or switch from using double quotes to single for `class` attribute: The code definitely should not look like: ```python -title = self._search_regex( # incorrect +title = self._search_regex( r'<span style="position: absolute; left: 910px; width: 90px; float: right; z-index: 9999;" class="title">(.*?)</span>', webpage, 'title', group='title') ``` -or even - -```python -title = self._search_regex( # incorrect - r'<span style=".*?" class="title">(.*?)</span>', - webpage, 'title', group='title') -``` - -Here the presence or absence of other attributes including `style` is irrelevent for the data we need, and so the regex must not depend on it - - ### Long lines policy -There is a soft limit to keep lines of code under 100 characters long. This means it should be respected if possible and if it does not make readability and code maintenance worse. Sometimes, it may be reasonable to go upto 120 characters and sometimes even 80 can be unreadable. Keep in mind that this is not a hard limit and is just one of many tools to make the code more readable. +There is a soft limit to keep lines of code under 80 characters long. This means it should be respected if possible and if it does not make readability and code maintenance worse. For example, you should **never** split long string literals like URLs or some other often copied entities over multiple lines to fit this limit: -Conversely, don't unecessarily split small lines further. As a rule of thumb, if removing the line split keeps the code under 80 characters, it should be a single line. - -##### Examples - Correct: ```python @@ -478,47 +339,6 @@ Incorrect: 'PLMYEtVRpaqY00V9W81Cwmzp6N6vZqfUKD4' ``` -Correct: - -```python -uploader = traverse_obj(info, ('uploader', 'name'), ('author', 'fullname')) -``` - -Incorrect: - -```python -uploader = traverse_obj( - info, - ('uploader', 'name'), - ('author', 'fullname')) -``` - -Correct: - -```python -formats = self._extract_m3u8_formats( - m3u8_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', - note='Downloading HD m3u8 information', errnote='Unable to download HD m3u8 information') -``` - -Incorrect: - -```python -formats = self._extract_m3u8_formats(m3u8_url, - video_id, - 'mp4', - 'm3u8_native', - m3u8_id='hls', - note='Downloading HD m3u8 information', - errnote='Unable to download HD m3u8 information') -``` - - -### Quotes - -Always use single quotes for strings (even if the string has `'`) and double quotes for docstrings. Use `'''` only for multi-line strings. An exception can be made if a string has multiple single quotes in it and escaping makes it significantly harder to read. For f-strings, use you can use double quotes on the inside. But avoid f-strings that have too many quotes inside. - - ### Inline values Extracting variables is acceptable for reducing code duplication and improving readability of complex expressions. However, you should avoid extracting variables used only once and moving them to opposite parts of the extractor file, which makes reading the linear flow difficult. @@ -539,7 +359,6 @@ TITLE_RE = r'<title>([^<]+)</title>' title = self._html_search_regex(TITLE_RE, webpage, 'title') ``` - ### Collapse fallbacks Multiple fallback values can quickly become unwieldy. Collapse multiple fallback values into a single expression via a list of patterns. @@ -565,101 +384,52 @@ description = ( Methods supporting list of patterns are: `_search_regex`, `_html_search_regex`, `_og_search_property`, `_html_search_meta`. - ### Trailing parentheses -Always move trailing parentheses used for grouping/functions after the last argument. On the other hand, literal list/tuple/dict/set should closed be in a new line. Generators and list/dict comprehensions may use either style +Always move trailing parentheses after the last argument. -#### Examples +#### Example Correct: ```python -url = try_get( - info, lambda x: x['ResultSet']['Result'][0]['VideoUrlSet']['VideoUrl'], list) ``` -Correct: - -```python -url = try_get(info, - lambda x: x['ResultSet']['Result'][0]['VideoUrlSet']['VideoUrl'], - list) -``` Incorrect: ```python -url = try_get( - info, lambda x: x['ResultSet']['Result'][0]['VideoUrlSet']['VideoUrl'], list, ) ``` -Correct: - -```python -f = { - 'url': url, - 'format_id': format_id, -} -``` - -Incorrect: - -```python -f = {'url': url, - 'format_id': format_id} -``` - -Correct: - -```python -formats = [process_formats(f) for f in format_data - if f.get('type') in ('hls', 'dash', 'direct') and f.get('downloadable')] -``` - -Correct: - -```python -formats = [ - process_formats(f) for f in format_data - if f.get('type') in ('hls', 'dash', 'direct') and f.get('downloadable') -] -``` - - ### Use convenience conversion and parsing functions -Wrap all extracted numeric data into safe functions from [`yt_dlp/utils.py`](yt_dlp/utils.py): `int_or_none`, `float_or_none`. Use them for string to number conversions as well. +Wrap all extracted numeric data into safe functions from [`youtube_dl/utils.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/utils.py): `int_or_none`, `float_or_none`. Use them for string to number conversions as well. Use `url_or_none` for safe URL processing. -Use `try_get`, `dict_get` and `traverse_obj` for safe metadata extraction from parsed JSON. +Use `try_get` for safe metadata extraction from parsed JSON. -Use `unified_strdate` for uniform `upload_date` or any `YYYYMMDD` meta field extraction, `unified_timestamp` for uniform `timestamp` extraction, `parse_filesize` for `filesize` extraction, `parse_count` for count meta fields extraction, `parse_resolution`, `parse_duration` for `duration` extraction, `parse_age_limit` for `age_limit` extraction. +Use `unified_strdate` for uniform `upload_date` or any `YYYYMMDD` meta field extraction, `unified_timestamp` for uniform `timestamp` extraction, `parse_filesize` for `filesize` extraction, `parse_count` for count meta fields extraction, `parse_resolution`, `parse_duration` for `duration` extraction, `parse_age_limit` for `age_limit` extraction. -Explore [`yt_dlp/utils.py`](yt_dlp/utils.py) for more useful convenience functions. +Explore [`youtube_dl/utils.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/utils.py) for more useful convenience functions. #### More examples ##### Safely extract optional description from parsed JSON + ```python -description = traverse_obj(response, ('result', 'video', 'summary'), expected_type=str) +description = try_get(response, lambda x: x['result']['video'][0]['summary'], compat_str) ``` ##### Safely extract more optional metadata + ```python -video = traverse_obj(response, ('result', 'video', 0), default={}, expected_type=dict) +video = try_get(response, lambda x: x['result']['video'][0], dict) or {} description = video.get('summary') duration = float_or_none(video.get('durationMs'), scale=1000) view_count = int_or_none(video.get('views')) ``` - - - - -# EMBEDDING YT-DLP -See [README.md#embedding-yt-dlp](README.md#embedding-yt-dlp) for instructions on how to embed yt-dlp in another Python program diff --git a/Changelog.md b/Changelog.md index 6f564caa0..0a76f65be 100644 --- a/Changelog.md +++ b/Changelog.md @@ -661,7 +661,7 @@ * [update] Check for new version even if not updateable * [build] Add more files to the tarball * [build] Allow building with py2exe (and misc fixes) -* [build] Use pycryptodomex by [shirt](https://github.com/shirt-dev), [pukkandan](https://github.com/pukkandan) +* [build] Use pycryptodome by [shirt](https://github.com/shirt-dev), [pukkandan](https://github.com/pukkandan) * [cleanup] Some minor refactoring, improve docs and misc cleanup @@ -1122,7 +1122,7 @@ * [youtube] Non-fatal alert reporting for unavailable videos page by [coletdjnz](https://github.com/coletdjnz) * [twitcasting] Websocket support by [nao20010128nao](https://github.com/nao20010128nao) * [mediasite] Extract slides by [fstirlitz](https://github.com/fstirlitz) -* [funimation] Extract subtitles +* [funimation] Extract subtitles * [pornhub] Extract `cast` * [hotstar] Use server time for authentication instead of local time * [EmbedThumbnail] Fix for already downloaded thumbnail @@ -1218,7 +1218,7 @@ ### 2021.05.20 -* **Youtube improvements**: +* **Youtube improvements**: * Support youtube music `MP`, `VL` and `browse` pages * Extract more formats for youtube music by [craftingmod](https://github.com/craftingmod), [coletdjnz](https://github.com/coletdjnz) and [pukkandan](https://github.com/pukkandan) * Extract multiple subtitles in same language by [pukkandan](https://github.com/pukkandan) and [tpikonen](https://github.com/tpikonen) diff --git a/Collaborators.md b/Collaborators.md deleted file mode 100644 index 29d320dd6..000000000 --- a/Collaborators.md +++ /dev/null @@ -1,50 +0,0 @@ -# Collaborators - -This is a list of the collaborators of the project and their major contributions. See the [Changelog](Changelog.md) for more details. - -You can also find lists of all [contributors of yt-dlp](CONTRIBUTORS) and [authors of youtube-dl](https://github.com/ytdl-org/youtube-dl/blob/master/AUTHORS) - - -## [pukkandan](https://github.com/pukkandan) - -[](https://ko-fi.com/pukkandan) - -* Owner of the fork - - - -## [shirt](https://github.com/shirt-dev) - -[](https://ko-fi.com/shirt) - -* Multithreading (`-N`) and aria2c support for fragment downloads -* Support for media initialization and discontinuity in HLS -* The self-updater (`-U`) - - - -## [coletdjnz](https://github.com/coletdjnz) - -[](https://github.com/sponsors/coletdjnz) - -* YouTube improvements including: age-gate bypass, private playlists, multiple-clients (to avoid throttling) and a lot of under-the-hood improvements -* Added support for downloading YoutubeWebArchive videos - - - -## [Ashish0804](https://github.com/Ashish0804) - -[](https://ko-fi.com/ashish0804) - -* Added support for new websites BiliIntl, DiscoveryPlusIndia, OlympicsReplay, PlanetMarathi, ShemarooMe, Utreon, Zee5 etc -* Added playlist/series downloads for Hotstar, ParamountPlus, Rumble, SonyLIV, Trovo, TubiTv, Voot etc -* Improved/fixed support for HiDive, HotStar, Hungama, LBRY, LinkedInLearning, Mxplayer, SonyLiv, TV2, Vimeo, VLive etc - - -## [Lesmiscore](https://github.com/Lesmiscore) (nao20010128nao) - -**Bitcoin**: bc1qfd02r007cutfdjwjmyy9w23rjvtls6ncve7r3s -**Monacoin**: mona1q3tf7dzvshrhfe3md379xtvt2n22duhglv5dskr - -* Download live from start to end for YouTube -* Added support for new websites mildom, PixivSketch, skeb, radiko, voicy, mirrativ, openrec, whowatch, damtomo, 17.live, mixch etc @@ -1,24 +1,121 @@ -This is free and unencumbered software released into the public domain. - -Anyone is free to copy, modify, publish, use, compile, sell, or -distribute this software, either in source code form or as a compiled -binary, for any purpose, commercial or non-commercial, and by any -means. - -In jurisdictions that recognize copyright laws, the author or authors -of this software dedicate any and all copyright interest in the -software to the public domain. We make this dedication for the benefit -of the public at large and to the detriment of our heirs and -successors. We intend this dedication to be an overt act of -relinquishment in perpetuity of all present and future rights to this -software under copyright law. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. -IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR -OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -OTHER DEALINGS IN THE SOFTWARE. - -For more information, please refer to <http://unlicense.org/> +Creative Commons Legal Code + +CC0 1.0 Universal + + CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE + LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN + ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS + INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES + REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS + PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM + THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED + HEREUNDER. + +Statement of Purpose + +The laws of most jurisdictions throughout the world automatically confer +exclusive Copyright and Related Rights (defined below) upon the creator +and subsequent owner(s) (each and all, an "owner") of an original work of +authorship and/or a database (each, a "Work"). + +Certain owners wish to permanently relinquish those rights to a Work for +the purpose of contributing to a commons of creative, cultural and +scientific works ("Commons") that the public can reliably and without fear +of later claims of infringement build upon, modify, incorporate in other +works, reuse and redistribute as freely as possible in any form whatsoever +and for any purposes, including without limitation commercial purposes. +These owners may contribute to the Commons to promote the ideal of a free +culture and the further production of creative, cultural and scientific +works, or to gain reputation or greater distribution for their Work in +part through the use and efforts of others. + +For these and/or other purposes and motivations, and without any +expectation of additional consideration or compensation, the person +associating CC0 with a Work (the "Affirmer"), to the extent that he or she +is an owner of Copyright and Related Rights in the Work, voluntarily +elects to apply CC0 to the Work and publicly distribute the Work under its +terms, with knowledge of his or her Copyright and Related Rights in the +Work and the meaning and intended legal effect of CC0 on those rights. + +1. Copyright and Related Rights. A Work made available under CC0 may be +protected by copyright and related or neighboring rights ("Copyright and +Related Rights"). Copyright and Related Rights include, but are not +limited to, the following: + + i. the right to reproduce, adapt, distribute, perform, display, + communicate, and translate a Work; + ii. moral rights retained by the original author(s) and/or performer(s); +iii. publicity and privacy rights pertaining to a person's image or + likeness depicted in a Work; + iv. rights protecting against unfair competition in regards to a Work, + subject to the limitations in paragraph 4(a), below; + v. rights protecting the extraction, dissemination, use and reuse of data + in a Work; + vi. database rights (such as those arising under Directive 96/9/EC of the + European Parliament and of the Council of 11 March 1996 on the legal + protection of databases, and under any national implementation + thereof, including any amended or successor version of such + directive); and +vii. other similar, equivalent or corresponding rights throughout the + world based on applicable law or treaty, and any national + implementations thereof. + +2. Waiver. To the greatest extent permitted by, but not in contravention +of, applicable law, Affirmer hereby overtly, fully, permanently, +irrevocably and unconditionally waives, abandons, and surrenders all of +Affirmer's Copyright and Related Rights and associated claims and causes +of action, whether now known or unknown (including existing as well as +future claims and causes of action), in the Work (i) in all territories +worldwide, (ii) for the maximum duration provided by applicable law or +treaty (including future time extensions), (iii) in any current or future +medium and for any number of copies, and (iv) for any purpose whatsoever, +including without limitation commercial, advertising or promotional +purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each +member of the public at large and to the detriment of Affirmer's heirs and +successors, fully intending that such Waiver shall not be subject to +revocation, rescission, cancellation, termination, or any other legal or +equitable action to disrupt the quiet enjoyment of the Work by the public +as contemplated by Affirmer's express Statement of Purpose. + +3. Public License Fallback. Should any part of the Waiver for any reason +be judged legally invalid or ineffective under applicable law, then the +Waiver shall be preserved to the maximum extent permitted taking into +account Affirmer's express Statement of Purpose. In addition, to the +extent the Waiver is so judged Affirmer hereby grants to each affected +person a royalty-free, non transferable, non sublicensable, non exclusive, +irrevocable and unconditional license to exercise Affirmer's Copyright and +Related Rights in the Work (i) in all territories worldwide, (ii) for the +maximum duration provided by applicable law or treaty (including future +time extensions), (iii) in any current or future medium and for any number +of copies, and (iv) for any purpose whatsoever, including without +limitation commercial, advertising or promotional purposes (the +"License"). The License shall be deemed effective as of the date CC0 was +applied by Affirmer to the Work. Should any part of the License for any +reason be judged legally invalid or ineffective under applicable law, such +partial invalidity or ineffectiveness shall not invalidate the remainder +of the License, and in such case Affirmer hereby affirms that he or she +will not (i) exercise any of his or her remaining Copyright and Related +Rights in the Work or (ii) assert any associated claims and causes of +action with respect to the Work, in either case contrary to Affirmer's +express Statement of Purpose. + +4. Limitations and Disclaimers. + + a. No trademark or patent rights held by Affirmer are waived, abandoned, + surrendered, licensed or otherwise affected by this document. + b. Affirmer offers the Work as-is and makes no representations or + warranties of any kind concerning the Work, express, implied, + statutory or otherwise, including without limitation warranties of + title, merchantability, fitness for a particular purpose, non + infringement, or the absence of latent or other defects, accuracy, or + the present or absence of errors, whether or not discoverable, all to + the greatest extent permissible under applicable law. + c. Affirmer disclaims responsibility for clearing rights of other persons + that may apply to the Work or any use thereof, including without + limitation any person's Copyright and Related Rights in the Work. + Further, Affirmer disclaims responsibility for obtaining any necessary + consents, permissions or other rights required for any use of the + Work. + d. Affirmer understands and acknowledges that Creative Commons is not a + party to this document and has no duty or obligation with respect to + this CC0 or use of the Work. @@ -1,28 +1,25 @@ -all: lazy-extractors yt-dlp doc pypi-files -clean: clean-test clean-dist -clean-all: clean clean-cache +all: yt-dlp doc pypi-files +clean: clean-test clean-dist clean-cache completions: completion-bash completion-fish completion-zsh -doc: README.md CONTRIBUTING.md issuetemplates supportedsites +doc: README.md CONTRIBUTING.md ot: offlinetest tar: yt-dlp.tar.gz # Keep this list in sync with MANIFEST.in # intended use: when building a source distribution, # make pypi-files && python setup.py sdist -pypi-files: AUTHORS Changelog.md LICENSE README.md README.txt supportedsites completions yt-dlp.1 devscripts/* test/* +pypi-files: AUTHORS Changelog.md LICENSE README.md completions devscripts/* test/* -.PHONY: all clean install test tar pypi-files completions ot offlinetest codetest supportedsites +.PHONY: all clean install test tar pypi-files completions ot offlinetest codetest clean-test: - rm -rf test/testdata/sigs/player-*.js tmp/ *.annotations.xml *.aria2 *.description *.dump *.frag \ - *.frag.aria2 *.frag.urls *.info.json *.live_chat.json *.meta *.part* *.tmp *.temp *.unknown_video *.ytdl \ - *.3gp *.ape *.avi *.desktop *.flac *.flv *.jpeg *.jpg *.m4a *.m4v *.mhtml *.mkv *.mov *.mp3 \ - *.mp4 *.ogg *.opus *.png *.sbv *.srt *.swf *.swp *.ttml *.url *.vtt *.wav *.webloc *.webm *.webp + rm -rf *.3gp *.annotations.xml *.ape *.avi *.description *.dump *.flac *.flv *.frag *.frag.aria2 *.frag.urls \ + *.info.json *.jpeg *.jpg *.live_chat.json *.m4a *.m4v *.mkv *.mp3 *.mp4 *.ogg *.opus *.part* *.png *.sbv *.srt \ + *.swf *.swp *.ttml *.vtt *.wav *.webm *.webp *.ytdl test/testdata/player-*.js clean-dist: - rm -rf yt-dlp.1.temp.md yt-dlp.1 README.txt MANIFEST build/ dist/ .coverage cover/ yt-dlp.tar.gz completions/ \ - yt_dlp/extractor/lazy_extractors.py *.spec CONTRIBUTING.md.tmp yt-dlp yt-dlp.exe yt_dlp.egg-info/ AUTHORS .mailmap + rm -rf MANIFEST build/ dist/ .coverage cover/ yt-dlp.tar.gz completions/ yt_dlp/extractor/lazy_extractors.py *.spec CONTRIBUTING.md.tmp yt-dlp yt-dlp.exe yt_dlp.egg-info/ AUTHORS .mailmap clean-cache: - find . \( -name "*.pyc" -o -name "*.class" \) -delete + find . -name "*.pyc" -o -name "*.class" -delete completion-bash: completions/bash/yt-dlp completion-fish: completions/fish/yt-dlp.fish @@ -39,12 +36,8 @@ PYTHON ?= /usr/bin/env python3 # set SYSCONFDIR to /etc if PREFIX=/usr or PREFIX=/usr/local SYSCONFDIR = $(shell if [ $(PREFIX) = /usr -o $(PREFIX) = /usr/local ]; then echo /etc; else echo $(PREFIX)/etc; fi) -# set markdown input format to "markdown-smart" for pandoc version 2 and to "markdown" for pandoc prior to version 2 -MARKDOWN = $(shell if [ `pandoc -v | head -n1 | cut -d" " -f2 | head -c1` = "2" ]; then echo markdown-smart; else echo markdown; fi) - -install: lazy-extractors yt-dlp yt-dlp.1 completions - install -Dm755 yt-dlp $(DESTDIR)$(BINDIR)/yt-dlp - install -Dm644 yt-dlp.1 $(DESTDIR)$(MANDIR)/man1/yt-dlp.1 +install: yt-dlp completions + install -Dm755 yt-dlp $(DESTDIR)$(BINDIR) install -Dm644 completions/bash/yt-dlp $(DESTDIR)$(SHAREDIR)/bash-completion/completions/yt-dlp install -Dm644 completions/zsh/_yt-dlp $(DESTDIR)$(SHAREDIR)/zsh/site-functions/_yt-dlp install -Dm644 completions/fish/yt-dlp.fish $(DESTDIR)$(SHAREDIR)/fish/vendor_completions.d/yt-dlp.fish @@ -80,25 +73,6 @@ README.md: yt_dlp/*.py yt_dlp/*/*.py CONTRIBUTING.md: README.md $(PYTHON) devscripts/make_contributing.py README.md CONTRIBUTING.md -issuetemplates: devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.yml .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.yml .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.yml .github/ISSUE_TEMPLATE_tmpl/4_bug_report.yml .github/ISSUE_TEMPLATE_tmpl/5_feature_request.yml yt_dlp/version.py - $(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.yml .github/ISSUE_TEMPLATE/1_broken_site.yml - $(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.yml .github/ISSUE_TEMPLATE/2_site_support_request.yml - $(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.yml .github/ISSUE_TEMPLATE/3_site_feature_request.yml - $(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/4_bug_report.yml .github/ISSUE_TEMPLATE/4_bug_report.yml - $(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/5_feature_request.yml .github/ISSUE_TEMPLATE/5_feature_request.yml - $(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/6_question.yml .github/ISSUE_TEMPLATE/6_question.yml - -supportedsites: - $(PYTHON) devscripts/make_supportedsites.py supportedsites.md - -README.txt: README.md - pandoc -f $(MARKDOWN) -t plain README.md -o README.txt - -yt-dlp.1: README.md - $(PYTHON) devscripts/prepare_manpage.py yt-dlp.1.temp.md - pandoc -s -f $(MARKDOWN) -t man yt-dlp.1.temp.md -o yt-dlp.1 - rm -f yt-dlp.1.temp.md - completions/bash/yt-dlp: yt_dlp/*.py yt_dlp/*/*.py devscripts/bash-completion.in mkdir -p completions/bash $(PYTHON) devscripts/bash-completion.py @@ -125,9 +99,9 @@ yt-dlp.tar.gz: all --exclude '__pycache__' \ --exclude '.git' \ -- \ - README.md supportedsites.md Changelog.md LICENSE \ - CONTRIBUTING.md Collaborators.md CONTRIBUTORS AUTHORS \ - Makefile MANIFEST.in yt-dlp.1 README.txt completions \ + bin README.md Changelog.md LICENSE \ + CONTRIBUTING.md CONTRIBUTORS AUTHORS \ + Makefile MANIFEST.in README.md completions \ setup.py setup.cfg yt-dlp yt_dlp requirements.txt \ devscripts test tox.ini pytest.ini @@ -1,1067 +1,465 @@ -<!-- MANPAGE: BEGIN EXCLUDED SECTION --> -<div align="center"> - -[](#readme) - -[](#release-files "Release") -[](LICENSE "License") -[](Collaborators.md#collaborators "Donate") -[](https://readthedocs.org/projects/yt-dlp/ "Docs") -[](supportedsites.md "Supported Sites") -[](https://pypi.org/project/yt-dlp "PyPi") -[](https://github.com/yt-dlp/yt-dlp/actions "CI Status") -[](https://discord.gg/H5MNcFW63r "Discord") -[](https://matrix.to/#/#yt-dlp:matrix.org "Matrix") -[](https://github.com/yt-dlp/yt-dlp/commits "Commit History") -[](https://github.com/yt-dlp/yt-dlp/commits "Commit History") - -</div> -<!-- MANPAGE: END EXCLUDED SECTION --> - -yt-dlp is a [youtube-dl](https://github.com/ytdl-org/youtube-dl) fork based on the now inactive [youtube-dlc](https://github.com/blackjack4494/yt-dlc). The main focus of this project is adding new features and patches while also keeping up to date with the original project - -<!-- MANPAGE: MOVE "USAGE AND OPTIONS" SECTION HERE --> - -<!-- MANPAGE: BEGIN EXCLUDED SECTION --> -* [NEW FEATURES](#new-features) - * [Differences in default behavior](#differences-in-default-behavior) -* [INSTALLATION](#installation) - * [Update](#update) - * [Release Files](#release-files) - * [Dependencies](#dependencies) - * [Compile](#compile) -* [USAGE AND OPTIONS](#usage-and-options) - * [General Options](#general-options) - * [Network Options](#network-options) - * [Geo-restriction](#geo-restriction) - * [Video Selection](#video-selection) - * [Download Options](#download-options) - * [Filesystem Options](#filesystem-options) - * [Thumbnail Options](#thumbnail-options) - * [Internet Shortcut Options](#internet-shortcut-options) - * [Verbosity and Simulation Options](#verbosity-and-simulation-options) - * [Workarounds](#workarounds) - * [Video Format Options](#video-format-options) - * [Subtitle Options](#subtitle-options) - * [Authentication Options](#authentication-options) - * [Post-processing Options](#post-processing-options) - * [SponsorBlock Options](#sponsorblock-options) - * [Extractor Options](#extractor-options) -* [CONFIGURATION](#configuration) - * [Authentication with .netrc file](#authentication-with-netrc-file) -* [OUTPUT TEMPLATE](#output-template) - * [Output template and Windows batch files](#output-template-and-windows-batch-files) - * [Output template examples](#output-template-examples) -* [FORMAT SELECTION](#format-selection) - * [Filtering Formats](#filtering-formats) - * [Sorting Formats](#sorting-formats) - * [Format Selection examples](#format-selection-examples) -* [MODIFYING METADATA](#modifying-metadata) - * [Modifying metadata examples](#modifying-metadata-examples) -* [EXTRACTOR ARGUMENTS](#extractor-arguments) -* [PLUGINS](#plugins) -* [EMBEDDING YT-DLP](#embedding-yt-dlp) -* [DEPRECATED OPTIONS](#deprecated-options) -* [CONTRIBUTING](CONTRIBUTING.md#contributing-to-yt-dlp) - * [Opening an Issue](CONTRIBUTING.md#opening-an-issue) - * [Developer Instructions](CONTRIBUTING.md#developer-instructions) -* [MORE](#more) -<!-- MANPAGE: END EXCLUDED SECTION --> - - -# NEW FEATURES - -* Based on **youtube-dl 2021.12.17 [commit/5add3f4](https://github.com/ytdl-org/youtube-dl/commit/5add3f4373287e6346ca3551239edab549284db3)** and **youtube-dlc 2020.11.11-3 [commit/f9401f2](https://github.com/blackjack4494/yt-dlc/commit/f9401f2a91987068139c5f757b12fc711d4c0cee)**: You get all the features and patches of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) in addition to the latest [youtube-dl](https://github.com/ytdl-org/youtube-dl) - -* **[SponsorBlock Integration](#sponsorblock-options)**: You can mark/remove sponsor sections in youtube videos by utilizing the [SponsorBlock](https://sponsor.ajay.app) API - -* **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection than what is possible by simply using `--format` ([examples](#format-selection-examples)) - -* **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--write-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/yt-dlp/yt-dlp/pull/31) for details. - -* **Youtube improvements**: - * All Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) and private playlists supports downloading multiple pages of content - * Search (`ytsearch:`, `ytsearchdate:`), search URLs and in-channel search works - * Mixes supports downloading multiple pages of content - * Some (but not all) age-gated content can be downloaded without cookies - * Fix for [n-sig based throttling](https://github.com/ytdl-org/youtube-dl/issues/29326) - * Redirect channel's home URL automatically to `/video` to preserve the old behaviour - * `255kbps` audio is extracted (if available) from youtube music when premium cookies are given - * Youtube music Albums, channels etc can be downloaded ([except self-uploaded music](https://github.com/yt-dlp/yt-dlp/issues/723)) - * Download livestreams from the start using `--live-from-start` (experimental) - -* **Cookies from browser**: Cookies can be automatically extracted from all major web browsers using `--cookies-from-browser BROWSER[+KEYRING][:PROFILE]` - -* **Split video by chapters**: Videos can be split into multiple files based on chapters using `--split-chapters` - -* **Multi-threaded fragment downloads**: Download multiple fragments of m3u8/mpd videos in parallel. Use `--concurrent-fragments` (`-N`) option to set the number of threads used - -* **Aria2c with HLS/DASH**: You can use `aria2c` as the external downloader for DASH(mpd) and HLS(m3u8) formats - -* **New and fixed extractors**: Many new extractors have been added and a lot of existing ones have been fixed. See the [changelog](Changelog.md) or the [list of supported sites](supportedsites.md) - -* **New MSOs**: Philo, Spectrum, SlingTV, Cablevision, RCN - -* **Subtitle extraction from manifests**: Subtitles can be extracted from streaming media manifests. See [commit/be6202f](https://github.com/yt-dlp/yt-dlp/commit/be6202f12b97858b9d716e608394b51065d0419f) for details - -* **Multiple paths and output templates**: You can give different [output templates](#output-template) and download paths for different types of files. You can also set a temporary path where intermediary files are downloaded to using `--paths` (`-P`) - -* **Portable Configuration**: Configuration files are automatically loaded from the home and root directories. See [configuration](#configuration) for details - -* **Output template improvements**: Output templates can now have date-time formatting, numeric offsets, object traversal etc. See [output template](#output-template) for details. Even more advanced operations can also be done with the help of `--parse-metadata` and `--replace-in-metadata` - -* **Other new options**: Many new options have been added such as `--concat-playlist`, `--print`, `--wait-for-video`, `--sleep-requests`, `--convert-thumbnails`, `--write-link`, `--force-download-archive`, `--force-overwrites`, `--break-on-reject` etc - -* **Improvements**: Regex and other operators in `--match-filter`, multiple `--postprocessor-args` and `--downloader-args`, faster archive checking, more [format selection options](#format-selection), merge multi-video/audio, multiple `--config-locations`, `--exec` at different stages, etc - -* **Plugins**: Extractors and PostProcessors can be loaded from an external file. See [plugins](#plugins) for details - -* **Self-updater**: The releases can be updated using `yt-dlp -U` - -See [changelog](Changelog.md) or [commits](https://github.com/yt-dlp/yt-dlp/commits) for the full list of changes - -### Differences in default behavior - -Some of yt-dlp's default options are different from that of youtube-dl and youtube-dlc: - -* The options `--auto-number` (`-A`), `--title` (`-t`) and `--literal` (`-l`), no longer work. See [removed options](#Removed) for details -* `avconv` is not supported as an alternative to `ffmpeg` -* The default [output template](#output-template) is `%(title)s [%(id)s].%(ext)s`. There is no real reason for this change. This was changed before yt-dlp was ever made public and now there are no plans to change it back to `%(title)s-%(id)s.%(ext)s`. Instead, you may use `--compat-options filename` -* The default [format sorting](#sorting-formats) is different from youtube-dl and prefers higher resolution and better codecs rather than higher bitrates. You can use the `--format-sort` option to change this to any order you prefer, or use `--compat-options format-sort` to use youtube-dl's sorting order -* The default format selector is `bv*+ba/b`. This means that if a combined video + audio format that is better than the best video-only format is found, the former will be preferred. Use `-f bv+ba/b` or `--compat-options format-spec` to revert this -* Unlike youtube-dlc, yt-dlp does not allow merging multiple audio/video streams into one file by default (since this conflicts with the use of `-f bv*+ba`). If needed, this feature must be enabled using `--audio-multistreams` and `--video-multistreams`. You can also use `--compat-options multistreams` to enable both -* `--ignore-errors` is enabled by default. Use `--abort-on-error` or `--compat-options abort-on-error` to abort on errors instead -* When writing metadata files such as thumbnails, description or infojson, the same information (if available) is also written for playlists. Use `--no-write-playlist-metafiles` or `--compat-options no-playlist-metafiles` to not write these files -* `--add-metadata` attaches the `infojson` to `mkv` files in addition to writing the metadata when used with `--write-info-json`. Use `--no-embed-info-json` or `--compat-options no-attach-info-json` to revert this -* Some metadata are embedded into different fields when using `--add-metadata` as compared to youtube-dl. Most notably, `comment` field contains the `webpage_url` and `synopsis` contains the `description`. You can [use `--parse-metadata`](#modifying-metadata) to modify this to your liking or use `--compat-options embed-metadata` to revert this -* `playlist_index` behaves differently when used with options like `--playlist-reverse` and `--playlist-items`. See [#302](https://github.com/yt-dlp/yt-dlp/issues/302) for details. You can use `--compat-options playlist-index` if you want to keep the earlier behavior -* The output of `-F` is listed in a new format. Use `--compat-options list-formats` to revert this -* All *experiences* of a funimation episode are considered as a single video. This behavior breaks existing archives. Use `--compat-options seperate-video-versions` to extract information from only the default player -* Youtube live chat (if available) is considered as a subtitle. Use `--sub-langs all,-live_chat` to download all subtitles except live chat. You can also use `--compat-options no-live-chat` to prevent live chat from downloading -* Youtube channel URLs are automatically redirected to `/video`. Append a `/featured` to the URL to download only the videos in the home page. If the channel does not have a videos tab, we try to download the equivalent `UU` playlist instead. For all other tabs, if the channel does not show the requested tab, an error will be raised. Also, `/live` URLs raise an error if there are no live videos instead of silently downloading the entire channel. You may use `--compat-options no-youtube-channel-redirect` to revert all these redirections -* Unavailable videos are also listed for youtube playlists. Use `--compat-options no-youtube-unavailable-videos` to remove this -* If `ffmpeg` is used as the downloader, the downloading and merging of formats happen in a single step when possible. Use `--compat-options no-direct-merge` to revert this -* Thumbnail embedding in `mp4` is done with mutagen if possible. Use `--compat-options embed-thumbnail-atomicparsley` to force the use of AtomicParsley instead -* Some private fields such as filenames are removed by default from the infojson. Use `--no-clean-infojson` or `--compat-options no-clean-infojson` to revert this -* When `--embed-subs` and `--write-subs` are used together, the subtitles are written to disk and also embedded in the media file. You can use just `--embed-subs` to embed the subs and automatically delete the separate file. See [#630 (comment)](https://github.com/yt-dlp/yt-dlp/issues/630#issuecomment-893659460) for more info. `--compat-options no-keep-subs` can be used to revert this - -For ease of use, a few more compat options are available: -* `--compat-options all`: Use all compat options -* `--compat-options youtube-dl`: Same as `--compat-options all,-multistreams` -* `--compat-options youtube-dlc`: Same as `--compat-options all,-no-live-chat,-no-youtube-channel-redirect` - +hypervideo - A fork of youtube-dl without nonfree parts + +- [INSTALLATION](#installation) +- [DESCRIPTION](#description) +- [OPTIONS](#options) +- [CONFIGURATION](#configuration) +- [OUTPUT TEMPLATE](#output-template) +- [FORMAT SELECTION](#format-selection) +- [VIDEO SELECTION](#video-selection) +- [FAQ](#faq) +- [DEVELOPER INSTRUCTIONS](#developer-instructions) +- [EMBEDDING YOUTUBE-DL](#embedding-hypervideo) +- [BUGS](#bugs) +- [COPYRIGHT](#copyright) # INSTALLATION -You can install yt-dlp using one of the following methods: - -### Using the release binary - -You can simply download the [correct binary file](#release-files) for your OS - -<!-- MANPAGE: BEGIN EXCLUDED SECTION --> -[](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp.exe) -[](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp) -[](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp.tar.gz) -[](#release-files) -[](https://github.com/yt-dlp/yt-dlp/releases) -<!-- MANPAGE: END EXCLUDED SECTION --> - -Note: The manpages, shell completion files etc. are available in the [source tarball](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp.tar.gz) - -<!-- TODO: Move to Wiki --> -In UNIX-like OSes (MacOS, Linux, BSD), you can also install the same in one of the following ways: - -``` -sudo curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp -sudo chmod a+rx /usr/local/bin/yt-dlp -``` +A fork of [youtube-dl](https://ytdl-org.github.io/youtube-dl/) without nonfree parts -``` -sudo wget https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -O /usr/local/bin/yt-dlp -sudo chmod a+rx /usr/local/bin/yt-dlp -``` - -``` -sudo aria2c https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp --dir /usr/local/bin -o yt-dlp -sudo chmod a+rx /usr/local/bin/yt-dlp -``` - - -### With [PIP](https://pypi.org/project/pip) - -You can install the [PyPI package](https://pypi.org/project/yt-dlp) with: -``` -python3 -m pip install -U yt-dlp -``` +Hypervideo is distributed for [Hyperbola GNU/Linux-libre](https://www.hyperbola.info/). -You can install without any of the optional dependencies using: -``` -python3 -m pip install --no-deps -U yt-dlp -``` +To install on Hyperbola run: -If you want to be on the cutting edge, you can also install the master branch with: +```console +# pacman -S hypervideo ``` -python3 -m pip install --force-reinstall https://github.com/yt-dlp/yt-dlp/archive/master.zip -``` - -Note that on some systems, you may need to use `py` or `python` instead of `python3` -<!-- TODO: Add to Wiki, Remove Taps --> -### With [Homebrew](https://brew.sh) +# DESCRIPTION +**hypervideo** A fork of youtube-dl without nonfree parts -macOS or Linux users that are using Homebrew can also install it by: - -``` -brew install yt-dlp/taps/yt-dlp +```console +$ hypervideo [OPTIONS] URL [URL...] ``` -## UPDATE -You can use `yt-dlp -U` to update if you are [using the provided release](#using-the-release-binary) - -If you [installed with pip](#with-pip), simply re-run the same command that was used to install the program - -If you [installed using Homebrew](#with-homebrew), run `brew upgrade yt-dlp/taps/yt-dlp` - -<!-- MANPAGE: BEGIN EXCLUDED SECTION --> -## RELEASE FILES - -#### Recommended - -File|Description -:---|:--- -[yt-dlp](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp)|Platform-independant binary. Needs Python (recommended for **UNIX-like systems**) -[yt-dlp.exe](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp.exe)|Windows (Win7 SP1+) standalone x64 binary (recommended for **Windows**) - -#### Alternatives - -File|Description -:---|:--- -[yt-dlp_macos](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_macos)|MacOS (10.15+) standalone executable -[yt-dlp_x86.exe](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_x86.exe)|Windows (Vista SP2+) standalone x86 (32-bit) binary -[yt-dlp_min.exe](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_min.exe)|Windows (Win7 SP1+) standalone x64 binary built with `py2exe`.<br/> Does not contain `pycryptodomex`, needs VC++14 -[yt-dlp_win.zip](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_win.zip)|Unpackaged Windows executable (no auto-update) -[yt-dlp_macos.zip](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_macos.zip)|Unpackaged MacOS (10.15+) executable (no auto-update) - -#### Misc - -File|Description -:---|:--- -[yt-dlp.tar.gz](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp.tar.gz)|Source tarball. Also contains manpages, completions, etc -[SHA2-512SUMS](https://github.com/yt-dlp/yt-dlp/releases/latest/download/SHA2-512SUMS)|GNU-style SHA512 sums -[SHA2-256SUMS](https://github.com/yt-dlp/yt-dlp/releases/latest/download/SHA2-256SUMS)|GNU-style SHA256 sums -<!-- MANPAGE: END EXCLUDED SECTION --> - -## DEPENDENCIES -Python versions 3.6+ (CPython and PyPy) are supported. Other versions and implementations may or may not work correctly. - -<!-- Python 3.5+ uses VC++14 and it is already embedded in the binary created -<!x-- https://www.microsoft.com/en-us/download/details.aspx?id=26999 --x> -On windows, [Microsoft Visual C++ 2010 SP1 Redistributable Package (x86)](https://download.microsoft.com/download/1/6/5/165255E7-1014-4D0A-B094-B6A430A6BFFC/vcredist_x86.exe) is also necessary to run yt-dlp. You probably already have this, but if the executable throws an error due to missing `MSVCR100.dll` you need to install it manually. ---> - -While all the other dependencies are optional, `ffmpeg` and `ffprobe` are highly recommended - -* [**ffmpeg** and **ffprobe**](https://www.ffmpeg.org) - Required for [merging separate video and audio files](#format-selection) as well as for various [post-processing](#post-processing-options) tasks. License [depends on the build](https://www.ffmpeg.org/legal.html) -* [**mutagen**](https://github.com/quodlibet/mutagen) - For embedding thumbnail in certain formats. Licensed under [GPLv2+](https://github.com/quodlibet/mutagen/blob/master/COPYING) -* [**pycryptodomex**](https://github.com/Legrandin/pycryptodome) - For decrypting AES-128 HLS streams and various other data. Licensed under [BSD2](https://github.com/Legrandin/pycryptodome/blob/master/LICENSE.rst) -* [**websockets**](https://github.com/aaugustin/websockets) - For downloading over websocket. Licensed under [BSD3](https://github.com/aaugustin/websockets/blob/main/LICENSE) -* [**secretstorage**](https://github.com/mitya57/secretstorage) - For accessing the Gnome keyring while decrypting cookies of Chromium-based browsers on Linux. Licensed under [BSD](https://github.com/mitya57/secretstorage/blob/master/LICENSE) -* [**AtomicParsley**](https://github.com/wez/atomicparsley) - For embedding thumbnail in mp4/m4a if mutagen is not present. Licensed under [GPLv2+](https://github.com/wez/atomicparsley/blob/master/COPYING) -* [**rtmpdump**](http://rtmpdump.mplayerhq.hu) - For downloading `rtmp` streams. ffmpeg will be used as a fallback. Licensed under [GPLv2+](http://rtmpdump.mplayerhq.hu) -* [**mplayer**](http://mplayerhq.hu/design7/info.html) or [**mpv**](https://mpv.io) - For downloading `rstp` streams. ffmpeg will be used as a fallback. Licensed under [GPLv2+](https://github.com/mpv-player/mpv/blob/master/Copyright) -* [**phantomjs**](https://github.com/ariya/phantomjs) - Used in extractors where javascript needs to be run. Licensed under [BSD3](https://github.com/ariya/phantomjs/blob/master/LICENSE.BSD) -* [**sponskrub**](https://github.com/faissaloo/SponSkrub) - For using the now **deprecated** [sponskrub options](#sponskrub-options). Licensed under [GPLv3+](https://github.com/faissaloo/SponSkrub/blob/master/LICENCE.md) -* Any external downloader that you want to use with `--downloader` - -To use or redistribute the dependencies, you must agree to their respective licensing terms. - -The Windows and MacOS standalone release binaries are already built with the python interpreter, mutagen, pycryptodomex and websockets included. - -**Note**: There are some regressions in newer ffmpeg versions that causes various issues when used alongside yt-dlp. Since ffmpeg is such an important dependency, we provide [custom builds](https://github.com/yt-dlp/FFmpeg-Builds#ffmpeg-static-auto-builds) with patches for these issues at [yt-dlp/FFmpeg-Builds](https://github.com/yt-dlp/FFmpeg-Builds). See [the readme](https://github.com/yt-dlp/FFmpeg-Builds#patches-applied) for details on the specific issues solved by these builds - - -## COMPILE - -**For Windows**: -To build the Windows executable, you must have pyinstaller (and optionally mutagen, pycryptodomex, websockets). Once you have all the necessary dependencies installed, (optionally) build lazy extractors using `devscripts/make_lazy_extractors.py`, and then just run `pyinst.py`. The executable will be built for the same architecture (32/64 bit) as the python used to build it. - - py -m pip install -U pyinstaller -r requirements.txt - py devscripts/make_lazy_extractors.py - py pyinst.py - -Note that pyinstaller [does not support](https://github.com/pyinstaller/pyinstaller#requirements-and-tested-platforms) Python installed from the Windows store without using a virtual environment - -**For Unix**: -You will need the required build tools: `python`, `make` (GNU), `pandoc`, `zip`, `pytest` -Then simply run `make`. You can also run `make yt-dlp` instead to compile only the binary without updating any of the additional files - -**Note**: In either platform, `devscripts/update-version.py` can be used to automatically update the version number - -You can also fork the project on github and run your fork's [build workflow](.github/workflows/build.yml) to automatically build a release - -# USAGE AND OPTIONS - -<!-- MANPAGE: BEGIN EXCLUDED SECTION --> - yt-dlp [OPTIONS] [--] URL [URL...] - -`Ctrl+F` is your friend :D -<!-- MANPAGE: END EXCLUDED SECTION --> - -<!-- Auto generated --> -## General Options: - -h, --help Print this help text and exit - --version Print program version and exit - -U, --update Update this program to latest version. Make - sure that you have sufficient permissions - (run with sudo if needed) - -i, --ignore-errors Ignore download and postprocessing errors. - The download will be considered successful - even if the postprocessing fails - --no-abort-on-error Continue with next video on download - errors; e.g. to skip unavailable videos in - a playlist (default) - --abort-on-error Abort downloading of further videos if an - error occurs (Alias: --no-ignore-errors) - --dump-user-agent Display the current user-agent and exit - --list-extractors List all supported extractors and exit - --extractor-descriptions Output descriptions of all supported - extractors and exit - --force-generic-extractor Force extraction to use the generic - extractor - --default-search PREFIX Use this prefix for unqualified URLs. For - example "gvsearch2:" downloads two videos - from google videos for the search term - "large apple". Use the value "auto" to let - yt-dlp guess ("auto_warning" to emit a - warning when guessing). "error" just throws - an error. The default value "fixup_error" - repairs broken URLs, but emits an error if - this is not possible instead of searching - --ignore-config Don't load any more configuration files - except those given by --config-locations. - For backward compatibility, if this option - is found inside the system configuration - file, the user configuration is not loaded. - (Alias: --no-config) - --no-config-locations Do not load any custom configuration files - (default). When given inside a - configuration file, ignore all previous - --config-locations defined in the current - file - --config-locations PATH Location of the main configuration file; - either the path to the config or its - containing directory. Can be used multiple - times and inside other configuration files - --flat-playlist Do not extract the videos of a playlist, - only list them - --no-flat-playlist Extract the videos of a playlist - --live-from-start Download livestreams from the start. - Currently only supported for YouTube - (Experimental) - --no-live-from-start Download livestreams from the current time - (default) - --wait-for-video MIN[-MAX] Wait for scheduled streams to become - available. Pass the minimum number of - seconds (or range) to wait between retries - --no-wait-for-video Do not wait for scheduled streams (default) - --mark-watched Mark videos watched (even with --simulate). - Currently only supported for YouTube - --no-mark-watched Do not mark videos watched (default) - --no-colors Do not emit color codes in output - --compat-options OPTS Options that can help keep compatibility - with youtube-dl or youtube-dlc - configurations by reverting some of the - changes made in yt-dlp. See "Differences in - default behavior" for details +# OPTIONS + -h, --help Print this help text and exit + --version Print program version and exit + -i, --ignore-errors Continue on download errors, for + example to skip unavailable videos in a + playlist + --abort-on-error Abort downloading of further videos (in + the playlist or the command line) if an + error occurs + --dump-user-agent Display the current browser + identification + --list-extractors List all supported extractors + --extractor-descriptions Output descriptions of all supported + extractors + --force-generic-extractor Force extraction to use the generic + extractor + --default-search PREFIX Use this prefix for unqualified URLs. + For example "gvsearch2:" downloads two + videos from google videos for youtube- + dl "large apple". Use the value "auto" + to let hypervideo guess ("auto_warning" + to emit a warning when guessing). + "error" just throws an error. The + default value "fixup_error" repairs + broken URLs, but emits an error if this + is not possible instead of searching. + --ignore-config Do not read configuration files. When + given in the global configuration file + /etc/hypervideo.conf: Do not read the + user configuration in + ~/.config/hypervideo/config + --config-location PATH Location of the configuration file; + either the path to the config or its + containing directory. + --flat-playlist Do not extract the videos of a + playlist, only list them. + --mark-watched Mark videos watched (YouTube only) + --no-mark-watched Do not mark videos watched (YouTube + only) + --no-color Do not emit color codes in output ## Network Options: - --proxy URL Use the specified HTTP/HTTPS/SOCKS proxy. - To enable SOCKS proxy, specify a proper - scheme. For example - socks5://user:pass@127.0.0.1:1080/. Pass in - an empty string (--proxy "") for direct - connection - --socket-timeout SECONDS Time to wait before giving up, in seconds - --source-address IP Client-side IP address to bind to - -4, --force-ipv4 Make all connections via IPv4 - -6, --force-ipv6 Make all connections via IPv6 - -## Geo-restriction: - --geo-verification-proxy URL Use this proxy to verify the IP address for - some geo-restricted sites. The default - proxy specified by --proxy (or none, if the - option is not present) is used for the - actual downloading - --geo-bypass Bypass geographic restriction via faking - X-Forwarded-For HTTP header (default) - --no-geo-bypass Do not bypass geographic restriction via - faking X-Forwarded-For HTTP header - --geo-bypass-country CODE Force bypass geographic restriction with - explicitly provided two-letter ISO 3166-2 - country code - --geo-bypass-ip-block IP_BLOCK Force bypass geographic restriction with - explicitly provided IP block in CIDR - notation + --proxy URL Use the specified HTTP/HTTPS/SOCKS + proxy. To enable SOCKS proxy, specify a + proper scheme. For example + socks5://127.0.0.1:1080/. Pass in an + empty string (--proxy "") for direct + connection + --socket-timeout SECONDS Time to wait before giving up, in + seconds + --source-address IP Client-side IP address to bind to + -4, --force-ipv4 Make all connections via IPv4 + -6, --force-ipv6 Make all connections via IPv6 + +## Geo Restriction: + --geo-verification-proxy URL Use this proxy to verify the IP address + for some geo-restricted sites. The + default proxy specified by --proxy (or + none, if the option is not present) is + used for the actual downloading. + --geo-bypass Bypass geographic restriction via + faking X-Forwarded-For HTTP header + --no-geo-bypass Do not bypass geographic restriction + via faking X-Forwarded-For HTTP header + --geo-bypass-country CODE Force bypass geographic restriction + with explicitly provided two-letter ISO + 3166-2 country code + --geo-bypass-ip-block IP_BLOCK Force bypass geographic restriction + with explicitly provided IP block in + CIDR notation ## Video Selection: - --playlist-start NUMBER Playlist video to start at (default is 1) - --playlist-end NUMBER Playlist video to end at (default is last) - --playlist-items ITEM_SPEC Playlist video items to download. Specify - indices of the videos in the playlist - separated by commas like: "--playlist-items - 1,2,5,8" if you want to download videos - indexed 1, 2, 5, 8 in the playlist. You can - specify range: "--playlist-items - 1-3,7,10-13", it will download the videos - at index 1, 2, 3, 7, 10, 11, 12 and 13 - --min-filesize SIZE Do not download any videos smaller than - SIZE (e.g. 50k or 44.6m) - --max-filesize SIZE Do not download any videos larger than SIZE - (e.g. 50k or 44.6m) - --date DATE Download only videos uploaded on this date. - The date can be "YYYYMMDD" or in the format - "(now|today)[+-][0-9](day|week|month|year)(s)?" - --datebefore DATE Download only videos uploaded on or before - this date. The date formats accepted is the - same as --date - --dateafter DATE Download only videos uploaded on or after - this date. The date formats accepted is the - same as --date - --match-filter FILTER Generic video filter. Any field (see - "OUTPUT TEMPLATE") can be compared with a - number or a string using the operators - defined in "Filtering formats". You can - also simply specify a field to match if the - field is present and "!field" to check if - the field is not present. In addition, - Python style regular expression matching - can be done using "~=", and multiple - filters can be checked with "&". Use a "\" - to escape "&" or quotes if needed. Eg: - --match-filter "!is_live & like_count>?100 - & description~='(?i)\bcats \& dogs\b'" - matches only videos that are not live, has - a like count more than 100 (or the like - field is not available), and also has a - description that contains the phrase "cats - & dogs" (ignoring case) - --no-match-filter Do not use generic video filter (default) - --no-playlist Download only the video, if the URL refers - to a video and a playlist - --yes-playlist Download the playlist, if the URL refers to - a video and a playlist - --age-limit YEARS Download only videos suitable for the given - age - --download-archive FILE Download only videos not listed in the - archive file. Record the IDs of all - downloaded videos in it - --no-download-archive Do not use archive file (default) - --max-downloads NUMBER Abort after downloading NUMBER files - --break-on-existing Stop the download process when encountering - a file that is in the archive - --break-on-reject Stop the download process when encountering - a file that has been filtered out - --break-per-input Make --break-on-existing and --break-on- - reject act only on the current input URL - --no-break-per-input --break-on-existing and --break-on-reject - terminates the entire download queue - --skip-playlist-after-errors N Number of allowed failures until the rest - of the playlist is skipped + --playlist-start NUMBER Playlist video to start at (default is + 1) + --playlist-end NUMBER Playlist video to end at (default is + last) + --playlist-items ITEM_SPEC Playlist video items to download. + Specify indices of the videos in the + playlist separated by commas like: "-- + playlist-items 1,2,5,8" if you want to + download videos indexed 1, 2, 5, 8 in + the playlist. You can specify range: " + --playlist-items 1-3,7,10-13", it will + download the videos at index 1, 2, 3, + 7, 10, 11, 12 and 13. + --match-title REGEX Download only matching titles (regex or + caseless sub-string) + --reject-title REGEX Skip download for matching titles + (regex or caseless sub-string) + --max-downloads NUMBER Abort after downloading NUMBER files + --min-filesize SIZE Do not download any videos smaller than + SIZE (e.g. 50k or 44.6m) + --max-filesize SIZE Do not download any videos larger than + SIZE (e.g. 50k or 44.6m) + --date DATE Download only videos uploaded in this + date + --datebefore DATE Download only videos uploaded on or + before this date (i.e. inclusive) + --dateafter DATE Download only videos uploaded on or + after this date (i.e. inclusive) + --min-views COUNT Do not download any videos with less + than COUNT views + --max-views COUNT Do not download any videos with more + than COUNT views + --match-filter FILTER Generic video filter. Specify any key + (see the "OUTPUT TEMPLATE" for a list + of available keys) to match if the key + is present, !key to check if the key is + not present, key > NUMBER (like + "comment_count > 12", also works with + >=, <, <=, !=, =) to compare against a + number, key = 'LITERAL' (like "uploader + = 'Mike Smith'", also works with !=) to + match against a string literal and & to + require multiple matches. Values which + are not known are excluded unless you + put a question mark (?) after the + operator. For example, to only match + videos that have been liked more than + 100 times and disliked less than 50 + times (or the dislike functionality is + not available at the given service), + but who also have a description, use + --match-filter "like_count > 100 & + dislike_count <? 50 & description" . + --no-playlist Download only the video, if the URL + refers to a video and a playlist. + --yes-playlist Download the playlist, if the URL + refers to a video and a playlist. + --age-limit YEARS Download only videos suitable for the + given age + --download-archive FILE Download only videos not listed in the + archive file. Record the IDs of all + downloaded videos in it. + --include-ads Download advertisements as well + (experimental) ## Download Options: - -N, --concurrent-fragments N Number of fragments of a dash/hlsnative - video that should be downloaded - concurrently (default is 1) - -r, --limit-rate RATE Maximum download rate in bytes per second - (e.g. 50K or 4.2M) - --throttled-rate RATE Minimum download rate in bytes per second - below which throttling is assumed and the - video data is re-extracted (e.g. 100K) - -R, --retries RETRIES Number of retries (default is 10), or - "infinite" - --file-access-retries RETRIES Number of times to retry on file access - error (default is 10), or "infinite" - --fragment-retries RETRIES Number of retries for a fragment (default - is 10), or "infinite" (DASH, hlsnative and - ISM) - --skip-unavailable-fragments Skip unavailable fragments for DASH, - hlsnative and ISM (default) - (Alias: --no-abort-on-unavailable-fragment) - --abort-on-unavailable-fragment Abort downloading if a fragment is unavailable - (Alias: --no-skip-unavailable-fragments) - --keep-fragments Keep downloaded fragments on disk after - downloading is finished - --no-keep-fragments Delete downloaded fragments after - downloading is finished (default) - --buffer-size SIZE Size of download buffer (e.g. 1024 or 16K) - (default is 1024) - --resize-buffer The buffer size is automatically resized - from an initial value of --buffer-size - (default) - --no-resize-buffer Do not automatically adjust the buffer size - --http-chunk-size SIZE Size of a chunk for chunk-based HTTP - downloading (e.g. 10485760 or 10M) (default - is disabled). May be useful for bypassing - bandwidth throttling imposed by a webserver - (experimental) - --playlist-reverse Download playlist videos in reverse order - --no-playlist-reverse Download playlist videos in default order - (default) - --playlist-random Download playlist videos in random order - --xattr-set-filesize Set file xattribute ytdl.filesize with - expected file size - --hls-use-mpegts Use the mpegts container for HLS videos; - allowing some players to play the video - while downloading, and reducing the chance - of file corruption if download is - interrupted. This is enabled by default for - live streams - --no-hls-use-mpegts Do not use the mpegts container for HLS - videos. This is default when not - downloading live streams - --downloader [PROTO:]NAME Name or path of the external downloader to - use (optionally) prefixed by the protocols - (http, ftp, m3u8, dash, rstp, rtmp, mms) to - use it for. Currently supports native, - aria2c, avconv, axel, curl, ffmpeg, httpie, - wget (Recommended: aria2c). You can use - this option multiple times to set different - downloaders for different protocols. For - example, --downloader aria2c --downloader - "dash,m3u8:native" will use aria2c for - http/ftp downloads, and the native - downloader for dash/m3u8 downloads (Alias: - --external-downloader) - --downloader-args NAME:ARGS Give these arguments to the external - downloader. Specify the downloader name and - the arguments separated by a colon ":". For - ffmpeg, arguments can be passed to - different positions using the same syntax - as --postprocessor-args. You can use this - option multiple times to give different - arguments to different downloaders (Alias: - --external-downloader-args) + -r, --limit-rate RATE Maximum download rate in bytes per + second (e.g. 50K or 4.2M) + -R, --retries RETRIES Number of retries (default is 10), or + "infinite". + --fragment-retries RETRIES Number of retries for a fragment + (default is 10), or "infinite" (DASH, + hlsnative and ISM) + --skip-unavailable-fragments Skip unavailable fragments (DASH, + hlsnative and ISM) + --abort-on-unavailable-fragment Abort downloading when some fragment is + not available + --keep-fragments Keep downloaded fragments on disk after + downloading is finished; fragments are + erased by default + --buffer-size SIZE Size of download buffer (e.g. 1024 or + 16K) (default is 1024) + --no-resize-buffer Do not automatically adjust the buffer + size. By default, the buffer size is + automatically resized from an initial + value of SIZE. + --http-chunk-size SIZE Size of a chunk for chunk-based HTTP + downloading (e.g. 10485760 or 10M) + (default is disabled). May be useful + for bypassing bandwidth throttling + imposed by a webserver (experimental) + --playlist-reverse Download playlist videos in reverse + order + --playlist-random Download playlist videos in random + order + --xattr-set-filesize Set file xattribute ytdl.filesize with + expected file size + --hls-prefer-native Use the native HLS downloader instead + of ffmpeg + --hls-prefer-ffmpeg Use ffmpeg instead of the native HLS + downloader + --hls-use-mpegts Use the mpegts container for HLS + videos, allowing to play the video + while downloading (some players may not + be able to play it) + --external-downloader COMMAND Use the specified external downloader. + Currently supports aria2c,avconv,axel,c + url,ffmpeg,httpie,wget + --external-downloader-args ARGS Give these arguments to the external + downloader ## Filesystem Options: - -a, --batch-file FILE File containing URLs to download ("-" for - stdin), one URL per line. Lines starting - with "#", ";" or "]" are considered as - comments and ignored - --no-batch-file Do not read URLs from batch file (default) - -P, --paths [TYPES:]PATH The paths where the files should be - downloaded. Specify the type of file and - the path separated by a colon ":". All the - same TYPES as --output are supported. - Additionally, you can also provide "home" - (default) and "temp" paths. All - intermediary files are first downloaded to - the temp path and then the final files are - moved over to the home path after download - is finished. This option is ignored if - --output is an absolute path - -o, --output [TYPES:]TEMPLATE Output filename template; see "OUTPUT - TEMPLATE" for details - --output-na-placeholder TEXT Placeholder value for unavailable meta - fields in output filename template - (default: "NA") - --restrict-filenames Restrict filenames to only ASCII - characters, and avoid "&" and spaces in - filenames - --no-restrict-filenames Allow Unicode characters, "&" and spaces in - filenames (default) - --windows-filenames Force filenames to be Windows-compatible - --no-windows-filenames Make filenames Windows-compatible only if - using Windows (default) - --trim-filenames LENGTH Limit the filename length (excluding - extension) to the specified number of - characters - -w, --no-overwrites Do not overwrite any files - --force-overwrites Overwrite all video and metadata files. - This option includes --no-continue - --no-force-overwrites Do not overwrite the video, but overwrite - related files (default) - -c, --continue Resume partially downloaded files/fragments - (default) - --no-continue Do not resume partially downloaded - fragments. If the file is not fragmented, - restart download of the entire file - --part Use .part files instead of writing directly - into output file (default) - --no-part Do not use .part files - write directly - into output file - --mtime Use the Last-modified header to set the - file modification time (default) - --no-mtime Do not use the Last-modified header to set - the file modification time - --write-description Write video description to a .description - file - --no-write-description Do not write video description (default) - --write-info-json Write video metadata to a .info.json file - (this may contain personal information) - --no-write-info-json Do not write video metadata (default) - --write-playlist-metafiles Write playlist metadata in addition to the - video metadata when using --write-info-json, - --write-description etc. (default) - --no-write-playlist-metafiles Do not write playlist metadata when using - --write-info-json, --write-description etc. - --clean-infojson Remove some private fields such as - filenames from the infojson. Note that it - could still contain some personal - information (default) - --no-clean-infojson Write all fields to the infojson - --write-comments Retrieve video comments to be placed in the - infojson. The comments are fetched even - without this option if the extraction is - known to be quick (Alias: --get-comments) - --no-write-comments Do not retrieve video comments unless the - extraction is known to be quick (Alias: - --no-get-comments) - --load-info-json FILE JSON file containing the video information - (created with the "--write-info-json" - option) - --cookies FILE Netscape formatted file to read cookies - from and dump cookie jar in - --no-cookies Do not read/dump cookies from/to file - (default) - --cookies-from-browser BROWSER[+KEYRING][:PROFILE] - The name of the browser and (optionally) - the name/path of the profile to load - cookies from, separated by a ":". Currently - supported browsers are: brave, chrome, - chromium, edge, firefox, opera, safari, - vivaldi. By default, the most recently - accessed profile is used. The keyring used - for decrypting Chromium cookies on Linux - can be (optionally) specified after the - browser name separated by a "+". Currently - supported keyrings are: basictext, - gnomekeyring, kwallet - --no-cookies-from-browser Do not load cookies from browser (default) - --cache-dir DIR Location in the filesystem where youtube-dl - can store some downloaded information (such - as client ids and signatures) permanently. - By default $XDG_CACHE_HOME/yt-dlp or - ~/.cache/yt-dlp - --no-cache-dir Disable filesystem caching - --rm-cache-dir Delete all filesystem cache files + -a, --batch-file FILE File containing URLs to download ('-' + for stdin), one URL per line. Lines + starting with '#', ';' or ']' are + considered as comments and ignored. + --id Use only video ID in file name + -o, --output TEMPLATE Output filename template, see the + "OUTPUT TEMPLATE" for all the info + --output-na-placeholder PLACEHOLDER Placeholder value for unavailable meta + fields in output filename template + (default is "NA") + --autonumber-start NUMBER Specify the start value for + %(autonumber)s (default is 1) + --restrict-filenames Restrict filenames to only ASCII + characters, and avoid "&" and spaces in + filenames + -w, --no-overwrites Do not overwrite files + -c, --continue Force resume of partially downloaded + files. By default, hypervideo will + resume downloads if possible. + --no-continue Do not resume partially downloaded + files (restart from beginning) + --no-part Do not use .part files - write directly + into output file + --no-mtime Do not use the Last-modified header to + set the file modification time + --write-description Write video description to a + .description file + --write-info-json Write video metadata to a .info.json + file + --write-annotations Write video annotations to a + .annotations.xml file + --load-info-json FILE JSON file containing the video + information (created with the "--write- + info-json" option) + --cookies FILE File to read cookies from and dump + cookie jar in + --cache-dir DIR Location in the filesystem where + hypervideo can store some downloaded + information permanently. By default + $XDG_CACHE_HOME/hypervideo or + ~/.cache/hypervideo . At the moment, + only YouTube player files (for videos + with obfuscated signatures) are cached, + but that may change. + --no-cache-dir Disable filesystem caching + --rm-cache-dir Delete all filesystem cache files ## Thumbnail Options: - --write-thumbnail Write thumbnail image to disk - --no-write-thumbnail Do not write thumbnail image to disk - (default) - --write-all-thumbnails Write all thumbnail image formats to disk - --list-thumbnails List available thumbnails of each video. - Simulate unless --no-simulate is used - -## Internet Shortcut Options: - --write-link Write an internet shortcut file, depending - on the current platform (.url, .webloc or - .desktop). The URL may be cached by the OS - --write-url-link Write a .url Windows internet shortcut. The - OS caches the URL based on the file path - --write-webloc-link Write a .webloc macOS internet shortcut - --write-desktop-link Write a .desktop Linux internet shortcut - -## Verbosity and Simulation Options: - -q, --quiet Activate quiet mode. If used with - --verbose, print the log to stderr - --no-warnings Ignore warnings - -s, --simulate Do not download the video and do not write - anything to disk - --no-simulate Download the video even if printing/listing - options are used - --ignore-no-formats-error Ignore "No video formats" error. Useful for - extracting metadata even if the videos are - not actually available for download - (experimental) - --no-ignore-no-formats-error Throw error when no downloadable video - formats are found (default) - --skip-download Do not download the video but write all - related files (Alias: --no-download) - -O, --print [WHEN:]TEMPLATE Field name or output template to print to - screen, optionally prefixed with when to - print it, separated by a ":". Supported - values of "WHEN" are the same as that of - --use-postprocessor, and "video" (default). - Implies --quiet and --simulate (unless - --no-simulate is used). This option can be - used multiple times - --print-to-file [WHEN:]TEMPLATE FILE - Append given template to the file. The - values of WHEN and TEMPLATE are same as - that of --print. FILE uses the same syntax - as the output template. This option can be - used multiple times - -j, --dump-json Quiet, but print JSON information for each - video. Simulate unless --no-simulate is - used. See "OUTPUT TEMPLATE" for a - description of available keys - -J, --dump-single-json Quiet, but print JSON information for each - url or infojson passed. Simulate unless - --no-simulate is used. If the URL refers to - a playlist, the whole playlist information - is dumped in a single line - --force-write-archive Force download archive entries to be - written as far as no errors occur, even if - -s or another simulation option is used - (Alias: --force-download-archive) - --newline Output progress bar as new lines - --no-progress Do not print progress bar - --progress Show progress bar, even if in quiet mode - --console-title Display progress in console titlebar - --progress-template [TYPES:]TEMPLATE - Template for progress outputs, optionally - prefixed with one of "download:" (default), - "download-title:" (the console title), - "postprocess:", or "postprocess-title:". - The video's fields are accessible under the - "info" key and the progress attributes are - accessible under "progress" key. E.g.: - --console-title --progress-template - "download-title:%(info.id)s-%(progress.eta)s" - -v, --verbose Print various debugging information - --dump-pages Print downloaded pages encoded using base64 - to debug problems (very verbose) - --write-pages Write downloaded intermediary pages to - files in the current directory to debug - problems - --print-traffic Display sent and read HTTP traffic + --write-thumbnail Write thumbnail image to disk + --write-all-thumbnails Write all thumbnail image formats to + disk + --list-thumbnails Simulate and list all available + thumbnail formats + +## Verbosity / Simulation Options: + -q, --quiet Activate quiet mode + --no-warnings Ignore warnings + -s, --simulate Do not download the video and do not + write anything to disk + --skip-download Do not download the video + -g, --get-url Simulate, quiet but print URL + -e, --get-title Simulate, quiet but print title + --get-id Simulate, quiet but print id + --get-thumbnail Simulate, quiet but print thumbnail URL + --get-description Simulate, quiet but print video + description + --get-duration Simulate, quiet but print video length + --get-filename Simulate, quiet but print output + filename + --get-format Simulate, quiet but print output format + -j, --dump-json Simulate, quiet but print JSON + information. See the "OUTPUT TEMPLATE" + for a description of available keys. + -J, --dump-single-json Simulate, quiet but print JSON + information for each command-line + argument. If the URL refers to a + playlist, dump the whole playlist + information in a single line. + --print-json Be quiet and print the video + information as JSON (video is still + being downloaded). + --newline Output progress bar as new lines + --no-progress Do not print progress bar + --console-title Display progress in console titlebar + -v, --verbose Print various debugging information + --dump-pages Print downloaded pages encoded using + base64 to debug problems (very verbose) + --write-pages Write downloaded intermediary pages to + files in the current directory to debug + problems + --print-traffic Display sent and read HTTP traffic + -C, --call-home Contact the hypervideo server for + debugging + --no-call-home Do NOT contact the hypervideo server + for debugging ## Workarounds: - --encoding ENCODING Force the specified encoding (experimental) - --legacy-server-connect Explicitly allow HTTPS connection to - servers that do not support RFC 5746 secure - renegotiation - --no-check-certificates Suppress HTTPS certificate validation - --prefer-insecure Use an unencrypted connection to retrieve - information about the video (Currently - supported only for YouTube) - --add-header FIELD:VALUE Specify a custom HTTP header and its value, - separated by a colon ":". You can use this - option multiple times - --bidi-workaround Work around terminals that lack - bidirectional text support. Requires bidiv - or fribidi executable in PATH - --sleep-requests SECONDS Number of seconds to sleep between requests - during data extraction - --sleep-interval SECONDS Number of seconds to sleep before each - download. This is the minimum time to sleep - when used along with --max-sleep-interval - (Alias: --min-sleep-interval) - --max-sleep-interval SECONDS Maximum number of seconds to sleep. Can - only be used along with --min-sleep-interval - --sleep-subtitles SECONDS Number of seconds to sleep before each - subtitle download + --encoding ENCODING Force the specified encoding + (experimental) + --no-check-certificate Suppress HTTPS certificate validation + --prefer-insecure Use an unencrypted connection to + retrieve information about the video. + (Currently supported only for YouTube) + --user-agent UA Specify a custom user agent + --referer URL Specify a custom referer, use if the + video access is restricted to one + domain + --add-header FIELD:VALUE Specify a custom HTTP header and its + value, separated by a colon ':'. You + can use this option multiple times + --bidi-workaround Work around terminals that lack + bidirectional text support. Requires + bidiv or fribidi executable in PATH + --sleep-interval SECONDS Number of seconds to sleep before each + download when used alone or a lower + bound of a range for randomized sleep + before each download (minimum possible + number of seconds to sleep) when used + along with --max-sleep-interval. + --max-sleep-interval SECONDS Upper bound of a range for randomized + sleep before each download (maximum + possible number of seconds to sleep). + Must only be used along with --min- + sleep-interval. ## Video Format Options: - -f, --format FORMAT Video format code, see "FORMAT SELECTION" - for more details - -S, --format-sort SORTORDER Sort the formats by the fields given, see - "Sorting Formats" for more details - --S-force, --format-sort-force Force user specified sort order to have - precedence over all fields, see "Sorting - Formats" for more details - --no-format-sort-force Some fields have precedence over the user - specified sort order (default), see - "Sorting Formats" for more details - --video-multistreams Allow multiple video streams to be merged - into a single file - --no-video-multistreams Only one video stream is downloaded for - each output file (default) - --audio-multistreams Allow multiple audio streams to be merged - into a single file - --no-audio-multistreams Only one audio stream is downloaded for - each output file (default) - --prefer-free-formats Prefer video formats with free containers - over non-free ones of same quality. Use - with "-S ext" to strictly prefer free - containers irrespective of quality - --no-prefer-free-formats Don't give any special preference to free - containers (default) - --check-formats Check that the selected formats are - actually downloadable - --check-all-formats Check all formats for whether they are - actually downloadable - --no-check-formats Do not check that the formats are actually - downloadable - -F, --list-formats List available formats of each video. - Simulate unless --no-simulate is used - --merge-output-format FORMAT If a merge is required (e.g. - bestvideo+bestaudio), output to given - container format. One of mkv, mp4, ogg, - webm, flv. Ignored if no merge is required + -f, --format FORMAT Video format code, see the "FORMAT + SELECTION" for all the info + --all-formats Download all available video formats + --prefer-free-formats Prefer free video formats unless a + specific one is requested + -F, --list-formats List all available formats of requested + videos + --youtube-skip-dash-manifest Do not download the DASH manifests and + related data on YouTube videos + --merge-output-format FORMAT If a merge is required (e.g. + bestvideo+bestaudio), output to given + container format. One of mkv, mp4, ogg, + webm, flv. Ignored if no merge is + required ## Subtitle Options: - --write-subs Write subtitle file - --no-write-subs Do not write subtitle file (default) - --write-auto-subs Write automatically generated subtitle file - (Alias: --write-automatic-subs) - --no-write-auto-subs Do not write auto-generated subtitles - (default) (Alias: --no-write-automatic-subs) - --list-subs List available subtitles of each video. - Simulate unless --no-simulate is used - --sub-format FORMAT Subtitle format, accepts formats - preference, for example: "srt" or - "ass/srt/best" - --sub-langs LANGS Languages of the subtitles to download (can - be regex) or "all" separated by commas. - (Eg: --sub-langs "en.*,ja") You can prefix - the language code with a "-" to exempt it - from the requested languages. (Eg: - --sub-langs all,-live_chat) Use --list-subs - for a list of available language tags + --write-sub Write subtitle file + --write-auto-sub Write automatically generated subtitle + file (YouTube only) + --all-subs Download all the available subtitles of + the video + --list-subs List all available subtitles for the + video + --sub-format FORMAT Subtitle format, accepts formats + preference, for example: "srt" or + "ass/srt/best" + --sub-lang LANGS Languages of the subtitles to download + (optional) separated by commas, use + --list-subs for available language tags ## Authentication Options: - -u, --username USERNAME Login with this account ID - -p, --password PASSWORD Account password. If this option is left - out, yt-dlp will ask interactively - -2, --twofactor TWOFACTOR Two-factor authentication code - -n, --netrc Use .netrc authentication data - --netrc-location PATH Location of .netrc authentication data; - either the path or its containing - directory. Defaults to ~/.netrc - --video-password PASSWORD Video password (vimeo, youku) - --ap-mso MSO Adobe Pass multiple-system operator (TV - provider) identifier, use --ap-list-mso for - a list of available MSOs - --ap-username USERNAME Multiple-system operator account login - --ap-password PASSWORD Multiple-system operator account password. - If this option is left out, yt-dlp will ask - interactively - --ap-list-mso List all supported multiple-system - operators - -## Post-Processing Options: - -x, --extract-audio Convert video files to audio-only files - (requires ffmpeg and ffprobe) - --audio-format FORMAT Specify audio format to convert the audio - to when -x is used. Currently supported - formats are: best (default) or one of - best|aac|flac|mp3|m4a|opus|vorbis|wav|alac - --audio-quality QUALITY Specify ffmpeg audio quality, insert a - value between 0 (best) and 10 (worst) for - VBR or a specific bitrate like 128K - (default 5) - --remux-video FORMAT Remux the video into another container if - necessary (currently supported: mp4|mkv|flv - |webm|mov|avi|mp3|mka|m4a|ogg|opus). If - target container does not support the - video/audio codec, remuxing will fail. You - can specify multiple rules; Eg. - "aac>m4a/mov>mp4/mkv" will remux aac to - m4a, mov to mp4 and anything else to mkv. - --recode-video FORMAT Re-encode the video into another format if - re-encoding is necessary. The syntax and - supported formats are the same as --remux-video - --postprocessor-args NAME:ARGS Give these arguments to the postprocessors. - Specify the postprocessor/executable name - and the arguments separated by a colon ":" - to give the argument to the specified - postprocessor/executable. Supported PP are: - Merger, ModifyChapters, SplitChapters, - ExtractAudio, VideoRemuxer, VideoConvertor, - Metadata, EmbedSubtitle, EmbedThumbnail, - SubtitlesConvertor, ThumbnailsConvertor, - FixupStretched, FixupM4a, FixupM3u8, - FixupTimestamp and FixupDuration. The - supported executables are: AtomicParsley, - FFmpeg and FFprobe. You can also specify - "PP+EXE:ARGS" to give the arguments to the - specified executable only when being used - by the specified postprocessor. - Additionally, for ffmpeg/ffprobe, "_i"/"_o" - can be appended to the prefix optionally - followed by a number to pass the argument - before the specified input/output file. Eg: - --ppa "Merger+ffmpeg_i1:-v quiet". You can - use this option multiple times to give - different arguments to different - postprocessors. (Alias: --ppa) - -k, --keep-video Keep the intermediate video file on disk - after post-processing - --no-keep-video Delete the intermediate video file after - post-processing (default) - --post-overwrites Overwrite post-processed files (default) - --no-post-overwrites Do not overwrite post-processed files - --embed-subs Embed subtitles in the video (only for mp4, - webm and mkv videos) - --no-embed-subs Do not embed subtitles (default) - --embed-thumbnail Embed thumbnail in the video as cover art - --no-embed-thumbnail Do not embed thumbnail (default) - --embed-metadata Embed metadata to the video file. Also - embeds chapters/infojson if present unless - --no-embed-chapters/--no-embed-info-json - are used (Alias: --add-metadata) - --no-embed-metadata Do not add metadata to file (default) - (Alias: --no-add-metadata) - --embed-chapters Add chapter markers to the video file - (Alias: --add-chapters) - --no-embed-chapters Do not add chapter markers (default) - (Alias: --no-add-chapters) - --embed-info-json Embed the infojson as an attachment to - mkv/mka video files - --no-embed-info-json Do not embed the infojson as an attachment - to the video file - --parse-metadata FROM:TO Parse additional metadata like title/artist - from other fields; see "MODIFYING METADATA" - for details - --replace-in-metadata FIELDS REGEX REPLACE - Replace text in a metadata field using the - given regex. This option can be used - multiple times - --xattrs Write metadata to the video file's xattrs - (using dublin core and xdg standards) - --concat-playlist POLICY Concatenate videos in a playlist. One of - "never", "always", or "multi_video" - (default; only when the videos form a - single show). All the video files must have - same codecs and number of streams to be - concatable. The "pl_video:" prefix can be - used with "--paths" and "--output" to set - the output filename for the split files. - See "OUTPUT TEMPLATE" for details - --fixup POLICY Automatically correct known faults of the - file. One of never (do nothing), warn (only - emit a warning), detect_or_warn (the - default; fix file if we can, warn - otherwise), force (try fixing even if file - already exists) - --ffmpeg-location PATH Location of the ffmpeg binary; either the - path to the binary or its containing - directory - --exec [WHEN:]CMD Execute a command, optionally prefixed with - when to execute it (after_move if - unspecified), separated by a ":". Supported - values of "WHEN" are the same as that of - --use-postprocessor. Same syntax as the - output template can be used to pass any - field as arguments to the command. After - download, an additional field "filepath" - that contains the final path of the - downloaded file is also available, and if - no fields are passed, %(filepath)q is - appended to the end of the command. This - option can be used multiple times - --no-exec Remove any previously defined --exec - --convert-subs FORMAT Convert the subtitles to another format - (currently supported: srt|vtt|ass|lrc) - (Alias: --convert-subtitles) - --convert-thumbnails FORMAT Convert the thumbnails to another format - (currently supported: jpg|png|webp) - --split-chapters Split video into multiple files based on - internal chapters. The "chapter:" prefix - can be used with "--paths" and "--output" - to set the output filename for the split - files. See "OUTPUT TEMPLATE" for details - --no-split-chapters Do not split video based on chapters - (default) - --remove-chapters REGEX Remove chapters whose title matches the - given regular expression. Time ranges - prefixed by a "*" can also be used in place - of chapters to remove the specified range. - Eg: --remove-chapters "*10:15-15:00" - --remove-chapters "intro". This option can - be used multiple times - --no-remove-chapters Do not remove any chapters from the file - (default) - --force-keyframes-at-cuts Force keyframes around the chapters before - removing/splitting them. Requires a - re-encode and thus is very slow, but the - resulting video may have fewer artifacts - around the cuts - --no-force-keyframes-at-cuts Do not force keyframes around the chapters - when cutting/splitting (default) - --use-postprocessor NAME[:ARGS] The (case sensitive) name of plugin - postprocessors to be enabled, and - (optionally) arguments to be passed to it, - separated by a colon ":". ARGS are a - semicolon ";" delimited list of NAME=VALUE. - The "when" argument determines when the - postprocessor is invoked. It can be one of - "pre_process" (after video extraction), - "after_filter" (after video passes filter), - "before_dl" (before each video download), - "post_process" (after each video download; - default), "after_move" (after moving video - file to it's final locations), - "after_video" (after downloading and - processing all formats of a video), or - "playlist" (at end of playlist). This - option can be used multiple times to add - different postprocessors - -## SponsorBlock Options: -Make chapter entries for, or remove various segments (sponsor, - introductions, etc.) from downloaded YouTube videos using the - [SponsorBlock API](https://sponsor.ajay.app) - - --sponsorblock-mark CATS SponsorBlock categories to create chapters - for, separated by commas. Available - categories are all, default(=all), sponsor, - intro, outro, selfpromo, preview, filler, - interaction, music_offtopic, poi_highlight. - You can prefix the category with a "-" to - exempt it. See [1] for description of the - categories. Eg: --sponsorblock-mark all,-preview - [1] https://wiki.sponsor.ajay.app/w/Segment_Categories - --sponsorblock-remove CATS SponsorBlock categories to be removed from - the video file, separated by commas. If a - category is present in both mark and - remove, remove takes precedence. The syntax - and available categories are the same as - for --sponsorblock-mark except that - "default" refers to "all,-filler" and - poi_highlight is not available - --sponsorblock-chapter-title TEMPLATE - The title template for SponsorBlock - chapters created by --sponsorblock-mark. - The same syntax as the output template is - used, but the only available fields are - start_time, end_time, category, categories, - name, category_names. Defaults to - "[SponsorBlock]: %(category_names)l" - --no-sponsorblock Disable both --sponsorblock-mark and - --sponsorblock-remove - --sponsorblock-api URL SponsorBlock API location, defaults to - https://sponsor.ajay.app - -## Extractor Options: - --extractor-retries RETRIES Number of retries for known extractor - errors (default is 3), or "infinite" - --allow-dynamic-mpd Process dynamic DASH manifests (default) - (Alias: --no-ignore-dynamic-mpd) - --ignore-dynamic-mpd Do not process dynamic DASH manifests - (Alias: --no-allow-dynamic-mpd) - --hls-split-discontinuity Split HLS playlists to different formats at - discontinuities such as ad breaks - --no-hls-split-discontinuity Do not split HLS playlists to different - formats at discontinuities such as ad - breaks (default) - --extractor-args KEY:ARGS Pass these arguments to the extractor. See - "EXTRACTOR ARGUMENTS" for details. You can - use this option multiple times to give - arguments for different extractors + -u, --username USERNAME Login with this account ID + -p, --password PASSWORD Account password. If this option is + left out, hypervideo will ask + interactively. + -2, --twofactor TWOFACTOR Two-factor authentication code + -n, --netrc Use .netrc authentication data + --video-password PASSWORD Video password (vimeo, youku) + +## Adobe Pass Options: + --ap-mso MSO Adobe Pass multiple-system operator (TV + provider) identifier, use --ap-list-mso + for a list of available MSOs + --ap-username USERNAME Multiple-system operator account login + --ap-password PASSWORD Multiple-system operator account + password. If this option is left out, + hypervideo will ask interactively. + --ap-list-mso List all supported multiple-system + operators + +## Post-processing Options: + -x, --extract-audio Convert video files to audio-only files + (requires ffmpeg/avconv and + ffprobe/avprobe) + --audio-format FORMAT Specify audio format: "best", "aac", + "flac", "mp3", "m4a", "opus", "vorbis", + or "wav"; "best" by default; No effect + without -x + --audio-quality QUALITY Specify ffmpeg/avconv audio quality, + insert a value between 0 (better) and 9 + (worse) for VBR or a specific bitrate + like 128K (default 5) + --recode-video FORMAT Encode the video to another format if + necessary (currently supported: + mp4|flv|ogg|webm|mkv|avi) + --postprocessor-args ARGS Give these arguments to the + postprocessor + -k, --keep-video Keep the video file on disk after the + post-processing; the video is erased by + default + --no-post-overwrites Do not overwrite post-processed files; + the post-processed files are + overwritten by default + --embed-subs Embed subtitles in the video (only for + mp4, webm and mkv videos) + --embed-thumbnail Embed thumbnail in the audio as cover + art + --add-metadata Write metadata to the video file + --metadata-from-title FORMAT Parse additional metadata like song + title / artist from the video title. + The format syntax is the same as + --output. Regular expression with named + capture groups may also be used. The + parsed parameters replace existing + values. Example: --metadata-from-title + "%(artist)s - %(title)s" matches a + title like "Coldplay - Paradise". + Example (regex): --metadata-from-title + "(?P<artist>.+?) - (?P<title>.+)" + --xattrs Write metadata to the video file's + xattrs (using dublin core and xdg + standards) + --fixup POLICY Automatically correct known faults of + the file. One of never (do nothing), + warn (only emit a warning), + detect_or_warn (the default; fix file + if we can, warn otherwise) + --prefer-avconv Prefer avconv over ffmpeg for running + the postprocessors + --prefer-ffmpeg Prefer ffmpeg over avconv for running + the postprocessors (default) + --ffmpeg-location PATH Location of the ffmpeg/avconv binary; + either the path to the binary or its + containing directory. + --exec CMD Execute a command on the file after + downloading and post-processing, + similar to find's -exec syntax. + Example: --exec 'adb push {} + /sdcard/Music/ && rm {}' + --convert-subs FORMAT Convert the subtitles to other format + (currently supported: srt|ass|vtt|lrc) # CONFIGURATION -You can configure yt-dlp by placing any supported command line option to a configuration file. The configuration is loaded from the following locations: +You can configure hypervideo by placing any supported command line option to a configuration file. On GNU+Linux and macOS, the system wide configuration file is located at `/etc/hypervideo.conf` and the user wide configuration file at `~/.config/hypervideo/config`. Note that by default configuration file may not exist so you may need to create it yourself. -1. **Main Configuration**: The file given by `--config-location` -1. **Portable Configuration**: `yt-dlp.conf` in the same directory as the bundled binary. If you are running from source-code (`<root dir>/yt_dlp/__main__.py`), the root directory is used instead. -1. **Home Configuration**: `yt-dlp.conf` in the home path given by `-P`, or in the current directory if no such path is given -1. **User Configuration**: - * `%XDG_CONFIG_HOME%/yt-dlp/config` (recommended on Linux/macOS) - * `%XDG_CONFIG_HOME%/yt-dlp.conf` - * `%APPDATA%/yt-dlp/config` (recommended on Windows) - * `%APPDATA%/yt-dlp/config.txt` - * `~/yt-dlp.conf` - * `~/yt-dlp.conf.txt` +For example, with the following configuration file hypervideo will always extract the audio, not copy the mtime, use a proxy and save all videos under `Movies` directory in your home directory: - `%XDG_CONFIG_HOME%` defaults to `~/.config` if undefined. On windows, `%APPDATA%` generally points to `C:\Users\<user name>\AppData\Roaming` and `~` points to `%HOME%` if present, `%USERPROFILE%` (generally `C:\Users\<user name>`), or `%HOMEDRIVE%%HOMEPATH%` -1. **System Configuration**: `/etc/yt-dlp.conf` - -For example, with the following configuration file yt-dlp will always extract the audio, not copy the mtime, use a proxy and save all videos under `YouTube` directory in your home directory: ``` # Lines starting with # are comments @@ -1074,108 +472,72 @@ For example, with the following configuration file yt-dlp will always extract th # Use this proxy --proxy 127.0.0.1:3128 -# Save all videos under YouTube directory in your home directory --o ~/YouTube/%(title)s.%(ext)s +# Save all videos under Movies directory in your home directory +-o ~/Movies/%(title)s.%(ext)s ``` -Note that options in configuration file are just the same options aka switches used in regular command line calls; thus there **must be no whitespace** after `-` or `--`, e.g. `-o` or `--proxy` but not `- o` or `-- proxy`. +Note that options in configuration file are just the same options aka switches used in regular command line calls thus there **must be no whitespace** after `-` or `--`, e.g. `-o` or `--proxy` but not `- o` or `-- proxy`. + +You can use `--ignore-config` if you want to disable the configuration file for a particular hypervideo run. -You can use `--ignore-config` if you want to disable all configuration files for a particular yt-dlp run. If `--ignore-config` is found inside any configuration file, no further configuration will be loaded. For example, having the option in the portable configuration file prevents loading of home, user, and system configurations. Additionally, (for backward compatibility) if `--ignore-config` is found inside the system configuration file, the user configuration is not loaded. +You can also use `--config-location` if you want to use custom configuration file for a particular hypervideo run. ### Authentication with `.netrc` file -You may also want to configure automatic credentials storage for extractors that support authentication (by providing login and password with `--username` and `--password`) in order not to pass credentials as command line arguments on every yt-dlp execution and prevent tracking plain text passwords in the shell command history. You can achieve this using a [`.netrc` file](https://stackoverflow.com/tags/.netrc/info) on a per extractor basis. For that you will need to create a `.netrc` file in `--netrc-location` and restrict permissions to read/write by only you: +You may also want to configure automatic credentials storage for extractors that support authentication (by providing login and password with `--username` and `--password`) in order not to pass credentials as command line arguments on every hypervideo execution and prevent tracking plain text passwords in the shell command history. You can achieve this using a [`.netrc` file](https://stackoverflow.com/tags/.netrc/info) on a per extractor basis. For that you will need to create a `.netrc` file in your `$HOME` and restrict permissions to read/write by only you: ``` touch $HOME/.netrc chmod a-rwx,u+rw $HOME/.netrc ``` After that you can add credentials for an extractor in the following format, where *extractor* is the name of the extractor in lowercase: ``` -machine <extractor> login <username> password <password> +machine <extractor> login <login> password <password> ``` For example: ``` machine youtube login myaccount@gmail.com password my_youtube_password machine twitch login my_twitch_account_name password my_twitch_password ``` -To activate authentication with the `.netrc` file you should pass `--netrc` to yt-dlp or place it in the [configuration file](#configuration). +To activate authentication with the `.netrc` file you should pass `--netrc` to hypervideo or place it in the [configuration file](#configuration). -The default location of the .netrc file is `$HOME` (`~`) in UNIX. On Windows, it is `%HOME%` if present, `%USERPROFILE%` (generally `C:\Users\<user name>`) or `%HOMEDRIVE%%HOMEPATH%` +On Windows you may also need to setup the `%HOME%` environment variable manually. For example: +``` +set HOME=%USERPROFILE% +``` # OUTPUT TEMPLATE -The `-o` option is used to indicate a template for the output file names while `-P` option is used to specify the path each type of file should be saved to. +The `-o` option allows users to indicate a template for the output file names. -<!-- MANPAGE: BEGIN EXCLUDED SECTION --> **tl;dr:** [navigate me to examples](#output-template-examples). -<!-- MANPAGE: END EXCLUDED SECTION --> - -The simplest usage of `-o` is not to set any template arguments when downloading a single file, like in `yt-dlp -o funny_video.flv "https://some/video"` (hard-coding file extension like this is _not_ recommended and could break some post-processing). - -It may however also contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [Python string formatting operations](https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by formatting operations. -The field names themselves (the part inside the parenthesis) can also have some special formatting: -1. **Object traversal**: The dictionaries and lists available in metadata can be traversed by using a `.` (dot) separator. You can also do python slicing using `:`. Eg: `%(tags.0)s`, `%(subtitles.en.-1.ext)s`, `%(id.3:7:-1)s`, `%(formats.:.format_id)s`. `%()s` refers to the entire infodict. Note that all the fields that become available using this method are not listed below. Use `-j` to see such fields - -1. **Addition**: Addition and subtraction of numeric fields can be done using `+` and `-` respectively. Eg: `%(playlist_index+10)03d`, `%(n_entries+1-playlist_index)d` - -1. **Date/time Formatting**: Date/time fields can be formatted according to [strftime formatting](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) by specifying it separated from the field name using a `>`. Eg: `%(duration>%H-%M-%S)s`, `%(upload_date>%Y-%m-%d)s`, `%(epoch-3600>%H-%M-%S)s` - -1. **Alternatives**: Alternate fields can be specified separated with a `,`. Eg: `%(release_date>%Y,upload_date>%Y|Unknown)s` - -1. **Replacement**: A replacement value can specified using a `&` separator. If the field is *not* empty, this replacement value will be used instead of the actual field content. This is done after alternate fields are considered; thus the replacement is used if *any* of the alternative fields is *not* empty. - -1. **Default**: A literal default value can be specified for when the field is empty using a `|` separator. This overrides `--output-na-template`. Eg: `%(uploader|Unknown)s` - -1. **More Conversions**: In addition to the normal format types `diouxXeEfFgGcrs`, `B`, `j`, `l`, `q`, `D`, `S` can be used for converting to **B**ytes, **j**son (flag `#` for pretty-printing), a comma separated **l**ist (flag `#` for `\n` newline-separated), a string **q**uoted for the terminal (flag `#` to split a list into different arguments), to add **D**ecimal suffixes (Eg: 10M) (flag `#` to use 1024 as factor), and to **S**anitize as filename (flag `#` for restricted), respectively - -1. **Unicode normalization**: The format type `U` can be used for NFC [unicode normalization](https://docs.python.org/3/library/unicodedata.html#unicodedata.normalize). The alternate form flag (`#`) changes the normalization to NFD and the conversion flag `+` can be used for NFKC/NFKD compatibility equivalence normalization. Eg: `%(title)+.100U` is NFKC - -To summarize, the general syntax for a field is: -``` -%(name[.keys][addition][>strf][,alternate][&replacement][|default])[flags][width][.precision][length]type -``` - -Additionally, you can set different output templates for the various metadata files separately from the general output template by specifying the type of file followed by the template separated by a colon `:`. The different file types supported are `subtitle`, `thumbnail`, `description`, `annotation` (deprecated), `infojson`, `link`, `pl_thumbnail`, `pl_description`, `pl_infojson`, `chapter`, `pl_video`. For example, `-o "%(title)s.%(ext)s" -o "thumbnail:%(title)s\%(title)s.%(ext)s"` will put the thumbnails in a folder with the same name as the video. If any of the templates is empty, that type of file will not be written. Eg: `--write-thumbnail -o "thumbnail:"` will write thumbnails only for playlists and not for video. - -The available fields are: +The basic usage is not to set any template arguments when downloading a single file, like in `hypervideo -o funny_video.flv "https://some/video"`. However, it may contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [python string formatting operations](https://docs.python.org/2/library/stdtypes.html#string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by formatting operations. Allowed names along with sequence type are: - `id` (string): Video identifier - `title` (string): Video title - - `fulltitle` (string): Video title ignoring live timestamp and generic title - `url` (string): Video URL - `ext` (string): Video filename extension - `alt_title` (string): A secondary title of the video - - `description` (string): The description of the video - `display_id` (string): An alternative identifier for the video - `uploader` (string): Full name of the video uploader - `license` (string): License name the video is licensed under - `creator` (string): The creator of the video + - `release_date` (string): The date (YYYYMMDD) when the video was released - `timestamp` (numeric): UNIX timestamp of the moment the video became available - `upload_date` (string): Video upload date (YYYYMMDD) - - `release_timestamp` (numeric): UNIX timestamp of the moment the video was released - - `release_date` (string): The date (YYYYMMDD) when the video was released - - `modified_timestamp` (numeric): UNIX timestamp of the moment the video was last modified - - `modified_date` (string): The date (YYYYMMDD) when the video was last modified - `uploader_id` (string): Nickname or id of the video uploader - `channel` (string): Full name of the channel the video is uploaded on - `channel_id` (string): Id of the channel - - `channel_follower_count` (numeric): Number of followers of the channel - `location` (string): Physical location where the video was filmed - `duration` (numeric): Length of the video in seconds - - `duration_string` (string): Length of the video (HH:mm:ss) - `view_count` (numeric): How many users have watched the video on the platform - `like_count` (numeric): Number of positive ratings of the video - `dislike_count` (numeric): Number of negative ratings of the video - `repost_count` (numeric): Number of reposts of the video - `average_rating` (numeric): Average rating give by users, the scale used depends on the webpage - - `comment_count` (numeric): Number of comments on the video (For some extractors, comments are only downloaded at the end, and so this field cannot be used) + - `comment_count` (numeric): Number of comments on the video - `age_limit` (numeric): Age restriction for the video (years) - - `live_status` (string): One of "is_live", "was_live", "is_upcoming", "not_live" - `is_live` (boolean): Whether this video is a live stream or a fixed-length video - - `was_live` (boolean): Whether this video was originally a live stream - - `playable_in_embed` (string): Whether this video is allowed to play in embedded players on other sites - - `availability` (string): Whether the video is "private", "premium_only", "subscriber_only", "needs_auth", "unlisted" or "public" - `start_time` (numeric): Time in seconds where the reproduction should start, as specified in the URL - `end_time` (numeric): Time in seconds where the reproduction should end, as specified in the URL - `format` (string): A human-readable description of the format @@ -1190,7 +552,6 @@ The available fields are: - `asr` (numeric): Audio sampling rate in Hertz - `vbr` (numeric): Average video bitrate in KBit/s - `fps` (numeric): Frame rate - - `dynamic_range` (string): The dynamic range of the video - `vcodec` (string): Name of the video codec in use - `container` (string): Name of the container format - `filesize` (numeric): The number of bytes, if known in advance @@ -1198,22 +559,14 @@ The available fields are: - `protocol` (string): The protocol that will be used for the actual download - `extractor` (string): Name of the extractor - `extractor_key` (string): Key name of the extractor - - `epoch` (numeric): Unix epoch of when the information extraction was completed + - `epoch` (numeric): Unix epoch when creating the file - `autonumber` (numeric): Number that will be increased with each download, starting at `--autonumber-start` - - `video_autonumber` (numeric): Number that will be increased with each video - - `n_entries` (numeric): Total number of extracted items in the playlist - - `playlist_id` (string): Identifier of the playlist that contains the video - - `playlist_title` (string): Name of the playlist that contains the video - - `playlist` (string): `playlist_id` or `playlist_title` - - `playlist_count` (numeric): Total number of items in the playlist. May not be known if entire playlist is not extracted - - `playlist_index` (numeric): Index of the video in the playlist padded with leading zeros according the final index - - `playlist_autonumber` (numeric): Position of the video in the playlist download queue padded with leading zeros according to the total length of the playlist + - `playlist` (string): Name or id of the playlist that contains the video + - `playlist_index` (numeric): Index of the video in the playlist padded with leading zeros according to the total length of the playlist + - `playlist_id` (string): Playlist identifier + - `playlist_title` (string): Playlist title - `playlist_uploader` (string): Full name of the playlist uploader - `playlist_uploader_id` (string): Nickname or id of the playlist uploader - - `webpage_url` (string): A URL to the video webpage which if given to yt-dlp should allow to get the same result again - - `webpage_url_basename` (string): The basename of the webpage URL - - `webpage_url_domain` (string): The domain of the webpage URL - - `original_url` (string): The URL given by the user (or same as `webpage_url` for playlist entries) Available for the video that belongs to some logical chapter or section: @@ -1244,107 +597,60 @@ Available for the media that is a track or a part of a music album: - `disc_number` (numeric): Number of the disc or other physical medium the track belongs to - `release_year` (numeric): Year (YYYY) when the album was released -Available for `chapter:` prefix when using `--split-chapters` for videos with internal chapters: - - - `section_title` (string): Title of the chapter - - `section_number` (numeric): Number of the chapter within the file - - `section_start` (numeric): Start time of the chapter in seconds - - `section_end` (numeric): End time of the chapter in seconds - -Available only when used in `--print`: - - - `urls` (string): The URLs of all requested formats, one in each line - - `filename` (string): Name of the video file. Note that the actual filename may be different due to post-processing. Use `--exec echo` to get the name after all postprocessing is complete - - `formats_table` (table): The video format table as printed by `--list-formats` - - `thumbnails_table` (table): The thumbnail format table as printed by `--list-thumbnails` - - `subtitles_table` (table): The subtitle format table as printed by `--list-subs` - - `automatic_captions_table` (table): The automatic subtitle format table as printed by `--list-subs` - - -Available only in `--sponsorblock-chapter-title`: - - - `start_time` (numeric): Start time of the chapter in seconds - - `end_time` (numeric): End time of the chapter in seconds - - `categories` (list): The SponsorBlock categories the chapter belongs to - - `category` (string): The smallest SponsorBlock category the chapter belongs to - - `category_names` (list): Friendly names of the categories - - `name` (string): Friendly name of the smallest category - -Each aforementioned sequence when referenced in an output template will be replaced by the actual value corresponding to the sequence name. For example for `-o %(title)s-%(id)s.%(ext)s` and an mp4 video with title `yt-dlp test video` and id `BaW_jenozKc`, this will result in a `yt-dlp test video-BaW_jenozKc.mp4` file created in the current directory. +Each aforementioned sequence when referenced in an output template will be replaced by the actual value corresponding to the sequence name. Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by a particular extractor. Such sequences will be replaced with placeholder value provided with `--output-na-placeholder` (`NA` by default). -Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by a particular extractor. Such sequences will be replaced with placeholder value provided with `--output-na-placeholder` (`NA` by default). +For example for `-o %(title)s-%(id)s.%(ext)s` and an mp4 video with title `hypervideo test video` and id `BaW_jenozKcj`, this will result in a `hypervideo test video-BaW_jenozKcj.mp4` file created in the current directory. -**Tip**: Look at the `-j` output to identify which fields are available for the particular URL +For numeric sequences you can use numeric related formatting, for example, `%(view_count)05d` will result in a string with view count padded with zeros up to 5 characters, like in `00042`. -For numeric sequences you can use [numeric related formatting](https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting), for example, `%(view_count)05d` will result in a string with view count padded with zeros up to 5 characters, like in `00042`. - -Output templates can also contain arbitrary hierarchical path, e.g. `-o "%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s"` which will result in downloading each video in a directory corresponding to this path template. Any missing directory will be automatically created for you. +Output templates can also contain arbitrary hierarchical path, e.g. `-o '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s'` which will result in downloading each video in a directory corresponding to this path template. Any missing directory will be automatically created for you. To use percent literals in an output template use `%%`. To output to stdout use `-o -`. -The current default template is `%(title)s [%(id)s].%(ext)s`. +The current default template is `%(title)s-%(id)s.%(ext)s`. -In some cases, you don't want special characters such as 中, spaces, or &, such as when transferring the downloaded filename to a Windows system or the filename through an 8bit-unsafe channel. In these cases, add the `--restrict-filenames` flag to get a shorter title. +In some cases, you don't want special characters such as 中, spaces, or &, such as when transferring the downloaded filename to a Windows system or the filename through an 8bit-unsafe channel. In these cases, add the `--restrict-filenames` flag to get a shorter title: -<!-- MANPAGE: BEGIN EXCLUDED SECTION --> #### Output template and Windows batch files If you are using an output template inside a Windows batch file then you must escape plain percent characters (`%`) by doubling, so that `-o "%(title)s-%(id)s.%(ext)s"` should become `-o "%%(title)s-%%(id)s.%%(ext)s"`. However you should not touch `%`'s that are not plain characters, e.g. environment variables for expansion should stay intact: `-o "C:\%HOMEPATH%\Desktop\%%(title)s.%%(ext)s"`. -<!-- MANPAGE: END EXCLUDED SECTION --> #### Output template examples -```bash -$ yt-dlp --get-filename -o "test video.%(ext)s" BaW_jenozKc -test video.webm # Literal name with correct extension +Note that on Windows you may need to use double quotes instead of single. -$ yt-dlp --get-filename -o "%(title)s.%(ext)s" BaW_jenozKc -youtube-dl test video ''_ä↭𝕐.webm # All kinds of weird characters +```bash +$ hypervideo --get-filename -o '%(title)s.%(ext)s' BaW_jenozKc +hypervideo test video ''_ä↭𝕐.mp4 # All kinds of weird characters -$ yt-dlp --get-filename -o "%(title)s.%(ext)s" BaW_jenozKc --restrict-filenames -youtube-dl_test_video_.webm # Restricted file name +$ hypervideo --get-filename -o '%(title)s.%(ext)s' BaW_jenozKc --restrict-filenames +hypervideo_test_video_.mp4 # A simple file name # Download YouTube playlist videos in separate directory indexed by video order in a playlist -$ yt-dlp -o "%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s" "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re" - -# Download YouTube playlist videos in separate directories according to their uploaded year -$ yt-dlp -o "%(upload_date>%Y)s/%(title)s.%(ext)s" "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re" - -# Prefix playlist index with " - " separator, but only if it is available -$ yt-dlp -o '%(playlist_index|)s%(playlist_index& - |)s%(title)s.%(ext)s' BaW_jenozKc "https://www.youtube.com/user/TheLinuxFoundation/playlists" +$ hypervideo -o '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re # Download all playlists of YouTube channel/user keeping each playlist in separate directory: -$ yt-dlp -o "%(uploader)s/%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s" "https://www.youtube.com/user/TheLinuxFoundation/playlists" +$ hypervideo -o '%(uploader)s/%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' https://www.youtube.com/user/TheLinuxFoundation/playlists # Download Udemy course keeping each chapter in separate directory under MyVideos directory in your home -$ yt-dlp -u user -p password -P "~/MyVideos" -o "%(playlist)s/%(chapter_number)s - %(chapter)s/%(title)s.%(ext)s" "https://www.udemy.com/java-tutorial" +$ hypervideo -u user -p password -o '~/MyVideos/%(playlist)s/%(chapter_number)s - %(chapter)s/%(title)s.%(ext)s' https://www.udemy.com/java-tutorial/ # Download entire series season keeping each series and each season in separate directory under C:/MyVideos -$ yt-dlp -P "C:/MyVideos" -o "%(series)s/%(season_number)s - %(season)s/%(episode_number)s - %(episode)s.%(ext)s" "https://videomore.ru/kino_v_detalayah/5_sezon/367617" - -# Download video as "C:\MyVideos\uploader\title.ext", subtitles as "C:\MyVideos\subs\uploader\title.ext" -# and put all temporary files in "C:\MyVideos\tmp" -$ yt-dlp -P "C:/MyVideos" -P "temp:tmp" -P "subtitle:subs" -o "%(uploader)s/%(title)s.%(ext)s" BaW_jenoz --write-subs - -# Download video as "C:\MyVideos\uploader\title.ext" and subtitles as "C:\MyVideos\uploader\subs\title.ext" -$ yt-dlp -P "C:/MyVideos" -o "%(uploader)s/%(title)s.%(ext)s" -o "subtitle:%(uploader)s/subs/%(title)s.%(ext)s" BaW_jenozKc --write-subs +$ hypervideo -o "C:/MyVideos/%(series)s/%(season_number)s - %(season)s/%(episode_number)s - %(episode)s.%(ext)s" https://videomore.ru/kino_v_detalayah/5_sezon/367617 # Stream the video being downloaded to stdout -$ yt-dlp -o - BaW_jenozKc +$ hypervideo -o - BaW_jenozKc ``` # FORMAT SELECTION -By default, yt-dlp tries to download the best available quality if you **don't** pass any options. -This is generally equivalent to using `-f bestvideo*+bestaudio/best`. However, if multiple audiostreams is enabled (`--audio-multistreams`), the default format changes to `-f bestvideo+bestaudio/best`. Similarly, if ffmpeg is unavailable, or if you use yt-dlp to stream to `stdout` (`-o -`), the default becomes `-f best/bestvideo+bestaudio`. +By default hypervideo tries to download the best available quality, i.e. if you want the best quality you **don't need** to pass any special options, hypervideo will guess it for you by **default**. -**Deprecation warning**: Latest versions of yt-dlp can stream multiple formats to the stdout simultaneously using ffmpeg. So, in future versions, the default for this will be set to `-f bv*+ba/b` similar to normal downloads. If you want to preserve the `-f b/bv+ba` setting, it is recommended to explicitly specify it in the configuration options. +But sometimes you may want to download in a different format, for example when you are on a slow or intermittent connection. The key mechanism for achieving this is so-called *format selection* based on which you can explicitly specify desired format, select formats based on some criterion or criteria, setup precedence and much more. -The general syntax for format selection is `-f FORMAT` (or `--format FORMAT`) where `FORMAT` is a *selector expression*, i.e. an expression that describes format or formats you would like to download. +The general syntax for format selection is `--format FORMAT` or shorter `-f FORMAT` where `FORMAT` is a *selector expression*, i.e. an expression that describes format or formats you would like to download. -<!-- MANPAGE: BEGIN EXCLUDED SECTION --> **tl;dr:** [navigate me to examples](#format-selection-examples). -<!-- MANPAGE: END EXCLUDED SECTION --> The simplest case is requesting a specific format, for example with `-f 22` you can download the format with format code equal to 22. You can get the list of available format codes for particular video using `--list-formats` or `-F`. Note that these format codes are extractor specific. @@ -1354,37 +660,19 @@ You can use `-f -` to interactively provide the format selector *for each video* You can also use special names to select particular edge case formats: - - `all`: Select **all formats** separately - - `mergeall`: Select and **merge all formats** (Must be used with `--audio-multistreams`, `--video-multistreams` or both) - - `b*`, `best*`: Select the best quality format that **contains either** a video or an audio - - `b`, `best`: Select the best quality format that **contains both** video and audio. Equivalent to `best*[vcodec!=none][acodec!=none]` - - `bv`, `bestvideo`: Select the best quality **video-only** format. Equivalent to `best*[acodec=none]` - - `bv*`, `bestvideo*`: Select the best quality format that **contains video**. It may also contain audio. Equivalent to `best*[vcodec!=none]` - - `ba`, `bestaudio`: Select the best quality **audio-only** format. Equivalent to `best*[vcodec=none]` - - `ba*`, `bestaudio*`: Select the best quality format that **contains audio**. It may also contain video. Equivalent to `best*[acodec!=none]` - - `w*`, `worst*`: Select the worst quality format that contains either a video or an audio - - `w`, `worst`: Select the worst quality format that contains both video and audio. Equivalent to `worst*[vcodec!=none][acodec!=none]` - - `wv`, `worstvideo`: Select the worst quality video-only format. Equivalent to `worst*[acodec=none]` - - `wv*`, `worstvideo*`: Select the worst quality format that contains video. It may also contain audio. Equivalent to `worst*[vcodec!=none]` - - `wa`, `worstaudio`: Select the worst quality audio-only format. Equivalent to `worst*[vcodec=none]` - - `wa*`, `worstaudio*`: Select the worst quality format that contains audio. It may also contain video. Equivalent to `worst*[acodec!=none]` - -For example, to download the worst quality video-only format you can use `-f worstvideo`. It is however recommended not to use `worst` and related options. When your format selector is `worst`, the format which is worst in all respects is selected. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps` instead of `-f worst`. See [sorting formats](#sorting-formats) for more details. + - `best`: Select the best quality format represented by a single file with video and audio. + - `worst`: Select the worst quality format represented by a single file with video and audio. + - `bestvideo`: Select the best quality video-only format (e.g. DASH video). May not be available. + - `worstvideo`: Select the worst quality video-only format. May not be available. + - `bestaudio`: Select the best quality audio only-format. May not be available. + - `worstaudio`: Select the worst quality audio only-format. May not be available. -You can select the n'th best format of a type by using `best<type>.<n>`. For example, `best.2` will select the 2nd best combined format. Similarly, `bv*.3` will select the 3rd best format that contains a video stream. +For example, to download the worst quality video-only format you can use `-f worstvideo`. -If you want to download multiple videos and they don't have the same formats available, you can specify the order of preference using slashes. Note that formats on the left hand side are preferred, for example `-f 22/17/18` will download format 22 if it's available, otherwise it will download format 17 if it's available, otherwise it will download format 18 if it's available, otherwise it will complain that no suitable formats are available for download. +If you want to download multiple videos and they don't have the same formats available, you can specify the order of preference using slashes. Note that slash is left-associative, i.e. formats on the left hand side are preferred, for example `-f 22/17/18` will download format 22 if it's available, otherwise it will download format 17 if it's available, otherwise it will download format 18 if it's available, otherwise it will complain that no suitable formats are available for download. If you want to download several formats of the same video use a comma as a separator, e.g. `-f 22,17,18` will download all these three formats, of course if they are available. Or a more sophisticated example combined with the precedence feature: `-f 136/137/mp4/bestvideo,140/m4a/bestaudio`. -You can merge the video and audio of multiple formats into a single file using `-f <format1>+<format2>+...` (requires ffmpeg installed), for example `-f bestvideo+bestaudio` will download the best video-only format, the best audio-only format and mux them together with ffmpeg. - -**Deprecation warning**: Since the *below* described behavior is complex and counter-intuitive, this will be removed and multistreams will be enabled by default in the future. A new operator will be instead added to limit formats to single audio/video - -Unless `--video-multistreams` is used, all formats with a video stream except the first one are ignored. Similarly, unless `--audio-multistreams` is used, all formats with an audio stream except the first one are ignored. For example, `-f bestvideo+best+bestaudio --video-multistreams --audio-multistreams` will download and merge all 3 given formats. The resulting file will have 2 video streams and 2 audio streams. But `-f bestvideo+best+bestaudio --no-video-multistreams` will download and merge only `bestvideo` and `bestaudio`. `best` is ignored since another format containing a video stream (`bestvideo`) has already been selected. The order of the formats is therefore important. `-f best+bestaudio --no-audio-multistreams` will download and merge both formats while `-f bestaudio+best --no-audio-multistreams` will ignore `best` and download only `bestaudio`. - -## Filtering Formats - You can also filter the video formats by putting a condition in brackets, as in `-f "best[height=720]"` (or `-f "[filesize>10M]"`). The following numeric meta fields can be used with comparisons `<`, `<=`, `>`, `>=`, `=` (equals), `!=` (not equals): @@ -1398,7 +686,7 @@ The following numeric meta fields can be used with comparisons `<`, `<=`, `>`, ` - `asr`: Audio sampling rate in Hertz - `fps`: Frame rate -Also filtering work for comparisons `=` (equals), `^=` (starts with), `$=` (ends with), `*=` (contains), `~=` (matches regex) and following string meta fields: +Also filtering work for comparisons `=` (equals), `^=` (starts with), `$=` (ends with), `*=` (contains) and following string meta fields: - `ext`: File extension - `acodec`: Name of the audio codec in use @@ -1408,351 +696,648 @@ Also filtering work for comparisons `=` (equals), `^=` (starts with), `$=` (ends - `format_id`: A short description of the format - `language`: Language code -Any string comparison may be prefixed with negation `!` in order to produce an opposite comparison, e.g. `!*=` (does not contain). The comparand of a string comparison needs to be quoted with either double or single quotes if it contains spaces or special characters other than `._-`. - -Note that none of the aforementioned meta fields are guaranteed to be present since this solely depends on the metadata obtained by particular extractor, i.e. the metadata offered by the website. Any other field made available by the extractor can also be used for filtering. - -Formats for which the value is not known are excluded unless you put a question mark (`?`) after the operator. You can combine format filters, so `-f "[height<=?720][tbr>500]"` selects up to 720p videos (or videos where the height is not known) with a bitrate of at least 500 KBit/s. You can also use the filters with `all` to download all formats that satisfy the filter. For example, `-f "all[vcodec=none]"` selects all audio-only formats. - -Format selectors can also be grouped using parentheses, for example if you want to download the best pre-merged mp4 and webm formats with a height lower than 480 you can use `-f "(mp4,webm)[height<480]"`. - -## Sorting Formats - -You can change the criteria for being considered the `best` by using `-S` (`--format-sort`). The general format for this is `--format-sort field1,field2...`. - -The available fields are: - - - `hasvid`: Gives priority to formats that has a video stream - - `hasaud`: Gives priority to formats that has a audio stream - - `ie_pref`: The format preference - - `lang`: The language preference - - `quality`: The quality of the format - - `source`: The preference of the source - - `proto`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8_native`/`m3u8` > `http_dash_segments`> `websocket_frag` > `mms`/`rtsp` > `f4f`/`f4m`) - - `vcodec`: Video Codec (`av01` > `vp9.2` > `vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other) - - `acodec`: Audio Codec (`flac`/`alac` > `wav`/`aiff` > `opus` > `vorbis` > `aac` > `mp4a` > `mp3` > `eac3` > `ac3` > `dts` > other) - - `codec`: Equivalent to `vcodec,acodec` - - `vext`: Video Extension (`mp4` > `webm` > `flv` > other). If `--prefer-free-formats` is used, `webm` is preferred. - - `aext`: Audio Extension (`m4a` > `aac` > `mp3` > `ogg` > `opus` > `webm` > other). If `--prefer-free-formats` is used, the order changes to `opus` > `ogg` > `webm` > `m4a` > `mp3` > `aac`. - - `ext`: Equivalent to `vext,aext` - - `filesize`: Exact filesize, if known in advance - - `fs_approx`: Approximate filesize calculated from the manifests - - `size`: Exact filesize if available, otherwise approximate filesize - - `height`: Height of video - - `width`: Width of video - - `res`: Video resolution, calculated as the smallest dimension. - - `fps`: Framerate of video - - `hdr`: The dynamic range of the video (`DV` > `HDR12` > `HDR10+` > `HDR10` > `HLG` > `SDR`) - - `tbr`: Total average bitrate in KBit/s - - `vbr`: Average video bitrate in KBit/s - - `abr`: Average audio bitrate in KBit/s - - `br`: Equivalent to using `tbr,vbr,abr` - - `asr`: Audio sample rate in Hz - -**Deprecation warning**: Many of these fields have (currently undocumented) aliases, that may be removed in a future version. It is recommended to use only the documented field names. +Any string comparison may be prefixed with negation `!` in order to produce an opposite comparison, e.g. `!*=` (does not contain). + +Note that none of the aforementioned meta fields are guaranteed to be present since this solely depends on the metadata obtained by particular extractor, i.e. the metadata offered by the video hoster. -All fields, unless specified otherwise, are sorted in descending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a preferred value for the fields, separated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two preferred values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp9.2` > `av01` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB. +Formats for which the value is not known are excluded unless you put a question mark (`?`) after the operator. You can combine format filters, so `-f "[height <=? 720][tbr>500]"` selects up to 720p videos (or videos where the height is not known) with a bitrate of at least 500 KBit/s. -The fields `hasvid` and `ie_pref` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--format-sort-force`. Apart from these, the default order used is: `lang,quality,res,fps,hdr:12,codec:vp9.2,size,br,asr,proto,ext,hasaud,source,id`. The extractors may override this default order, but they cannot override the user-provided order. +You can merge the video and audio of two formats into a single file using `-f <video-format>+<audio-format>` (requires ffmpeg or avconv installed), for example `-f bestvideo+bestaudio` will download the best video-only format, the best audio-only format and mux them together with ffmpeg/avconv. -Note that the default has `codec:vp9.2`; i.e. `av1` is not preferred. Similarly, the default for hdr is `hdr:12`; i.e. dolby vision is not preferred. These choices are made since DV and AV1 formats are not yet fully compatible with most devices. This may be changed in the future as more devices become capable of smoothly playing back these formats. +Format selectors can also be grouped using parentheses, for example if you want to download the best mp4 and webm formats with a height lower than 480 you can use `-f '(mp4,webm)[height<480]'`. -If your format selector is `worst`, the last item is selected after sorting. This means it will select the format that is worst in all respects. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps`. +Since the end of April 2015 and version 2015.04.26, hypervideo uses `-f bestvideo+bestaudio/best` as the default format selection (see [#5447](https://github.com/ytdl-org/youtube-dl/issues/5447), [#5456](https://github.com/ytdl-org/youtube-dl/issues/5456)). If ffmpeg or avconv are installed this results in downloading `bestvideo` and `bestaudio` separately and muxing them together into a single file giving the best overall quality available. Otherwise it falls back to `best` and results in downloading the best available quality served as a single file. `best` is also needed for videos that don't come from YouTube because they don't provide the audio and video in two different files. If you want to only download some DASH formats (for example if you are not interested in getting videos with a resolution higher than 1080p), you can add `-f bestvideo[height<=?1080]+bestaudio/best` to your configuration file. Note that if you use youtube-dl to stream to `stdout` (and most likely to pipe it to your media player then), i.e. you explicitly specify output template as `-o -`, youtube-dl still uses `-f best` format selection in order to start content delivery immediately to your player and not to wait until `bestvideo` and `bestaudio` are downloaded and muxed. -**Tip**: You can use the `-v -F` to see how the formats have been sorted (worst to best). +If you want to preserve the old format selection behavior (prior to youtube-dl 2015.04.26), i.e. you want to download the best available quality media served as a single file, you should explicitly specify your choice with `-f best`. You may want to add it to the [configuration file](#configuration) in order not to type it every time you run hypervideo. -## Format Selection examples +#### Format selection examples + +Note that on Windows you may need to use double quotes instead of single. ```bash -# Download and merge the best video-only format and the best audio-only format, -# or download the best combined format if video-only format is not available -$ yt-dlp -f "bv+ba/b" +# Download best mp4 format available or any other best if no mp4 available +$ hypervideo -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best' + +# Download best format available but no better than 480p +$ hypervideo -f 'bestvideo[height<=480]+bestaudio/best[height<=480]' + +# Download best video only format but no bigger than 50 MB +$ hypervideo -f 'best[filesize<50M]' + +# Download best format available via direct link over HTTP/HTTPS protocol +$ hypervideo -f '(bestvideo+bestaudio/best)[protocol^=http]' -# Download best format that contains video, -# and if it doesn't already have an audio stream, merge it with best audio-only format -$ yt-dlp -f "bv*+ba/b" +# Download the best video format and the best audio format without merging them +$ hypervideo -f 'bestvideo,bestaudio' -o '%(title)s.f%(format_id)s.%(ext)s' +``` +Note that in the last example, an output template is recommended as bestvideo and bestaudio may have the same file name. + + +# VIDEO SELECTION -# Same as above -$ yt-dlp +Videos can be filtered by their upload date using the options `--date`, `--datebefore` or `--dateafter`. They accept dates in two formats: -# Download the best video-only format and the best audio-only format without merging them -# For this case, an output template should be used since -# by default, bestvideo and bestaudio will have the same file name. -$ yt-dlp -f "bv,ba" -o "%(title)s.f%(format_id)s.%(ext)s" + - Absolute dates: Dates in the format `YYYYMMDD`. + - Relative dates: Dates in the format `(now|today)[+-][0-9](day|week|month|year)(s)?` -# Download and merge the best format that has a video stream, -# and all audio-only formats into one file -$ yt-dlp -f "bv*+mergeall[vcodec=none]" --audio-multistreams +Examples: + +```bash +# Download only the videos uploaded in the last 6 months +$ hypervideo --dateafter now-6months -# Download and merge the best format that has a video stream, -# and the best 2 audio-only formats into one file -$ yt-dlp -f "bv*+ba+ba.2" --audio-multistreams +# Download only the videos uploaded on January 1, 1970 +$ hypervideo --date 19700101 +$ # Download only the videos uploaded in the 200x decade +$ hypervideo --dateafter 20000101 --datebefore 20091231 +``` -# The following examples show the old method (without -S) of format selection -# and how to use -S to achieve a similar but (generally) better result +# FAQ -# Download the worst video available (old method) -$ yt-dlp -f "wv*+wa/w" +### How do I update hypervideo? -# Download the best video available but with the smallest resolution -$ yt-dlp -S "+res" +If you have used pacman, a simple `doas pacman -Syu hypervideo` is sufficient to update. -# Download the smallest video available -$ yt-dlp -S "+size,+br" +As a last resort, you can also uninstall the version installed by your package manager and follow our manual installation instructions. For that, remove the distribution's package, with a line like +```console +$ doas pacman -Rs hypervideo +``` +### I'm getting an error `Unable to extract OpenGraph title` on YouTube playlists -# Download the best mp4 video available, or the best video if no mp4 available -$ yt-dlp -f "bv*[ext=mp4]+ba[ext=m4a]/b[ext=mp4] / bv*+ba/b" +YouTube changed their playlist format in March 2014 and later on, so you'll need at least hypervideo to download all YouTube videos. -# Download the best video with the best extension -# (For video, mp4 > webm > flv. For audio, m4a > aac > mp3 ...) -$ yt-dlp -S "ext" +### I'm getting an error when trying to use output template: `error: using output template conflicts with using title, video ID or auto number` +Make sure you are not using `-o` with any of these options `-t`, `--title`, `--id`, `-A` or `--auto-number` set in command line or in a configuration file. Remove the latter if any. +### Do I always have to pass `-citw`? -# Download the best video available but no better than 480p, -# or the worst video if there is no video under 480p -$ yt-dlp -f "bv*[height<=480]+ba/b[height<=480] / wv*+ba/w" +By default, hypervideo intends to have the best options (incidentally, if you have a convincing case that these should be different, [please file an issue where you explain that](https://yt-dl.org/bug)). Therefore, it is unnecessary and sometimes harmful to copy long option strings from webpages. In particular, the only option out of `-citw` that is regularly useful is `-i`. -# Download the best video available with the largest height but no better than 480p, -# or the best video with the smallest resolution if there is no video under 480p -$ yt-dlp -S "height:480" +### Can you please put the `-b` option back? -# Download the best video available with the largest resolution but no better than 480p, -# or the best video with the smallest resolution if there is no video under 480p -# Resolution is determined by using the smallest dimension. -# So this works correctly for vertical videos as well -$ yt-dlp -S "res:480" +Most people asking this question are not aware that hypervideo now defaults to downloading the highest available quality as reported by YouTube, which will be 1080p or 720p in some cases, so you no longer need the `-b` option. For some specific videos, maybe YouTube does not report them to be available in a specific high quality format you're interested in. In that case, simply request it with the `-f` option and hypervideo will try to download it. +### I get HTTP error 402 when trying to download a video. What's this? +Apparently YouTube requires you to pass a CAPTCHA test if you download too much. We're [considering to provide a way to let you solve the CAPTCHA](https://github.com/ytdl-org/youtube-dl/issues/154), but at the moment, your best course of action is pointing a web browser to the youtube URL, solving the CAPTCHA, and restart hypervideo. -# Download the best video (that also has audio) but no bigger than 50 MB, -# or the worst video (that also has audio) if there is no video under 50 MB -$ yt-dlp -f "b[filesize<50M] / w" +### Do I need any other programs? -# Download largest video (that also has audio) but no bigger than 50 MB, -# or the smallest video (that also has audio) if there is no video under 50 MB -$ yt-dlp -f "b" -S "filesize:50M" +hypervideo works fine on its own on most sites. However, if you want to convert video/audio, you'll need [avconv](https://libav.org/) or [ffmpeg](https://www.ffmpeg.org/). On some sites - most notably YouTube - videos can be retrieved in a higher quality format without sound. hypervideo will detect whether avconv/ffmpeg is present and automatically pick the best option. -# Download best video (that also has audio) that is closest in size to 50 MB -$ yt-dlp -f "b" -S "filesize~50M" +Videos or video formats streamed via RTMP protocol can only be downloaded when [rtmpdump](https://rtmpdump.mplayerhq.hu/) is installed. Downloading MMS and RTSP videos requires either [mplayer](https://mplayerhq.hu/) or [mpv](https://mpv.io/) to be installed. +### I have downloaded a video but how can I play it? +Once the video is fully downloaded, use any video player, such as [mpv](https://mpv.io/), [vlc](https://www.videolan.org/) or [mplayer](https://www.mplayerhq.hu/). -# Download best video available via direct link over HTTP/HTTPS protocol, -# or the best video available via any protocol if there is no such video -$ yt-dlp -f "(bv*+ba/b)[protocol^=http][protocol!*=dash] / (bv*+ba/b)" +### I extracted a video URL with `-g`, but it does not play on another machine / in my web browser. -# Download best video available via the best protocol -# (https/ftps > http/ftp > m3u8_native > m3u8 > http_dash_segments ...) -$ yt-dlp -S "proto" +It depends a lot on the service. In many cases, requests for the video (to download/play it) must come from the same IP address and with the same cookies and/or HTTP headers. Use the `--cookies` option to write the required cookies into a file, and advise your downloader to read cookies from that file. Some sites also require a common user agent to be used, use `--dump-user-agent` to see the one in use by hypervideo. You can also get necessary cookies and HTTP headers from JSON output obtained with `--dump-json`. +It may be beneficial to use IPv6; in some cases, the restrictions are only applied to IPv4. Some services (sometimes only for a subset of videos) do not restrict the video URL by IP address, cookie, or user-agent, but these are the exception rather than the rule. +Please bear in mind that some URL protocols are **not** supported by browsers out of the box, including RTMP. If you are using `-g`, your own downloader must support these as well. -# Download the best video with either h264 or h265 codec, -# or the best video if there is no such video -$ yt-dlp -f "(bv*[vcodec~='^((he|a)vc|h26[45])']+ba) / (bv*+ba/b)" +If you want to play the video on a machine that is not running hypervideo, you can relay the video content from the machine that runs hypervideo. You can use `-o -` to let hypervideo stream a video to stdout, or simply allow the player to download the files written by hypervideo in turn. -# Download the best video with best codec no better than h264, -# or the best video with worst codec if there is no such video -$ yt-dlp -S "codec:h264" +### ERROR: no fmt_url_map or conn information found in video info -# Download the best video with worst codec no worse than h264, -# or the best video with best codec if there is no such video -$ yt-dlp -S "+codec:h264" +YouTube has switched to a new video info format in July 2011 which is not supported by old versions of hypervideo. See [above](#how-do-i-update-hypervideo) for how to update hypervideo. +### ERROR: unable to download video +YouTube requires an additional signature since September 2012 which is not supported by old versions of hypervideo. See [above](#how-do-i-update-hypervideo) for how to hypervideo. -# More complex examples +### Video URL contains an ampersand and I'm getting some strange output `[1] 2839` or `'v' is not recognized as an internal or external command` -# Download the best video no better than 720p preferring framerate greater than 30, -# or the worst video (still preferring framerate greater than 30) if there is no such video -$ yt-dlp -f "((bv*[fps>30]/bv*)[height<=720]/(wv*[fps>30]/wv*)) + ba / (b[fps>30]/b)[height<=720]/(w[fps>30]/w)" +That's actually the output from your shell. Since ampersand is one of the special shell characters it's interpreted by the shell preventing you from passing the whole URL to hypervideo. To disable your shell from interpreting the ampersands (or any other special characters) you have to either put the whole URL in quotes or escape them with a backslash (which approach will work depends on your shell). -# Download the video with the largest resolution no better than 720p, -# or the video with the smallest resolution available if there is no such video, -# preferring larger framerate for formats with the same resolution -$ yt-dlp -S "res:720,fps" +For example if your URL is https://www.youtube.com/watch?t=4&v=BaW_jenozKc you should end up with following command: +```console +$ hypervideo 'https://www.youtube.com/watch?t=4&v=BaW_jenozKc' +``` +or -# Download the video with smallest resolution no worse than 480p, -# or the video with the largest resolution available if there is no such video, -# preferring better codec and then larger total bitrate for the same resolution -$ yt-dlp -S "+res:480,codec,br" +```console +$ hypervideo https://www.youtube.com/watch?t=4\&v=BaW_jenozKc ``` -# MODIFYING METADATA +### ExtractorError: Could not find JS function u'OF' -The metadata obtained by the extractors can be modified by using `--parse-metadata` and `--replace-in-metadata` +In February 2015, the new YouTube player contained a character sequence in a string that was misinterpreted by old versions of hypervideo. See [above](#how-do-i-update-hypervideo) for how to update hypervideo. -`--replace-in-metadata FIELDS REGEX REPLACE` is used to replace text in any metadata field using [python regular expression](https://docs.python.org/3/library/re.html#regular-expression-syntax). [Backreferences](https://docs.python.org/3/library/re.html?highlight=backreferences#re.sub) can be used in the replace string for advanced use. +### HTTP Error 429: Too Many Requests or 402: Payment Required -The general syntax of `--parse-metadata FROM:TO` is to give the name of a field or an [output template](#output-template) to extract data from, and the format to interpret it as, separated by a colon `:`. Either a [python regular expression](https://docs.python.org/3/library/re.html#regular-expression-syntax) with named capture groups or a similar syntax to the [output template](#output-template) (only `%(field)s` formatting is supported) can be used for `TO`. The option can be used multiple times to parse and modify various fields. +These two error codes indicate that the service is blocking your IP address because of overuse. Usually this is a soft block meaning that you can gain access again after solving CAPTCHA. Just open a browser and solve a CAPTCHA the service suggests you and after that [pass cookies](#how-do-i-pass-cookies-to-hypervideo) to hypervideo. Note that if your machine has multiple external IPs then you should also pass exactly the same IP you've used for solving CAPTCHA with [`--source-address`](#network-options). Also you may need to pass a `User-Agent` HTTP header of your browser with [`--user-agent`](#workarounds). -Note that any field created by this can be used in the [output template](#output-template) and will also affect the media file's metadata added when using `--add-metadata`. +If this is not the case (no CAPTCHA suggested to solve by the service) then you can contact the service and ask them to unblock your IP address, or - if you have acquired a whitelisted IP address already - use the [`--proxy` or `--source-address` options](#network-options) to select another IP address. -This option also has a few special uses: -* You can download an additional URL based on the metadata of the currently downloaded video. To do this, set the field `additional_urls` to the URL that you want to download. Eg: `--parse-metadata "description:(?P<additional_urls>https?://www\.vimeo\.com/\d+)` will download the first vimeo video found in the description -* You can use this to change the metadata that is embedded in the media file. To do this, set the value of the corresponding field with a `meta_` prefix. For example, any value you set to `meta_description` field will be added to the `description` field in the file. For example, you can use this to set a different "description" and "synopsis". To modify the metadata of individual streams, use the `meta<n>_` prefix (Eg: `meta1_language`). Any value set to the `meta_` field will overwrite all default values. +### SyntaxError: Non-ASCII character -For reference, these are the fields yt-dlp adds by default to the file metadata: +The error -Metadata fields|From -:---|:--- -`title`|`track` or `title` -`date`|`upload_date` -`description`, `synopsis`|`description` -`purl`, `comment`|`webpage_url` -`track`|`track_number` -`artist`|`artist`, `creator`, `uploader` or `uploader_id` -`genre`|`genre` -`album`|`album` -`album_artist`|`album_artist` -`disc`|`disc_number` -`show`|`series` -`season_number`|`season_number` -`episode_id`|`episode` or `episode_id` -`episode_sort`|`episode_number` -`language` of each stream|From the format's `language` -**Note**: The file format may not support some of these fields + File "hypervideo", line 2 + SyntaxError: Non-ASCII character '\x93' ... +means you're using an outdated version of Python. Please update to Python 2.6 or 2.7. -## Modifying metadata examples +### What is this binary file? Where has the code gone? -```bash -# Interpret the title as "Artist - Title" -$ yt-dlp --parse-metadata "title:%(artist)s - %(title)s" +Since June 2012 ([#342](https://github.com/ytdl-org/youtube-dl/issues/342)) hypervideo is packed as an executable zipfile, simply unzip it (might need renaming to `hypervideo.zip` first on some systems) or clone the git repository, as laid out above. If you modify the code, you can run it by executing the `__main__.py` file. To recompile the executable, run `make hypervideo`. + +### The exe throws an error due to missing `MSVCR100.dll` + +To run the exe you need to install first the [Microsoft Visual C++ 2010 Service Pack 1 Redistributable Package (x86)](https://download.microsoft.com/download/1/6/5/165255E7-1014-4D0A-B094-B6A430A6BFFC/vcredist_x86.exe). + +### On Windows, how should I set up ffmpeg and hypervideo? Where should I put the exe files? + +If you put hypervideo and ffmpeg in the same directory that you're running the command from, it will work, but that's rather cumbersome. + +To make a different directory work - either for ffmpeg, or for hypervideo, or for both - simply create the directory (say, `C:\bin`, or `C:\Users\<User name>\bin`), put all the executables directly in there, and then [set your PATH environment variable](https://www.java.com/en/download/help/path.xml) to include that directory. + +From then on, after restarting your shell, you will be able to access both hypervideo and ffmpeg (and hypervideo will be able to find ffmpeg) by simply typing `hypervideo` or `ffmpeg`, no matter what directory you're in. + +### How do I put downloads into a specific folder? + +Use the `-o` to specify an [output template](#output-template), for example `-o "/home/user/videos/%(title)s-%(id)s.%(ext)s"`. If you want this for all of your downloads, put the option into your [configuration file](#configuration). + +### How do I download a video starting with a `-`? + +Either prepend `https://www.youtube.com/watch?v=` or separate the ID from the options with `--`: + + $ hypervideo -- -wNyEUrxzFU + $ hypervideo "https://www.youtube.com/watch?v=-wNyEUrxzFU" + +### How do I pass cookies to hypervideo? + +Use the `--cookies` option, for example `--cookies /path/to/cookies/file.txt`. + +In order to extract cookies from browser use any conforming browser extension for exporting cookies. For example, [Get cookies.txt](https://chrome.google.com/webstore/detail/get-cookiestxt/bgaddhkoddajcdgocldbbfleckgcbcid/) (for Chrome) or [cookies.txt](https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/) (for Firefox). + +Note that the cookies file must be in Mozilla/Netscape format and the first line of the cookies file must be either `# HTTP Cookie File` or `# Netscape HTTP Cookie File`. Make sure you have correct [newline format](https://en.wikipedia.org/wiki/Newline) in the cookies file and convert newlines if necessary to correspond with your OS, namely `CRLF` (`\r\n`) for Windows and `LF` (`\n`) for Unix and Unix-like systems (Linux, macOS, etc.). `HTTP Error 400: Bad Request` when using `--cookies` is a good sign of invalid newline format. + +Passing cookies to hypervideo is a good way to workaround login when a particular extractor does not implement it explicitly. Another use case is working around [CAPTCHA](https://en.wikipedia.org/wiki/CAPTCHA) some websites require you to solve in particular cases in order to get access (e.g. YouTube, CloudFlare). + +### How do I stream directly to media player? + +You will first need to tell hypervideo to stream media to stdout with `-o -`, and also tell your media player to read from stdin (it must be capable of this for streaming) and then pipe former to latter. For example, streaming to [vlc](https://www.videolan.org/) can be achieved with: + + $ hypervideo -o - "https://www.youtube.com/watch?v=BaW_jenozKcj" | vlc - + +### How do I download only new videos from a playlist? + +Use download-archive feature. With this feature you should initially download the complete playlist with `--download-archive /path/to/download/archive/file.txt` that will record identifiers of all the videos in a special file. Each subsequent run with the same `--download-archive` will download only new videos and skip all videos that have been downloaded before. Note that only successful downloads are recorded in the file. + +For example, at first, + + $ hypervideo --download-archive archive.txt "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re" + +will download the complete `PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re` playlist and create a file `archive.txt`. Each subsequent run will only download new videos if any: + + $ hypervideo --download-archive archive.txt "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re" + +### Should I add `--hls-prefer-native` into my config? + +When hypervideo detects an HLS video, it can download it either with the built-in downloader or ffmpeg. Since many HLS streams are slightly invalid and ffmpeg/hypervideo each handle some invalid cases better than the other, there is an option to switch the downloader if needed. + +When hypervideo knows that one particular downloader works better for a given website, that downloader will be picked. Otherwise, hypervideo will pick the best downloader for general compatibility, which at the moment happens to be ffmpeg. This choice may change in future versions of hypervideo, with improvements of the built-in downloader and/or ffmpeg. + +In particular, the generic extractor (used when your website is not in the [list of supported sites by youtube-dl](https://ytdl-org.github.io/youtube-dl/supportedsites.html) cannot mandate one specific downloader. + +If you put either `--hls-prefer-native` or `--hls-prefer-ffmpeg` into your configuration, a different subset of videos will fail to download correctly. Instead, it is much better to [file an issue](https://yt-dl.org/bug) or a pull request which details why the native or the ffmpeg HLS downloader is a better choice for your use case. + +### Can you add support for this anime video site, or site which shows current movies for free? + +As a matter of policy (as well as legality), hypervideo does not include support for services that specialize in infringing copyright. As a rule of thumb, if you cannot easily find a video that the service is quite obviously allowed to distribute (i.e. that has been uploaded by the creator, the creator's distributor, or is published under a free license), the service is probably unfit for inclusion to hypervideo. + +A note on the service that they don't host the infringing content, but just link to those who do, is evidence that the service should **not** be included into hypervideo. The same goes for any DMCA note when the whole front page of the service is filled with videos they are not allowed to distribute. A "fair use" note is equally unconvincing if the service shows copyright-protected videos in full without authorization. + +Support requests for services that **do** purchase the rights to distribute their content are perfectly fine though. If in doubt, you can simply include a source that mentions the legitimate purchase of content. + +### How can I speed up work on my issue? + +(Also known as: Help, my important issue not being solved!) The hypervideo core developer team is quite small. While we do our best to solve as many issues as possible, sometimes that can take quite a while. To speed up your issue, here's what you can do: + +First of all, please do report the issue [at our issue tracker](https://yt-dl.org/bugs). That allows us to coordinate all efforts by users and developers, and serves as a unified point. Unfortunately, the hypervideo project has grown too large to use personal email as an effective communication channel. + +Please read the [bug reporting instructions](#bugs) below. A lot of bugs lack all the necessary information. If you can, offer proxy, VPN, or shell access to the hypervideo developers. If you are able to, test the issue from multiple computers in multiple countries to exclude local censorship or misconfiguration issues. + +If nobody is interested in solving your issue, you are welcome to take matters into your own hands and submit a pull request (or coerce/pay somebody else to do so). + +Feel free to bump the issue from time to time by writing a small comment ("Issue is still present in hypervideo version ...from France, but fixed from Belgium"), but please not more than once a month. Please do not declare your issue as `important` or `urgent`. + +### How can I detect whether a given URL is supported by hypervideo? + +For one, have a look at the [list of supported sites](docs/supportedsites.md). Note that it can sometimes happen that the site changes its URL scheme (say, from https://example.com/video/1234567 to https://example.com/v/1234567 ) and hypervideo reports an URL of a service in that list as unsupported. In that case, simply report a bug. + +It is *not* possible to detect whether a URL is supported or not. That's because hypervideo contains a generic extractor which matches **all** URLs. You may be tempted to disable, exclude, or remove the generic extractor, but the generic extractor not only allows users to extract videos from lots of websites that embed a video from another service, but may also be used to extract video from a service that it's hosting itself. Therefore, we neither recommend nor support disabling, excluding, or removing the generic extractor. + +If you want to find out whether a given URL is supported, simply call hypervideo with it. If you get no videos back, chances are the URL is either not referring to a video or unsupported. You can find out which by examining the output (if you run hypervideo on the console) or catching an `UnsupportedError` exception if you run it from a Python program. + +# Why do I need to go through that much red tape when filing bugs? + +Before we had the issue template, despite our extensive [bug reporting instructions](#bugs), about 80% of the issue reports we got were useless, for instance because people used ancient versions hundreds of releases old, because of simple syntactic errors (not in hypervideo but in general shell usage), because the problem was already reported multiple times before, because people did not actually read an error message, even if it said "please install ffmpeg", because people did not mention the URL they were trying to download and many more simple, easy-to-avoid problems, many of whom were totally unrelated to hypervideo. + +hypervideo is an free software project manned by too few volunteers, so we'd rather spend time fixing bugs where we are certain none of those simple problems apply, and where we can be reasonably confident to be able to reproduce the issue without asking the reporter repeatedly. As such, the output of `hypervideo -v YOUR_URL_HERE` is really all that's required to file an issue. The issue template also guides you through some basic steps you can do, such as checking that your version of hypervideo is current. + +# DEVELOPER INSTRUCTIONS + +Most users do not need to build hypervideo and can [download the builds](https://git.conocimientoslibres.ga/software/hypervideo.git) or get them from their distribution. + +To run hypervideo as a developer, you don't need to build anything either. Simply execute + + $ python -m youtube_dl + +To run the test, simply invoke your favorite test runner, or execute a test file directly; any of the following work: + + $ python -m unittest discover + $ python test/test_download.py + $ nosetests + +See item 6 of [new extractor tutorial](#adding-support-for-a-new-site) for how to run extractor specific test cases. + +If you want to create a build of hypervideo yourself, you'll need + +* python +* make (only GNU make is supported) +* pandoc +* zip +* nosetests + +### Adding support for a new site + +If you want to add support for a new site, first of all **make sure** this site is **not dedicated to [copyright infringement](README.md#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free)**. hypervideo does **not support** such sites thus pull requests adding support for them **will be rejected**. + +After you have ensured this site is distributing its content legally, you can follow this quick list (assuming your service is called `yourextractor`): + +1. [Fork this repository](https://git.conocimientoslibres.ga/software/hypervideo.git) +2. Check out the source code with: + + $ git clone https://git.conocimientoslibres.ga/software/hypervideo.git + +3. Start a new git branch with + + $ cd hypervideo + $ git checkout -b yourextractor + +4. Start with this simple template and save it to `youtube_dl/extractor/yourextractor.py`: + +```python + # coding: utf-8 + from __future__ import unicode_literals + + from .common import InfoExtractor + + + class YourExtractorIE(InfoExtractor): + _VALID_URL = r'https?://(?:www\.)?yourextractor\.com/watch/(?P<id>[0-9]+)' + _TEST = { + 'url': 'https://yourextractor.com/watch/42', + 'md5': 'TODO: md5 sum of the first 10241 bytes of the video file (use --test)', + 'info_dict': { + 'id': '42', + 'ext': 'mp4', + 'title': 'Video title goes here', + 'thumbnail': r're:^https?://.*\.jpg$', + # TODO more properties, either as: + # * A value + # * MD5 checksum; start the string with md5: + # * A regular expression; start the string with re: + # * Any Python type (for example int or float) + } + } + + def _real_extract(self, url): + video_id = self._match_id(url) + webpage = self._download_webpage(url, video_id) + + # TODO more code goes here, for example ... + title = self._html_search_regex(r'<h1>(.+?)</h1>', webpage, 'title') + + return { + 'id': video_id, + 'title': title, + 'description': self._og_search_description(webpage), + 'uploader': self._search_regex(r'<div[^>]+id="uploader"[^>]*>([^<]+)<', webpage, 'uploader', fatal=False), + # TODO more properties (see youtube_dl/extractor/common.py) + } +``` -# Regex example -$ yt-dlp --parse-metadata "description:Artist - (?P<artist>.+)" +5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/extractors.py). +6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc. Note that tests with `only_matching` key in test's dict are not counted in. +7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/ytdl-org/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L94-L303). Add tests and code for as many as you want. +8. Make sure your code follows [hypervideo coding conventions](#hypervideo-coding-conventions) and check the code with [flake8](https://flake8.pycqa.org/en/latest/index.html#quickstart): -# Set title as "Series name S01E05" -$ yt-dlp --parse-metadata "%(series)s S%(season_number)02dE%(episode_number)02d:%(title)s" + $ flake8 youtube_dl/extractor/yourextractor.py -# Set "comment" field in video metadata using description instead of webpage_url -$ yt-dlp --parse-metadata "description:(?s)(?P<meta_comment>.+)" --add-metadata +9. Make sure your code works under all [Python](https://www.python.org/) versions claimed supported by hypervideo, namely 2.6, 2.7, and 3.2+. +10. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files and [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this: -# Remove "formats" field from the infojson by setting it to an empty string -$ yt-dlp --parse-metadata ":(?P<formats>)" -j + $ git add youtube_dl/extractor/extractors.py + $ git add youtube_dl/extractor/yourextractor.py + $ git commit -m '[yourextractor] Add new extractor' + $ git push origin yourextractor -# Replace all spaces and "_" in title and uploader with a `-` -$ yt-dlp --replace-in-metadata "title,uploader" "[ _]" "-" +11. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it. +In any case, thank you very much for your contributions! + +## hypervideo coding conventions + +This section introduces a guide lines for writing idiomatic, robust and future-proof extractor code. + +Extractors are very fragile by nature since they depend on the layout of the source data provided by 3rd party media hosters out of your control and this layout tends to change. As an extractor implementer your task is not only to write code that will extract media links and metadata correctly but also to minimize dependency on the source's layout and even to make the code foresee potential future changes and be ready for that. This is important because it will allow the extractor not to break on minor layout changes thus keeping old hypervideo versions working. Even though this breakage issue is easily fixed by emitting a new version of hypervideo with a fix incorporated, all the previous versions become broken in all repositories and distros' packages that may not be so prompt in fetching the update from us. Needless to say, some non rolling release distros may never receive an update at all. + +### Mandatory and optional metafields + +For extraction to work hypervideo relies on metadata your extractor extracts and provides to hypervideo expressed by an [information dictionary](https://github.com/ytdl-org/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L94-L303) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by youtube-dl: + + - `id` (media identifier) + - `title` (media title) + - `url` (media download URL) or `formats` + +In fact only the last option is technically mandatory (i.e. if you can't figure out the download location of the media the extraction does not make any sense). But by convention hypervideo also treats `id` and `title` as mandatory. Thus the aforementioned metafields are the critical data that the extraction does not make any sense without and if any of them fail to be extracted then the extractor is considered completely broken. + +[Any field](https://github.com/ytdl-org/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L188-L303) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerant** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields. + +#### Example + +Say you have some source dictionary `meta` that you've fetched as JSON with HTTP request and it has a key `summary`: + +```python +meta = self._download_json(url, video_id) +``` + +Assume at this point `meta`'s layout is: + +```python +{ + ... + "summary": "some fancy summary text", + ... +} +``` + +Assume you want to extract `summary` and put it into the resulting info dict as `description`. Since `description` is an optional meta field you should be ready that this key may be missing from the `meta` dict, so that you should extract it like: + +```python +description = meta.get('summary') # correct +``` + +and not like: + +```python +description = meta['summary'] # incorrect +``` + +The latter will break extraction process with `KeyError` if `summary` disappears from `meta` at some later time but with the former approach extraction will just go ahead with `description` set to `None` which is perfectly fine (remember `None` is equivalent to the absence of data). + +Similarly, you should pass `fatal=False` when extracting optional data from a webpage with `_search_regex`, `_html_search_regex` or similar methods, for instance: + +```python +description = self._search_regex( + r'<span[^>]+id="title"[^>]*>([^<]+)<', + webpage, 'description', fatal=False) +``` + +With `fatal` set to `False` if `_search_regex` fails to extract `description` it will emit a warning and continue extraction. + +You can also pass `default=<some fallback value>`, for example: + +```python +description = self._search_regex( + r'<span[^>]+id="title"[^>]*>([^<]+)<', + webpage, 'description', default=None) +``` + +On failure this code will silently continue the extraction with `description` set to `None`. That is useful for metafields that may or may not be present. + +### Provide fallbacks + +When extracting metadata try to do so from multiple sources. For example if `title` is present in several places, try extracting from at least some of them. This makes it more future-proof in case some of the sources become unavailable. + +#### Example + +Say `meta` from the previous example has a `title` and you are about to extract it. Since `title` is a mandatory meta field you should end up with something like: + +```python +title = meta['title'] +``` + +If `title` disappears from `meta` in future due to some changes on the hoster's side the extraction would fail since `title` is mandatory. That's expected. + +Assume that you have some another source you can extract `title` from, for example `og:title` HTML meta of a `webpage`. In this case you can provide a fallback scenario: + +```python +title = meta.get('title') or self._og_search_title(webpage) +``` + +This code will try to extract from `meta` first and if it fails it will try extracting `og:title` from a `webpage`. + +### Regular expressions + +#### Don't capture groups you don't use + +Capturing group must be an indication that it's used somewhere in the code. Any group that is not used must be non capturing. + +##### Example + +Don't capture id attribute name here since you can't use it for anything anyway. + +Correct: + +```python +r'(?:id|ID)=(?P<id>\d+)' +``` + +Incorrect: +```python +r'(id|ID)=(?P<id>\d+)' +``` + + +#### Make regular expressions relaxed and flexible + +When using regular expressions try to write them fuzzy, relaxed and flexible, skipping insignificant parts that are more likely to change, allowing both single and double quotes for quoted values and so on. + +##### Example + +Say you need to extract `title` from the following HTML code: + +```html +<span style="position: absolute; left: 910px; width: 90px; float: right; z-index: 9999;" class="title">some fancy title</span> +``` + +The code for that task should look similar to: + +```python +title = self._search_regex( + r'<span[^>]+class="title"[^>]*>([^<]+)', webpage, 'title') +``` + +Or even better: + +```python +title = self._search_regex( + r'<span[^>]+class=(["\'])title\1[^>]*>(?P<title>[^<]+)', + webpage, 'title', group='title') ``` -# EXTRACTOR ARGUMENTS +Note how you tolerate potential changes in the `style` attribute's value or switch from using double quotes to single for `class` attribute: -Some extractors accept additional arguments which can be passed using `--extractor-args KEY:ARGS`. `ARGS` is a `;` (semicolon) separated string of `ARG=VAL1,VAL2`. Eg: `--extractor-args "youtube:player-client=android_agegate,web;include_live_dash" --extractor-args "funimation:version=uncut"` +The code definitely should not look like: -The following extractors use this feature: +```python +title = self._search_regex( + r'<span style="position: absolute; left: 910px; width: 90px; float: right; z-index: 9999;" class="title">(.*?)</span>', + webpage, 'title', group='title') +``` + +### Long lines policy + +There is a soft limit to keep lines of code under 80 characters long. This means it should be respected if possible and if it does not make readability and code maintenance worse. + +For example, you should **never** split long string literals like URLs or some other often copied entities over multiple lines to fit this limit: + +Correct: + +```python +'https://www.youtube.com/watch?v=FqZTN594JQw&list=PLMYEtVRpaqY00V9W81Cwmzp6N6vZqfUKD4' +``` + +Incorrect: + +```python +'https://www.youtube.com/watch?v=FqZTN594JQw&list=' +'PLMYEtVRpaqY00V9W81Cwmzp6N6vZqfUKD4' +``` + +### Inline values + +Extracting variables is acceptable for reducing code duplication and improving readability of complex expressions. However, you should avoid extracting variables used only once and moving them to opposite parts of the extractor file, which makes reading the linear flow difficult. + +#### Example + +Correct: + +```python +title = self._html_search_regex(r'<title>([^<]+)</title>', webpage, 'title') +``` -#### youtube -* `skip`: `hls` or `dash` (or both) to skip download of the respective manifests -* `player_client`: Clients to extract video data from. The main clients are `web`, `android`, `ios`, `mweb`. These also have `_music`, `_embedded`, `_agegate`, and `_creator` variants (Eg: `web_embedded`) (`mweb` has only `_agegate`). By default, `android,web` is used, but the agegate and creator variants are added as required for age-gated videos. Similarly the music variants are added for `music.youtube.com` urls. You can also use `all` to use all the clients, and `default` for the default clients. -* `player_skip`: Skip some network requests that are generally needed for robust extraction. One or more of `configs` (skip client configs), `webpage` (skip initial webpage), `js` (skip js player). While these options can help reduce the number of requests needed or avoid some rate-limiting, they could cause some issues. See [#860](https://github.com/yt-dlp/yt-dlp/pull/860) for more details -* `include_live_dash`: Include live dash formats even without `--live-from-start` (These formats don't download properly) -* `comment_sort`: `top` or `new` (default) - choose comment sorting mode (on YouTube's side) -* `max_comments`: Limit the amount of comments to gather. Comma-separated list of integers representing `max-comments,max-parents,max-replies,max-replies-per-thread`. Default is `all,all,all,all`. - * E.g. `all,all,1000,10` will get a maximum of 1000 replies total, with up to 10 replies per thread. `1000,all,100` will get a maximum of 1000 comments, with a maximum of 100 replies total. -* `max_comment_depth` Maximum depth for nested comments. YouTube supports depths 1 or 2 (default) - * **Deprecated**: Set `max-replies` to `0` or `all` in `max_comments` instead (e.g. `max_comments=all,all,0` to get no replies) +Incorrect: -#### youtubetab (YouTube playlists, channels, feeds, etc.) -* `skip`: One or more of `webpage` (skip initial webpage download), `authcheck` (allow the download of playlists requiring authentication when no initial webpage is downloaded. This may cause unwanted behavior, see [#1122](https://github.com/yt-dlp/yt-dlp/pull/1122) for more details) -* `approximate_date`: Extract approximate `upload_date` in flat-playlist. This may cause date-based filters to be slightly off +```python +TITLE_RE = r'<title>([^<]+)</title>' +# ...some lines of code... +title = self._html_search_regex(TITLE_RE, webpage, 'title') +``` -#### funimation -* `language`: Languages to extract. Eg: `funimation:language=english,japanese` -* `version`: The video version to extract - `uncut` or `simulcast` +### Collapse fallbacks -#### crunchyroll -* `language`: Languages to extract. Eg: `crunchyroll:language=jaJp` -* `hardsub`: Which hard-sub versions to extract. Eg: `crunchyroll:hardsub=None,enUS` +Multiple fallback values can quickly become unwieldy. Collapse multiple fallback values into a single expression via a list of patterns. -#### crunchyroll:beta -* `format`: Which stream type(s) to extract. Default is `adaptive_hls` Eg: `crunchyrollbeta:format=vo_adaptive_hls` - * Potentially useful values include `adaptive_hls`, `adaptive_dash`, `vo_adaptive_hls`, `vo_adaptive_dash`, `download_hls`, `trailer_hls`, `trailer_dash` -* `hardsub`: Preference order for which hardsub versions to extract. Default is `None` (no hardsubs). Eg: `crunchyrollbeta:hardsub=en-US,None` +#### Example -#### vikichannel -* `video_types`: Types of videos to download - one or more of `episodes`, `movies`, `clips`, `trailers` +Good: -#### youtubewebarchive -* `check_all`: Try to check more at the cost of more requests. One or more of `thumbnails`, `captures` +```python +description = self._html_search_meta( + ['og:description', 'description', 'twitter:description'], + webpage, 'description', default=None) +``` -#### gamejolt -* `comment_sort`: `hot` (default), `you` (cookies needed), `top`, `new` - choose comment sorting mode (on GameJolt's side) +Unwieldy: -#### hotstar -* `res`: resolution to ignore - one or more of `sd`, `hd`, `fhd` -* `vcodec`: vcodec to ignore - one or more of `h264`, `h265`, `dvh265` -* `dr`: dynamic range to ignore - one or more of `sdr`, `hdr10`, `dv` +```python +description = ( + self._og_search_description(webpage, default=None) + or self._html_search_meta('description', webpage, default=None) + or self._html_search_meta('twitter:description', webpage, default=None)) +``` -#### tiktok -* `app_version`: App version to call mobile APIs with - should be set along with `manifest_app_version`. (e.g. `20.2.1`) -* `manifest_app_version`: Numeric app version to call mobile APIs with. (e.g. `221`) +Methods supporting list of patterns are: `_search_regex`, `_html_search_regex`, `_og_search_property`, `_html_search_meta`. -#### rokfinchannel -* `tab`: Which tab to download. One of `new`, `top`, `videos`, `podcasts`, `streams`, `stacks`. (E.g. `rokfinchannel:tab=streams`) +### Trailing parentheses +Always move trailing parentheses after the last argument. -NOTE: These options may be changed/removed in the future without concern for backward compatibility +#### Example -<!-- MANPAGE: MOVE "INSTALLATION" SECTION HERE --> +Correct: +```python + lambda x: x['ResultSet']['Result'][0]['VideoUrlSet']['VideoUrl'], + list) +``` -# PLUGINS +Incorrect: -Plugins are loaded from `<root-dir>/ytdlp_plugins/<type>/__init__.py`; where `<root-dir>` is the directory of the binary (`<root-dir>/yt-dlp`), or the root directory of the module if you are running directly from source-code (`<root dir>/yt_dlp/__main__.py`). Plugins are currently not supported for the `pip` version +```python + lambda x: x['ResultSet']['Result'][0]['VideoUrlSet']['VideoUrl'], + list, +) +``` -Plugins can be of `<type>`s `extractor` or `postprocessor`. Extractor plugins do not need to be enabled from the CLI and are automatically invoked when the input URL is suitable for it. Postprocessor plugins can be invoked using `--use-postprocessor NAME`. +### Use convenience conversion and parsing functions -See [ytdlp_plugins](ytdlp_plugins) for example plugins. +Wrap all extracted numeric data into safe functions from [`youtube_dl/utils.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/utils.py): `int_or_none`, `float_or_none`. Use them for string to number conversions as well. -Note that **all** plugins are imported even if not invoked, and that **there are no checks** performed on plugin code. Use plugins at your own risk and only if you trust the code +Use `url_or_none` for safe URL processing. -If you are a plugin author, add [ytdlp-plugins](https://github.com/topics/ytdlp-plugins) as a topic to your repository for discoverability +Use `try_get` for safe metadata extraction from parsed JSON. +Use `unified_strdate` for uniform `upload_date` or any `YYYYMMDD` meta field extraction, `unified_timestamp` for uniform `timestamp` extraction, `parse_filesize` for `filesize` extraction, `parse_count` for count meta fields extraction, `parse_resolution`, `parse_duration` for `duration` extraction, `parse_age_limit` for `age_limit` extraction. +Explore [`youtube_dl/utils.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/utils.py) for more useful convenience functions. -# EMBEDDING YT-DLP +#### More examples -yt-dlp makes the best effort to be a good command-line program, and thus should be callable from any programming language. +##### Safely extract optional description from parsed JSON +```python +description = try_get(response, lambda x: x['result']['video'][0]['summary'], compat_str) +``` -Your program should avoid parsing the normal stdout since they may change in future versions. Instead they should use options such as `-J`, `--print`, `--progress-template`, `--exec` etc to create console output that you can reliably reproduce and parse. +##### Safely extract more optional metadata +```python +video = try_get(response, lambda x: x['result']['video'][0], dict) or {} +description = video.get('summary') +duration = float_or_none(video.get('durationMs'), scale=1000) +view_count = int_or_none(video.get('views')) +``` -From a Python program, you can embed yt-dlp in a more powerful fashion, like this: +# EMBEDDING YOUTUBE-DL + +hypervideo makes the best effort to be a good command-line program, and thus should be callable from any programming language. If you encounter any problems parsing its output, feel free to [create a report](https://github.com/ytdl-org/youtube-dl/issues/new). + +From a Python program, you can embed hypervideo in a more powerful fashion, like this: ```python -from yt_dlp import YoutubeDL +from __future__ import unicode_literals +import youtube_dl -ydl_opts = {'format': 'bestaudio'} -with YoutubeDL(ydl_opts) as ydl: +ydl_opts = {} +with youtube_dl.YoutubeDL(ydl_opts) as ydl: ydl.download(['https://www.youtube.com/watch?v=BaW_jenozKc']) ``` -Most likely, you'll want to use various options. For a list of options available, have a look at [`yt_dlp/YoutubeDL.py`](yt_dlp/YoutubeDL.py#L191). +Most likely, you'll want to use various options. For a list of options available, have a look at [`youtube_dl/YoutubeDL.py`](https://github.com/ytdl-org/youtube-dl/blob/3e4cedf9e8cd3157df2457df7274d0c842421945/youtube_dl/YoutubeDL.py#L137-L312). For a start, if you want to intercept youtube-dl's output, set a `logger` object. -Here's a more complete example demonstrating various functionality: +Here's a more complete example of a program that outputs only errors (and a short message after the download is finished), and downloads/converts the video to an mp3 file: ```python -import json -import yt_dlp +from __future__ import unicode_literals +import youtube_dl -class MyLogger: +class MyLogger(object): def debug(self, msg): - # For compatibility with youtube-dl, both debug and info are passed into debug - # You can distinguish them by the prefix '[debug] ' - if msg.startswith('[debug] '): - pass - else: - self.info(msg) - - def info(self, msg): pass def warning(self, msg): @@ -1762,196 +1347,103 @@ class MyLogger: print(msg) -# ℹ️ See the docstring of yt_dlp.postprocessor.common.PostProcessor -class MyCustomPP(yt_dlp.postprocessor.PostProcessor): - # ℹ️ See docstring of yt_dlp.postprocessor.common.PostProcessor.run - def run(self, info): - self.to_screen('Doing stuff') - return [], info - - -# ℹ️ See "progress_hooks" in the docstring of yt_dlp.YoutubeDL def my_hook(d): if d['status'] == 'finished': print('Done downloading, now converting ...') -def format_selector(ctx): - """ Select the best video and the best audio that won't result in an mkv. - This is just an example and does not handle all cases """ - - # formats are already sorted worst to best - formats = ctx.get('formats')[::-1] - - # acodec='none' means there is no audio - best_video = next(f for f in formats - if f['vcodec'] != 'none' and f['acodec'] == 'none') - - # find compatible audio extension - audio_ext = {'mp4': 'm4a', 'webm': 'webm'}[best_video['ext']] - # vcodec='none' means there is no video - best_audio = next(f for f in formats if ( - f['acodec'] != 'none' and f['vcodec'] == 'none' and f['ext'] == audio_ext)) - - yield { - # These are the minimum required fields for a merged format - 'format_id': f'{best_video["format_id"]}+{best_audio["format_id"]}', - 'ext': best_video['ext'], - 'requested_formats': [best_video, best_audio], - # Must be + separated list of protocols - 'protocol': f'{best_video["protocol"]}+{best_audio["protocol"]}' - } - - -# ℹ️ See docstring of yt_dlp.YoutubeDL for a description of the options ydl_opts = { - 'format': format_selector, + 'format': 'bestaudio/best', 'postprocessors': [{ - # Embed metadata in video using ffmpeg. - # ℹ️ See yt_dlp.postprocessor.FFmpegMetadataPP for the arguments it accepts - 'key': 'FFmpegMetadata', - 'add_chapters': True, - 'add_metadata': True, + 'key': 'FFmpegExtractAudio', + 'preferredcodec': 'mp3', + 'preferredquality': '192', }], 'logger': MyLogger(), 'progress_hooks': [my_hook], } +with youtube_dl.YoutubeDL(ydl_opts) as ydl: + ydl.download(['https://www.youtube.com/watch?v=BaW_jenozKc']) +``` +# BUGS -# Add custom headers -yt_dlp.utils.std_headers.update({'Referer': 'https://www.google.com'}) +Bugs and suggestions should be reported at: <https://github.com/ytdl-org/youtube-dl/issues>. Unless you were prompted to or there is another pertinent reason (e.g. GitHub fails to accept the bug report), please do not send bug reports via personal email. For discussions, join us in the IRC channel [#youtube-dl](irc://chat.freenode.net/#youtube-dl) on freenode ([webchat](https://webchat.freenode.net/?randomnick=1&channels=youtube-dl)). -# ℹ️ See the public functions in yt_dlp.YoutubeDL for for other available functions. -# Eg: "ydl.download", "ydl.download_with_info_file" -with yt_dlp.YoutubeDL(ydl_opts) as ydl: - ydl.add_post_processor(MyCustomPP()) - info = ydl.extract_info('https://www.youtube.com/watch?v=BaW_jenozKc') +**Please include the full output of hypervideo when run with `-v`**, i.e. **add** `-v` flag to **your command line**, copy the **whole** output and post it in the issue body wrapped in \`\`\` for better formatting. It should look similar to this: - # ℹ️ ydl.sanitize_info makes the info json-serializable - print(json.dumps(ydl.sanitize_info(info))) ``` +$ hypervideo -v <your command line> +[debug] System config: [] +[debug] User config: [] +[debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=BaW_jenozKcj'] +[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251 +[debug] hypervideo version 1.1.11 +[debug] Git HEAD: 135392e +[debug] Python version 2.6.6 - Windows-2003Server-5.2.3790-SP2 +[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4 +[debug] Proxy map: {} +... +``` + +**Do not post screenshots of verbose logs; only plain text is acceptable.** + +The output (including the first lines) contains important debugging information. Issues without the full output are often not reproducible and therefore do not get solved in short order, if ever. + +Please re-read your issue once again to avoid a couple of common mistakes (you can and should use this as a checklist): + +### Is the description of the issue itself sufficient? + +We often get issue reports that we cannot really decipher. While in most cases we eventually get the required information after asking back multiple times, this poses an unnecessary drain on our resources. Many contributors, including myself, are also not native speakers, so we may misread some parts. + +So please elaborate on what feature you are requesting, or what bug you want to be fixed. Make sure that it's obvious + +- What the problem is +- How it could be fixed +- How your proposed solution would look like + +If your report is shorter than two lines, it is almost certainly missing some of these, which makes it hard for us to respond to it. We're often too polite to close the issue outright, but the missing info makes misinterpretation likely. As a committer myself, I often get frustrated by these issues, since the only possible way for me to move forward on them is to ask for clarification over and over. + +For bug reports, this means that your report should contain the *complete* output of hypervideo when called with the `-v` flag. The error message you get for (most) bugs even says so, but you would not believe how many of our bug reports do not contain this information. + +If your server has multiple IPs or you suspect censorship, adding `--call-home` may be a good idea to get more diagnostics. If the error is `ERROR: Unable to extract ...` and you cannot reproduce it from multiple countries, add `--dump-pages` (warning: this will yield a rather large output, redirect it to the file `log.txt` by adding `>log.txt 2>&1` to your command-line) or upload the `.dump` files you get when you add `--write-pages` [somewhere](https://gist.github.com/). + +**Site support requests must contain an example URL**. An example URL is a URL you might want to download, like `https://www.youtube.com/watch?v=BaW_jenozKc`. There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. `https://www.youtube.com/`) is *not* an example URL. + +### Are you using the latest version? + +Before reporting any issue, type `doas pacman -Sy hypervideo`. This should report that you're up-to-date. About 20% of the reports we receive are already fixed, but people are using outdated versions. This goes for feature requests as well. + +### Is the issue already documented? + +Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or browse the [GitHub Issues](https://github.com/ytdl-org/youtube-dl/search?type=Issues) of this repository. If there is an issue, feel free to write something along the lines of "This affects me as well, with version 2015.01.01. Here is some more information on the issue: ...". While some issues may be old, a new post into them often spurs rapid activity. + +### Why are existing options not enough? + +Before requesting a new feature, please have a quick peek at [the list of supported options](https://github.com/ytdl-org/youtube-dl/blob/master/README.md#options). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem. + +### Is there enough context in your bug report? + +People want to solve problems, and often think they do us a favor by breaking down their larger problems (e.g. wanting to skip already downloaded files) to a specific request (e.g. requesting us to look whether the file exists before downloading the info page). However, what often happens is that they break down the problem into two steps: One simple, and one impossible (or extremely complicated one). + +We are then presented with a very complicated request when the original problem could be solved far easier, e.g. by recording the downloaded video IDs in a separate file. To avoid this, you must include the greater context where it is non-obvious. In particular, every feature request that does not consist of adding support for a new site should contain a use case scenario that explains in what situation the missing feature would be useful. + +### Does the issue involve one problem, and one problem only? + +Some of our users seem to think there is a limit of issues they can or should open. There is no limit of issues they can or should open. While it may seem appealing to be able to dump all your issues into one ticket, that means that someone who solves one of your issues cannot mark the issue as closed. Typically, reporting a bunch of issues leads to the ticket lingering since nobody wants to attack that behemoth, until someone mercifully splits the issue into multiple ones. + +In particular, every site support request issue should only pertain to services at one site (generally under a common domain, but always using the same backend technology). Do not request support for vimeo user videos, White house podcasts, and Google Plus pages in the same issue. Also, make sure that you don't post bug reports alongside feature requests. As a rule of thumb, a feature request does not include outputs of hypervideo that are not immediately related to the feature at hand. Do not post reports of a network error alongside the request for a new video service. + +### Is anyone going to need the feature? + +Only post features that you (or an incapacitated friend you can personally talk to) require. Do not post features because they seem like a good idea. If they are really useful, they will be requested by someone who requires them. + +### Is your question about hypervideo? + +It may sound strange, but some bug reports we receive are completely unrelated to hypervideo and relate to a different, or even the reporter's own, application. Please make sure that you are actually using hypervideo. If you are using a UI for hypervideo, report the bug to the maintainer of the actual application providing the UI. On the other hand, if your UI for hypervideo fails in some way you believe is related to hypervideo, by all means, go ahead and report the bug. + +# COPYRIGHT + +hypervideo is released into the [CC0 1.0 Universal](LICENSE) by the copyright holders. -**Tip**: If you are porting your code from youtube-dl to yt-dlp, one important point to look out for is that we do not guarantee the return value of `YoutubeDL.extract_info` to be json serializable, or even be a dictionary. It will be dictionary-like, but if you want to ensure it is a serializable dictionary, pass it through `YoutubeDL.sanitize_info` as shown in the example above - - -<!-- MANPAGE: MOVE "NEW FEATURES" SECTION HERE --> - -# DEPRECATED OPTIONS - -These are all the deprecated options and the current alternative to achieve the same effect - -#### Almost redundant options -While these options are almost the same as their new counterparts, there are some differences that prevents them being redundant - - -j, --dump-json --print "%()j" - -F, --list-formats --print formats_table - --list-thumbnails --print thumbnails_table --print playlist:thumbnails_table - --list-subs --print automatic_captions_table --print subtitles_table - -#### Redundant options -While these options are redundant, they are still expected to be used due to their ease of use - - --get-description --print description - --get-duration --print duration_string - --get-filename --print filename - --get-format --print format - --get-id --print id - --get-thumbnail --print thumbnail - -e, --get-title --print title - -g, --get-url --print urls - --match-title REGEX --match-filter "title ~= (?i)REGEX" - --reject-title REGEX --match-filter "title !~= (?i)REGEX" - --min-views COUNT --match-filter "view_count >=? COUNT" - --max-views COUNT --match-filter "view_count <=? COUNT" - --user-agent UA --add-header "User-Agent:UA" - --referer URL --add-header "Referer:URL" - - -#### Not recommended -While these options still work, their use is not recommended since there are other alternatives to achieve the same - - --exec-before-download CMD --exec "before_dl:CMD" - --no-exec-before-download --no-exec - --all-formats -f all - --all-subs --sub-langs all --write-subs - --print-json -j --no-simulate - --autonumber-size NUMBER Use string formatting. Eg: %(autonumber)03d - --autonumber-start NUMBER Use internal field formatting like %(autonumber+NUMBER)s - --id -o "%(id)s.%(ext)s" - --metadata-from-title FORMAT --parse-metadata "%(title)s:FORMAT" - --hls-prefer-native --downloader "m3u8:native" - --hls-prefer-ffmpeg --downloader "m3u8:ffmpeg" - --list-formats-old --compat-options list-formats (Alias: --no-list-formats-as-table) - --list-formats-as-table --compat-options -list-formats [Default] (Alias: --no-list-formats-old) - --youtube-skip-dash-manifest --extractor-args "youtube:skip=dash" (Alias: --no-youtube-include-dash-manifest) - --youtube-skip-hls-manifest --extractor-args "youtube:skip=hls" (Alias: --no-youtube-include-hls-manifest) - --youtube-include-dash-manifest Default (Alias: --no-youtube-skip-dash-manifest) - --youtube-include-hls-manifest Default (Alias: --no-youtube-skip-hls-manifest) - - -#### Developer options -These options are not intended to be used by the end-user - - --test Download only part of video for testing extractors - --youtube-print-sig-code For testing youtube signatures - --allow-unplayable-formats List unplayable formats also - --no-allow-unplayable-formats Default - - -#### Old aliases -These are aliases that are no longer documented for various reasons - - --avconv-location --ffmpeg-location - --clean-infojson --clean-info-json - --cn-verification-proxy URL --geo-verification-proxy URL - --dump-headers --print-traffic - --dump-intermediate-pages --dump-pages - --force-write-download-archive --force-write-archive - --load-info --load-info-json - --no-clean-infojson --no-clean-info-json - --no-split-tracks --no-split-chapters - --no-write-srt --no-write-subs - --prefer-unsecure --prefer-insecure - --rate-limit RATE --limit-rate RATE - --split-tracks --split-chapters - --srt-lang LANGS --sub-langs LANGS - --trim-file-names LENGTH --trim-filenames LENGTH - --write-srt --write-subs - --yes-overwrites --force-overwrites - -#### Sponskrub Options -Support for [SponSkrub](https://github.com/faissaloo/SponSkrub) has been deprecated in favor of the `--sponsorblock` options - - --sponskrub --sponsorblock-mark all - --no-sponskrub --no-sponsorblock - --sponskrub-cut --sponsorblock-remove all - --no-sponskrub-cut --sponsorblock-remove -all - --sponskrub-force Not applicable - --no-sponskrub-force Not applicable - --sponskrub-location Not applicable - --sponskrub-args Not applicable - -#### No longer supported -These options may no longer work as intended - - --prefer-avconv avconv is not officially supported by yt-dlp (Alias: --no-prefer-ffmpeg) - --prefer-ffmpeg Default (Alias: --no-prefer-avconv) - -C, --call-home Not implemented - --no-call-home Default - --include-ads No longer supported - --no-include-ads Default - --write-annotations No supported site has annotations now - --no-write-annotations Default - -#### Removed -These options were deprecated since 2014 and have now been entirely removed - - -A, --auto-number -o "%(autonumber)s-%(id)s.%(ext)s" - -t, --title -o "%(title)s-%(id)s.%(ext)s" - -l, --literal -o accepts literal names - -# CONTRIBUTING -See [CONTRIBUTING.md](CONTRIBUTING.md#contributing-to-yt-dlp) for instructions on [Opening an Issue](CONTRIBUTING.md#opening-an-issue) and [Contributing code to the project](CONTRIBUTING.md#developer-instructions) - -# MORE -For FAQ see the [youtube-dl README](https://github.com/ytdl-org/youtube-dl#faq) +This README file was originally written by [Daniel Bolton](https://github.com/dbbolton) and is likewise released into the public domain. diff --git a/bin/yt-dlp b/bin/yt-dlp new file mode 100644 index 000000000..baecdeb0a --- /dev/null +++ b/bin/yt-dlp @@ -0,0 +1,6 @@ +#!/usr/bin/python + +import hypervideo_dl + +if __name__ == '__main__': + hypervideo_dl.main() diff --git a/build.sh b/build.sh new file mode 100644 index 000000000..8726833db --- /dev/null +++ b/build.sh @@ -0,0 +1,104 @@ +#!/bin/bash + +# ============================================================================== +# FUNCTIONS - START +# ============================================================================== +VERSION='1.1.12' +BASEDIR="${2-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)}" + +run_rebrand() +{ + printf '%b%s%b%s%b\n' '\e[1;32m' '==> ' '\e[0m\033[1m' 'Rebrand...' '\e[m' + + mv -T yt_dlp hypervideo_dl + mv bin/yt-dlp bin/hypervideo + + find . -type f \( \ + -iname "*" \ + ! -iname "build.sh" \ + ! -iname ".travis.yml" \ + ! -iname ".gitlab-ci.yml" \ + ! -path "./.git*" \) \ + -exec grep -rIl 'yt_dlp' {} + -exec sed -i 's|yt_dlp|hypervideo_dl|g' {} \; + + find . -name "tox.ini" -type f -exec sed -i 's|yt-dlp|hypervideo|g' {} \; + find . -name "setup.cfg" -type f -exec sed -i 's|yt-dlp|hypervideo|g' {} \; + find . -name "Makefile" -type f -exec sed -i 's|yt-dlp|hypervideo|g' {} \; + find . -name "*.py" -type f -exec sed -i 's|yt-dlp|hypervideo|g' {} \; + find . -name "*.in" -type f -exec sed -i 's|yt-dlp|hypervideo|g' {} \; + find docs -type f -exec sed -i 's|yt-dlp|hypervideo|g' {} \; + + # fixes URLs + find . -name "*.py" -type f -exec sed -i 's|hypervideo+test+video|yt-dlp+test+video|g' {} \; + find . -name "*.py" -type f -exec sed -i 's|https://github.com/ytdl-org/hypervideo/|https://github.com/ytdl-org/yt-dlp/|g' {} \; + find . -name "*.py" -type f -exec sed -i 's|https://github.com/rg3/hypervideo/issues/|https://github.com/rg3/yt-dlp/issues/|g' {} \; + find . -name "*.py" -type f -exec sed -i 's|http://hypervideo.bandcamp.com/|http://yt-dlp.bandcamp.com/|g' {} \; + find . -name "*.py" -type f -exec sed -i 's|http://yt-dlp.bandcamp.com/track/hypervideo-test-song|http://yt-dlp.bandcamp.com/track/yt-dlp-test-song|g' {} \; + find . -name "*.py" -type f -exec sed -i 's|https://www.dropbox.com/s/nelirfsxnmcfbfh/hypervideo|https://www.dropbox.com/s/nelirfsxnmcfbfh/yt-dlp|g' {} \; + find . -name "*.py" -type f -exec sed -i 's|http://8tracks.com/ytdl/hypervideo-test-tracks-a|http://8tracks.com/ytdl/yt-dlp-test-tracks-a|g' {} \; + find . -name "*.py" -type f -exec sed -i 's|http://phihag.de/2014/hypervideo/|http://phihag.de/2014/yt-dlp/|g' {} \; + find . -name "*.py" -type f -exec sed -i 's|https://hypervideo-demo.neocities.org/|https://yt-dlp-demo.neocities.org/|g' {} \; + find . -name "*.py" -type f -exec sed -i 's|https://travis-ci.org/ytdl-org/hypervideo/|https://travis-ci.org/ytdl-org/yt-dlp/|g' {} \; + find . -name "*.py" -type f -exec sed -i 's|https://soundcloud.com/jaimemf/hypervideo-test|https://soundcloud.com/jaimemf/yt-dlp-test|g' {} \; + find . -name "*.py" -type f -exec sed -i 's|http://streamcloud.eu/skp9j99s4bpz/hypervideo_test_video|http://streamcloud.eu/skp9j99s4bpz/yt_dlp_test_video|g' {} \; + + find . -type f \( \ + -iname "*" \ + ! -iname "build.sh" \ + ! -iname ".travis.yml" \ + ! -iname ".gitlab-ci.yml" \ + ! -path "./.git*" \) \ + -exec grep -rIl 'YT-DLP' {} + -exec sed -i 's|YT-DLP|HYPERVIDEO|g' {} \; + + # Set current version + sed -i "s|__version.*|__version__ = '${VERSION}'|" hypervideo_dl/version.py + + # Set license + sed -i "s|UNLICENSE|CC0-1.0|" docs/conf.py +} + +run_clean() +{ + printf '%b%s%b%s%b\n' '\e[1;32m' '==> ' '\e[0m\033[1m' 'Clean tempfiles...' '\e[m' + rm -rf -- hypervideo.1.temp.md hypervideo.1 hypervideo.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ hypervideo.tar.gz hypervideo_dl/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png CONTRIBUTING.md.tmp hypervideo hypervideo.exe + find . -name "*.pyc" -delete + find . -name "*.class" -delete +} + +run_make() +{ + printf '%b%s%b%s%b\n' '\e[1;32m' '==> ' '\e[0m\033[1m' 'Making hypervideo...' '\e[m' + make hypervideo.tar.gz +} + +run_copy() +{ + printf '%b%s%b%s%b\n' '\e[1;32m' '==> ' '\e[0m\033[1m' 'Genering build...' '\e[m' + install -d -m755 build + mv hypervideo.tar.gz "./build/hypervideo.tar.gz" + cd build && tar xzf hypervideo.tar.gz && + mv hypervideo "hypervideo-${VERSION}" && + tar -czvf "hypervideo-${VERSION}.tar.gz" "hypervideo-${VERSION}" && + rm -rf "hypervideo-${VERSION}" hypervideo.tar.gz + cd "$BASEDIR" || return + printf '%b%s%b%s%b\n' '\e[1;32m' '==> ' '\e[0m\033[1m' 'Tarball in build/ directory' '\e[m' +} + +# Restore directory, compatibility reasons +run_reset() +{ + printf '%b%s%b%s%b\n' '\e[1;32m' '==> ' '\e[0m\033[1m' 'Restore compatibility...' '\e[m' + mv -T hypervideo_dl yt_dlp + mv bin/hypervideo bin/yt-dlp + rm -v hypervideo + printf '%b%s%b%s%b\n' '\e[1;32m' '==> ' '\e[0m\033[1m' 'Please execute: "git checkout ." for complete restore' '\e[m' +} + +# ============================================================================== +# EXECUTION - START +# ============================================================================== +run_rebrand "$@" && run_clean "$@" + +run_make "$@" + +run_copy "$@" && run_reset "$@" diff --git a/devscripts/create-github-release.py b/devscripts/create-github-release.py deleted file mode 100644 index 53b3e0f48..000000000 --- a/devscripts/create-github-release.py +++ /dev/null @@ -1,112 +0,0 @@ -# Unused - -#!/usr/bin/env python3 -from __future__ import unicode_literals - -import io -import json -import mimetypes -import netrc -import optparse -import os -import re -import sys - -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) - -from yt_dlp.compat import ( - compat_basestring, - compat_getpass, - compat_print, - compat_urllib_request, -) -from yt_dlp.utils import ( - make_HTTPS_handler, - sanitized_Request, -) - - -class GitHubReleaser(object): - _API_URL = 'https://api.github.com/repos/ytdl-org/youtube-dl/releases' - _UPLOADS_URL = 'https://uploads.github.com/repos/ytdl-org/youtube-dl/releases/%s/assets?name=%s' - _NETRC_MACHINE = 'github.com' - - def __init__(self, debuglevel=0): - self._init_github_account() - https_handler = make_HTTPS_handler({}, debuglevel=debuglevel) - self._opener = compat_urllib_request.build_opener(https_handler) - - def _init_github_account(self): - try: - info = netrc.netrc().authenticators(self._NETRC_MACHINE) - if info is not None: - self._token = info[2] - compat_print('Using GitHub credentials found in .netrc...') - return - else: - compat_print('No GitHub credentials found in .netrc') - except (IOError, netrc.NetrcParseError): - compat_print('Unable to parse .netrc') - self._token = compat_getpass( - 'Type your GitHub PAT (personal access token) and press [Return]: ') - - def _call(self, req): - if isinstance(req, compat_basestring): - req = sanitized_Request(req) - req.add_header('Authorization', 'token %s' % self._token) - response = self._opener.open(req).read().decode('utf-8') - return json.loads(response) - - def list_releases(self): - return self._call(self._API_URL) - - def create_release(self, tag_name, name=None, body='', draft=False, prerelease=False): - data = { - 'tag_name': tag_name, - 'target_commitish': 'master', - 'name': name, - 'body': body, - 'draft': draft, - 'prerelease': prerelease, - } - req = sanitized_Request(self._API_URL, json.dumps(data).encode('utf-8')) - return self._call(req) - - def create_asset(self, release_id, asset): - asset_name = os.path.basename(asset) - url = self._UPLOADS_URL % (release_id, asset_name) - # Our files are small enough to be loaded directly into memory. - data = open(asset, 'rb').read() - req = sanitized_Request(url, data) - mime_type, _ = mimetypes.guess_type(asset_name) - req.add_header('Content-Type', mime_type or 'application/octet-stream') - return self._call(req) - - -def main(): - parser = optparse.OptionParser(usage='%prog CHANGELOG VERSION BUILDPATH') - options, args = parser.parse_args() - if len(args) != 3: - parser.error('Expected a version and a build directory') - - changelog_file, version, build_path = args - - with io.open(changelog_file, encoding='utf-8') as inf: - changelog = inf.read() - - mobj = re.search(r'(?s)version %s\n{2}(.+?)\n{3}' % version, changelog) - body = mobj.group(1) if mobj else '' - - releaser = GitHubReleaser() - - new_release = releaser.create_release( - version, name='yt-dlp %s' % version, body=body) - release_id = new_release['id'] - - for asset in os.listdir(build_path): - compat_print('Uploading %s...' % asset) - releaser.create_asset(release_id, os.path.join(build_path, asset)) - - -if __name__ == '__main__': - main() diff --git a/devscripts/gh-pages.unused/add-version.py b/devscripts/gh-pages.unused/add-version.py deleted file mode 100644 index 9ea01374d..000000000 --- a/devscripts/gh-pages.unused/add-version.py +++ /dev/null @@ -1,43 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import unicode_literals - -import json -import sys -import hashlib -import os.path - - -if len(sys.argv) <= 1: - print('Specify the version number as parameter') - sys.exit() -version = sys.argv[1] - -with open('update/LATEST_VERSION', 'w') as f: - f.write(version) - -versions_info = json.load(open('update/versions.json')) -if 'signature' in versions_info: - del versions_info['signature'] - -new_version = {} - -filenames = { - 'bin': 'yt-dlp', - 'exe': 'yt-dlp.exe', - 'tar': 'yt-dlp-%s.tar.gz' % version} -build_dir = os.path.join('..', '..', 'build', version) -for key, filename in filenames.items(): - url = 'https://yt-dl.org/downloads/%s/%s' % (version, filename) - fn = os.path.join(build_dir, filename) - with open(fn, 'rb') as f: - data = f.read() - if not data: - raise ValueError('File %s is empty!' % fn) - sha256sum = hashlib.sha256(data).hexdigest() - new_version[key] = (url, sha256sum) - -versions_info['versions'][version] = new_version -versions_info['latest'] = version - -with open('update/versions.json', 'w') as jsonf: - json.dump(versions_info, jsonf, indent=4, sort_keys=True) diff --git a/devscripts/gh-pages.unused/generate-download.py b/devscripts/gh-pages.unused/generate-download.py deleted file mode 100644 index a873d32ee..000000000 --- a/devscripts/gh-pages.unused/generate-download.py +++ /dev/null @@ -1,22 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import unicode_literals - -import json - -versions_info = json.load(open('update/versions.json')) -version = versions_info['latest'] -version_dict = versions_info['versions'][version] - -# Read template page -with open('download.html.in', 'r', encoding='utf-8') as tmplf: - template = tmplf.read() - -template = template.replace('@PROGRAM_VERSION@', version) -template = template.replace('@PROGRAM_URL@', version_dict['bin'][0]) -template = template.replace('@PROGRAM_SHA256SUM@', version_dict['bin'][1]) -template = template.replace('@EXE_URL@', version_dict['exe'][0]) -template = template.replace('@EXE_SHA256SUM@', version_dict['exe'][1]) -template = template.replace('@TAR_URL@', version_dict['tar'][0]) -template = template.replace('@TAR_SHA256SUM@', version_dict['tar'][1]) -with open('download.html', 'w', encoding='utf-8') as dlf: - dlf.write(template) diff --git a/devscripts/gh-pages.unused/sign-versions.py b/devscripts/gh-pages.unused/sign-versions.py deleted file mode 100644 index fa389c358..000000000 --- a/devscripts/gh-pages.unused/sign-versions.py +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import unicode_literals, with_statement - -import rsa -import json -from binascii import hexlify - -try: - input = raw_input -except NameError: - pass - -versions_info = json.load(open('update/versions.json')) -if 'signature' in versions_info: - del versions_info['signature'] - -print('Enter the PKCS1 private key, followed by a blank line:') -privkey = b'' -while True: - try: - line = input() - except EOFError: - break - if line == '': - break - privkey += line.encode('ascii') + b'\n' -privkey = rsa.PrivateKey.load_pkcs1(privkey) - -signature = hexlify(rsa.pkcs1.sign(json.dumps(versions_info, sort_keys=True).encode('utf-8'), privkey, 'SHA-256')).decode() -print('signature: ' + signature) - -versions_info['signature'] = signature -with open('update/versions.json', 'w') as versionsf: - json.dump(versions_info, versionsf, indent=4, sort_keys=True) diff --git a/devscripts/gh-pages.unused/update-copyright.py b/devscripts/gh-pages.unused/update-copyright.py deleted file mode 100644 index e122d0283..000000000 --- a/devscripts/gh-pages.unused/update-copyright.py +++ /dev/null @@ -1,21 +0,0 @@ -#!/usr/bin/env python3 -# coding: utf-8 - -from __future__ import with_statement, unicode_literals - -import datetime -import glob -import io # For Python 2 compatibility -import os -import re - -year = str(datetime.datetime.now().year) -for fn in glob.glob('*.html*'): - with io.open(fn, encoding='utf-8') as f: - content = f.read() - newc = re.sub(r'(?P<copyright>Copyright © 2011-)(?P<year>[0-9]{4})', 'Copyright © 2011-' + year, content) - if content != newc: - tmpFn = fn + '.part' - with io.open(tmpFn, 'wt', encoding='utf-8') as outf: - outf.write(newc) - os.rename(tmpFn, fn) diff --git a/devscripts/gh-pages.unused/update-feed.py b/devscripts/gh-pages.unused/update-feed.py deleted file mode 100644 index c9f2fdb07..000000000 --- a/devscripts/gh-pages.unused/update-feed.py +++ /dev/null @@ -1,76 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import unicode_literals - -import datetime -import io -import json -import textwrap - - -atom_template = textwrap.dedent("""\ - <?xml version="1.0" encoding="utf-8"?> - <feed xmlns="http://www.w3.org/2005/Atom"> - <link rel="self" href="http://ytdl-org.github.io/youtube-dl/update/releases.atom" /> - <title>yt-dlp releases</title> - <id>https://yt-dl.org/feed/yt-dlp-updates-feed</id> - <updated>@TIMESTAMP@</updated> - @ENTRIES@ - </feed>""") - -entry_template = textwrap.dedent(""" - <entry> - <id>https://yt-dl.org/feed/yt-dlp-updates-feed/yt-dlp-@VERSION@</id> - <title>New version @VERSION@</title> - <link href="http://ytdl-org.github.io/yt-dlp" /> - <content type="xhtml"> - <div xmlns="http://www.w3.org/1999/xhtml"> - Downloads available at <a href="https://yt-dl.org/downloads/@VERSION@/">https://yt-dl.org/downloads/@VERSION@/</a> - </div> - </content> - <author> - <name>The yt-dlp maintainers</name> - </author> - <updated>@TIMESTAMP@</updated> - </entry> - """) - -now = datetime.datetime.now() -now_iso = now.isoformat() + 'Z' - -atom_template = atom_template.replace('@TIMESTAMP@', now_iso) - -versions_info = json.load(open('update/versions.json')) -versions = list(versions_info['versions'].keys()) -versions.sort() - -entries = [] -for v in versions: - fields = v.split('.') - year, month, day = map(int, fields[:3]) - faked = 0 - patchlevel = 0 - while True: - try: - datetime.date(year, month, day) - except ValueError: - day -= 1 - faked += 1 - assert day > 0 - continue - break - if len(fields) >= 4: - try: - patchlevel = int(fields[3]) - except ValueError: - patchlevel = 1 - timestamp = '%04d-%02d-%02dT00:%02d:%02dZ' % (year, month, day, faked, patchlevel) - - entry = entry_template.replace('@TIMESTAMP@', timestamp) - entry = entry.replace('@VERSION@', v) - entries.append(entry) - -entries_str = textwrap.indent(''.join(entries), '\t') -atom_template = atom_template.replace('@ENTRIES@', entries_str) - -with io.open('update/releases.atom', 'w', encoding='utf-8') as atom_file: - atom_file.write(atom_template) diff --git a/devscripts/gh-pages.unused/update-sites.py b/devscripts/gh-pages.unused/update-sites.py deleted file mode 100644 index b53685fcc..000000000 --- a/devscripts/gh-pages.unused/update-sites.py +++ /dev/null @@ -1,37 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import unicode_literals - -import sys -import os -import textwrap - -# We must be able to import yt_dlp -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))) - -import yt_dlp - - -def main(): - with open('supportedsites.html.in', 'r', encoding='utf-8') as tmplf: - template = tmplf.read() - - ie_htmls = [] - for ie in yt_dlp.list_extractors(age_limit=None): - ie_html = '<b>{}</b>'.format(ie.IE_NAME) - ie_desc = getattr(ie, 'IE_DESC', None) - if ie_desc is False: - continue - elif ie_desc is not None: - ie_html += ': {}'.format(ie.IE_DESC) - if not ie.working(): - ie_html += ' (Currently broken)' - ie_htmls.append('<li>{}</li>'.format(ie_html)) - - template = template.replace('@SITES@', textwrap.indent('\n'.join(ie_htmls), '\t')) - - with open('supportedsites.html', 'w', encoding='utf-8') as sitesf: - sitesf.write(template) - - -if __name__ == '__main__': - main() diff --git a/devscripts/make_contributing.py b/devscripts/make_contributing.py index 6b1b8219c..8b3670d85 100755 --- a/devscripts/make_contributing.py +++ b/devscripts/make_contributing.py @@ -22,7 +22,7 @@ def main(): bug_text = re.search( r'(?s)#\s*BUGS\s*[^\n]*\s*(.*?)#\s*COPYRIGHT', readme).group(1) dev_text = re.search( - r'(?s)(#\s*DEVELOPER INSTRUCTIONS.*?)#\s*EMBEDDING yt-dlp', readme).group(1) + r'(?s)(#\s*DEVELOPER INSTRUCTIONS.*?)#\s*EMBEDDING YT-DLP', readme).group(1) out = bug_text + dev_text diff --git a/devscripts/make_issue_template.py b/devscripts/make_issue_template.py deleted file mode 100644 index 902059231..000000000 --- a/devscripts/make_issue_template.py +++ /dev/null @@ -1,29 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import unicode_literals - -import io -import optparse - - -def main(): - parser = optparse.OptionParser(usage='%prog INFILE OUTFILE') - options, args = parser.parse_args() - if len(args) != 2: - parser.error('Expected an input and an output filename') - - infile, outfile = args - - with io.open(infile, encoding='utf-8') as inf: - issue_template_tmpl = inf.read() - - # Get the version from yt_dlp/version.py without importing the package - exec(compile(open('yt_dlp/version.py').read(), - 'yt_dlp/version.py', 'exec')) - - out = issue_template_tmpl % {'version': locals()['__version__']} - - with io.open(outfile, 'w', encoding='utf-8') as outf: - outf.write(out) - -if __name__ == '__main__': - main() diff --git a/devscripts/make_readme.py b/devscripts/make_readme.py index 3f56af744..47d6d27b7 100755 --- a/devscripts/make_readme.py +++ b/devscripts/make_readme.py @@ -18,12 +18,12 @@ if isinstance(helptext, bytes): with io.open(README_FILE, encoding='utf-8') as f: oldreadme = f.read() -header = oldreadme[:oldreadme.index('## General Options:')] +header = oldreadme[:oldreadme.index('# OPTIONS')] footer = oldreadme[oldreadme.index('# CONFIGURATION'):] -options = helptext[helptext.index(' General Options:'):] +options = helptext[helptext.index(' General Options:') + 19:] options = re.sub(r'(?m)^ (\w.+)$', r'## \1', options) -options = options + '\n' +options = '# OPTIONS\n' + options + '\n' with io.open(README_FILE, 'w', encoding='utf-8') as f: f.write(header) diff --git a/devscripts/prepare_manpage.py b/devscripts/prepare_manpage.py index 6612723f7..10b0aec3e 100644 --- a/devscripts/prepare_manpage.py +++ b/devscripts/prepare_manpage.py @@ -9,7 +9,7 @@ import re ROOT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) README_FILE = os.path.join(ROOT_DIR, 'README.md') -PREFIX = r'''%yt-dlp(1) +PREFIX = r'''%YT-DLP(1) # NAME diff --git a/devscripts/release.sh b/devscripts/release.sh deleted file mode 100755 index 188b166e6..000000000 --- a/devscripts/release.sh +++ /dev/null @@ -1,143 +0,0 @@ -# Unused - -#!/bin/bash - -# IMPORTANT: the following assumptions are made -# * the GH repo is on the origin remote -# * the gh-pages branch is named so locally -# * the git config user.signingkey is properly set - -# You will need -# pip install coverage nose rsa wheel - -# TODO -# release notes -# make hash on local files - -set -e - -skip_tests=true -gpg_sign_commits="" -buildserver='localhost:8142' - -while true -do -case "$1" in - --run-tests) - skip_tests=false - shift - ;; - --gpg-sign-commits|-S) - gpg_sign_commits="-S" - shift - ;; - --buildserver) - buildserver="$2" - shift 2 - ;; - --*) - echo "ERROR: unknown option $1" - exit 1 - ;; - *) - break - ;; -esac -done - -if [ -z "$1" ]; then echo "ERROR: specify version number like this: $0 1994.09.06"; exit 1; fi -version="$1" -major_version=$(echo "$version" | sed -n 's#^\([0-9]*\.[0-9]*\.[0-9]*\).*#\1#p') -if test "$major_version" '!=' "$(date '+%Y.%m.%d')"; then - echo "$version does not start with today's date!" - exit 1 -fi - -if [ ! -z "`git tag | grep "$version"`" ]; then echo 'ERROR: version already present'; exit 1; fi -if [ ! -z "`git status --porcelain | grep -v CHANGELOG`" ]; then echo 'ERROR: the working directory is not clean; commit or stash changes'; exit 1; fi -useless_files=$(find yt_dlp -type f -not -name '*.py') -if [ ! -z "$useless_files" ]; then echo "ERROR: Non-.py files in yt_dlp: $useless_files"; exit 1; fi -if [ ! -f "updates_key.pem" ]; then echo 'ERROR: updates_key.pem missing'; exit 1; fi -if ! type pandoc >/dev/null 2>/dev/null; then echo 'ERROR: pandoc is missing'; exit 1; fi -if ! python3 -c 'import rsa' 2>/dev/null; then echo 'ERROR: python3-rsa is missing'; exit 1; fi -if ! python3 -c 'import wheel' 2>/dev/null; then echo 'ERROR: wheel is missing'; exit 1; fi - -read -p "Is Changelog up to date? (y/n) " -n 1 -if [[ ! $REPLY =~ ^[Yy]$ ]]; then exit 1; fi - -/bin/echo -e "\n### First of all, testing..." -make clean -if $skip_tests ; then - echo 'SKIPPING TESTS' -else - nosetests --verbose --with-coverage --cover-package=yt_dlp --cover-html test --stop || exit 1 -fi - -/bin/echo -e "\n### Changing version in version.py..." -sed -i "s/__version__ = '.*'/__version__ = '$version'/" yt_dlp/version.py - -/bin/echo -e "\n### Changing version in Changelog..." -sed -i "s/<unreleased>/$version/" Changelog.md - -/bin/echo -e "\n### Committing documentation, templates and yt_dlp/version.py..." -make README.md CONTRIBUTING.md issuetemplates supportedsites -git add README.md CONTRIBUTING.md .github/ISSUE_TEMPLATE/1_broken_site.md .github/ISSUE_TEMPLATE/2_site_support_request.md .github/ISSUE_TEMPLATE/3_site_feature_request.md .github/ISSUE_TEMPLATE/4_bug_report.md .github/ISSUE_TEMPLATE/5_feature_request.md .github/ISSUE_TEMPLATE/6_question.md docs/supportedsites.md yt_dlp/version.py Changelog.md -git commit $gpg_sign_commits -m "release $version" - -/bin/echo -e "\n### Now tagging, signing and pushing..." -git tag -s -m "Release $version" "$version" -git show "$version" -read -p "Is it good, can I push? (y/n) " -n 1 -if [[ ! $REPLY =~ ^[Yy]$ ]]; then exit 1; fi -echo -MASTER=$(git rev-parse --abbrev-ref HEAD) -git push origin $MASTER:master -git push origin "$version" - -/bin/echo -e "\n### OK, now it is time to build the binaries..." -REV=$(git rev-parse HEAD) -make yt-dlp yt-dlp.tar.gz -read -p "VM running? (y/n) " -n 1 -wget "http://$buildserver/build/ytdl-org/youtube-dl/yt-dlp.exe?rev=$REV" -O yt-dlp.exe -mkdir -p "build/$version" -mv yt-dlp yt-dlp.exe "build/$version" -mv yt-dlp.tar.gz "build/$version/yt-dlp-$version.tar.gz" -RELEASE_FILES="yt-dlp yt-dlp.exe yt-dlp-$version.tar.gz" -(cd build/$version/ && md5sum $RELEASE_FILES > MD5SUMS) -(cd build/$version/ && sha1sum $RELEASE_FILES > SHA1SUMS) -(cd build/$version/ && sha256sum $RELEASE_FILES > SHA2-256SUMS) -(cd build/$version/ && sha512sum $RELEASE_FILES > SHA2-512SUMS) - -/bin/echo -e "\n### Signing and uploading the new binaries to GitHub..." -for f in $RELEASE_FILES; do gpg --passphrase-repeat 5 --detach-sig "build/$version/$f"; done - -ROOT=$(pwd) -python devscripts/create-github-release.py Changelog.md $version "$ROOT/build/$version" - -ssh ytdl@yt-dl.org "sh html/update_latest.sh $version" - -/bin/echo -e "\n### Now switching to gh-pages..." -git clone --branch gh-pages --single-branch . build/gh-pages -( - set -e - ORIGIN_URL=$(git config --get remote.origin.url) - cd build/gh-pages - "$ROOT/devscripts/gh-pages/add-version.py" $version - "$ROOT/devscripts/gh-pages/update-feed.py" - "$ROOT/devscripts/gh-pages/sign-versions.py" < "$ROOT/updates_key.pem" - "$ROOT/devscripts/gh-pages/generate-download.py" - "$ROOT/devscripts/gh-pages/update-copyright.py" - "$ROOT/devscripts/gh-pages/update-sites.py" - git add *.html *.html.in update - git commit $gpg_sign_commits -m "release $version" - git push "$ROOT" gh-pages - git push "$ORIGIN_URL" gh-pages -) -rm -rf build - -make pypi-files -echo "Uploading to PyPi ..." -python setup.py sdist bdist_wheel upload -make clean - -/bin/echo -e "\n### DONE!" diff --git a/devscripts/show-downloads-statistics.py b/devscripts/show-downloads-statistics.py deleted file mode 100644 index 4855aa7c8..000000000 --- a/devscripts/show-downloads-statistics.py +++ /dev/null @@ -1,49 +0,0 @@ -# Unused - -#!/usr/bin/env python3 -from __future__ import unicode_literals - -import itertools -import json -import os -import re -import sys - -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) - -from yt_dlp.compat import ( - compat_print, - compat_urllib_request, -) -from yt_dlp.utils import format_bytes - - -def format_size(bytes): - return '%s (%d bytes)' % (format_bytes(bytes), bytes) - - -total_bytes = 0 - -for page in itertools.count(1): - releases = json.loads(compat_urllib_request.urlopen( - 'https://api.github.com/repos/ytdl-org/youtube-dl/releases?page=%s' % page - ).read().decode('utf-8')) - - if not releases: - break - - for release in releases: - compat_print(release['name']) - for asset in release['assets']: - asset_name = asset['name'] - total_bytes += asset['download_count'] * asset['size'] - if all(not re.match(p, asset_name) for p in ( - r'^yt-dlp$', - r'^yt-dlp-\d{4}\.\d{2}\.\d{2}(?:\.\d+)?\.tar\.gz$', - r'^yt-dlp\.exe$')): - continue - compat_print( - ' %s size: %s downloads: %d' - % (asset_name, format_size(asset['size']), asset['download_count'])) - -compat_print('total downloads traffic: %s' % format_size(total_bytes)) diff --git a/devscripts/update-formulae.py b/devscripts/update-formulae.py deleted file mode 100644 index 41bc1ac7a..000000000 --- a/devscripts/update-formulae.py +++ /dev/null @@ -1,37 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import unicode_literals - -import json -import os -import re -import sys - -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) - -from yt_dlp.compat import compat_urllib_request - - -# usage: python3 ./devscripts/update-formulae.py <path-to-formulae-rb> <version> -# version can be either 0-aligned (yt-dlp version) or normalized (PyPl version) - -filename, version = sys.argv[1:] - -normalized_version = '.'.join(str(int(x)) for x in version.split('.')) - -pypi_release = json.loads(compat_urllib_request.urlopen( - 'https://pypi.org/pypi/yt-dlp/%s/json' % normalized_version -).read().decode('utf-8')) - -tarball_file = next(x for x in pypi_release['urls'] if x['filename'].endswith('.tar.gz')) - -sha256sum = tarball_file['digests']['sha256'] -url = tarball_file['url'] - -with open(filename, 'r') as r: - formulae_text = r.read() - -formulae_text = re.sub(r'sha256 "[0-9a-f]*?"', 'sha256 "%s"' % sha256sum, formulae_text) -formulae_text = re.sub(r'url "[^"]*?"', 'url "%s"' % url, formulae_text) - -with open(filename, 'w') as w: - w.write(formulae_text) diff --git a/devscripts/update-version.py b/devscripts/update-version.py deleted file mode 100644 index 0ee7bf291..000000000 --- a/devscripts/update-version.py +++ /dev/null @@ -1,42 +0,0 @@ -#!/usr/bin/env python3 -from datetime import datetime -import sys -import subprocess - - -with open('yt_dlp/version.py', 'rt') as f: - exec(compile(f.read(), 'yt_dlp/version.py', 'exec')) -old_version = locals()['__version__'] - -old_version_list = old_version.split('.') - -old_ver = '.'.join(old_version_list[:3]) -old_rev = old_version_list[3] if len(old_version_list) > 3 else '' - -ver = datetime.utcnow().strftime("%Y.%m.%d") - -rev = (sys.argv[1:] or [''])[0] # Use first argument, if present as revision number -if not rev: - rev = str(int(old_rev or 0) + 1) if old_ver == ver else '' - -VERSION = '.'.join((ver, rev)) if rev else ver - -try: - sp = subprocess.Popen(['git', 'rev-parse', '--short', 'HEAD'], stdout=subprocess.PIPE) - GIT_HEAD = sp.communicate()[0].decode().strip() or None -except Exception: - GIT_HEAD = None - -VERSION_FILE = f'''\ -# Autogenerated by devscripts/update-version.py - -__version__ = {VERSION!r} - -RELEASE_GIT_HEAD = {GIT_HEAD!r} -''' - -with open('yt_dlp/version.py', 'wt') as f: - f.write(VERSION_FILE) - -print('::set-output name=ytdlp_version::' + VERSION) -print(f'\nVersion = {VERSION}, Git HEAD = {GIT_HEAD}') diff --git a/devscripts/wine-py2exe.sh b/devscripts/wine-py2exe.sh deleted file mode 100755 index 8bc8ce55b..000000000 --- a/devscripts/wine-py2exe.sh +++ /dev/null @@ -1,58 +0,0 @@ -# UNUSED - -#!/bin/bash - -# Run with as parameter a setup.py that works in the current directory -# e.g. no os.chdir() -# It will run twice, the first time will crash - -set -e - -SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )" - -if [ ! -d wine-py2exe ]; then - - sudo apt-get install wine1.3 axel bsdiff - - mkdir wine-py2exe - cd wine-py2exe - export WINEPREFIX=`pwd` - - axel -a "http://www.python.org/ftp/python/2.7/python-2.7.msi" - axel -a "http://downloads.sourceforge.net/project/py2exe/py2exe/0.6.9/py2exe-0.6.9.win32-py2.7.exe" - #axel -a "http://winetricks.org/winetricks" - - # http://appdb.winehq.org/objectManager.php?sClass=version&iId=21957 - echo "Follow python setup on screen" - wine msiexec /i python-2.7.msi - - echo "Follow py2exe setup on screen" - wine py2exe-0.6.9.win32-py2.7.exe - - #echo "Follow Microsoft Visual C++ 2008 Redistributable Package setup on screen" - #bash winetricks vcrun2008 - - rm py2exe-0.6.9.win32-py2.7.exe - rm python-2.7.msi - #rm winetricks - - # http://bugs.winehq.org/show_bug.cgi?id=3591 - - mv drive_c/Python27/Lib/site-packages/py2exe/run.exe drive_c/Python27/Lib/site-packages/py2exe/run.exe.backup - bspatch drive_c/Python27/Lib/site-packages/py2exe/run.exe.backup drive_c/Python27/Lib/site-packages/py2exe/run.exe "$SCRIPT_DIR/SizeOfImage.patch" - mv drive_c/Python27/Lib/site-packages/py2exe/run_w.exe drive_c/Python27/Lib/site-packages/py2exe/run_w.exe.backup - bspatch drive_c/Python27/Lib/site-packages/py2exe/run_w.exe.backup drive_c/Python27/Lib/site-packages/py2exe/run_w.exe "$SCRIPT_DIR/SizeOfImage_w.patch" - - cd - - -else - - export WINEPREFIX="$( cd wine-py2exe && pwd )" - -fi - -wine "C:\\Python27\\python.exe" "$1" py2exe > "py2exe.log" 2>&1 || true -echo '# Copying python27.dll' >> "py2exe.log" -cp "$WINEPREFIX/drive_c/windows/system32/python27.dll" build/bdist.win32/winexe/bundle-2.7/ -wine "C:\\Python27\\python.exe" "$1" py2exe >> "py2exe.log" 2>&1 - @@ -88,7 +88,7 @@ def pycryptodome_module(): try: import Crypto # noqa: F401 print('WARNING: Using Crypto since Cryptodome is not available. ' - 'Install with: pip install pycryptodomex', file=sys.stderr) + 'Install with: pip install pycryptodome', file=sys.stderr) return 'Crypto' except ImportError: pass diff --git a/requirements.txt b/requirements.txt index cecd08eae..6a982fa36 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,3 +1,3 @@ mutagen -pycryptodomex +pycryptodome websockets @@ -23,7 +23,7 @@ LONG_DESCRIPTION = '\n\n'.join(( '**PS**: Some links in this document will not work since this is a copy of the README.md from Github', open('README.md', 'r', encoding='utf-8').read())) -REQUIREMENTS = ['mutagen', 'pycryptodomex', 'websockets'] +REQUIREMENTS = ['mutagen', 'pycryptodome', 'websockets'] if sys.argv[1:2] == ['py2exe']: @@ -107,20 +107,15 @@ else: setup( name='yt-dlp', version=__version__, - maintainer='pukkandan', - maintainer_email='pukkandan.ytdlp@gmail.com', + maintainer='Jesús E..', + maintainer_email='heckyel@hyperbola.info', + license='CC0-1.0', description=DESCRIPTION, long_description=LONG_DESCRIPTION, long_description_content_type='text/markdown', - url='https://github.com/yt-dlp/yt-dlp', + url='https://git.conocimientoslibres.ga/software/hypervideo.git', packages=packages, install_requires=REQUIREMENTS, - project_urls={ - 'Documentation': 'https://yt-dlp.readthedocs.io', - 'Source': 'https://github.com/yt-dlp/yt-dlp', - 'Tracker': 'https://github.com/yt-dlp/yt-dlp/issues', - 'Funding': 'https://github.com/yt-dlp/yt-dlp/blob/master/Collaborators.md#collaborators', - }, classifiers=[ 'Topic :: Multimedia :: Video', 'Development Status :: 5 - Production/Stable', diff --git a/test/parameters.json b/test/parameters.json index bc4561374..06fe3e31b 100644 --- a/test/parameters.json +++ b/test/parameters.json @@ -44,5 +44,6 @@ "writesubtitles": false, "allsubtitles": false, "listsubtitles": false, + "socket_timeout": 20, "fixup": "never" } diff --git a/test/swftests.unused/.gitignore b/test/swftests.unused/.gitignore deleted file mode 100644 index da97ff7ca..000000000 --- a/test/swftests.unused/.gitignore +++ /dev/null @@ -1 +0,0 @@ -*.swf diff --git a/test/swftests.unused/ArrayAccess.as b/test/swftests.unused/ArrayAccess.as deleted file mode 100644 index e22caa386..000000000 --- a/test/swftests.unused/ArrayAccess.as +++ /dev/null @@ -1,19 +0,0 @@ -// input: [["a", "b", "c", "d"]] -// output: ["c", "b", "a", "d"] - -package { -public class ArrayAccess { - public static function main(ar:Array):Array { - var aa:ArrayAccess = new ArrayAccess(); - return aa.f(ar, 2); - } - - private function f(ar:Array, num:Number):Array{ - var x:String = ar[0]; - var y:String = ar[num % ar.length]; - ar[0] = y; - ar[num] = x; - return ar; - } -} -} diff --git a/test/swftests.unused/ClassCall.as b/test/swftests.unused/ClassCall.as deleted file mode 100644 index aef58daf3..000000000 --- a/test/swftests.unused/ClassCall.as +++ /dev/null @@ -1,17 +0,0 @@ -// input: [] -// output: 121 - -package { -public class ClassCall { - public static function main():int{ - var f:OtherClass = new OtherClass(); - return f.func(100,20); - } -} -} - -class OtherClass { - public function func(x: int, y: int):int { - return x+y+1; - } -} diff --git a/test/swftests.unused/ClassConstruction.as b/test/swftests.unused/ClassConstruction.as deleted file mode 100644 index 436479f8f..000000000 --- a/test/swftests.unused/ClassConstruction.as +++ /dev/null @@ -1,15 +0,0 @@ -// input: [] -// output: 0 - -package { -public class ClassConstruction { - public static function main():int{ - var f:Foo = new Foo(); - return 0; - } -} -} - -class Foo { - -} diff --git a/test/swftests.unused/ConstArrayAccess.as b/test/swftests.unused/ConstArrayAccess.as deleted file mode 100644 index 07dc3f460..000000000 --- a/test/swftests.unused/ConstArrayAccess.as +++ /dev/null @@ -1,18 +0,0 @@ -// input: [] -// output: 4 - -package { -public class ConstArrayAccess { - private static const x:int = 2; - private static const ar:Array = ["42", "3411"]; - - public static function main():int{ - var c:ConstArrayAccess = new ConstArrayAccess(); - return c.f(); - } - - public function f(): int { - return ar[1].length; - } -} -} diff --git a/test/swftests.unused/ConstantInt.as b/test/swftests.unused/ConstantInt.as deleted file mode 100644 index e0bbb6166..000000000 --- a/test/swftests.unused/ConstantInt.as +++ /dev/null @@ -1,12 +0,0 @@ -// input: [] -// output: 2 - -package { -public class ConstantInt { - private static const x:int = 2; - - public static function main():int{ - return x; - } -} -} diff --git a/test/swftests.unused/DictCall.as b/test/swftests.unused/DictCall.as deleted file mode 100644 index c2d174cc2..000000000 --- a/test/swftests.unused/DictCall.as +++ /dev/null @@ -1,10 +0,0 @@ -// input: [{"x": 1, "y": 2}] -// output: 3 - -package { -public class DictCall { - public static function main(d:Object):int{ - return d.x + d.y; - } -} -} diff --git a/test/swftests.unused/EqualsOperator.as b/test/swftests.unused/EqualsOperator.as deleted file mode 100644 index 837a69a46..000000000 --- a/test/swftests.unused/EqualsOperator.as +++ /dev/null @@ -1,10 +0,0 @@ -// input: [] -// output: false - -package { -public class EqualsOperator { - public static function main():Boolean{ - return 1 == 2; - } -} -} diff --git a/test/swftests.unused/LocalVars.as b/test/swftests.unused/LocalVars.as deleted file mode 100644 index b2911a9f3..000000000 --- a/test/swftests.unused/LocalVars.as +++ /dev/null @@ -1,13 +0,0 @@ -// input: [1, 2] -// output: 3 - -package { -public class LocalVars { - public static function main(a:int, b:int):int{ - var c:int = a + b + b; - var d:int = c - b; - var e:int = d; - return e; - } -} -} diff --git a/test/swftests.unused/MemberAssignment.as b/test/swftests.unused/MemberAssignment.as deleted file mode 100644 index dcba5e3ff..000000000 --- a/test/swftests.unused/MemberAssignment.as +++ /dev/null @@ -1,22 +0,0 @@ -// input: [1] -// output: 2 - -package { -public class MemberAssignment { - public var v:int; - - public function g():int { - return this.v; - } - - public function f(a:int):int{ - this.v = a; - return this.v + this.g(); - } - - public static function main(a:int): int { - var v:MemberAssignment = new MemberAssignment(); - return v.f(a); - } -} -} diff --git a/test/swftests.unused/NeOperator.as b/test/swftests.unused/NeOperator.as deleted file mode 100644 index 61dcbc4e9..000000000 --- a/test/swftests.unused/NeOperator.as +++ /dev/null @@ -1,24 +0,0 @@ -// input: [] -// output: 123 - -package { -public class NeOperator { - public static function main(): int { - var res:int = 0; - if (1 != 2) { - res += 3; - } else { - res += 4; - } - if (2 != 2) { - res += 10; - } else { - res += 20; - } - if (9 == 9) { - res += 100; - } - return res; - } -} -} diff --git a/test/swftests.unused/PrivateCall.as b/test/swftests.unused/PrivateCall.as deleted file mode 100644 index f1c110a37..000000000 --- a/test/swftests.unused/PrivateCall.as +++ /dev/null @@ -1,21 +0,0 @@ -// input: [] -// output: 9 - -package { -public class PrivateCall { - public static function main():int{ - var f:OtherClass = new OtherClass(); - return f.func(); - } -} -} - -class OtherClass { - private function pf():int { - return 9; - } - - public function func():int { - return this.pf(); - } -} diff --git a/test/swftests.unused/PrivateVoidCall.as b/test/swftests.unused/PrivateVoidCall.as deleted file mode 100644 index 2cc016797..000000000 --- a/test/swftests.unused/PrivateVoidCall.as +++ /dev/null @@ -1,22 +0,0 @@ -// input: [] -// output: 9 - -package { -public class PrivateVoidCall { - public static function main():int{ - var f:OtherClass = new OtherClass(); - f.func(); - return 9; - } -} -} - -class OtherClass { - private function pf():void { - ; - } - - public function func():void { - this.pf(); - } -} diff --git a/test/swftests.unused/StaticAssignment.as b/test/swftests.unused/StaticAssignment.as deleted file mode 100644 index b061c219d..000000000 --- a/test/swftests.unused/StaticAssignment.as +++ /dev/null @@ -1,13 +0,0 @@ -// input: [1] -// output: 1 - -package { -public class StaticAssignment { - public static var v:int; - - public static function main(a:int):int{ - v = a; - return v; - } -} -} diff --git a/test/swftests.unused/StaticRetrieval.as b/test/swftests.unused/StaticRetrieval.as deleted file mode 100644 index c8352d819..000000000 --- a/test/swftests.unused/StaticRetrieval.as +++ /dev/null @@ -1,16 +0,0 @@ -// input: [] -// output: 1 - -package { -public class StaticRetrieval { - public static var v:int; - - public static function main():int{ - if (v) { - return 0; - } else { - return 1; - } - } -} -} diff --git a/test/swftests.unused/StringBasics.as b/test/swftests.unused/StringBasics.as deleted file mode 100644 index d27430b13..000000000 --- a/test/swftests.unused/StringBasics.as +++ /dev/null @@ -1,11 +0,0 @@ -// input: [] -// output: 3 - -package { -public class StringBasics { - public static function main():int{ - var s:String = "abc"; - return s.length; - } -} -} diff --git a/test/swftests.unused/StringCharCodeAt.as b/test/swftests.unused/StringCharCodeAt.as deleted file mode 100644 index c20d74d65..000000000 --- a/test/swftests.unused/StringCharCodeAt.as +++ /dev/null @@ -1,11 +0,0 @@ -// input: [] -// output: 9897 - -package { -public class StringCharCodeAt { - public static function main():int{ - var s:String = "abc"; - return s.charCodeAt(1) * 100 + s.charCodeAt(); - } -} -} diff --git a/test/swftests.unused/StringConversion.as b/test/swftests.unused/StringConversion.as deleted file mode 100644 index c976f5042..000000000 --- a/test/swftests.unused/StringConversion.as +++ /dev/null @@ -1,11 +0,0 @@ -// input: [] -// output: 2 - -package { -public class StringConversion { - public static function main():int{ - var s:String = String(99); - return s.length; - } -} -} diff --git a/test/test_iqiyi_sdk_interpreter.py b/test/test_iqiyi_sdk_interpreter.py deleted file mode 100644 index ee039f898..000000000 --- a/test/test_iqiyi_sdk_interpreter.py +++ /dev/null @@ -1,49 +0,0 @@ -#!/usr/bin/env python3 - -from __future__ import unicode_literals - -# Allow direct execution -import os -import sys -import unittest -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) - -from test.helper import FakeYDL, is_download_test -from yt_dlp.extractor import IqiyiIE - - -class IqiyiIEWithCredentials(IqiyiIE): - def _get_login_info(self): - return 'foo', 'bar' - - -class WarningLogger(object): - def __init__(self): - self.messages = [] - - def warning(self, msg): - self.messages.append(msg) - - def debug(self, msg): - pass - - def error(self, msg): - pass - - -@is_download_test -class TestIqiyiSDKInterpreter(unittest.TestCase): - def test_iqiyi_sdk_interpreter(self): - ''' - Test the functionality of IqiyiSDKInterpreter by trying to log in - - If `sign` is incorrect, /validate call throws an HTTP 556 error - ''' - logger = WarningLogger() - ie = IqiyiIEWithCredentials(FakeYDL({'logger': logger})) - ie._login() - self.assertTrue('unable to log in:' in logger.messages[0]) - - -if __name__ == '__main__': - unittest.main() diff --git a/test/test_swfinterp.py.disabled b/test/test_swfinterp.py.disabled deleted file mode 100644 index 5d5b21e6d..000000000 --- a/test/test_swfinterp.py.disabled +++ /dev/null @@ -1,80 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import unicode_literals - -# Allow direct execution -import os -import sys -import unittest -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) - - -import errno -import io -import json -import re -import subprocess - -from yt_dlp.swfinterp import SWFInterpreter - - -TEST_DIR = os.path.join( - os.path.dirname(os.path.abspath(__file__)), 'swftests') - - -class TestSWFInterpreter(unittest.TestCase): - pass - - -def _make_testfunc(testfile): - m = re.match(r'^(.*)\.(as)$', testfile) - if not m: - return - test_id = m.group(1) - - def test_func(self): - as_file = os.path.join(TEST_DIR, testfile) - swf_file = os.path.join(TEST_DIR, test_id + '.swf') - if ((not os.path.exists(swf_file)) - or os.path.getmtime(swf_file) < os.path.getmtime(as_file)): - # Recompile - try: - subprocess.check_call([ - 'mxmlc', '-output', swf_file, - '-static-link-runtime-shared-libraries', as_file]) - except OSError as ose: - if ose.errno == errno.ENOENT: - print('mxmlc not found! Skipping test.') - return - raise - - with open(swf_file, 'rb') as swf_f: - swf_content = swf_f.read() - swfi = SWFInterpreter(swf_content) - - with io.open(as_file, 'r', encoding='utf-8') as as_f: - as_content = as_f.read() - - def _find_spec(key): - m = re.search( - r'(?m)^//\s*%s:\s*(.*?)\n' % re.escape(key), as_content) - if not m: - raise ValueError('Cannot find %s in %s' % (key, testfile)) - return json.loads(m.group(1)) - - input_args = _find_spec('input') - output = _find_spec('output') - - swf_class = swfi.extract_class(test_id) - func = swfi.extract_function(swf_class, 'main') - res = func(input_args) - self.assertEqual(res, output) - - test_func.__name__ = str('test_swf_' + test_id) - setattr(TestSWFInterpreter, test_func.__name__, test_func) - - -for testfile in os.listdir(TEST_DIR): - _make_testfunc(testfile) - -if __name__ == '__main__': - unittest.main() diff --git a/test/test_unicode_literals.py.disabled b/test/test_unicode_literals.py.disabled deleted file mode 100644 index 6c1b7ec91..000000000 --- a/test/test_unicode_literals.py.disabled +++ /dev/null @@ -1,63 +0,0 @@ -from __future__ import unicode_literals - -# Allow direct execution -import os -import sys -import unittest -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) - -import io -import re - -rootDir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) - -IGNORED_FILES = [ - 'setup.py', # http://bugs.python.org/issue13943 - 'conf.py', - 'buildserver.py', -] - -IGNORED_DIRS = [ - '.git', - '.tox', -] - -from test.helper import assertRegexpMatches - - -class TestUnicodeLiterals(unittest.TestCase): - def test_all_files(self): - for dirpath, dirnames, filenames in os.walk(rootDir): - for ignore_dir in IGNORED_DIRS: - if ignore_dir in dirnames: - # If we remove the directory from dirnames os.walk won't - # recurse into it - dirnames.remove(ignore_dir) - for basename in filenames: - if not basename.endswith('.py'): - continue - if basename in IGNORED_FILES: - continue - - fn = os.path.join(dirpath, basename) - with io.open(fn, encoding='utf-8') as inf: - code = inf.read() - - if "'" not in code and '"' not in code: - continue - assertRegexpMatches( - self, - code, - r'(?:(?:#.*?|\s*)\n)*from __future__ import (?:[a-z_]+,\s*)*unicode_literals', - 'unicode_literals import missing in %s' % fn) - - m = re.search(r'(?<=\s)u[\'"](?!\)|,|$)', code) - if m is not None: - self.assertTrue( - m is None, - 'u present in %s, around %s' % ( - fn, code[m.start() - 10:m.end() + 10])) - - -if __name__ == '__main__': - unittest.main() diff --git a/test/test_update.py.disabled b/test/test_update.py.disabled deleted file mode 100644 index 1e8edf0f6..000000000 --- a/test/test_update.py.disabled +++ /dev/null @@ -1,30 +0,0 @@ -#!/usr/bin/env python3 - -from __future__ import unicode_literals - -# Allow direct execution -import os -import sys -import unittest -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) - - -import json -from yt_dlp.update import rsa_verify - - -class TestUpdate(unittest.TestCase): - def test_rsa_verify(self): - UPDATES_RSA_KEY = (0x9d60ee4d8f805312fdb15a62f87b95bd66177b91df176765d13514a0f1754bcd2057295c5b6f1d35daa6742c3ffc9a82d3e118861c207995a8031e151d863c9927e304576bc80692bc8e094896fcf11b66f3e29e04e3a71e9a11558558acea1840aec37fc396fb6b65dc81a1c4144e03bd1c011de62e3f1357b327d08426fe93, 65537) - with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'versions.json'), 'rb') as f: - versions_info = f.read().decode() - versions_info = json.loads(versions_info) - signature = versions_info['signature'] - del versions_info['signature'] - self.assertTrue(rsa_verify( - json.dumps(versions_info, sort_keys=True).encode('utf-8'), - signature, UPDATES_RSA_KEY)) - - -if __name__ == '__main__': - unittest.main() diff --git a/test/test_write_annotations.py.disabled b/test/test_write_annotations.py.disabled deleted file mode 100644 index 7e4d8bc5a..000000000 --- a/test/test_write_annotations.py.disabled +++ /dev/null @@ -1,81 +0,0 @@ -#!/usr/bin/env python3 -# coding: utf-8 -from __future__ import unicode_literals - -# Allow direct execution -import os -import sys -import unittest -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) - -from test.helper import get_params, try_rm, is_download_test - - -import io - -import xml.etree.ElementTree - -import yt_dlp.YoutubeDL -import yt_dlp.extractor - - -class YoutubeDL(yt_dlp.YoutubeDL): - def __init__(self, *args, **kwargs): - super(YoutubeDL, self).__init__(*args, **kwargs) - self.to_stderr = self.to_screen - - -params = get_params({ - 'writeannotations': True, - 'skip_download': True, - 'writeinfojson': False, - 'format': 'flv', -}) - - -TEST_ID = 'gr51aVj-mLg' -ANNOTATIONS_FILE = TEST_ID + '.annotations.xml' -EXPECTED_ANNOTATIONS = ['Speech bubble', 'Note', 'Title', 'Spotlight', 'Label'] - - -@is_download_test -class TestAnnotations(unittest.TestCase): - def setUp(self): - # Clear old files - self.tearDown() - - def test_info_json(self): - expected = list(EXPECTED_ANNOTATIONS) # Two annotations could have the same text. - ie = yt_dlp.extractor.YoutubeIE() - ydl = YoutubeDL(params) - ydl.add_info_extractor(ie) - ydl.download([TEST_ID]) - self.assertTrue(os.path.exists(ANNOTATIONS_FILE)) - annoxml = None - with io.open(ANNOTATIONS_FILE, 'r', encoding='utf-8') as annof: - annoxml = xml.etree.ElementTree.parse(annof) - self.assertTrue(annoxml is not None, 'Failed to parse annotations XML') - root = annoxml.getroot() - self.assertEqual(root.tag, 'document') - annotationsTag = root.find('annotations') - self.assertEqual(annotationsTag.tag, 'annotations') - annotations = annotationsTag.findall('annotation') - - # Not all the annotations have TEXT children and the annotations are returned unsorted. - for a in annotations: - self.assertEqual(a.tag, 'annotation') - if a.get('type') == 'text': - textTag = a.find('TEXT') - text = textTag.text - self.assertTrue(text in expected) # assertIn only added in python 2.7 - # remove the first occurrence, there could be more than one annotation with the same text - expected.remove(text) - # We should have seen (and removed) all the expected annotation texts. - self.assertEqual(len(expected), 0, 'Not all expected annotations were found.') - - def tearDown(self): - try_rm(ANNOTATIONS_FILE) - - -if __name__ == '__main__': - unittest.main() diff --git a/test/versions.json b/test/versions.json deleted file mode 100644 index 6cccc2259..000000000 --- a/test/versions.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "latest": "2013.01.06", - "signature": "72158cdba391628569ffdbea259afbcf279bbe3d8aeb7492690735dc1cfa6afa754f55c61196f3871d429599ab22f2667f1fec98865527b32632e7f4b3675a7ef0f0fbe084d359256ae4bba68f0d33854e531a70754712f244be71d4b92e664302aa99653ee4df19800d955b6c4149cd2b3f24288d6e4b40b16126e01f4c8ce6", - "versions": { - "2013.01.02": { - "bin": [ - "http://youtube-dl.org/downloads/2013.01.02/youtube-dl", - "f5b502f8aaa77675c4884938b1e4871ebca2611813a0c0e74f60c0fbd6dcca6b" - ], - "exe": [ - "http://youtube-dl.org/downloads/2013.01.02/youtube-dl.exe", - "75fa89d2ce297d102ff27675aa9d92545bbc91013f52ec52868c069f4f9f0422" - ], - "tar": [ - "http://youtube-dl.org/downloads/2013.01.02/youtube-dl-2013.01.02.tar.gz", - "6a66d022ac8e1c13da284036288a133ec8dba003b7bd3a5179d0c0daca8c8196" - ] - }, - "2013.01.06": { - "bin": [ - "http://youtube-dl.org/downloads/2013.01.06/youtube-dl", - "64b6ed8865735c6302e836d4d832577321b4519aa02640dc508580c1ee824049" - ], - "exe": [ - "http://youtube-dl.org/downloads/2013.01.06/youtube-dl.exe", - "58609baf91e4389d36e3ba586e21dab882daaaee537e4448b1265392ae86ff84" - ], - "tar": [ - "http://youtube-dl.org/downloads/2013.01.06/youtube-dl-2013.01.06.tar.gz", - "fe77ab20a95d980ed17a659aa67e371fdd4d656d19c4c7950e7b720b0c2f1a86" - ] - } - } -}
\ No newline at end of file @@ -10,7 +10,6 @@ deps = passenv = HOME defaultargs = test --exclude test_download.py --exclude test_age_restriction.py --exclude test_subtitles.py --exclude test_write_annotations.py - --exclude test_youtube_lists.py --exclude test_iqiyi_sdk_interpreter.py - --exclude test_socks.py + --exclude test_youtube_lists.py --exclude test_socks.py commands = nosetests --verbose {posargs:{[testenv]defaultargs}} # --with-coverage --cover-package=yt_dlp --cover-html # test.test_download:TestDownload.test_NowVideo diff --git a/yt-dlp.sh b/yt-dlp.sh index 71a9aa163..71a9aa163 100755..100644 --- a/yt-dlp.sh +++ b/yt-dlp.sh diff --git a/yt_dlp/YoutubeDL.py b/yt_dlp/YoutubeDL.py index 49143cb16..99c280ffc 100644 --- a/yt_dlp/YoutubeDL.py +++ b/yt_dlp/YoutubeDL.py @@ -158,8 +158,7 @@ from .postprocessor import ( MoveFilesAfterDownloadPP, _PLUGIN_CLASSES as plugin_postprocessors ) -from .update import detect_variant -from .version import __version__, RELEASE_GIT_HEAD +from .version import __version__ if compat_os_name == 'nt': import ctypes diff --git a/yt_dlp/__init__.py b/yt_dlp/__init__.py index 926b5cad3..524130807 100644 --- a/yt_dlp/__init__.py +++ b/yt_dlp/__init__.py @@ -1,9 +1,7 @@ -#!/usr/bin/env python3 +#!/usr/bin/python # coding: utf-8 -f'You are using an unsupported version of Python. Only Python versions 3.6 and above are supported by yt-dlp' # noqa: F541 - -__license__ = 'Public Domain' +__license__ = 'CC0-1.0' import codecs import io @@ -44,7 +42,6 @@ from .utils import ( traverse_obj, write_string, ) -from .update import run_update from .downloader import ( FileDownloader, ) @@ -819,19 +816,8 @@ def _real_main(argv=None): if opts.rm_cachedir: ydl.cache.remove() - # Update version - if opts.update_self: - # If updater returns True, exit. Required for windows - if run_update(ydl): - if actual_use: - sys.exit('ERROR: The program must exit for the update to complete') - sys.exit() - # Maybe do nothing if not actual_use: - if opts.update_self or opts.rm_cachedir: - sys.exit() - ydl.warn_if_short_id(sys.argv[1:] if argv is None else argv) parser.error( 'You must provide at least one URL.\n' diff --git a/yt_dlp/downloader/hls.py b/yt_dlp/downloader/hls.py index e932fd6ae..f3f32b514 100644 --- a/yt_dlp/downloader/hls.py +++ b/yt_dlp/downloader/hls.py @@ -72,9 +72,9 @@ class HlsFD(FragmentFD): can_download, message = self.can_download(s, info_dict, self.params.get('allow_unplayable_formats')), None if can_download and not compat_pycrypto_AES and '#EXT-X-KEY:METHOD=AES-128' in s: if FFmpegFD.available(): - can_download, message = False, 'The stream has AES-128 encryption and pycryptodomex is not available' + can_download, message = False, 'The stream has AES-128 encryption and pycryptodome is not available' else: - message = ('The stream has AES-128 encryption and neither ffmpeg nor pycryptodomex are available; ' + message = ('The stream has AES-128 encryption and neither ffmpeg nor pycryptodome are available; ' 'Decryption will be performed natively, but will be extremely slow') if not can_download: has_drm = re.search('|'.join([ diff --git a/yt_dlp/extractor/iqiyi.py b/yt_dlp/extractor/iqiyi.py index 74e20a54a..b13b9f4cf 100644 --- a/yt_dlp/extractor/iqiyi.py +++ b/yt_dlp/extractor/iqiyi.py @@ -9,30 +9,14 @@ import time from .common import InfoExtractor from ..compat import ( compat_str, - compat_urllib_parse_urlencode, - compat_urllib_parse_unquote ) -from .openload import PhantomJSwrapper from ..utils import ( clean_html, - decode_packed_codes, - ExtractorError, - float_or_none, - format_field, get_element_by_id, get_element_by_attribute, - int_or_none, - js_to_json, + ExtractorError, ohdave_rsa_encrypt, - parse_age_limit, - parse_duration, - parse_iso8601, - parse_resolution, - qualities, remove_start, - str_or_none, - traverse_obj, - urljoin, ) @@ -40,135 +24,6 @@ def md5_text(text): return hashlib.md5(text.encode('utf-8')).hexdigest() -class IqiyiSDK(object): - def __init__(self, target, ip, timestamp): - self.target = target - self.ip = ip - self.timestamp = timestamp - - @staticmethod - def split_sum(data): - return compat_str(sum(map(lambda p: int(p, 16), list(data)))) - - @staticmethod - def digit_sum(num): - if isinstance(num, int): - num = compat_str(num) - return compat_str(sum(map(int, num))) - - def even_odd(self): - even = self.digit_sum(compat_str(self.timestamp)[::2]) - odd = self.digit_sum(compat_str(self.timestamp)[1::2]) - return even, odd - - def preprocess(self, chunksize): - self.target = md5_text(self.target) - chunks = [] - for i in range(32 // chunksize): - chunks.append(self.target[chunksize * i:chunksize * (i + 1)]) - if 32 % chunksize: - chunks.append(self.target[32 - 32 % chunksize:]) - return chunks, list(map(int, self.ip.split('.'))) - - def mod(self, modulus): - chunks, ip = self.preprocess(32) - self.target = chunks[0] + ''.join(map(lambda p: compat_str(p % modulus), ip)) - - def split(self, chunksize): - modulus_map = { - 4: 256, - 5: 10, - 8: 100, - } - - chunks, ip = self.preprocess(chunksize) - ret = '' - for i in range(len(chunks)): - ip_part = compat_str(ip[i] % modulus_map[chunksize]) if i < 4 else '' - if chunksize == 8: - ret += ip_part + chunks[i] - else: - ret += chunks[i] + ip_part - self.target = ret - - def handle_input16(self): - self.target = md5_text(self.target) - self.target = self.split_sum(self.target[:16]) + self.target + self.split_sum(self.target[16:]) - - def handle_input8(self): - self.target = md5_text(self.target) - ret = '' - for i in range(4): - part = self.target[8 * i:8 * (i + 1)] - ret += self.split_sum(part) + part - self.target = ret - - def handleSum(self): - self.target = md5_text(self.target) - self.target = self.split_sum(self.target) + self.target - - def date(self, scheme): - self.target = md5_text(self.target) - d = time.localtime(self.timestamp) - strings = { - 'y': compat_str(d.tm_year), - 'm': '%02d' % d.tm_mon, - 'd': '%02d' % d.tm_mday, - } - self.target += ''.join(map(lambda c: strings[c], list(scheme))) - - def split_time_even_odd(self): - even, odd = self.even_odd() - self.target = odd + md5_text(self.target) + even - - def split_time_odd_even(self): - even, odd = self.even_odd() - self.target = even + md5_text(self.target) + odd - - def split_ip_time_sum(self): - chunks, ip = self.preprocess(32) - self.target = compat_str(sum(ip)) + chunks[0] + self.digit_sum(self.timestamp) - - def split_time_ip_sum(self): - chunks, ip = self.preprocess(32) - self.target = self.digit_sum(self.timestamp) + chunks[0] + compat_str(sum(ip)) - - -class IqiyiSDKInterpreter(object): - def __init__(self, sdk_code): - self.sdk_code = sdk_code - - def run(self, target, ip, timestamp): - self.sdk_code = decode_packed_codes(self.sdk_code) - - functions = re.findall(r'input=([a-zA-Z0-9]+)\(input', self.sdk_code) - - sdk = IqiyiSDK(target, ip, timestamp) - - other_functions = { - 'handleSum': sdk.handleSum, - 'handleInput8': sdk.handle_input8, - 'handleInput16': sdk.handle_input16, - 'splitTimeEvenOdd': sdk.split_time_even_odd, - 'splitTimeOddEven': sdk.split_time_odd_even, - 'splitIpTimeSum': sdk.split_ip_time_sum, - 'splitTimeIpSum': sdk.split_time_ip_sum, - } - for function in functions: - if re.match(r'mod\d+', function): - sdk.mod(int(function[3:])) - elif re.match(r'date[ymd]{3}', function): - sdk.date(function[4:]) - elif re.match(r'split\d+', function): - sdk.split(int(function[5:])) - elif function in other_functions: - other_functions[function]() - else: - raise ExtractorError('Unknown function %s' % function) - - return sdk.target - - class IqiyiIE(InfoExtractor): IE_NAME = 'iqiyi' IE_DESC = '爱奇艺' @@ -253,51 +108,7 @@ class IqiyiIE(InfoExtractor): return ohdave_rsa_encrypt(data, e, N) def _login(self): - username, password = self._get_login_info() - - # No authentication to be performed - if not username: - return True - - data = self._download_json( - 'http://kylin.iqiyi.com/get_token', None, - note='Get token for logging', errnote='Unable to get token for logging') - sdk = data['sdk'] - timestamp = int(time.time()) - target = '/apis/reglogin/login.action?lang=zh_TW&area_code=null&email=%s&passwd=%s&agenttype=1&from=undefined&keeplogin=0&piccode=&fromurl=&_pos=1' % ( - username, self._rsa_fun(password.encode('utf-8'))) - - interp = IqiyiSDKInterpreter(sdk) - sign = interp.run(target, data['ip'], timestamp) - - validation_params = { - 'target': target, - 'server': 'BEA3AA1908656AABCCFF76582C4C6660', - 'token': data['token'], - 'bird_src': 'f8d91d57af224da7893dd397d52d811a', - 'sign': sign, - 'bird_t': timestamp, - } - validation_result = self._download_json( - 'http://kylin.iqiyi.com/validate?' + compat_urllib_parse_urlencode(validation_params), None, - note='Validate credentials', errnote='Unable to validate credentials') - - MSG_MAP = { - 'P00107': 'please login via the web interface and enter the CAPTCHA code', - 'P00117': 'bad username or password', - } - - code = validation_result['code'] - if code != 'A00000': - msg = MSG_MAP.get(code) - if not msg: - msg = 'error %s' % code - if validation_result.get('msg'): - msg += ': ' + validation_result['msg'] - self.report_warning('unable to log in: ' + msg) - return False - - return True + raise ExtractorError("iQiyi's non-free authentication algorithm has made login impossible", expected=True) def get_raw_data(self, tvid, video_id): tm = int(time.time() * 1000) @@ -406,359 +217,3 @@ class IqiyiIE(InfoExtractor): 'title': title, 'formats': formats, } - - -class IqIE(InfoExtractor): - IE_NAME = 'iq.com' - IE_DESC = 'International version of iQiyi' - _VALID_URL = r'https?://(?:www\.)?iq\.com/play/(?:[\w%-]*-)?(?P<id>\w+)' - _TESTS = [{ - 'url': 'https://www.iq.com/play/one-piece-episode-1000-1ma1i6ferf4', - 'md5': '2d7caf6eeca8a32b407094b33b757d39', - 'info_dict': { - 'ext': 'mp4', - 'id': '1ma1i6ferf4', - 'title': '航海王 第1000集', - 'description': 'Subtitle available on Sunday 4PM（GMT+8）.', - 'duration': 1430, - 'timestamp': 1637488203, - 'upload_date': '20211121', - 'episode_number': 1000, - 'episode': 'Episode 1000', - 'series': 'One Piece', - 'age_limit': 13, - 'average_rating': float, - }, - 'params': { - 'format': '500', - }, - 'expected_warnings': ['format is restricted'] - }, { - # VIP-restricted video - 'url': 'https://www.iq.com/play/mermaid-in-the-fog-2021-gbdpx13bs4', - 'only_matching': True - }] - _BID_TAGS = { - '100': '240P', - '200': '360P', - '300': '480P', - '500': '720P', - '600': '1080P', - '610': '1080P50', - '700': '2K', - '800': '4K', - } - _LID_TAGS = { - '1': 'zh_CN', - '2': 'zh_TW', - '3': 'en', - '18': 'th', - '21': 'my', - '23': 'vi', - '24': 'id', - '26': 'es', - '28': 'ar', - } - - _DASH_JS = ''' - console.log(page.evaluate(function() { - var tvid = "%(tvid)s"; var vid = "%(vid)s"; var src = "%(src)s"; - var uid = "%(uid)s"; var dfp = "%(dfp)s"; var mode = "%(mode)s"; var lang = "%(lang)s"; - var bid_list = %(bid_list)s; var ut_list = %(ut_list)s; var tm = new Date().getTime(); - var cmd5x_func = %(cmd5x_func)s; var cmd5x_exporter = {}; cmd5x_func({}, cmd5x_exporter, {}); var cmd5x = cmd5x_exporter.cmd5x; - var authKey = cmd5x(cmd5x('') + tm + '' + tvid); - var k_uid = Array.apply(null, Array(32)).map(function() {return Math.floor(Math.random() * 15).toString(16)}).join(''); - var dash_paths = {}; - bid_list.forEach(function(bid) { - var query = { - 'tvid': tvid, - 'bid': bid, - 'ds': 1, - 'vid': vid, - 'src': src, - 'vt': 0, - 'rs': 1, - 'uid': uid, - 'ori': 'pcw', - 'ps': 1, - 'k_uid': k_uid, - 'pt': 0, - 'd': 0, - 's': '', - 'lid': '', - 'slid': 0, - 'cf': '', - 'ct': '', - 'authKey': authKey, - 'k_tag': 1, - 'ost': 0, - 'ppt': 0, - 'dfp': dfp, - 'prio': JSON.stringify({ - 'ff': 'f4v', - 'code': 2 - }), - 'k_err_retries': 0, - 'up': '', - 'su': 2, - 'applang': lang, - 'sver': 2, - 'X-USER-MODE': mode, - 'qd_v': 2, - 'tm': tm, - 'qdy': 'a', - 'qds': 0, - 'k_ft1': 141287244169348, - 'k_ft4': 34359746564, - 'k_ft5': 1, - 'bop': JSON.stringify({ - 'version': '10.0', - 'dfp': dfp - }), - }; - var enc_params = []; - for (var prop in query) { - enc_params.push(encodeURIComponent(prop) + '=' + encodeURIComponent(query[prop])); - } - ut_list.forEach(function(ut) { - enc_params.push('ut=' + ut); - }) - var dash_path = '/dash?' + enc_params.join('&'); dash_path += '&vf=' + cmd5x(dash_path); - dash_paths[bid] = dash_path; - }); - return JSON.stringify(dash_paths); - })); - saveAndExit(); - ''' - - def _extract_vms_player_js(self, webpage, video_id): - player_js_cache = self._downloader.cache.load('iq', 'player_js') - if player_js_cache: - return player_js_cache - webpack_js_url = self._proto_relative_url(self._search_regex( - r'<script src="((?:https?)?//stc.iqiyipic.com/_next/static/chunks/webpack-\w+\.js)"', webpage, 'webpack URL')) - webpack_js = self._download_webpage(webpack_js_url, video_id, note='Downloading webpack JS', errnote='Unable to download webpack JS') - webpack_map1, webpack_map2 = [self._parse_json(js_map, video_id, transform_source=js_to_json) for js_map in self._search_regex( - r'\(({[^}]*})\[\w+\][^\)]*\)\s*\+\s*["\']\.["\']\s*\+\s*({[^}]*})\[\w+\]\+["\']\.js', webpack_js, 'JS locations', group=(1, 2))] - for module_index in reversed(list(webpack_map2.keys())): - module_js = self._download_webpage( - f'https://stc.iqiyipic.com/_next/static/chunks/{webpack_map1.get(module_index, module_index)}.{webpack_map2[module_index]}.js', - video_id, note=f'Downloading #{module_index} module JS', errnote='Unable to download module JS', fatal=False) or '' - if 'vms request' in module_js: - self._downloader.cache.store('iq', 'player_js', module_js) - return module_js - raise ExtractorError('Unable to extract player JS') - - def _extract_cmd5x_function(self, webpage, video_id): - return self._search_regex(r',\s*(function\s*\([^\)]*\)\s*{\s*var _qda.+_qdc\(\)\s*})\s*,', - self._extract_vms_player_js(webpage, video_id), 'signature function') - - def _update_bid_tags(self, webpage, video_id): - extracted_bid_tags = self._parse_json( - self._search_regex( - r'arguments\[1\][^,]*,\s*function\s*\([^\)]*\)\s*{\s*"use strict";?\s*var \w=({.+}})\s*,\s*\w\s*=\s*{\s*getNewVd', - self._extract_vms_player_js(webpage, video_id), 'video tags', default=''), - video_id, transform_source=js_to_json, fatal=False) - if not extracted_bid_tags: - return - self._BID_TAGS = { - bid: traverse_obj(extracted_bid_tags, (bid, 'value'), expected_type=str, default=self._BID_TAGS.get(bid)) - for bid in extracted_bid_tags.keys() - } - - def _get_cookie(self, name, default=None): - cookie = self._get_cookies('https://iq.com/').get(name) - return cookie.value if cookie else default - - def _real_extract(self, url): - video_id = self._match_id(url) - webpage = self._download_webpage(url, video_id) - self._update_bid_tags(webpage, video_id) - - next_props = self._search_nextjs_data(webpage, video_id)['props'] - page_data = next_props['initialState']['play'] - video_info = page_data['curVideoInfo'] - - uid = traverse_obj( - self._parse_json( - self._get_cookie('I00002', '{}'), video_id, transform_source=compat_urllib_parse_unquote, fatal=False), - ('data', 'uid'), default=0) - - if uid: - vip_data = self._download_json( - 'https://pcw-api.iq.com/api/vtype', video_id, note='Downloading VIP data', errnote='Unable to download VIP data', query={ - 'batch': 1, - 'platformId': 3, - 'modeCode': self._get_cookie('mod', 'intl'), - 'langCode': self._get_cookie('lang', 'en_us'), - 'deviceId': self._get_cookie('QC005', '') - }, fatal=False) - ut_list = traverse_obj(vip_data, ('data', 'all_vip', ..., 'vipType'), expected_type=str_or_none, default=[]) - else: - ut_list = ['0'] - - # bid 0 as an initial format checker - dash_paths = self._parse_json(PhantomJSwrapper(self).get( - url, html='<!DOCTYPE html>', video_id=video_id, note2='Executing signature code', jscode=self._DASH_JS % { - 'tvid': video_info['tvId'], - 'vid': video_info['vid'], - 'src': traverse_obj(next_props, ('initialProps', 'pageProps', 'ptid'), - expected_type=str, default='04022001010011000000'), - 'uid': uid, - 'dfp': self._get_cookie('dfp', ''), - 'mode': self._get_cookie('mod', 'intl'), - 'lang': self._get_cookie('lang', 'en_us'), - 'bid_list': '[' + ','.join(['0', *self._BID_TAGS.keys()]) + ']', - 'ut_list': '[' + ','.join(ut_list) + ']', - 'cmd5x_func': self._extract_cmd5x_function(webpage, video_id), - })[1].strip(), video_id) - - formats, subtitles = [], {} - initial_format_data = self._download_json( - urljoin('https://cache-video.iq.com', dash_paths['0']), video_id, - note='Downloading initial video format info', errnote='Unable to download initial video format info')['data'] - - preview_time = traverse_obj( - initial_format_data, ('boss_ts', (None, 'data'), ('previewTime', 'rtime')), expected_type=float_or_none, get_all=False) - if traverse_obj(initial_format_data, ('boss_ts', 'data', 'prv'), expected_type=int_or_none): - self.report_warning('This preview video is limited%s' % format_field(preview_time, template='to %s seconds')) - - # TODO: Extract audio-only formats - for bid in set(traverse_obj(initial_format_data, ('program', 'video', ..., 'bid'), expected_type=str_or_none, default=[])): - dash_path = dash_paths.get(bid) - if not dash_path: - self.report_warning(f'Unknown format id: {bid}. It is currently not being extracted') - continue - format_data = traverse_obj(self._download_json( - urljoin('https://cache-video.iq.com', dash_path), video_id, - note=f'Downloading format data for {self._BID_TAGS[bid]}', errnote='Unable to download format data', - fatal=False), 'data', expected_type=dict) - - video_format = next((video_format for video_format in traverse_obj( - format_data, ('program', 'video', ...), expected_type=dict, default=[]) if str(video_format['bid']) == bid), {}) - extracted_formats = [] - if video_format.get('m3u8Url'): - extracted_formats.extend(self._extract_m3u8_formats( - urljoin(format_data.get('dm3u8', 'https://cache-m.iq.com/dc/dt/'), video_format['m3u8Url']), - 'mp4', m3u8_id=bid, fatal=False)) - if video_format.get('mpdUrl'): - # TODO: Properly extract mpd hostname - extracted_formats.extend(self._extract_mpd_formats( - urljoin(format_data.get('dm3u8', 'https://cache-m.iq.com/dc/dt/'), video_format['mpdUrl']), - mpd_id=bid, fatal=False)) - if video_format.get('m3u8'): - ff = video_format.get('ff', 'ts') - if ff == 'ts': - m3u8_formats, _ = self._parse_m3u8_formats_and_subtitles( - video_format['m3u8'], ext='mp4', m3u8_id=bid, fatal=False) - extracted_formats.extend(m3u8_formats) - elif ff == 'm4s': - mpd_data = traverse_obj( - self._parse_json(video_format['m3u8'], video_id, fatal=False), ('payload', ..., 'data'), expected_type=str) - if not mpd_data: - continue - mpd_formats, _ = self._parse_mpd_formats_and_subtitles( - mpd_data, bid, format_data.get('dm3u8', 'https://cache-m.iq.com/dc/dt/')) - extracted_formats.extend(mpd_formats) - else: - self.report_warning(f'{ff} formats are currently not supported') - - if not extracted_formats: - if video_format.get('s'): - self.report_warning(f'{self._BID_TAGS[bid]} format is restricted') - else: - self.report_warning(f'Unable to extract {self._BID_TAGS[bid]} format') - for f in extracted_formats: - f.update({ - 'quality': qualities(list(self._BID_TAGS.keys()))(bid), - 'format_note': self._BID_TAGS[bid], - **parse_resolution(video_format.get('scrsz')) - }) - formats.extend(extracted_formats) - - self._sort_formats(formats) - - for sub_format in traverse_obj(initial_format_data, ('program', 'stl', ...), expected_type=dict, default=[]): - lang = self._LID_TAGS.get(str_or_none(sub_format.get('lid')), sub_format.get('_name')) - subtitles.setdefault(lang, []).extend([{ - 'ext': format_ext, - 'url': urljoin(initial_format_data.get('dstl', 'http://meta.video.iqiyi.com'), sub_format[format_key]) - } for format_key, format_ext in [('srt', 'srt'), ('webvtt', 'vtt')] if sub_format.get(format_key)]) - - extra_metadata = page_data.get('albumInfo') if video_info.get('albumId') and page_data.get('albumInfo') else video_info - return { - 'id': video_id, - 'title': video_info['name'], - 'formats': formats, - 'subtitles': subtitles, - 'description': video_info.get('mergeDesc'), - 'duration': parse_duration(video_info.get('len')), - 'age_limit': parse_age_limit(video_info.get('rating')), - 'average_rating': traverse_obj(page_data, ('playScoreInfo', 'score'), expected_type=float_or_none), - 'timestamp': parse_iso8601(video_info.get('isoUploadDate')), - 'categories': traverse_obj(extra_metadata, ('videoTagMap', ..., ..., 'name'), expected_type=str), - 'cast': traverse_obj(extra_metadata, ('actorArr', ..., 'name'), expected_type=str), - 'episode_number': int_or_none(video_info.get('order')) or None, - 'series': video_info.get('albumName'), - } - - -class IqAlbumIE(InfoExtractor): - IE_NAME = 'iq.com:album' - _VALID_URL = r'https?://(?:www\.)?iq\.com/album/(?:[\w%-]*-)?(?P<id>\w+)' - _TESTS = [{ - 'url': 'https://www.iq.com/album/one-piece-1999-1bk9icvr331', - 'info_dict': { - 'id': '1bk9icvr331', - 'title': 'One Piece', - 'description': 'Subtitle available on Sunday 4PM（GMT+8）.' - }, - 'playlist_mincount': 238 - }, { - # Movie/single video - 'url': 'https://www.iq.com/album/九龙城寨-2021-22yjnij099k', - 'info_dict': { - 'ext': 'mp4', - 'id': '22yjnij099k', - 'title': '九龙城寨', - 'description': 'md5:8a09f50b8ba0db4dc69bc7c844228044', - 'duration': 5000, - 'timestamp': 1641911371, - 'upload_date': '20220111', - 'series': '九龙城寨', - 'cast': ['Shi Yan Neng', 'Yu Lang', 'Peter lv', 'Sun Zi Jun', 'Yang Xiao Bo'], - 'age_limit': 13, - 'average_rating': float, - }, - 'expected_warnings': ['format is restricted'] - }] - - def _entries(self, album_id_num, page_ranges, album_id=None, mode_code='intl', lang_code='en_us'): - for page_range in page_ranges: - page = self._download_json( - f'https://pcw-api.iq.com/api/episodeListSource/{album_id_num}', album_id, - note=f'Downloading video list episodes {page_range.get("msg", "")}', - errnote='Unable to download video list', query={ - 'platformId': 3, - 'modeCode': mode_code, - 'langCode': lang_code, - 'endOrder': page_range['to'], - 'startOrder': page_range['from'] - }) - for video in page['data']['epg']: - yield self.url_result('https://www.iq.com/play/%s' % (video.get('playLocSuffix') or video['qipuIdStr']), - IqIE.ie_key(), video.get('qipuIdStr'), video.get('name')) - - def _real_extract(self, url): - album_id = self._match_id(url) - webpage = self._download_webpage(url, album_id) - next_data = self._search_nextjs_data(webpage, album_id) - album_data = next_data['props']['initialState']['album']['videoAlbumInfo'] - - if album_data.get('videoType') == 'singleVideo': - return self.url_result('https://www.iq.com/play/%s' % album_id, IqIE.ie_key()) - return self.playlist_result( - self._entries(album_data['albumId'], album_data['totalPageRange'], album_id, - traverse_obj(next_data, ('props', 'initialProps', 'pageProps', 'modeCode')), - traverse_obj(next_data, ('props', 'initialProps', 'pageProps', 'langCode'))), - album_id, album_data.get('name'), album_data.get('desc')) diff --git a/yt_dlp/extractor/ivi.py b/yt_dlp/extractor/ivi.py index 5f8a046e0..098ab6665 100644 --- a/yt_dlp/extractor/ivi.py +++ b/yt_dlp/extractor/ivi.py @@ -141,7 +141,7 @@ class IviIE(InfoExtractor): elif site == 353: continue elif not pycryptodome_found: - raise ExtractorError('pycryptodomex not found. Please install', expected=True) + raise ExtractorError('pycryptodome not found. Please install', expected=True) elif message: extractor_msg += ': ' + message raise ExtractorError(extractor_msg % video_id, expected=True) diff --git a/yt_dlp/options.py b/yt_dlp/options.py index 17d8d5da6..9f1f31974 100644 --- a/yt_dlp/options.py +++ b/yt_dlp/options.py @@ -240,10 +240,6 @@ def create_parser(): action='version', help='Print program version and exit') general.add_option( - '-U', '--update', - action='store_true', dest='update_self', - help='Update this program to latest version. Make sure that you have sufficient permissions (run with sudo if needed)') - general.add_option( '-i', '--ignore-errors', action='store_true', dest='ignoreerrors', help='Ignore download and postprocessing errors. The download will be considered successful even if the postprocessing fails') diff --git a/yt_dlp/swfinterp.py.disabled b/yt_dlp/swfinterp.py.disabled deleted file mode 100644 index 0c7158575..000000000 --- a/yt_dlp/swfinterp.py.disabled +++ /dev/null @@ -1,834 +0,0 @@ -from __future__ import unicode_literals - -import collections -import io -import zlib - -from .compat import ( - compat_str, - compat_struct_unpack, -) -from .utils import ( - ExtractorError, -) - - -def _extract_tags(file_contents): - if file_contents[1:3] != b'WS': - raise ExtractorError( - 'Not an SWF file; header is %r' % file_contents[:3]) - if file_contents[:1] == b'C': - content = zlib.decompress(file_contents[8:]) - else: - raise NotImplementedError( - 'Unsupported compression format %r' % - file_contents[:1]) - - # Determine number of bits in framesize rectangle - framesize_nbits = compat_struct_unpack('!B', content[:1])[0] >> 3 - framesize_len = (5 + 4 * framesize_nbits + 7) // 8 - - pos = framesize_len + 2 + 2 - while pos < len(content): - header16 = compat_struct_unpack('<H', content[pos:pos + 2])[0] - pos += 2 - tag_code = header16 >> 6 - tag_len = header16 & 0x3f - if tag_len == 0x3f: - tag_len = compat_struct_unpack('<I', content[pos:pos + 4])[0] - pos += 4 - assert pos + tag_len <= len(content), \ - ('Tag %d ends at %d+%d - that\'s longer than the file (%d)' - % (tag_code, pos, tag_len, len(content))) - yield (tag_code, content[pos:pos + tag_len]) - pos += tag_len - - -class _AVMClass_Object(object): - def __init__(self, avm_class): - self.avm_class = avm_class - - def __repr__(self): - return '%s#%x' % (self.avm_class.name, id(self)) - - -class _ScopeDict(dict): - def __init__(self, avm_class): - super(_ScopeDict, self).__init__() - self.avm_class = avm_class - - def __repr__(self): - return '%s__Scope(%s)' % ( - self.avm_class.name, - super(_ScopeDict, self).__repr__()) - - -class _AVMClass(object): - def __init__(self, name_idx, name, static_properties=None): - self.name_idx = name_idx - self.name = name - self.method_names = {} - self.method_idxs = {} - self.methods = {} - self.method_pyfunctions = {} - self.static_properties = static_properties if static_properties else {} - - self.variables = _ScopeDict(self) - self.constants = {} - - def make_object(self): - return _AVMClass_Object(self) - - def __repr__(self): - return '_AVMClass(%s)' % (self.name) - - def register_methods(self, methods): - self.method_names.update(methods.items()) - self.method_idxs.update(dict( - (idx, name) - for name, idx in methods.items())) - - -class _Multiname(object): - def __init__(self, kind): - self.kind = kind - - def __repr__(self): - return '[MULTINAME kind: 0x%x]' % self.kind - - -def _read_int(reader): - res = 0 - shift = 0 - for _ in range(5): - buf = reader.read(1) - assert len(buf) == 1 - b = compat_struct_unpack('<B', buf)[0] - res = res | ((b & 0x7f) << shift) - if b & 0x80 == 0: - break - shift += 7 - return res - - -def _u30(reader): - res = _read_int(reader) - assert res & 0xf0000000 == 0 - return res - - -_u32 = _read_int - - -def _s32(reader): - v = _read_int(reader) - if v & 0x80000000 != 0: - v = - ((v ^ 0xffffffff) + 1) - return v - - -def _s24(reader): - bs = reader.read(3) - assert len(bs) == 3 - last_byte = b'\xff' if (ord(bs[2:3]) >= 0x80) else b'\x00' - return compat_struct_unpack('<i', bs + last_byte)[0] - - -def _read_string(reader): - slen = _u30(reader) - resb = reader.read(slen) - assert len(resb) == slen - return resb.decode('utf-8') - - -def _read_bytes(count, reader): - assert count >= 0 - resb = reader.read(count) - assert len(resb) == count - return resb - - -def _read_byte(reader): - resb = _read_bytes(1, reader=reader) - res = compat_struct_unpack('<B', resb)[0] - return res - - -StringClass = _AVMClass('(no name idx)', 'String') -ByteArrayClass = _AVMClass('(no name idx)', 'ByteArray') -TimerClass = _AVMClass('(no name idx)', 'Timer') -TimerEventClass = _AVMClass('(no name idx)', 'TimerEvent', {'TIMER': 'timer'}) -_builtin_classes = { - StringClass.name: StringClass, - ByteArrayClass.name: ByteArrayClass, - TimerClass.name: TimerClass, - TimerEventClass.name: TimerEventClass, -} - - -class _Undefined(object): - def __bool__(self): - return False - __nonzero__ = __bool__ - - def __hash__(self): - return 0 - - def __str__(self): - return 'undefined' - __repr__ = __str__ - - -undefined = _Undefined() - - -class SWFInterpreter(object): - def __init__(self, file_contents): - self._patched_functions = { - (TimerClass, 'addEventListener'): lambda params: undefined, - } - code_tag = next(tag - for tag_code, tag in _extract_tags(file_contents) - if tag_code == 82) - p = code_tag.index(b'\0', 4) + 1 - code_reader = io.BytesIO(code_tag[p:]) - - # Parse ABC (AVM2 ByteCode) - - # Define a couple convenience methods - u30 = lambda *args: _u30(*args, reader=code_reader) - s32 = lambda *args: _s32(*args, reader=code_reader) - u32 = lambda *args: _u32(*args, reader=code_reader) - read_bytes = lambda *args: _read_bytes(*args, reader=code_reader) - read_byte = lambda *args: _read_byte(*args, reader=code_reader) - - # minor_version + major_version - read_bytes(2 + 2) - - # Constant pool - int_count = u30() - self.constant_ints = [0] - for _c in range(1, int_count): - self.constant_ints.append(s32()) - self.constant_uints = [0] - uint_count = u30() - for _c in range(1, uint_count): - self.constant_uints.append(u32()) - double_count = u30() - read_bytes(max(0, (double_count - 1)) * 8) - string_count = u30() - self.constant_strings = [''] - for _c in range(1, string_count): - s = _read_string(code_reader) - self.constant_strings.append(s) - namespace_count = u30() - for _c in range(1, namespace_count): - read_bytes(1) # kind - u30() # name - ns_set_count = u30() - for _c in range(1, ns_set_count): - count = u30() - for _c2 in range(count): - u30() - multiname_count = u30() - MULTINAME_SIZES = { - 0x07: 2, # QName - 0x0d: 2, # QNameA - 0x0f: 1, # RTQName - 0x10: 1, # RTQNameA - 0x11: 0, # RTQNameL - 0x12: 0, # RTQNameLA - 0x09: 2, # Multiname - 0x0e: 2, # MultinameA - 0x1b: 1, # MultinameL - 0x1c: 1, # MultinameLA - } - self.multinames = [''] - for _c in range(1, multiname_count): - kind = u30() - assert kind in MULTINAME_SIZES, 'Invalid multiname kind %r' % kind - if kind == 0x07: - u30() # namespace_idx - name_idx = u30() - self.multinames.append(self.constant_strings[name_idx]) - elif kind == 0x09: - name_idx = u30() - u30() - self.multinames.append(self.constant_strings[name_idx]) - else: - self.multinames.append(_Multiname(kind)) - for _c2 in range(MULTINAME_SIZES[kind]): - u30() - - # Methods - method_count = u30() - MethodInfo = collections.namedtuple( - 'MethodInfo', - ['NEED_ARGUMENTS', 'NEED_REST']) - method_infos = [] - for method_id in range(method_count): - param_count = u30() - u30() # return type - for _ in range(param_count): - u30() # param type - u30() # name index (always 0 for youtube) - flags = read_byte() - if flags & 0x08 != 0: - # Options present - option_count = u30() - for c in range(option_count): - u30() # val - read_bytes(1) # kind - if flags & 0x80 != 0: - # Param names present - for _ in range(param_count): - u30() # param name - mi = MethodInfo(flags & 0x01 != 0, flags & 0x04 != 0) - method_infos.append(mi) - - # Metadata - metadata_count = u30() - for _c in range(metadata_count): - u30() # name - item_count = u30() - for _c2 in range(item_count): - u30() # key - u30() # value - - def parse_traits_info(): - trait_name_idx = u30() - kind_full = read_byte() - kind = kind_full & 0x0f - attrs = kind_full >> 4 - methods = {} - constants = None - if kind == 0x00: # Slot - u30() # Slot id - u30() # type_name_idx - vindex = u30() - if vindex != 0: - read_byte() # vkind - elif kind == 0x06: # Const - u30() # Slot id - u30() # type_name_idx - vindex = u30() - vkind = 'any' - if vindex != 0: - vkind = read_byte() - if vkind == 0x03: # Constant_Int - value = self.constant_ints[vindex] - elif vkind == 0x04: # Constant_UInt - value = self.constant_uints[vindex] - else: - return {}, None # Ignore silently for now - constants = {self.multinames[trait_name_idx]: value} - elif kind in (0x01, 0x02, 0x03): # Method / Getter / Setter - u30() # disp_id - method_idx = u30() - methods[self.multinames[trait_name_idx]] = method_idx - elif kind == 0x04: # Class - u30() # slot_id - u30() # classi - elif kind == 0x05: # Function - u30() # slot_id - function_idx = u30() - methods[function_idx] = self.multinames[trait_name_idx] - else: - raise ExtractorError('Unsupported trait kind %d' % kind) - - if attrs & 0x4 != 0: # Metadata present - metadata_count = u30() - for _c3 in range(metadata_count): - u30() # metadata index - - return methods, constants - - # Classes - class_count = u30() - classes = [] - for class_id in range(class_count): - name_idx = u30() - - cname = self.multinames[name_idx] - avm_class = _AVMClass(name_idx, cname) - classes.append(avm_class) - - u30() # super_name idx - flags = read_byte() - if flags & 0x08 != 0: # Protected namespace is present - u30() # protected_ns_idx - intrf_count = u30() - for _c2 in range(intrf_count): - u30() - u30() # iinit - trait_count = u30() - for _c2 in range(trait_count): - trait_methods, trait_constants = parse_traits_info() - avm_class.register_methods(trait_methods) - if trait_constants: - avm_class.constants.update(trait_constants) - - assert len(classes) == class_count - self._classes_by_name = dict((c.name, c) for c in classes) - - for avm_class in classes: - avm_class.cinit_idx = u30() - trait_count = u30() - for _c2 in range(trait_count): - trait_methods, trait_constants = parse_traits_info() - avm_class.register_methods(trait_methods) - if trait_constants: - avm_class.constants.update(trait_constants) - - # Scripts - script_count = u30() - for _c in range(script_count): - u30() # init - trait_count = u30() - for _c2 in range(trait_count): - parse_traits_info() - - # Method bodies - method_body_count = u30() - Method = collections.namedtuple('Method', ['code', 'local_count']) - self._all_methods = [] - for _c in range(method_body_count): - method_idx = u30() - u30() # max_stack - local_count = u30() - u30() # init_scope_depth - u30() # max_scope_depth - code_length = u30() - code = read_bytes(code_length) - m = Method(code, local_count) - self._all_methods.append(m) - for avm_class in classes: - if method_idx in avm_class.method_idxs: - avm_class.methods[avm_class.method_idxs[method_idx]] = m - exception_count = u30() - for _c2 in range(exception_count): - u30() # from - u30() # to - u30() # target - u30() # exc_type - u30() # var_name - trait_count = u30() - for _c2 in range(trait_count): - parse_traits_info() - - assert p + code_reader.tell() == len(code_tag) - - def patch_function(self, avm_class, func_name, f): - self._patched_functions[(avm_class, func_name)] = f - - def extract_class(self, class_name, call_cinit=True): - try: - res = self._classes_by_name[class_name] - except KeyError: - raise ExtractorError('Class %r not found' % class_name) - - if call_cinit and hasattr(res, 'cinit_idx'): - res.register_methods({'$cinit': res.cinit_idx}) - res.methods['$cinit'] = self._all_methods[res.cinit_idx] - cinit = self.extract_function(res, '$cinit') - cinit([]) - - return res - - def extract_function(self, avm_class, func_name): - p = self._patched_functions.get((avm_class, func_name)) - if p: - return p - if func_name in avm_class.method_pyfunctions: - return avm_class.method_pyfunctions[func_name] - if func_name in self._classes_by_name: - return self._classes_by_name[func_name].make_object() - if func_name not in avm_class.methods: - raise ExtractorError('Cannot find function %s.%s' % ( - avm_class.name, func_name)) - m = avm_class.methods[func_name] - - def resfunc(args): - # Helper functions - coder = io.BytesIO(m.code) - s24 = lambda: _s24(coder) - u30 = lambda: _u30(coder) - - registers = [avm_class.variables] + list(args) + [None] * m.local_count - stack = [] - scopes = collections.deque([ - self._classes_by_name, avm_class.constants, avm_class.variables]) - while True: - opcode = _read_byte(coder) - if opcode == 9: # label - pass # Spec says: "Do nothing." - elif opcode == 16: # jump - offset = s24() - coder.seek(coder.tell() + offset) - elif opcode == 17: # iftrue - offset = s24() - value = stack.pop() - if value: - coder.seek(coder.tell() + offset) - elif opcode == 18: # iffalse - offset = s24() - value = stack.pop() - if not value: - coder.seek(coder.tell() + offset) - elif opcode == 19: # ifeq - offset = s24() - value2 = stack.pop() - value1 = stack.pop() - if value2 == value1: - coder.seek(coder.tell() + offset) - elif opcode == 20: # ifne - offset = s24() - value2 = stack.pop() - value1 = stack.pop() - if value2 != value1: - coder.seek(coder.tell() + offset) - elif opcode == 21: # iflt - offset = s24() - value2 = stack.pop() - value1 = stack.pop() - if value1 < value2: - coder.seek(coder.tell() + offset) - elif opcode == 32: # pushnull - stack.append(None) - elif opcode == 33: # pushundefined - stack.append(undefined) - elif opcode == 36: # pushbyte - v = _read_byte(coder) - stack.append(v) - elif opcode == 37: # pushshort - v = u30() - stack.append(v) - elif opcode == 38: # pushtrue - stack.append(True) - elif opcode == 39: # pushfalse - stack.append(False) - elif opcode == 40: # pushnan - stack.append(float('NaN')) - elif opcode == 42: # dup - value = stack[-1] - stack.append(value) - elif opcode == 44: # pushstring - idx = u30() - stack.append(self.constant_strings[idx]) - elif opcode == 48: # pushscope - new_scope = stack.pop() - scopes.append(new_scope) - elif opcode == 66: # construct - arg_count = u30() - args = list(reversed( - [stack.pop() for _ in range(arg_count)])) - obj = stack.pop() - res = obj.avm_class.make_object() - stack.append(res) - elif opcode == 70: # callproperty - index = u30() - mname = self.multinames[index] - arg_count = u30() - args = list(reversed( - [stack.pop() for _ in range(arg_count)])) - obj = stack.pop() - - if obj == StringClass: - if mname == 'String': - assert len(args) == 1 - assert isinstance(args[0], ( - int, compat_str, _Undefined)) - if args[0] == undefined: - res = 'undefined' - else: - res = compat_str(args[0]) - stack.append(res) - continue - else: - raise NotImplementedError( - 'Function String.%s is not yet implemented' - % mname) - elif isinstance(obj, _AVMClass_Object): - func = self.extract_function(obj.avm_class, mname) - res = func(args) - stack.append(res) - continue - elif isinstance(obj, _AVMClass): - func = self.extract_function(obj, mname) - res = func(args) - stack.append(res) - continue - elif isinstance(obj, _ScopeDict): - if mname in obj.avm_class.method_names: - func = self.extract_function(obj.avm_class, mname) - res = func(args) - else: - res = obj[mname] - stack.append(res) - continue - elif isinstance(obj, compat_str): - if mname == 'split': - assert len(args) == 1 - assert isinstance(args[0], compat_str) - if args[0] == '': - res = list(obj) - else: - res = obj.split(args[0]) - stack.append(res) - continue - elif mname == 'charCodeAt': - assert len(args) <= 1 - idx = 0 if len(args) == 0 else args[0] - assert isinstance(idx, int) - res = ord(obj[idx]) - stack.append(res) - continue - elif isinstance(obj, list): - if mname == 'slice': - assert len(args) == 1 - assert isinstance(args[0], int) - res = obj[args[0]:] - stack.append(res) - continue - elif mname == 'join': - assert len(args) == 1 - assert isinstance(args[0], compat_str) - res = args[0].join(obj) - stack.append(res) - continue - raise NotImplementedError( - 'Unsupported property %r on %r' - % (mname, obj)) - elif opcode == 71: # returnvoid - res = undefined - return res - elif opcode == 72: # returnvalue - res = stack.pop() - return res - elif opcode == 73: # constructsuper - # Not yet implemented, just hope it works without it - arg_count = u30() - args = list(reversed( - [stack.pop() for _ in range(arg_count)])) - obj = stack.pop() - elif opcode == 74: # constructproperty - index = u30() - arg_count = u30() - args = list(reversed( - [stack.pop() for _ in range(arg_count)])) - obj = stack.pop() - - mname = self.multinames[index] - assert isinstance(obj, _AVMClass) - - # We do not actually call the constructor for now; - # we just pretend it does nothing - stack.append(obj.make_object()) - elif opcode == 79: # callpropvoid - index = u30() - mname = self.multinames[index] - arg_count = u30() - args = list(reversed( - [stack.pop() for _ in range(arg_count)])) - obj = stack.pop() - if isinstance(obj, _AVMClass_Object): - func = self.extract_function(obj.avm_class, mname) - res = func(args) - assert res is undefined - continue - if isinstance(obj, _ScopeDict): - assert mname in obj.avm_class.method_names - func = self.extract_function(obj.avm_class, mname) - res = func(args) - assert res is undefined - continue - if mname == 'reverse': - assert isinstance(obj, list) - obj.reverse() - else: - raise NotImplementedError( - 'Unsupported (void) property %r on %r' - % (mname, obj)) - elif opcode == 86: # newarray - arg_count = u30() - arr = [] - for i in range(arg_count): - arr.append(stack.pop()) - arr = arr[::-1] - stack.append(arr) - elif opcode == 93: # findpropstrict - index = u30() - mname = self.multinames[index] - for s in reversed(scopes): - if mname in s: - res = s - break - else: - res = scopes[0] - if mname not in res and mname in _builtin_classes: - stack.append(_builtin_classes[mname]) - else: - stack.append(res[mname]) - elif opcode == 94: # findproperty - index = u30() - mname = self.multinames[index] - for s in reversed(scopes): - if mname in s: - res = s - break - else: - res = avm_class.variables - stack.append(res) - elif opcode == 96: # getlex - index = u30() - mname = self.multinames[index] - for s in reversed(scopes): - if mname in s: - scope = s - break - else: - scope = avm_class.variables - - if mname in scope: - res = scope[mname] - elif mname in _builtin_classes: - res = _builtin_classes[mname] - else: - # Assume uninitialized - # TODO warn here - res = undefined - stack.append(res) - elif opcode == 97: # setproperty - index = u30() - value = stack.pop() - idx = self.multinames[index] - if isinstance(idx, _Multiname): - idx = stack.pop() - obj = stack.pop() - obj[idx] = value - elif opcode == 98: # getlocal - index = u30() - stack.append(registers[index]) - elif opcode == 99: # setlocal - index = u30() - value = stack.pop() - registers[index] = value - elif opcode == 102: # getproperty - index = u30() - pname = self.multinames[index] - if pname == 'length': - obj = stack.pop() - assert isinstance(obj, (compat_str, list)) - stack.append(len(obj)) - elif isinstance(pname, compat_str): # Member access - obj = stack.pop() - if isinstance(obj, _AVMClass): - res = obj.static_properties[pname] - stack.append(res) - continue - - assert isinstance(obj, (dict, _ScopeDict)),\ - 'Accessing member %r on %r' % (pname, obj) - res = obj.get(pname, undefined) - stack.append(res) - else: # Assume attribute access - idx = stack.pop() - assert isinstance(idx, int) - obj = stack.pop() - assert isinstance(obj, list) - stack.append(obj[idx]) - elif opcode == 104: # initproperty - index = u30() - value = stack.pop() - idx = self.multinames[index] - if isinstance(idx, _Multiname): - idx = stack.pop() - obj = stack.pop() - obj[idx] = value - elif opcode == 115: # convert_ - value = stack.pop() - intvalue = int(value) - stack.append(intvalue) - elif opcode == 128: # coerce - u30() - elif opcode == 130: # coerce_a - value = stack.pop() - # um, yes, it's any value - stack.append(value) - elif opcode == 133: # coerce_s - assert isinstance(stack[-1], (type(None), compat_str)) - elif opcode == 147: # decrement - value = stack.pop() - assert isinstance(value, int) - stack.append(value - 1) - elif opcode == 149: # typeof - value = stack.pop() - return { - _Undefined: 'undefined', - compat_str: 'String', - int: 'Number', - float: 'Number', - }[type(value)] - elif opcode == 160: # add - value2 = stack.pop() - value1 = stack.pop() - res = value1 + value2 - stack.append(res) - elif opcode == 161: # subtract - value2 = stack.pop() - value1 = stack.pop() - res = value1 - value2 - stack.append(res) - elif opcode == 162: # multiply - value2 = stack.pop() - value1 = stack.pop() - res = value1 * value2 - stack.append(res) - elif opcode == 164: # modulo - value2 = stack.pop() - value1 = stack.pop() - res = value1 % value2 - stack.append(res) - elif opcode == 168: # bitand - value2 = stack.pop() - value1 = stack.pop() - assert isinstance(value1, int) - assert isinstance(value2, int) - res = value1 & value2 - stack.append(res) - elif opcode == 171: # equals - value2 = stack.pop() - value1 = stack.pop() - result = value1 == value2 - stack.append(result) - elif opcode == 175: # greaterequals - value2 = stack.pop() - value1 = stack.pop() - result = value1 >= value2 - stack.append(result) - elif opcode == 192: # increment_i - value = stack.pop() - assert isinstance(value, int) - stack.append(value + 1) - elif opcode == 208: # getlocal_0 - stack.append(registers[0]) - elif opcode == 209: # getlocal_1 - stack.append(registers[1]) - elif opcode == 210: # getlocal_2 - stack.append(registers[2]) - elif opcode == 211: # getlocal_3 - stack.append(registers[3]) - elif opcode == 212: # setlocal_0 - registers[0] = stack.pop() - elif opcode == 213: # setlocal_1 - registers[1] = stack.pop() - elif opcode == 214: # setlocal_2 - registers[2] = stack.pop() - elif opcode == 215: # setlocal_3 - registers[3] = stack.pop() - else: - raise NotImplementedError( - 'Unsupported opcode %d' % opcode) - - avm_class.method_pyfunctions[func_name] = resfunc - return resfunc diff --git a/yt_dlp/update.py b/yt_dlp/update.py deleted file mode 100644 index a208e163c..000000000 --- a/yt_dlp/update.py +++ /dev/null @@ -1,288 +0,0 @@ -from __future__ import unicode_literals - -import hashlib -import json -import os -import platform -import subprocess -import sys -import traceback -from zipimport import zipimporter - -from .compat import compat_realpath -from .utils import encode_compat_str, Popen, write_string - -from .version import __version__ - - -''' # Not signed -def rsa_verify(message, signature, key): - from hashlib import sha256 - assert isinstance(message, bytes) - byte_size = (len(bin(key[0])) - 2 + 8 - 1) // 8 - signature = ('%x' % pow(int(signature, 16), key[1], key[0])).encode() - signature = (byte_size * 2 - len(signature)) * b'0' + signature - asn1 = b'3031300d060960864801650304020105000420' - asn1 += sha256(message).hexdigest().encode() - if byte_size < len(asn1) // 2 + 11: - return False - expected = b'0001' + (byte_size - len(asn1) // 2 - 3) * b'ff' + b'00' + asn1 - return expected == signature -''' - - -def detect_variant(): - if hasattr(sys, 'frozen'): - prefix = 'mac' if sys.platform == 'darwin' else 'win' - if getattr(sys, '_MEIPASS', None): - if sys._MEIPASS == os.path.dirname(sys.executable): - return f'{prefix}_dir' - return f'{prefix}_exe' - return 'py2exe' - elif isinstance(globals().get('__loader__'), zipimporter): - return 'zip' - elif os.path.basename(sys.argv[0]) == '__main__.py': - return 'source' - return 'unknown' - - -_NON_UPDATEABLE_REASONS = { - 'win_exe': None, - 'zip': None, - 'mac_exe': None, - 'py2exe': None, - 'win_dir': 'Auto-update is not supported for unpackaged windows executable; Re-download the latest release', - 'mac_dir': 'Auto-update is not supported for unpackaged MacOS executable; Re-download the latest release', - 'source': 'You cannot update when running from source code; Use git to pull the latest changes', - 'unknown': 'It looks like you installed yt-dlp with a package manager, pip or setup.py; Use that to update', -} - - -def is_non_updateable(): - return _NON_UPDATEABLE_REASONS.get(detect_variant(), _NON_UPDATEABLE_REASONS['unknown']) - - -def run_update(ydl): - """ - Update the program file with the latest version from the repository - Returns whether the program should terminate - """ - - JSON_URL = 'https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest' - - def report_error(msg, expected=False): - ydl.report_error(msg, tb='' if expected else None) - - def report_unable(action, expected=False): - report_error(f'Unable to {action}', expected) - - def report_permission_error(file): - report_unable(f'write to {file}; Try running as administrator', True) - - def report_network_error(action, delim=';'): - report_unable(f'{action}{delim} Visit https://github.com/yt-dlp/yt-dlp/releases/latest', True) - - def calc_sha256sum(path): - h = hashlib.sha256() - b = bytearray(128 * 1024) - mv = memoryview(b) - with open(os.path.realpath(path), 'rb', buffering=0) as f: - for n in iter(lambda: f.readinto(mv), 0): - h.update(mv[:n]) - return h.hexdigest() - - # Download and check versions info - try: - version_info = ydl._opener.open(JSON_URL).read().decode('utf-8') - version_info = json.loads(version_info) - except Exception: - return report_network_error('obtain version info', delim='; Please try again later or') - - def version_tuple(version_str): - return tuple(map(int, version_str.split('.'))) - - version_id = version_info['tag_name'] - ydl.to_screen(f'Latest version: {version_id}, Current version: {__version__}') - if version_tuple(__version__) >= version_tuple(version_id): - ydl.to_screen(f'yt-dlp is up to date ({__version__})') - return - - err = is_non_updateable() - if err: - return report_error(err, True) - - # sys.executable is set to the full pathname of the exe-file for py2exe - # though symlinks are not followed so that we need to do this manually - # with help of realpath - filename = compat_realpath(sys.executable if hasattr(sys, 'frozen') else sys.argv[0]) - ydl.to_screen(f'Current Build Hash {calc_sha256sum(filename)}') - ydl.to_screen(f'Updating to version {version_id} ...') - - version_labels = { - 'zip_3': '', - 'win_exe_64': '.exe', - 'py2exe_64': '_min.exe', - 'win_exe_32': '_x86.exe', - 'mac_exe_64': '_macos', - } - - def get_bin_info(bin_or_exe, version): - label = version_labels['%s_%s' % (bin_or_exe, version)] - return next((i for i in version_info['assets'] if i['name'] == 'yt-dlp%s' % label), {}) - - def get_sha256sum(bin_or_exe, version): - filename = 'yt-dlp%s' % version_labels['%s_%s' % (bin_or_exe, version)] - urlh = next( - (i for i in version_info['assets'] if i['name'] in ('SHA2-256SUMS')), - {}).get('browser_download_url') - if not urlh: - return None - hash_data = ydl._opener.open(urlh).read().decode('utf-8') - return dict(ln.split()[::-1] for ln in hash_data.splitlines()).get(filename) - - if not os.access(filename, os.W_OK): - return report_permission_error(filename) - - # PyInstaller - variant = detect_variant() - if variant in ('win_exe', 'py2exe'): - directory = os.path.dirname(filename) - if not os.access(directory, os.W_OK): - return report_permission_error(directory) - try: - if os.path.exists(filename + '.old'): - os.remove(filename + '.old') - except (IOError, OSError): - return report_unable('remove the old version') - - try: - arch = platform.architecture()[0][:2] - url = get_bin_info(variant, arch).get('browser_download_url') - if not url: - return report_network_error('fetch updates') - urlh = ydl._opener.open(url) - newcontent = urlh.read() - urlh.close() - except (IOError, OSError): - return report_network_error('download latest version') - - try: - with open(filename + '.new', 'wb') as outf: - outf.write(newcontent) - except (IOError, OSError): - return report_permission_error(f'{filename}.new') - - expected_sum = get_sha256sum(variant, arch) - if not expected_sum: - ydl.report_warning('no hash information found for the release') - elif calc_sha256sum(filename + '.new') != expected_sum: - report_network_error('verify the new executable') - try: - os.remove(filename + '.new') - except OSError: - return report_unable('remove corrupt download') - - try: - os.rename(filename, filename + '.old') - except (IOError, OSError): - return report_unable('move current version') - try: - os.rename(filename + '.new', filename) - except (IOError, OSError): - report_unable('overwrite current version') - os.rename(filename + '.old', filename) - return - try: - # Continues to run in the background - Popen( - 'ping 127.0.0.1 -n 5 -w 1000 & del /F "%s.old"' % filename, - shell=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) - ydl.to_screen('Updated yt-dlp to version %s' % version_id) - return True # Exit app - except OSError: - report_unable('delete the old version') - - elif variant in ('zip', 'mac_exe'): - pack_type = '3' if variant == 'zip' else '64' - try: - url = get_bin_info(variant, pack_type).get('browser_download_url') - if not url: - return report_network_error('fetch updates') - urlh = ydl._opener.open(url) - newcontent = urlh.read() - urlh.close() - except (IOError, OSError): - return report_network_error('download the latest version') - - expected_sum = get_sha256sum(variant, pack_type) - if not expected_sum: - ydl.report_warning('no hash information found for the release') - elif hashlib.sha256(newcontent).hexdigest() != expected_sum: - return report_network_error('verify the new package') - - try: - with open(filename, 'wb') as outf: - outf.write(newcontent) - except (IOError, OSError): - return report_unable('overwrite current version') - - ydl.to_screen('Updated yt-dlp to version %s; Restart yt-dlp to use the new version' % version_id) - return - - assert False, f'Unhandled variant: {variant}' - - -''' # UNUSED -def get_notes(versions, fromVersion): - notes = [] - for v, vdata in sorted(versions.items()): - if v > fromVersion: - notes.extend(vdata.get('notes', [])) - return notes - - -def print_notes(to_screen, versions, fromVersion=__version__): - notes = get_notes(versions, fromVersion) - if notes: - to_screen('PLEASE NOTE:') - for note in notes: - to_screen(note) -''' - - -# Deprecated -def update_self(to_screen, verbose, opener): - - printfn = to_screen - - write_string( - 'DeprecationWarning: "yt_dlp.update.update_self" is deprecated and may be removed in a future version. ' - 'Use "yt_dlp.update.run_update(ydl)" instead\n') - - class FakeYDL(): - _opener = opener - to_screen = printfn - - @staticmethod - def report_warning(msg, *args, **kwargs): - return printfn('WARNING: %s' % msg, *args, **kwargs) - - @staticmethod - def report_error(msg, tb=None): - printfn('ERROR: %s' % msg) - if not verbose: - return - if tb is None: - # Copied from YoutubeDl.trouble - if sys.exc_info()[0]: - tb = '' - if hasattr(sys.exc_info()[1], 'exc_info') and sys.exc_info()[1].exc_info[0]: - tb += ''.join(traceback.format_exception(*sys.exc_info()[1].exc_info)) - tb += encode_compat_str(traceback.format_exc()) - else: - tb_data = traceback.format_list(traceback.extract_stack()) - tb = ''.join(tb_data) - if tb: - printfn(tb) - - return run_update(FakeYDL()) diff --git a/ytdlp_plugins/extractor/__init__.py b/ytdlp_plugins/extractor/__init__.py deleted file mode 100644 index 3045a590b..000000000 --- a/ytdlp_plugins/extractor/__init__.py +++ /dev/null @@ -1,4 +0,0 @@ -# flake8: noqa: F401 - -# ℹ️ The imported name must end in "IE" -from .sample import SamplePluginIE diff --git a/ytdlp_plugins/extractor/sample.py b/ytdlp_plugins/extractor/sample.py deleted file mode 100644 index d99b7ca33..000000000 --- a/ytdlp_plugins/extractor/sample.py +++ /dev/null @@ -1,16 +0,0 @@ -# coding: utf-8 - -# ⚠ Don't use relative imports -from yt_dlp.extractor.common import InfoExtractor - - -# ℹ️ Instructions on making extractors can be found at: -# 🔗 https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#adding-support-for-a-new-site - -class SamplePluginIE(InfoExtractor): - _WORKING = False - IE_DESC = False - _VALID_URL = r'^sampleplugin:' - - def _real_extract(self, url): - self.to_screen('URL "%s" sucessfully captured' % url) diff --git a/ytdlp_plugins/postprocessor/__init__.py b/ytdlp_plugins/postprocessor/__init__.py deleted file mode 100644 index 61099abbc..000000000 --- a/ytdlp_plugins/postprocessor/__init__.py +++ /dev/null @@ -1,4 +0,0 @@ -# flake8: noqa: F401 - -# ℹ️ The imported name must end in "PP" and is the name to be used in --use-postprocessor -from .sample import SamplePluginPP diff --git a/ytdlp_plugins/postprocessor/sample.py b/ytdlp_plugins/postprocessor/sample.py deleted file mode 100644 index 6ba49266e..000000000 --- a/ytdlp_plugins/postprocessor/sample.py +++ /dev/null @@ -1,28 +0,0 @@ -# coding: utf-8 - -# ⚠ Don't use relative imports -from yt_dlp.postprocessor.common import PostProcessor - - -# ℹ️ See the docstring of yt_dlp.postprocessor.common.PostProcessor -class SamplePluginPP(PostProcessor): - def __init__(self, downloader=None, **kwargs): - # ⚠ Only kwargs can be passed from the CLI, and all argument values will be string - # Also, "downloader", "when" and "key" are reserved names - super().__init__(downloader) - self._kwargs = kwargs - - # ℹ️ See docstring of yt_dlp.postprocessor.common.PostProcessor.run - def run(self, info): - if info.get('_type', 'video') != 'video': # PP was called for playlist - self.to_screen(f'Post-processing playlist {info.get("id")!r} with {self._kwargs}') - elif info.get('filepath'): # PP was called after download (default) - filepath = info.get('filepath') - self.to_screen(f'Post-processed {filepath!r} with {self._kwargs}') - elif info.get('requested_downloads'): # PP was called after_video - filepaths = [f.get('filepath') for f in info.get('requested_downloads')] - self.to_screen(f'Post-processed {filepaths!r} with {self._kwargs}') - else: # PP was called before actual download - filepath = info.get('_filename') - self.to_screen(f'Pre-processed {filepath!r} with {self._kwargs}') - return [], info # return list_of_files_to_delete, info_dict |
