aboutsummaryrefslogtreecommitdiffstats
path: root/youtube/util.py
Commit message (Collapse)AuthorAgeFilesLines
* Fix error during exit blockage detection when Set-Cookie missingJames Taylor2022-02-151-1/+1
| | | | Signed-off-by: Jesús <heckyel@hyperbola.info>
* Fix MaxRetryErrors due to Tor exit node blockageJames Taylor2022-02-151-3/+16
| | | | | | | | Sometimes YouTube redirects to a google.com/sorry page, seemingly setting up redirect loops. Other times the url redirects to itself. Signed-off-by: Jesús <heckyel@hyperbola.info>
* Prefix youtube URLs in video descriptions and channel about pageJames Taylor2021-08-091-0/+7
| | | | | | Closes #75 Signed-off-by: Jesús <heckyel@hyperbola.info>
* Switch to new comments api now that old one is being disabledJames Taylor2021-08-091-0/+13
| | | | | | | | | | | watch_comment api periodically gives the error "Top level comments mweb servlet is turned down." The continuation items for the new api are in a different arrangement in the json, so changes were necessary to the extract_items function. Signed-off-by: Jesús <heckyel@hyperbola.info>
* Cleaner channel-checking console error messages when Tor is closedJesús2021-07-281-1/+5
| | | | by James Taylor <user234683@users.noreply.github.com>
* Restart tor connection pool when the tor port setting is changedJames Taylor2021-07-281-0/+4
| | | | | | | Otherwise the old port continued to be used until the program was restarted. Signed-off-by: Jesús <heckyel@hyperbola.info>
* Friendlier error message when Tor is closed or network is downJames Taylor2021-07-041-3/+22
| | | | Signed-off-by: Jesús <heckyel@hyperbola.info>
* Capitalize name appJesús2021-06-101-1/+1
|
* Save uploader id in local playlistsJames Taylor2021-05-061-2/+5
| | | | Signed-off-by: Jesús <heckyel@hyperbola.info>
* util.py: Encode data using utf-8, not asciiJames Taylor2021-03-031-2/+2
| | | | Signed-off-by: Jesús <heckyel@hyperbola.info>
* util.py: Fix data not being passed in requestsJames Taylor2021-03-031-2/+2
| | | | Signed-off-by: Jesús <heckyel@hyperbola.info>
* Debugging response saving: save page even if it's an HTTP errorJames Taylor2021-02-261-8/+9
| | | | | | | This will help debug new types of exit node blockage or other errors. Signed-off-by: Jesús <heckyel@hyperbola.info>
* TorMgr: Change new identity cooldown to 7 seconds instead of 6James Taylor2021-02-131-4/+4
| | | | | | Because sometimes a new identity was not being respected Signed-off-by: Jesús <heckyel@hyperbola.info>
* fetch_url: 429: get new identity regardless of page contentJames Taylor2021-01-131-3/+1
| | | | | | | | | | | New 429 captcha page doesn't have IP. This new page appears to match the 429 code plus the json of {"redirect": ...} which would be occasionally received when the pbj json endpoint was used in the past. Closes #22 Signed-off-by: Jesús <heckyel@hyperbola.info>
* General theme: fix syntax W3C markup and add improve 'Published' dateJesús2020-12-301-0/+7
| | | | Signed-off-by: Jesús <heckyel@hyperbola.info>
* strip_non_ascii in comments-author-nameJesús2020-12-301-0/+6
|
* util: Remove obsolete parse_info_prepare_for_html functionJames Taylor2020-12-281-9/+0
| | | | | | | | Info parsing is handled by yt_data_extract, and html post-processing is done with util.prefix_urls and util.add_extra_html_info Signed-off-by: Jesús <heckyel@hyperbola.info>
* Remove obsolete uppercase_escape functionJames Taylor2020-12-281-6/+0
| | | | | | | This function was only necessary with the old ajax format, which was removed in 4d7bba92eb62518e2273d030235214f4a7605444 Signed-off-by: Jesús <heckyel@hyperbola.info>
* TorManager: Add delay before subsequent new identity retriesJames Taylor2020-12-251-1/+18
| | | | | | | | | | | | | | | The request can be retried immediately after the first new identity, but if we do more new identities, we have to wait for at least 6 seconds before doing the request, otherwise it won't be done on a new ip based on my experiments. Potential issue: If after getting third new identity, request takes > 12 seconds (since timeout is 15) and returns 429, then the Tor Manager will let it do a 4th try instead of giving up (meaning request is taking forever from user's perspective). Should be a very rare occurence however. Signed-off-by: Jesús <heckyel@hyperbola.info>
* Exit node retrying: Retry 3 times. Also add tests for it.James Taylor2020-12-211-4/+17
| | | | | | Closes #20 Signed-off-by: Jesús <heckyel@hyperbola.info>
* Fix youtube mixesJames Taylor2020-12-181-2/+8
| | | | | | | They cannot be viewed on their own, so change url in items to go to the video+playlist instead Signed-off-by: Jesús <heckyel@hyperbola.info>
* pep8Jesús2020-12-151-26/+40
|
* fetch_url: Use time.monotonic for comptability w/ tor_mgrJames Taylor2020-11-291-5/+8
| | | | | Buggy exit node retry behavior due to fetch_url still using time.time() instead of the time.monotonic() used by new_identity
* use Tor to resolve domain namesuser2346842020-11-121-2/+2
|
* new_identity: authenticate with blank passwordJames Taylor2020-10-251-1/+1
| | | | See https://github.com/user234683/youtube-local/issues/20#issuecomment-716198162
* Automatically change tor circuit once if ip is blockedJames Taylor2020-10-251-45/+117
| | | | | | | Use stem library to send a new identity signal via the tor control port. See #20
* fetch_url: move content decoding to before 429 checkJames Taylor2020-10-251-1/+3
| | | | | Content should be decoded before we try reading page to check for 429. Lucky that it worked at all before.
* Put vid title at end of download urls so downloads w/ that filenameJames Taylor2020-10-221-0/+54
|
* fix stylezrose5842020-10-091-1/+1
|
* settings.py: add 'proxy_images'zrose5842020-10-071-6/+7
|
* Increase max redirects for video routingJames Taylor2020-09-221-3/+13
|
* Add setting for tor_portuser9381202020-09-191-3/+3
|
* Add tor video routingJames Taylor2020-09-181-14/+23
| | | | | | | | | | Includes non-tor video routing by default, so no more chances of the browser leaking headers or user agent to googlevideo Adjust settings upgrade system to facilitate change to route_tor setting. Add some more space on settings page for dropdown settings so does not overflow due to options with long names. Closes #7
* Don't use brotli if import failsJames Taylor2020-08-111-2/+9
|
* Fix failure to parse comments when there's one from deleted channelJames Taylor2020-03-081-0/+9
| | | | | | Specifically, fix failures when any of the fields from the parsed comment are None, such as author, author_url, etc. (failure due to string concatenation when building urls).
* Ensure urllib3 and python version not leaked during video file checkJames Taylor2020-02-181-1/+2
| | | | | This is likely not a big deal since it is already assumed that video file server logs are not plugged into Google's tracking infrastructure, but it doesn't hurt to give less info.
* Fix MaxRetryError when checking for video URL accessJames Taylor2020-02-181-2/+10
| | | | | | | | The default urllib3 max redirect amount was set to 3. Change it to 10 and do not fail if there is a problem with checking for URL access. Just print the error to the console and proceed. Also add an unrelated remark about the bcptr=9999999999 parameter in watch.py
* Check for 403 errors and fallback on InvidiousJames Taylor2020-02-011-0/+8
| | | | 403 errors on the video urls happen typically when a video has copyrighted content or was livestreamed originally. They appear to not happen (or at least happen less frequently) if the Tor exit node used ipv6, however.
* Give a proper error message for 429 errorsJames Taylor2020-01-311-0/+23
| | | | | | These occur when too many requests are coming from a Tor exit node. Before, there would be an error page with an exception instructing users to report the issue. But this is an expected and persistent issue.
* Extraction: Move html post processing stuff from yt_data_extract to utilJames Taylor2019-12-191-0/+38
|
* Extraction: Rewrite item_extraction for better error handling and ↵James Taylor2019-12-181-0/+2
| | | | readability, rename extracted names for more consistency
* Extraction: Replace youtube-dl with custom-built watch page extractionJames Taylor2019-10-171-1/+8
|
* Extraction: Move comment extraction to yt_data_extractJames Taylor2019-09-191-9/+0
|
* Extraction: Fix url prefixingJames Taylor2019-09-081-0/+4
|
* Merge flask framework and other stuff from masterJames Taylor2019-08-091-2/+10
|\
| * Remove ad-hoc response saving from code, create a debug setting for fetch_urlJames Taylor2019-07-231-2/+11
| |
* | Overhaul refresh system, make it asynchronousJames Taylor2019-06-091-2/+52
| |
* | Make thumbnails work and other stuffJames Taylor2019-06-051-0/+32
|/
* prevent urllib3 from decoding contentJames Taylor2019-02-261-1/+1
|
* Use persistent connectionsJames Taylor2019-02-251-11/+87
|
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-22 21:34:25 +0000