Merge branch 'modular-data-extract'

Commits in this branch are prefixed with "Extraction:"
This branch refactors data extraction. All such functionality has been moved to the yt_data_extract module.
Responses from requests are given to the module and it parses them into a consistent, more useful format.
The dependency on youtube-dl has also been dropped and this functionality has been built from scratch for these reasons:
(1) I've noticed youtube-dl breaks more often than invidious (which uses watch page extraction built from scratch) in response to changes from Youtube, so I'm hoping what I wrote will also be less brittle.
(2) Such breakage is inconvenient because I have to manually merge the fixes since I had to make changes to youtube-dl to make it do things such as extracting related videos.
(3) I have no control over error handling and request pooling with youtube-dl, since it does all the requests (these would require intrusive changes I don't want to maintain).
(4) I will now be able to finally display the number of comments and whether comments are disabled without making additional requests.

1 files changed, 16 insertions, 15 deletions

diff --git a/youtube/playlist.py b/youtube/playlist.py
index 3e5b0d2..3ca235a 100644
--- a/youtube/playlist.py
+++ b/youtube/playlist.py
@@ -89,28 +89,29 @@ def get_playlist_page():
         )
         gevent.joinall(tasks)
         first_page_json, this_page_json = tasks[0].value, tasks[1].value
-    
-    try:    # first page
-        video_list = this_page_json['response']['contents']['singleColumnBrowseResultsRenderer']['tabs'][0]['tabRenderer']['content']['sectionListRenderer']['contents'][0]['itemSectionRenderer']['contents'][0]['playlistVideoListRenderer']['contents']
-    except KeyError:    # other pages
-        video_list = this_page_json['response']['continuationContents']['playlistVideoListContinuation']['contents']
-
-    parsed_video_list = [yt_data_extract.parse_info_prepare_for_html(video_json) for video_json in video_list]
 
+    info = yt_data_extract.extract_playlist_info(this_page_json)
+    if info['error']:
+        return flask.render_template('error.html', error_message = info['error'])
 
-    metadata = yt_data_extract.renderer_info(first_page_json['response']['header'])
-    yt_data_extract.prefix_urls(metadata)
+    if page != '1':
+        info['metadata'] = yt_data_extract.extract_playlist_metadata(first_page_json)
 
-    if 'description' not in metadata:
-        metadata['description'] = ''
+    util.prefix_urls(info['metadata'])
+    for item in info.get('items', ()):
+        util.prefix_urls(item)
+        util.add_extra_html_info(item)
+        if 'id' in item:
+            item['thumbnail'] = '/https://i.ytimg.com/vi/' + item['id'] + '/default.jpg'
 
-    video_count = int(metadata['size'].replace(',', ''))
-    metadata['size'] += ' videos'
+    video_count = yt_data_extract.deep_get(info, 'metadata', 'video_count')
+    if video_count is None:
+        video_count = 40
 
     return flask.render_template('playlist.html',
-        video_list = parsed_video_list,
+        video_list = info.get('items', []),
         num_pages = math.ceil(video_count/20),
         parameters_dictionary = request.args,
 
-        **metadata
+        **info['metadata']
     ).encode('utf-8')