PlugIn: YouTube (2015)

[Beaker]

Originally Posted by robogeek

Just a note to let everyone know the YouTube API sorting is fixed and the YouTube plugin appears to be working normally again

It is not working correctly for me. My Wiz.bin still climbs to ludicrous size very quickly. Perhaps this is rolling out slowly to certain individuals?

[robogeek]

In the cases where the plugin still doesn't seem to be working correctly, I suspect the plugin may be picking up the previously queued videos from when the sorting issue first started. You may have to go into the YouTube Recording Engine configuration (Setup->SageTV Plugins->Installed Plugins->General->YouTube Recording Engine->Configure Plugin) and find the option in there to purge the download queue.

<EDIT>
Nevermind, I see what you guys are talking about now. My YouTube favorites seem to be downloading new videos normally. But I see the video lists from within the YouTube (2015) plugin seem to be messed up for some YouTube channels, but not all. Not sure yet about what is going on there. But I think the growing wiz.bin is likely related to the download queue that needs to be purged.

<EDIT 2>
Upon further checking... my messed up looking favorite YouTube channels had the favorite option 'Fetch Watched Videos' set to Disable instead of Enabled, so there were gaps with only a couple of the most recent unwatched videos at the top of the list which were followed by much older unwatched videos. After ensuring the 'Fetch Watched Videos' was set to Enabled for all of my favorite YouTube channels, it appears the plugin is properly fetching the video lists which are properly sorted by date (newest to oldest). My wiz.bin also looks like it's maintaining a stable and expected size between 140MB and 150MB.

[robogeek]

I did notice one other issue with about a dozen YouTube videos from channels that I have set as Favorites. These videos won't download and will drop off of the queue and get queued again later, and they keep repeating that cycle. I don't know if it's a problem with youtube-dl.exe taking too long to get to the video download or if it's a problem with the YouTube plugin not waiting long enough and declaring the job dead and killing it before it starts downloading:

Sat 3/30 2:00:39.925 [Timer-28@4b8f546a] YT: JobManager.startFirstInQueue: Starting job.
Sat 3/30 2:00:39.926 [Timer-28@4b8f546a] YT: JobManager.startFirstInQueue: Started gS-rVPbVGYk
Sat 3/30 2:00:39.926 [Timer-28@4b8f546a] YT: MonitorJob: Job is done. 2f-gZwdTFyo
Sat 3/30 2:00:39.926 [Thread-113388@32a5f61e] YT: DownloadVideo: Starting new download.
Sat 3/30 2:00:39.926 [Timer-28@4b8f546a] YT: Videos.getVideoInfo: Returning results from cache.
Sat 3/30 2:00:39.926 [Thread-113388@32a5f61e] YT: DownloadVideo.executeDownloadWindows: targetDirectory specified <youtube_downloads\>
Sat 3/30 2:00:39.927 [Timer-28@4b8f546a] YT: Util.getAllDownloadedFiles: No files in directory <C:\Program Files\SageTV\SageTV\youtube_downloads>
Sat 3/30 2:00:39.927 [Timer-28@4b8f546a] YT: MonitorJob.renameAndMoveFile: No files found.
Sat 3/30 2:00:39.931 [Thread-113388@32a5f61e] YT: DownloadVideo.executeDownloadWindows: Command [youtube-dl.exe, --restrict-filenames, --no-part, -f, best, -o"youtube_downloads\%(id)s.%(ext)s", gS-rVPbVGYk]
Sat 3/30 2:00:39.932 [Timer-28@4b8f546a] YT: MonitorJob: renamedFile is null. Exiting.
Sat 3/30 2:00:39.935 [Thread-113383@92d741c] YT: DownloadVideo.executeDownloadWindows: youtube-dl has exited.
Sat 3/30 2:00:42.328 [Thread-113390@786866b1] YT: StreamGetter: stdout: [youtube] gS-rVPbVGYk: Downloading webpage
Sat 3/30 2:01:03.974 [Thread-113390@786866b1] YT: StreamGetter: stdout: [youtube] gS-rVPbVGYk: Downloading video info webpage
Sat 3/30 2:01:09.925 [Timer-28@4b8f546a] YT: MonitorJob: Looking for Client requests.
Sat 3/30 2:01:09.925 [Timer-28@4b8f546a] YT: MonitorJob: Found a runnning job. gS-rVPbVGYk
Sat 3/30 2:01:09.925 [Timer-28@4b8f546a] YT: MonitorJob: Job has not yet completed.
Sat 3/30 2:01:09.925 [Timer-28@4b8f546a] YT: MonitorJob: Job is progressing.
Sat 3/30 2:01:39.925 [Timer-28@4b8f546a] YT: MonitorJob: Looking for Client requests.
Sat 3/30 2:01:39.925 [Timer-28@4b8f546a] YT: MonitorJob: Found a runnning job. gS-rVPbVGYk
Sat 3/30 2:01:39.925 [Timer-28@4b8f546a] YT: MonitorJob: Job has not yet completed.
Sat 3/30 2:01:39.925 [Timer-28@4b8f546a] YT: MonitorJob: Job is not progressing. Stopping it.
Sat 3/30 2:01:39.925 [Timer-28@4b8f546a] YT: DownloadVideo.stop: Stopping.
Sat 3/30 2:01:39.926 [Timer-28@4b8f546a] YT: JobManager.updateUserRecord: jobQueue sise = 9
Sat 3/30 2:01:39.926 [Timer-28@4b8f546a] YT: MonitorJob: Finished job removed from queue gS-rVPbVGYk

If I run the youtube-dl command with the parameters specified in the log snippet above, it does download the video. The only thing is, it takes about 90 to 120 seconds from initiating the command until the video download actually begins. It looks like the YouTube plugin is giving up and killing the youtube-dl process after 60 seconds? I did run youtube-dl with the -U update switch to update to the latest version to see if that would help, but this issue still remains. And I'm not sure why this is only happening to a fairly small portion of videos.

[robogeek]

OK, here's what I've been able to figure out...

Most downloads of Favorites work without issue, but some aren't working through the YouTube Download Engine because the youtube-dl program is taking too long to proceed to the actual downloading step and the plugin appears to be interpreting that long delay as a failure and killing the process. (See log snippet in previous post). Playing videos from within the YouTube (2015) plugin works, it's just the favorites and manual recording of the videos that appears to be failing for a certain subset of videos.

I've traced the delay down to the downloading of the DASH manifest that is done by the youtube-dl program. The good news is there is a command line switch (--youtube-skip-dash-manifest) that disables the downloading of the DASH manifest and greatly reduces the delay between the time youtube-dl is started and the time it starts downloading the requested video. Ideally, this command switch would be added to the YouTube Download Engine plugin, or perhaps a new plugin config option added to allow custom youtube-dl command line switches to be specified. Alternatively, instead of adding this command line switch, the YouTube Download Engine could be updated so the youtube-dl watchdog timer is longer or possibly made configurable as a new plugin config option.

Caveat for the following workaround... I don't know if this is or will adversely affect other parts of the YouTube plugin, but everything appears to be working normally with no apparent errors:

Fortunately the youtube-dl program can accept additional command line switches via a config file. (https://github.com/ytdl-org/youtube-...#configuration) Linux should be fairly straight forward. I use Windows so I'll document the Windows stuff here. The documentation on github is slightly incorrect in that the C:\User\..... location specified is incomplete. If you are experiencing this issue of some favorites not being downloaded, create a file called config.txt and put in this command line switch:

--youtube-skip-dash-manifest

Then you need to figure out where to put this config.txt file. Whether you run SageTV (standalone) or SageTVService, you need to know which user account it is running under (the youtube-dl process will start under the same user account the SageTV process is running under). You can find this in the Services control panel or by checking the appropriate SageTV process in the Details tab of the Windows Task Manager. If running under a real user account, the config.txt file would be stored in this directory (create youtube-dl directory if it doesn't exist):

C:\Users\<user account>\AppData\Roaming\youtube-dl

Replace <user account> with the actual name of the user account. If you are running SageTVService under the LocalSystem or some other exotic account type, you'll have to use Google to figure out the appropriate %APPDATA% location for that account, unless someone else contributes that knowledge in a reply here.

Also, in the process of working through this, I have found the Video Format config option in the YouTube Recording Engine plugin always remains set to best. I can't seem to change it from my HD300 extenders or from the plugin manager in the web UI. I can only change it by shutting down the service and manually altering the property in the sage.properties file.

[tmiranda]

I received notification from Google that the bug impacting this plugin has been fixed.

[gary201]

This morning I think I noticed a new problem. When you go to watch a video, the plug-in displays "downloading video info webpage" and doesn't proceed any further. It happens to all of my Favorites. I assume Google is mucking around some more on their site again. Does anyone know if this is a temporary thing (another Google mess up) or something where the plug-in will need updating?

[tmiranda]

Originally Posted by gary201

This morning I think I noticed a new problem. When you go to watch a video, the plug-in displays "downloading video info webpage" and doesn't proceed any further. It happens to all of my Favorites. I assume Google is mucking around some more on their site again. Does anyone know if this is a temporary thing (another Google mess up) or something where the plug-in will need updating?

Whenever something like this happens the first thing you should do is update youtube-dl. Just replace the one already in your sage directory.

https://ytdl-org.github.io/youtube-dl/

[gary201]

That worked. Thanks!

[Rob23]

I've encountered an issue with this plugin in which videos begin playing for five seconds at which time the audio continues as normal, but the video is frozen. I can fast forward and the video will resume for three to five seconds and freezes again with the audio continuing. This only occurs on my HD200 extender. I am able to play videos as normal on my server. I have updated to the latest youtube-dl (2019.07.16) as well as uninstalled the plugin and reinstalled with no change in the video freezing behavior.

Does anyone have any idea what could be causing this?

[SteveW]

Originally Posted by Rob23

I've encountered an issue with this plugin in which videos begin playing for five seconds at which time the audio continues as normal, but the video is frozen. I can fast forward and the video will resume for three to five seconds and freezes again with the audio continuing. This only occurs on my HD200 extender. I am able to play videos as normal on my server. I have updated to the latest youtube-dl (2019.07.16) as well as uninstalled the plugin and reinstalled with no change in the video freezing behavior.

Does anyone have any idea what could be causing this?

I’ve had this happen lately on my HD300s.

[Rob23]

Originally Posted by SteveW

I’ve had this happen lately on my HD300s.

In the past year, it would happen to me occassionally, but not consistently. Usually, updating the youtube-dl file would resolve the issue. For the past couple of months, it happens on almost every video, and every update to the youtube-dl file I've installed has not resolved the issue.

[Rob23]

Can anyone confirm for me that the plugin is working correctly for them? I am trying to determine if it is something on my end or an issue with the current youtube-dl.

[MattHelm]

Originally Posted by Rob23

Can anyone confirm for me that the plugin is working correctly for them? I am trying to determine if it is something on my end or an issue with the current youtube-dl.

I'm having the same issue that you describe above.

[NoodleNT]

Mine hasn’t worked correctly since it went on the download craze a few months ago. My queue will say over 280 shows but nothing ever downloads. I have reset it a few times using the purge option but nothing changes. When I manually try and watch a show I get the message content must be downloaded before playback and then a random Info:[youtube] message.

If a new DLL fixes this, can you please update the repository so SageTV can see there is a plugin update?

Thanks.

[tmiranda]

Originally Posted by NoodleNT

Mine hasn’t worked correctly since it went on the download craze a few months ago. My queue will say over 280 shows but nothing ever downloads. I have reset it a few times using the purge option but nothing changes. When I manually try and watch a show I get the message content must be downloaded before playback and then a random Info:[youtube] message.

If a new DLL fixes this, can you please update the repository so SageTV can see there is a plugin update?

Thanks.

Sorry but I will not be able to look at this for a while. I'm currently in the cross hairs of Dorian and then I'll be out of the country for a week or so. I'll look at it when I get back.

The first thing you should always do is download the latest youtube-dl and overwrite the existing one. https://ytdl-org.github.io/youtube-dl/index.html

[Thomas Wischgol]

Sorry to hear you are affected by the storm. I hope you stay safe down there. Regarding the previous post, downloading a new youtubedll.exe might help (see post #767). Youtube is working fine over here but I did have to update the youtubedll.exe manually. It will not work with the one downloaded from Sage's repository.

[gary201]

I've been noticing a new problem viewing YouTube videos over the past few weeks that seems to affect some, but not all videos. It looks like the YouTube download is aborting, and although there is an indication that a retry is going to happen, that doesn't actually seem to be taking place.

Here is a log snippet from a short bit ago.

Sat 10/12 16:10:57.370 [EventRouter-0023a50006bc@473ece1c] YT: About to play PH0myD3Jbno
Sat 10/12 16:10:57.372 [EventRouter-0023a50006bc@473ece1c] YT: Start downloading.
Sat 10/12 16:10:57.375 [ReProcessHook@350a1798] YT: GetFormat: Returning results from cache.
Sat 10/12 16:10:57.376 [ReProcessHook@350a1798] YT: GetFormat.startFetch: In cache.
Sat 10/12 16:10:57.376 [ReProcessHook@350a1798] YT: GetFormat.isReady: In cache.
Sat 10/12 16:10:57.376 [ReProcessHook@350a1798] YT: GetUrl.startFetch: In cache.
Sat 10/12 16:10:57.376 [ReProcessHook@350a1798] YT: Format = mp4
Sat 10/12 16:10:57.377 [ReProcessHook@350a1798] YT: Expected file name PH0myD3Jbno.mp4
Sat 10/12 16:10:57.377 [Thread-2291@3bbc8895] YT: DownloadVideo: Starting new download.
Sat 10/12 16:10:57.378 [Thread-2291@3bbc8895] YT: DownloadVideo.executeDownloadLinux: Command ./youtube-dl --restrict-filenames --no-part -f best -o%(id)s.%(ext)s PH0myD3Jbno
Sat 10/12 16:10:57.466 [ReProcessHook@d518957] YT: Saving screen saver timeout.
Sat 10/12 16:11:01.195 [Thread-2293@2eab3e9f] YT: StreamGetter: stdout: [youtube] PH0myD3Jbno: Downloading webpage
Sat 10/12 16:11:02.062 [Thread-2293@2eab3e9f] YT: StreamGetter: stdout: [youtube] PH0myD3Jbno: Downloading video info webpage
Sat 10/12 16:11:02.727 [Thread-2293@2eab3e9f] YT: StreamGetter: stdout: [download] Destination: PH0myD3Jbno.mp4
Sat 10/12 16:11:02.991 [Thread-2293@2eab3e9f] YT: StreamGetter: stdout:
Sat 10/12 16:11:02.991 [Thread-2293@2eab3e9f] YT: StreamGetter: stdout: [download] 0.0% of 293.97MiB at Unknown speed ETA Unknown ETA
Sat 10/12 16:11:02.992 [Thread-2293@2eab3e9f] YT: StreamGetter: stdout: [download] 0.0% of 293.97MiB at 26.47KiB/s ETA 03:09:32
Sat 10/12 16:11:02.992 [Thread-2293@2eab3e9f] YT: StreamGetter: stdout: [download] 0.0% of 293.97MiB at 56.59KiB/s ETA 01:28:39
Sat 10/12 16:11:02.992 [Thread-2293@2eab3e9f] YT: StreamGetter: stdout: [download] 0.0% of 293.97MiB at 116.77KiB/s ETA 42:57
Sat 10/12 16:11:02.993 [Thread-2293@2eab3e9f] YT: StreamGetter: stdout: [download] 0.0% of 293.97MiB at 236.94KiB/s ETA 21:10
Sat 10/12 16:11:03.020 [Thread-2293@2eab3e9f] YT: StreamGetter: stdout: [download] 0.0% of 293.97MiB at 476.65KiB/s ETA 10:31
Sat 10/12 16:11:03.224 [Thread-2293@2eab3e9f] YT: StreamGetter: stdout: [download] 0.1% of 293.97MiB at 870.07KiB/s ETA 05:45
Sat 10/12 16:11:03.224 [Thread-2293@2eab3e9f] YT: StreamGetter: stdout: [download] 0.2% of 293.97MiB at 968.10KiB/s ETA 05:10[download] Got server HTTP error: Downloaded 493043 bytes, expected 308253755 bytes. Retrying (attempt 1 of 10)...
Sat 10/12 16:11:03.477 [Thread-2293@2eab3e9f] YT: StreamGetter: stdout:
Sat 10/12 16:11:03.500 [Thread-2293@2eab3e9f] YT: StreamGetter: Duplicate lines = 2:
Sat 10/12 16:11:03.500 [Thread-2291@3bbc8895] YT: DownloadVideo.executeDownloadLinux: youtube-dl has exited.
Sat 10/12 16:11:10.120 [EventRouter-0023a50006bc@473ece1c] YT: DownloadVideo.stop: Stopping.
Sat 10/12 16:11:10.469 [ReProcessHook@d518957] YT: DownloadVideo.stop: Stopping.
Sat 10/12 16:11:10.469 [ReProcessHook@d518957] YT: Deleted PH0myD3Jbno.mp4
Sat 10/12 16:11:10.470 [ReProcessHook@d518957] YT: Restoring screen saver timeout.


The problem does seem to persist for those videos that are failing (e.g. after a week or so, the same problem occurs). The videos are vieable from a browser or from the YouTube app on my iPhone. I gather this is an issue w/youtube-dl? I'm using the current version from 19.09.28. Anyone else observe this?

[gary201]

I'm going to assume this is an issue with youtube-dl, since I can repeat the problem (doing this from the youtube_downloads subdir since I didn't know what would get downloaded):

gary@media3 /opt/sagetv/server/youtube_downloads $ ../youtube-dl --restrict-filenames --no-part -f best PH0myD3Jbno
[youtube] PH0myD3Jbno: Downloading webpage
[youtube] PH0myD3Jbno: Downloading video info webpage
[download] Destination: First_McDonalds_Emma_s_Cafe_Trigger_the_Horse_More_Route_66_Finds-PH0myD3Jbno.mp4
[download] 0.2% of 293.97MiB at 2.36MiB/s ETA 02:04[download] Got server HTTP error: Downloaded 466871 bytes, expected 308253755 bytes. Retrying (attempt 1 of 10)...


ERROR: Did not get any data blocks
gary@media3 /opt/sagetv/server/youtube_downloads $

Then trying it again (w/o deleting what had previously been downloaded):

gary@media3 /opt/sagetv/server/youtube_downloads $ ../youtube-dl --restrict-filenames --no-part -f best PH0myD3Jbno
[youtube] PH0myD3Jbno: Downloading webpage
[youtube] PH0myD3Jbno: Downloading video info webpage
[download] Resuming download at byte 466871


ERROR: Did not get any data blocks
gary@media3 /opt/sagetv/server/youtube_downloads $

[gary201]

Found a new issue today with the YouTube plugin (likely something Google has done). When I scroll down the initial list to "Browse favorite channels", I see a short list of my favorites on the right. When I then click on that entry, I get a message "There are no favorite channels". Well, that's not right since a list of some of them are displayed on the main menu screen.

I extracted the sagetv log entries that occur when that error occurs, which is pasted below. I should note that I went through the process of logging off Youtube, logging back in and entering the device code as prompted, just to make sure this wasn't a login issue. I also went to the URL string listed just to see what was there, but got a "failed to load" error on that web page.

Sat 2/22 8:22:39.624 [EventRouter-0023a50006bc@650e6633] processUserEvent-UserEvent[select] evtTime=Sat 2/22 8:22:39.617
Sat 2/22 8:22:39.624 [EventRouter-0023a50006bc@650e6633] processOptionsMenu optionsMenu=default:TMIR-599561|OptionsMenu:FavChanWait
Sat 2/22 8:22:39.626 [Fork-TMIR-599568@14e616da] YT: Channels.getAllFavoriteVideos: ChannelIds(encrypted strings removed)
Sat 2/22 8:22:39.627 [Fork-TMIR-599568@14e616da] YT: DeviceAuth.getUsableAccessToken: Returned valid access code:(encrypted string removed)
Sat 2/22 8:22:39.627 [Fork-TMIR-599568@14e616da] YT: Quota.addQuota: Total quota usage is now 1284302
Sat 2/22 8:22:39.629 [FinalRender-0023a50006bc@774c44c6] MiniClient did not succeed with menu hint change to:menuName:YouTube Using the V3 API, popupName:FavChanWait, hasTextInput:false, errcode=1
Sat 2/22 8:22:39.743 [Fork-TMIR-599568@14e616da] YT: Channels.getChannelForId: JSON error. 403 : Access Not Configured. YouTube Data API has not been used in project 679168243199 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/youtube.googleapis.com/overview?project=679168243199 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
Sat 2/22 8:22:39.743 [Fork-TMIR-599568@14e616da] YT: Favorites 0
Sat 2/22 8:22:39.744 [Fork-TMIR-599568@14e616da] YT: DeviceAuth.getUsableAccessToken: Returned valid access code:(encrypted string removed)
Sat 2/22 8:22:39.744 [Fork-TMIR-599568@14e616da] YT: DeviceAuth.getUsableAccessToken: Returned valid access code:(encrypted string removed)
Sat 2/22 8:22:39.744 [Fork-TMIR-599568@14e616da] YT: Quota.addQuota: Total quota usage is now 1284308
Sat 2/22 8:22:39.844 [Fork-TMIR-599568@14e616da] YT: Subscriptions.getSubscriptionsForAuthorizedUser: JSON error. 403 : Access Not Configured. YouTube Data API has not been used in project 679168243199 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/youtube.googleapis.com/overview?project=679168243199 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
Sat 2/22 8:22:39.844 [Fork-TMIR-599568@14e616da] YT: Subscriptions.getSubscriptionsForAuthorizedUser: Size of items 0
Sat 2/22 8:22:39.844 [Fork-TMIR-599568@14e616da] YT: User is subscribed to 0
Sat 2/22 8:22:39.844 [Fork-TMIR-599568@14e616da] YT: Total 0
Sat 2/22 8:22:39.897 [ReProcessHook@268b259a] processOptionsMenu optionsMenu=default:TMIR-91968|OptionsMenu:Display Message Dialog
Sat 2/22 8:22:39.903 [FinalRender-0023a50006bc@774c44c6] MiniClient did not succeed with menu hint change to:menuName:YouTube Using the V3 API, popupName:Display Message Dialog, hasTextInput:false, errcode=1

Edit: As a followup, I also discovered that the same issue may be occurring with the PlayOn.tv app as it can't see YouTube video content either for any of my subscribed channels.

[Beaker]

This seems to be the pertinent bit from the log...

Sat 2/22 9:35:15.439 [Timer-1@98de03] YT: Subscriptions.getSubscriptionsForAuthorizedUser: JSON error. 403 : Access Not Configured. YouTube Data API has not been used in project 679168243199 before or it is disabled. Enable it by visiting https://console.developers.google.co...t=679168243199 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

Sat 2/22 9:35:15.441 [Timer-1@98de03] YT: Subscriptions.getSubscriptionsForAuthorizedUser: Size of items 0

Sat 2/22 9:35:15.443 [Timer-1@98de03] YT: FavoriteManager: Terminating due to exception null

I tried visiting the URL specified. It returns the following...

There was an error while loading /apis/api/youtube.googleapis.com/overview?project=679168243199. Please try again.

It isn't my project, so I'm not surprised.

I believe google has been changing things lately. I saw something about a desktop change set for early Feb2020. Could be related, I suppose.
https://support.google.com/youtube/t...27596769?hl=en