PlugIn: YouTube (2015)

[tmiranda]

Originally Posted by Rob23

Thank you for your taking time to look at this and for your patience.

A point of clarification - I have never logged into a YouTube account as I don't use the download function of this plugin. I just stream the videos as I go.

Before the last log file that I uploaded yesterday, I had grabbed the latest youtube-dl and loaded it in the SageTV install directory, replacing the previous version. So, I don't think that is the issue.

It is definitely displaying the latest videos on my favorite channels; I just can't get anything to play.

Only other thing I can think of is that it could be a Windows 7 firewall issue. I have all of the Sage-related ports open and appear to be able to see videos that are on YouTube, but they just won't play.

Any other thoughts?

The log shows me that youtube-dl is not getting the URL for the video. I'll look into it more closely tomorrow (no time today.) What channel are you trying to watch?

[Rob23]

Originally Posted by tmiranda

The log shows me that youtube-dl is not getting the URL for the video. I'll look into it more closely tomorrow (no time today.) What channel are you trying to watch?

I think I may have attempted the MotorTrend channel when I generated this log, but I don't recall. I have attempted to watch many different channels, all with the same resulting error. I'm beginning to suspect it may be a permissions issue with Windows 7, but then again I would think that it wouldn't even be displaying the list of videos in my favorite channels if Sage wasn't able to get out through the firewall.

Thank you again for looking into this.

[tmiranda]

Originally Posted by Rob23

I think I may have attempted the MotorTrend channel when I generated this log, but I don't recall. I have attempted to watch many different channels, all with the same resulting error. I'm beginning to suspect it may be a permissions issue with Windows 7, but then again I would think that it wouldn't even be displaying the list of videos in my favorite channels if Sage wasn't able to get out through the firewall.

Thank you again for looking into this.

I also think it's something with your system. I just brought up Motor Trend and was able to watch the videos without difficulty.

[Rob23]

Originally Posted by tmiranda

I also think it's something with your system. I just brought up Motor Trend and was able to watch the videos without difficulty.

You were correct. I ended up tracking it down to an internet options setting that was preventing the communication with YouTube. Once I set it to allow an anonymous login, the plugin is working as expected. Thanks again for looking at my log file.

[Rob23]

Originally Posted by tmiranda

OK, I know what's going on. Since you have downloaded it and not imported it as a video or TV recording the plugin does not know where to look for the downloaded video. I remember this case when I was coding the plugin and decided that finding the downloaded video would be "hit or miss" (because the user can choose to download the video anywhere) so I decided just to re-download it.

After spending a great deal of time pouring through a verbose log file (trace wasn't telling the whole story), I think I now better understand why I continue to have intermittent error messages indicating that previously watched video files are no longer available on disk.

I am using this in the same way starfire was (i.e. not setting the videos to download and import as recordings). Often, if I return to watch a video that I have watched before, I get an error indicating the file is no longer available on disk.

I believe the issue is that the files are being downloaded into a temporary file, that I understand should be deleted when Sage exits. However, in reading the error log, it appears that the YT plugin is attempting to locate the original downloaded file:

Wed 1/11 21:44:15.999 [AWT-EventQueue-0@1b8935b] YT: About to play 7mYK4o_kK3E
Wed 1/11 21:44:16.001 [AWT-EventQueue-0@1b8935b] YT: Util.fileIsDownloaded: Found file: OnlineVideo-7mYK4o_kK3E-335951711658898144.mp4
Wed 1/11 21:44:16.001 [AWT-EventQueue-0@1b8935b] YT: Video has been previously downloaded. C:\Users\Rob\AppData\Local\Temp\OnlineVideo-7mYK4o_kK3E-335951711658898144.mp4
Wed 1/11 21:44:16.008 [AsyncWatch@1228794] Returning null format for zero-length file
Wed 1/11 21:44:16.008 [AsyncWatch@1228794] Returning null format for zero-length file
Wed 1/11 21:44:16.009 [AsyncWatch@1228794] Increased StringBuffer pool to size=1
Wed 1/11 21:44:16.009 [AsyncWatch@1228794] VideoFrame.watch(A[1000000004,1000000002,"OnlineVideo-7mYK4o_kK3E-335951711658898144",0@0108.11:57,0,V])
Wed 1/11 21:44:16.010 [AsyncWatch@1228794] watchThisFile=MediaFile[id=0 A[1000000004,1000000002,"OnlineVideo-7mYK4o_kK3E-335951711658898144",0@0108.11:57,0,V] mask=V host=Rob-PC encodedBy= format=null C:\Users\Rob\AppData\Local\Temp\OnlineVideo-7mYK4o_kK3E-335951711658898144.mp4, Seg0[Sun 1/8 11:57:51.320-Sun 1/8 11:57:51.321]]
Wed 1/11 21:44:16.014 [AsyncWatch@1228794] MediaFile failing verify testFile=C:\Users\Rob\AppData\Local\Temp\OnlineVideo-7mYK4o_kK3E-335951711658898144.mp4 isFile=true len=0 isTV=false isRecording=false
Wed 1/11 21:44:16.014 [AsyncWatch@1228794] BAD WATCH Selected watch failed file verification.
Wed 1/11 21:44:16.014 [AsyncWatch@1228794] cleanupLocalFile called for MediaFile[id=0 A[1000000004,1000000002,"OnlineVideo-7mYK4o_kK3E-335951711658898144",0@0108.11:57,0,V] mask=V host=Rob-PC encodedBy= format=null C:\Users\Rob\AppData\Local\Temp\OnlineVideo-7mYK4o_kK3E-335951711658898144.mp4, Seg0[Sun 1/8 11:57:51.320-Sun 1/8 11:57:51.321]]
Wed 1/11 21:44:16.014 [BGLoader2-SAGETV_PROCESS_LOCAL_UI@d62450] Loading RawImage of size 1280x720 for url http://i.ytimg.com/vi/yFiBVIelJsQ/maxresdefault.jpg fileSize=118636
Wed 1/11 21:44:16.015 [AsyncWatch@1228794] processOptionsMenu optionsMenu=default:TMIR-91968|OptionsMenu: Display Message Dialog

Is there a way that the plugin can be modified such that the YT: Util.fileIsDownloaded process is never run? My objective is for the plugin to never look to determine if the file was previously downloaded and to always re-download a file when a video is selected for playback. Is this possible in the API? Or, should I be running some type of properties cleanup to ensure that the YT: Util.fileIsDownloaded process never sees that a file has been previously downloaded?

[tmiranda]

Originally Posted by Rob23

After spending a great deal of time pouring through a verbose log file (trace wasn't telling the whole story), I think I now better understand why I continue to have intermittent error messages indicating that previously watched video files are no longer available on disk.

I am using this in the same way starfire was (i.e. not setting the videos to download and import as recordings). Often, if I return to watch a video that I have watched before, I get an error indicating the file is no longer available on disk.

I believe the issue is that the files are being downloaded into a temporary file, that I understand should be deleted when Sage exits. However, in reading the error log, it appears that the YT plugin is attempting to locate the original downloaded file:

Wed 1/11 21:44:15.999 [AWT-EventQueue-0@1b8935b] YT: About to play 7mYK4o_kK3E
Wed 1/11 21:44:16.001 [AWT-EventQueue-0@1b8935b] YT: Util.fileIsDownloaded: Found file: OnlineVideo-7mYK4o_kK3E-335951711658898144.mp4
Wed 1/11 21:44:16.001 [AWT-EventQueue-0@1b8935b] YT: Video has been previously downloaded. C:\Users\Rob\AppData\Local\Temp\OnlineVideo-7mYK4o_kK3E-335951711658898144.mp4
Wed 1/11 21:44:16.008 [AsyncWatch@1228794] Returning null format for zero-length file
Wed 1/11 21:44:16.008 [AsyncWatch@1228794] Returning null format for zero-length file
Wed 1/11 21:44:16.009 [AsyncWatch@1228794] Increased StringBuffer pool to size=1
Wed 1/11 21:44:16.009 [AsyncWatch@1228794] VideoFrame.watch(A[1000000004,1000000002,"OnlineVideo-7mYK4o_kK3E-335951711658898144",0@0108.11:57,0,V])
Wed 1/11 21:44:16.010 [AsyncWatch@1228794] watchThisFile=MediaFile[id=0 A[1000000004,1000000002,"OnlineVideo-7mYK4o_kK3E-335951711658898144",0@0108.11:57,0,V] mask=V host=Rob-PC encodedBy= format=null C:\Users\Rob\AppData\Local\Temp\OnlineVideo-7mYK4o_kK3E-335951711658898144.mp4, Seg0[Sun 1/8 11:57:51.320-Sun 1/8 11:57:51.321]]
Wed 1/11 21:44:16.014 [AsyncWatch@1228794] MediaFile failing verify testFile=C:\Users\Rob\AppData\Local\Temp\OnlineVideo-7mYK4o_kK3E-335951711658898144.mp4 isFile=true len=0 isTV=false isRecording=false
Wed 1/11 21:44:16.014 [AsyncWatch@1228794] BAD WATCH Selected watch failed file verification.
Wed 1/11 21:44:16.014 [AsyncWatch@1228794] cleanupLocalFile called for MediaFile[id=0 A[1000000004,1000000002,"OnlineVideo-7mYK4o_kK3E-335951711658898144",0@0108.11:57,0,V] mask=V host=Rob-PC encodedBy= format=null C:\Users\Rob\AppData\Local\Temp\OnlineVideo-7mYK4o_kK3E-335951711658898144.mp4, Seg0[Sun 1/8 11:57:51.320-Sun 1/8 11:57:51.321]]
Wed 1/11 21:44:16.014 [BGLoader2-SAGETV_PROCESS_LOCAL_UI@d62450] Loading RawImage of size 1280x720 for url http://i.ytimg.com/vi/yFiBVIelJsQ/maxresdefault.jpg fileSize=118636
Wed 1/11 21:44:16.015 [AsyncWatch@1228794] processOptionsMenu optionsMenu=default:TMIR-91968|OptionsMenu: Display Message Dialog

Is there a way that the plugin can be modified such that the YT: Util.fileIsDownloaded process is never run? My objective is for the plugin to never look to determine if the file was previously downloaded and to always re-download a file when a video is selected for playback. Is this possible in the API? Or, should I be running some type of properties cleanup to ensure that the YT: Util.fileIsDownloaded process never sees that a file has been previously downloaded?

The plugin will always try to play it if it's been previously downloaded. I'll have to update the plugin to add the option to skip that check.

It looks like the file is still on the disk but is zero length?

One thing you can do is make sure the file is completely deleted from the disk and then start an imported video re-scan. This should make sure the plugin does not try to play the deleted file.

[Rob23]

Originally Posted by tmiranda

The plugin will always try to play it if it's been previously downloaded. I'll have to update the plugin to add the option to skip that check.

It looks like the file is still on the disk but is zero length?

One thing you can do is make sure the file is completely deleted from the disk and then start an imported video re-scan. This should make sure the plugin does not try to play the deleted file.

Thank you. I would like the option to skip that check.

I'm not sure I'm clear as to why the imported video re-scan would apply as I'm not importing the videos as recordings. I just navigate to my favorite channels and play the videos from there.

Regarding the zero length file, it almost seems like there is an issue with the way the temp files are being deleted when Sage exits. Clearly, the file name still resides in the temp directory, but it seems as though the bulk of the data is deleted. Perhaps the solution lies in looking at how the temp file deletion process is being executed. Im not sure.

[tmiranda]

Originally Posted by Rob23

Thank you. I would like the option to skip that check.

I'm not sure I'm clear as to why the imported video re-scan would apply as I'm not importing the videos as recordings. I just navigate to my favorite channels and play the videos from there.

Regarding the zero length file, it almost seems like there is an issue with the way the temp files are being deleted when Sage exits. Clearly, the file name still resides in the temp directory, but it seems as though the bulk of the data is deleted. Perhaps the solution lies in looking at how the temp file deletion process is being executed. Im not sure.

The re-scan may help because in order to play a video the plugin creates a temporary MediaFile Object, plays the video, then deletes the MediaFile Object. I'm thinking the re-scan may be needed to clear the MediaFile Object from the sage database. If the plugin doesn't find the MediaFile Object, it won't look for the physical file.

The temp files are being created using standard java methods. They should be deleted when the sage java VM exits, if not there is a bug with java or Windows.

If you manually delete the files from the temp directory the plugin should work. That's the theory anyway

[Rob23]

Originally Posted by tmiranda

The re-scan may help because in order to play a video the plugin creates a temporary MediaFile Object, plays the video, then deletes the MediaFile Object. I'm thinking the re-scan may be needed to clear the MediaFile Object from the sage database. If the plugin doesn't find the MediaFile Object, it won't look for the physical file.

The temp files are being created using standard java methods. They should be deleted when the sage java VM exits, if not there is a bug with java or Windows.

If you manually delete the files from the temp directory the plugin should work. That's the theory anyway

Interesting. When I went to the temp file directory, I found a handful of MPEG-4 files with file sizes of 0 KB, including the one in the error log referenced above. After manually deleting these files and running a Scan Imported Media in Sage, the above video is able to play in Sage. I am going to monitor the temp file directory to see if I can identify a build-up of these zero length files. Thank you again for your help.

[tmiranda]

Originally Posted by Rob23

Interesting. When I went to the temp file directory, I found a handful of MPEG-4 files with file sizes of 0 KB, including the one in the error log referenced above. After manually deleting these files and running a Scan Imported Media in Sage, the above video is able to play in Sage. I am going to monitor the temp file directory to see if I can identify a build-up of these zero length files. Thank you again for your help.

Glad it's now working. Let me know if you see a pattern with those zero length files. I should probably put a check for that in the code.

[EnterNoEscape]

I was checking the SageTV folder structure because it was reporting that it was over 270GB. The culprit was the youtube_downloads folder. I realize this is supposed to be a temporary place for videos as they are being streamed or downloaded, but it seems that it doesn't always clean up after itself. I know this kind of thing can be a little tricky because you don't want to delete something someone is still playing back for example, but some active management of this folder like removal of cached videos that are over a few days old if the used disk space exceeds a threshold would be desirable at least to me.

[SomeWhatLost]

Originally Posted by EnterNoEscape

I was checking the SageTV folder structure because it was reporting that it was over 270GB. The culprit was the youtube_downloads folder.

you must be doing something wrong...
my Sagetv/youtube_downloads folder is always empty...
not a file in sight...
spotless even...

of course, it seems for some reason that the plugin is just dumping the files in /sageTV for me...

why did it even make a folder if it wasn't going to use it?

is the location of where Youtub(2015) dumps files configurable? not that it really matters, as long as it plays the files, I am happy

[EnterNoEscape]

This is really just a humble request. I normally don't download much from YouTube and like this plugin a lot for the few times I visit. I think some of the problems came from the fact that I had added a favorite channel and set it to download everything when I meant to tell it to only download new things. Needless to say I got a lot of content rather quickly. Since this content was being copied into my recordings, I didn't think about the temporary files hanging around.

[MattHelm]

Originally Posted by EnterNoEscape

This is really just a humble request. I normally don't download much from YouTube and like this plugin a lot for the few times I visit. I think some of the problems came from the fact that I had added a favorite channel and set it to download everything when I meant to tell it to only download new things. Needless to say I got a lot of content rather quickly. Since this content was being copied into my recordings, I didn't think about the temporary files hanging around.

I use it for a few shows a week, and once in a while it just hangs (I think because they can't finish the download?) and leaves files there.

I do wish he'd add a custom "recording" path, so my SSD wouldn't be used. ;-)

[tmiranda]

Originally Posted by EnterNoEscape

This is really just a humble request. I normally don't download much from YouTube and like this plugin a lot for the few times I visit. I think some of the problems came from the fact that I had added a favorite channel and set it to download everything when I meant to tell it to only download new things. Needless to say I got a lot of content rather quickly. Since this content was being copied into my recordings, I didn't think about the temporary files hanging around.

It should cleanup after itself. Do you have any logfiles that show something being downloaded?

[EnterNoEscape]

Originally Posted by tmiranda

It should cleanup after itself. Do you have any logfiles that show something being downloaded?

The downloads all happened last week, so I might not have corresponding logs, but I'll check. Should it be cleaning itself up immediately after downloading the files or when SageTV shuts down?

Edit: I don't have the logs from when the files were downloaded.

[tmiranda]

Originally Posted by EnterNoEscape

The downloads all happened last week, so I might not have corresponding logs, but I'll check. Should it be cleaning itself up immediately after downloading the files or when SageTV shuts down?

Edit: I don't have the logs from when the files were downloaded.

It should delete them immediately. It downloads the video to the youtube_downloads directory then moves or copies the file to the final destination. If it copies the file the original should be deleted.

Is your target download directory on the same or different filesystem?

[EnterNoEscape]

Originally Posted by tmiranda

It should delete them immediately. It downloads the video to the youtube_downloads directory then moves or copies the file to the final destination. If it copies the file the original should be deleted.

Is your target download directory on the same or different filesystem?

Same system. I'm using the Docker container on unRAID. The recordings definitely showed up, the temporary files just didn't get removed. Would restarting SageTV while a video is downloading be enough to trip things up?

[tmiranda]

Originally Posted by EnterNoEscape

Same system. I'm using the Docker container on unRAID. The recordings definitely showed up, the temporary files just didn't get removed. Would restarting SageTV while a video is downloading be enough to trip things up?

Restarting should not matter. I wonder if the issue is docker specific? Can you download something and send me the log?

[EnterNoEscape]

Originally Posted by tmiranda

Restarting should not matter. I wonder if the issue is docker specific? Can you download something and send me the log?

No problem. It will probably be tomorrow. I'm getting some target invocation errors related to the plugin that implies that for some reason the .jar didn't load, so I'll be restarting SageTV after tonight's recordings to see if it fixes itself. I don't think this means anything is wrong with the plugin, but in case you're interested, here's the logs from that.

Edit: It looks like the plugin isn't loading. I uploaded another attachment with the failure log.