Plex Scanner and BMT Agent for SageTV

[wildgoose]

Originally Posted by wildgoose

Let me look into how we can get this data dump. You can do it with sageplex_sync but it's not exactly straightforward, especially for large libraries.

You can get the data dump from the browser, something like this:

http://user:pwd@localhost:8080/sagex/api?c=plex:GetMediaFileForName&1=XXXSummerOlympics-OpeningCeremony-1950709-0.mpg&encoder=json

user/pwd is the username/password to sagex.

XXXSummerOlympics-OpeningCeremony-1950709-0.mpg is the actual filename of the show you are interested in getting sage data from.

You can see a bunch of these URL lines in the sageplex_scanner.log file.

What you'll get back will be a JSON string. Find 2 episodes that should belong to the same show, but are being categorized into different shows. Look at the JSON data to see if you can spot any issues (show name, season, episode, etc..). You can google for online JSON formatter to convert the JSON string into a prettier format, I think there are also chrome plugin that would do this automatically.

[wildgoose]

Originally Posted by fresnoboy

For the Movies, Plex seems to find higher quality Fanart than BMT does. This is probably not a plugin question, but have you found a way to have plex do all the fanart or BMT to use a higher res source? I can fix it manually in plex, but I'd rather like something that was automatic.

Hmmm not sure. Personally I let PLEX handle my movies directly, and only import SageTV recordings into PLEX. I don't use SageTV to watch movies much now, as I wanted to extend the life of my HD300 (less of an issue now because Sage is now open source).

Originally Posted by fresnoboy

Is there a log file I can look at that would have the times to complete each scan, etc...? The plex instance is running in a different VM than SageTV does on a different hardware (I have 2 2xE5-2670's that my guests run on - the NAS runs on bare metal on a separate system), with tons of RAM and 8 cores, so it's should be a pretty fast platform for doing the processing.
Mike

It should be easy to figure out the scanning time from the scanner log. As there's a clear start and finish and they are time stamped. The time should not be long.

Figuring out the agent time is more difficult. PLEX calls into the plugin repeatedly for each recording that it wants metadata (after the scanner has completed). So there's no clear start/end (the plugin doesn't know when it has ended). There are also multiple threads, this is why I prefix the agent log with show/season/episode info as sometimes the entries are interleaved between 2 different threads.

Getting this metadata for 9k entries is probably what's taking up most of the time. However, on subsequent re-scan, PLEX should only be calling the agent on changed files. So if you perform a manual rescan without any change, com.plexapp.agents.bmtagenttvshows.log should not show any new activity on files.

[fresnoboy]

Guys, things were running well for quite awhile, but something has changed and plex is no longer getting any metadata from the scanner anymore. I first thought this was triggered by a plex media server update I did, but in checking the log files, the plugin and plex seem to be working fine - it is calling sagex on the sage server, but every entry is showing an error, with no media info returned.

Sage is working fine, recordings are going on fine, as is playback. I can go to the BMT webpage and it's working fine, and I can't find any errors in sage logfiles.

How can I turn on debugging on the sagex interface and see the other side of the queries the plex scanner is making?

I have restarted sage and rebooted the sageTV server a few times to no avail.

Thanks!
Mike

[fresnoboy]

Well, I figured it out. It was a java update I did on the server that I thought only was going to update the 64 bit version of java, but updated the 32 bit version as well. It didn't remove the old version 7 java, but it overrode it, and that was enough to cause the failure.

I worry about the security issues in older java implementations. Will we ever be able to run only a modern java with v9?

thanks!
Mike

[stuckless]

Originally Posted by fresnoboy

I worry about the security issues in older java implementations. Will we ever be able to run only a modern java with v9?

Unless you are using your sagetv server to browse the net with the same version of java that you are using for sagetv... You don't really need to worry about the security issues.

I run my Java server with Java 8, but, some parts of jetty (and likely other plugins) are not working. This isn't a sagetv thing, but a plugin thing. ie, nothing has to change with sagetv, but plugin authors (of which I'm one of them) need to look how the plugins are failing under v8 and fix it. Now that I'm running v8 on my server, I'll likely start this process.

[stuckless]

@wildgoose - I have SageTV docker container that I'm using for unRAID, and I've been a plex pass member for years, but due to my love/hate relationship with plex, I've never used it. I thought I'd give it another shot, and I figured that I would add an option to install the plex plugin in both the plex server and sagetv to make it easier to get the integration working under unRAID.

A couple of questions...
1. I tried running 'make' under linux and it fails. Is the Makefile windows only?
2. I see the synctool in the plex folder.. is this a plex integration thing (ie, plex knows what a sync tool is) or is this a commandline tool that needs to run periodically, etc.
3. The installer script seems to make some assumptions about the locations of plex, can this be overridden on the commandline?
4. The installer seems to prevent running as a user other than plex, can this be overridden on the command line?

Thanks.

[fresnoboy]

Originally Posted by stuckless

Unless you are using your sagetv server to browse the net with the same version of java that you are using for sagetv... You don't really need to worry about the security issues.

I run my Java server with Java 8, but, some parts of jetty (and likely other plugins) are not working. This isn't a sagetv thing, but a plugin thing. ie, nothing has to change with sagetv, but plugin authors (of which I'm one of them) need to look how the plugins are failing under v8 and fix it. Now that I'm running v8 on my server, I'll likely start this process.

Thanks Stuckless. You make a good point on the security issues. I do run a few video related services on the guest VM that SageTV runs on, so I always worry there is some vulnerability that I am unaware of if I have the old java on the system.

Is V9 of SageTV still 32 bit java based, or does it run under the x64 java runtime? If not, is it waiting for the plugins to go x64 as well? I'm unclear as to where all the various dependencies come from.

Thanks again for all your work on the plugins and what you have contributed. It is so useful to me and many others!

Plex works pretty well these days, esp with this plugin. The main reason I use it is not so much a TV client replacement but for mobile access to sync and auto transcode to ipads and android phones so the kids can watch it on long trips in the car and I can watch all the shows I never watch with my wife on airplanes.

I used to move the MPEG2 recordings to a flash drive I stick on my laptop to watch, and still do that somewhat, but it takes up a lot of space, and transcoding by hand is pretty cumbersome and takes time I don't usually have.

With plex, I can just select a season or a set of movies, and it'll transcode overnight and sync and then I can hop on the plane and everything is there. And being able to sync remotely when I am in a hotel or remote office with a fast connection to get a recently recorded show is very cool - I never could do that the old way.

There are some issues with syncing on iOS that occasionally bite me, but in general it works pretty well.

I just wish plex could do auto-skip on commercials, or even jump to the end of a commercial, but the +30/-10 skips work ok.

Thx
Mike

[stuckless]

Originally Posted by fresnoboy

Is V9 of SageTV still 32 bit java based, or does it run under the x64 java runtime? If not, is it waiting for the plugins to go x64 as well? I'm unclear as to where all the various dependencies come from.

So the 32/64 bit issues are not java related, but sagetv native related.

On linux, I only produce 64bit versions and as such, I need a 64bit jvm (Java 7 or 8 and eventualy 9 when it's released).

On Windows, there is only 32bit versions of the SageTV native parts, and as such, you also need to use the 32bit version of Java. Windows does not have 64bit version of the sagetv native parts, and it's likely that it never will, unless someone takes ownership of doing that.

The Jetty and SageTV Plugin issues are completely different. ie, on Java 8 (doesn't matter 32 bit or 64bit) some of the SageTV plugins that use jetty don't quite work well. I have yet to investigate this, but likely I will have to, since I'm running Java 8 on my server.

[Taddeusz]

Originally Posted by fresnoboy

Thanks Stuckless. You make a good point on the security issues. I do run a few video related services on the guest VM that SageTV runs on, so I always worry there is some vulnerability that I am unaware of if I have the old java on the system.

Is V9 of SageTV still 32 bit java based, or does it run under the x64 java runtime? If not, is it waiting for the plugins to go x64 as well? I'm unclear as to where all the various dependencies come from.

Thanks again for all your work on the plugins and what you have contributed. It is so useful to me and many others!

Plex works pretty well these days, esp with this plugin. The main reason I use it is not so much a TV client replacement but for mobile access to sync and auto transcode to ipads and android phones so the kids can watch it on long trips in the car and I can watch all the shows I never watch with my wife on airplanes.

I used to move the MPEG2 recordings to a flash drive I stick on my laptop to watch, and still do that somewhat, but it takes up a lot of space, and transcoding by hand is pretty cumbersome and takes time I don't usually have.

With plex, I can just select a season or a set of movies, and it'll transcode overnight and sync and then I can hop on the plane and everything is there. And being able to sync remotely when I am in a hotel or remote office with a fast connection to get a recently recorded show is very cool - I never could do that the old way.

There are some issues with syncing on iOS that occasionally bite me, but in general it works pretty well.

I just wish plex could do auto-skip on commercials, or even jump to the end of a commercial, but the +30/-10 skips work ok.

Thx
Mike

I wish the web UI would have skip buttons.

[wildgoose]

Originally Posted by stuckless

1. I tried running 'make' under linux and it fails. Is the Makefile windows only?

Yes I've only tested make on Windows. The main reason for the makefile is to build the cxfreeze python binary on windows (installer, synctool etc). Since mac/linux comes with python this is not needed. Plus copying some files and setting up some folder structure.

2. I see the synctool in the plex folder.. is this a plex integration thing (ie, plex knows what a sync tool is) or is this a commandline tool that needs to run periodically, etc.

PLEX knows nothing about this. It is just a command line tool that talks to both SageTV/PLEX via rest API to try to sync the library.

3. The installer script seems to make some assumptions about the locations of plex, can this be overridden on the commandline?

Try "--plexdir" to see if its what you need. When I wrote the installer I tested a PLEX install on Ubuntu server and set some defaults based on what I saw.

4. The installer seems to prevent running as a user other than plex, can this be overridden on the command line?

When I tested PLEX on ubuntu, I noticed its installed as the PLEX user and files belong to PLEX. If I run the installer as regular user via sudo to write to the plex users location, PLEX have trouble with the files due to permission issues. So I work around the issue by requiring the user to run the installer as PLEX user.

You can bypass the issue by change the code here:
https://github.com/ai7/sagetv-for-pl...nstall.py#L967

I don't know if PLEX is always installed as plex user on Linux or not. If not, then this part would need to be updated/fixed.

For the container setup, it might be easier to simply perform the install steps manually by copying the files to the right location, and configure the sageplex_cfg.json file. The main github page has the manual install instructions (again for Windows ) but it should be simple to figure out the linux equivalent.

[stuckless]

I did try this yesterday, but I didn't have much success I've set this up before so I know I can get it working.

But, I'm wondering... I just added my /var/media/tv to Plex with it's default scanner and metadata lookup, and it was able to accurately find and add all the shows without this plugin. So, I guess I'm wondering why use the plugin? Obviously the syncing tool is important, but even for me, that's less important right now as I just wanted to provide a web interface to my media, for my kids, so they can watch shows/movies on their computers (as they do with netflix right now).

In general I'm not a fan of the plex clients for watching tv, but the web ui is impressive.

[Taddeusz]

Originally Posted by stuckless

I did try this yesterday, but I didn't have much success I've set this up before so I know I can get it working.

But, I'm wondering... I just added my /var/media/tv to Plex with it's default scanner and metadata lookup, and it was able to accurately find and add all the shows without this plugin. So, I guess I'm wondering why use the plugin? Obviously the syncing tool is important, but even for me, that's less important right now as I just wanted to provide a web interface to my media, for my kids, so they can watch shows/movies on their computers (as they do with netflix right now).

In general I'm not a fan of the plex clients for watching tv, but the web ui is impressive.

The default scanner works fine as long as you have the option in SageTV set to put the season and episode numbers in the file name. That also assumes they are accurate. I ran across a recording last week that at the time of recording had no season or episode data. Only after did it get it and so is missing from the file name.

[panteragstk]

For some reason since converting plex to a docker in unraid I can no longer get the BMT Series Agent to work. Log is below.

2016-05-28 15:12:13,248 (2b5a7dca8480) :  INFO (core:349) - Starting framework core - Version: 2.6.3, Build: 5089475 (Fri Apr 22 21:25:00 UTC 2016)
2016-05-28 15:12:13,248 (2b5a7dca8480) :  DEBUG (core:361) - Using the standard policy
2016-05-28 15:12:13,248 (2b5a7dca8480) :  DEBUG (core:450) - Starting runtime component.
2016-05-28 15:12:13,251 (2b5a7dca8480) :  DEBUG (core:450) - Starting caching component.
2016-05-28 15:12:13,251 (2b5a7dca8480) :  DEBUG (core:450) - Starting data component.
2016-05-28 15:12:13,251 (2b5a7dca8480) :  DEBUG (core:450) - Starting networking component.
2016-05-28 15:12:13,251 (2b5a7dca8480) :  DEBUG (networking:286) - No cookie jar found
2016-05-28 15:12:13,252 (2b5a7dca8480) :  DEBUG (networking:450) - Setting the default network timeout to 20.0
2016-05-28 15:12:13,252 (2b5a7dca8480) :  DEBUG (core:450) - Starting localization component.
2016-05-28 15:12:13,253 (2b5a7dca8480) :  INFO (localization:409) - Setting the default locale to en-us
2016-05-28 15:12:13,253 (2b5a7dca8480) :  DEBUG (core:450) - Starting messaging component.
2016-05-28 15:12:13,253 (2b5a7dca8480) :  DEBUG (core:450) - Starting debugging component.
2016-05-28 15:12:13,254 (2b5a7dca8480) :  DEBUG (core:450) - Starting services component.
2016-05-28 15:12:13,254 (2b5a7dca8480) :  DEBUG (core:450) - Starting myplex component.
2016-05-28 15:12:13,255 (2b5a8607a700) :  DEBUG (networking:166) - Requesting 'http://127.0.0.1:32400/system/messaging/clear_events/com.plexapp.agents.bmtagenttvshows'
2016-05-28 15:12:13,255 (2b5a7dca8480) :  DEBUG (core:450) - Starting notifications component.
2016-05-28 15:12:13,406 (2b5a7dca8480) :  DEBUG (accessor:68) - Creating a new model access point for provider com.plexapp.agents.bmtagenttvshows in namespace 'metadata'
2016-05-28 15:12:13,412 (2b5a7dca8480) :  DEBUG (networking:166) - Requesting 'http://127.0.0.1:32400/:/plugins/com.plexapp.system/resourceHashes'
2016-05-28 15:12:17,645 (2b5a8607a700) :  DEBUG (services:265) - Plug-in is not daemonized - loading services from system
2016-05-28 15:12:17,646 (2b5a7dca8480) :  DEBUG (runtime:1117) - Created a thread named 'load_all_services'
2016-05-28 15:12:17,648 (2b5a7dca8480) :  DEBUG (runtime:1117) - Created a thread named 'get_server_info'
2016-05-28 15:12:17,650 (2b5a8607a700) :  DEBUG (networking:166) - Requesting 'http://127.0.0.1:32400/:/plugins/com.plexapp.system/messaging/function/X0J1bmRsZVNlcnZpY2U6QWxsU2VydmljZXM_/Y2VyZWFsMQoxCmxpc3QKMApyMAo_/Y2VyZWFsMQoxCmRpY3QKMApyMAo_'
2016-05-28 15:12:17,652 (2b5a8647c700) :  DEBUG (networking:166) - Requesting 'http://127.0.0.1:32400'
2016-05-28 15:12:17,652 (2b5a7dca8480) :  DEBUG (core:150) - Finished starting framework core
2016-05-28 15:12:17,654 (2b5a7dca8480) :  DEBUG (core:560) - Loading plug-in code
2016-05-28 15:12:17,818 (2b5a7dca8480) :  DEBUG (core:566) - Finished loading plug-in code
2016-05-28 15:12:17,819 (2b5a8647c700) :  DEBUG (core:538) - Machine identifier is c427c0270572c0d27ac36625cf4812d8fd4953c0
2016-05-28 15:12:17,820 (2b5a8647c700) :  DEBUG (core:539) - Server version is 0.9.16.6.1993-5089475
2016-05-28 15:12:17,820 (2b5a7dca8480) :  CRITICAL (core:615) - Exception starting plug-in (most recent call last):
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-5089475/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/core.py", line 608, in start
    self.sandbox.execute(self.init_code)
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-5089475/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/code/sandbox.py", line 256, in execute
    exec(code) in self.environment
  File "/config/Library/Application Support/Plex Media Server/Plug-ins/BMTAgentTVShows.bundle/Contents/Code/__init__.py", line 19, in <module>
    import plexlog              # log wrapper
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-5089475/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/code/sandbox.py", line 345, in __import__
    raise e
ImportError: No module named plexlog

2016-05-28 15:12:17,911 (2b5a8607a700) :  DEBUG (services:362) - Loaded services
2016-05-28 15:12:17,921 (2b5a8627b700) :  DEBUG (services:438) - No shared code to load

[wildgoose]

It's not finding plexlog, this is plexlog.py in the folder Plug-ins/BMTAgentTVShows.bundle/Contents/Code/.

Based on the error message it should be in

/config/Library/Application Support/Plex Media Server/Plug-ins/BMTAgentTVShows.bundle/Contents/Code/

Do you see it in this folder? It should contain the following additional files besides __init__.py

config.py
plexapi.py
plexlog.py
sagex.py
spvideo.py

[reccitron]

For the integration between SageTv and Plex for recordings, do both Sage and Plex have to be installed on the same server?

[wildgoose]

Originally Posted by reccitron

For the integration between SageTv and Plex for recordings, do both Sage and Plex have to be installed on the same server?

They don't need to be on the same server.

[tvmaster2]

Originally Posted by wildgoose

In the PLEX web UI, there should be an option to "clean bundle", and "empty trash" (It should be on the context menu of the library in the main page...)

Check out this link for some info:
https://support.plex.tv/hc/en-us/art...ibrary-Actions

Hi - I see that you may now be the go-to person of the original SageTV Movie Scanner, which I use with Plex Media Server on a PC.
There's a log file in the appdata area called sageplex_scanner.log which has grown to 15gb's. Does that seem correct, or if not, can it be deleted and it will just start regenerating again?
thanks

[Fuzzy]

Originally Posted by tvmaster2

Hi - I see that you may now be the go-to person of the original SageTV Movie Scanner, which I use with Plex Media Server on a PC.
There's a log file in the appdata area called sageplex_scanner.log which has grown to 15gb's. Does that seem correct, or if not, can it be deleted and it will just start regenerating again?
thanks

Log files can almost always be deleted. That said, it wind bee with finding it why it's so big, but most text editors will choke on that file. I'd delete or move it, then pay close attention to it over a few days to see what is being written to it so much.

[tvmaster2]

Originally Posted by Fuzzy

Log files can almost always be deleted. That said, it wind bee with finding it why it's so big, but most text editors will choke on that file. I'd delete or move it, then pay close attention to it over a few days to see what is being written to it so much.

yeah, it's a real mystery how it got that big, since all the Plex generated logs are a reasonable size. I'm assuming there may be a bug, which is why I brought it up here. At worst, I can always uninstall the SageTV Scanners and just use the Plex ones. For some reason, over a year ago, the SageTV Scanner was better at finding certain artwork, so maybe Plex has gotten better in that regard...
thanks

[wildgoose]

Originally Posted by tvmaster2

Hi - I see that you may now be the go-to person of the original SageTV Movie Scanner, which I use with Plex Media Server on a PC.
There's a log file in the appdata area called sageplex_scanner.log which has grown to 15gb's. Does that seem correct, or if not, can it be deleted and it will just start regenerating again?
thanks

No this doesn't look right. Yes you can safely delete it. This is the scanner log.

I have code that suppose to rotate the log at 4MB, and keep at most 5 logs, so around 20mb total. Looks like its not working for some reason. Any chance you are running an older version of the plugin?

https://github.com/ai7/sagetv-for-pl...plexlog.py#L84

Search for this file on your system, open it and see if you can find the string "maxsize" in there. According to the history the rotating feature was added around 2015-04-10.

Anyone else having similar issue? The sageplex_scanner.log should be less than 4MB in size.