Plugin: Schedules Direct EPG (not for v9)

[Slugger]

So as most of you may know, all personal projects tend to go on hold once the nice weather rolls around and although rather late this year, we're on the verge of spring breaking out up here in southern Ontario. My baseball season starts in a week and that along with other things chews up most of my summer. Once that starts up, my techy projects go on hiatus (or at least a very, very reduced pace) until the fall. Don't worry, major bug fixes on the EPG plugin are always on the table and I'll address those as needed (especially because I use this feed myself for all my systems).

I was thinking about adding support for Tribune's fanart into the EPG plugin, but am wondering if anyone would use it? Personally, I use BMT to handle all my fanart needs so I'm unlikely to use it myself, but I also see some benefits to it. For example, it appears Tribune has images for things that BMT does not have access to (i.e. sports). Not sure if that's something people are interested in?

Looking at the raw data, it seems most shows provide a single image so it wouldn't be as extensive as thetvdb's collection or whatever it is BMT is pulling from. I haven't actually downloaded any of the images from Tribune to look at them, but they seem to be 240(w)x360px or 360(w)x270px so I'm guessing that'd be bluray covers or something?

If no one's interested in this fanart then I won't bother with the effort. If that's the case, is there anything missing in this plugin that needs addressing before the fall? If not, I may just start my summer vacation early this year.

[JerryB]

FYI, by this morning program listings had gradually filled in for all channels in my lineups. I updated the plugin and forced a refresh of guide data. There were the expected info and warning messages in the log files but no major errors. I have no idea why it took so long to process the data and fill in the program listings but the error I was experiencing yesterday appears to be an artifact of deleting all my video sources and then rebuilding them from scratch.

With respect to the Tribune fanart question, the main problem I currently have with the SD plugin compared to the regular SageTV guide data is caused by the difference way the 2 sources process IDs (ie SageTV inappropriately strips leading zeros and SDepg doesn't). As a result, when I view program listings, fanart is automatically retrieved and displayed with the listing when I use the native SageTV data but not when I use the SD data. Please be aware that this issue only affects SageTV views of program listings/guide data. Once I record a show, BMT kicks in and retrieves the appropriate fanart. The issue is that BMT doesn't retrieve fanart until after the show is recorded so views such as "upcoming movies" or "Malores menus" don't have fanart when I use the SD plugin. If adding Tribune fanart to your plugin is something that is easy to do and would resolve this issue, then I'd definitely use it. If adding Tribune fanart would only affect recorded shows, then I have no use for it since I already get this fanart through BMT.

[Slugger]

Interesting... see my original thought on this was to only download fanart for recorded programs. I don't think you'd actually want to download every piece of fanart Tribune has available for every show in your epg listings? I have no idea just how much space that would require, but I'm thinking it's on the order of 10s of GBs. And then I'd have to be concerned with how that affects the plugin's speed of access to the data in the local zip file I generate with all of the data (epg data, tv logos, and now fanart).

I'd only download the fanart and make it available thru some kind of API; someone/something else would still have to load it whereever it belongs. Like BMT uses Phoenix APIs to read fanart; other plugins probably read from other sources.

Anyway, we'll let this sit for a few days and see if anyone else has thoughts before deciding.

[iumgrad]

Slugger, I upgraded this morning to the .63 upgrade. After upgrade Sage reported that the plugin was missing the "sdepg.jar" file. Investigating I see a file in the directory with this name "sdepg.jar.162". It has a time stamp of 4/27/2014 3:02 PM. Will changing this "sdepg.jar" solve the error report and allow everything to function?

[Slugger]

Most likely that will work, but it assumes that it's the latest jar file sitting there. What I suggest is:

Delete that file, download this and put the jar inside that zip into your JARs folder. Also, check your JARs folder:

dir sdjson*.jar

Ensure there's only one jar file listed: sdjson-api-20131021.1.jar

If not, remove any extras.

The Sage plugin manager screwed up during the upgrade -- I've seen this happen periodically. So you need to make sure the proper versions of the jars are now installed.

[JerryB]

Originally Posted by Slugger

Interesting... see my original thought on this was to only download fanart for recorded programs. I don't think you'd actually want to download every piece of fanart Tribune has available for every show in your epg listings? I have no idea just how much space that would require, but I'm thinking it's on the order of 10s of GBs. And then I'd have to be concerned with how that affects the plugin's speed of access to the data in the local zip file I generate with all of the data (epg data, tv logos, and now fanart).

That's correct. I definitely don't want to download fanart for every program. With the SageTV epg service, fanart is only downloaded for the program that is highlighted/selected in the guide, not all the programs in the guide. This occurs when you select the program, not when the guide data is downloaded, and the fanart that is retrieved appears to then be cached somewhere so that it's immediately available if you return to that program in the guide. The Default SageTV interface and the various interface mods/menues (Malore's menues, Gemstone, and Phoenix) all have the ability to quickly obtain and display the fanart for the selected show in the program guide/menu.

I believe that this automatic retrieval of fanart for listings in the program guide is built into SageTV but requires the default stripping of leading zero's in the program ID that occurs when you use the default SageTV guide service. Since these zero's are not stripped by the SD plugin, SageTV and associated interface mods don't find a match when they attempt to retrieve fanart for the selected item in the program guide/menu.

I previously opened a Gemstone Issue which goes into this in more detail if you'd like to look at it.

http://code.google.com/p/gemstone/is...=&sort=&id=100

Unfortunately, the issue was never resolved and is still open. I was hoping the availability of Tribune Fanart might provide a simple way to resolve it. If it doesn't, that's fine. I'll just continue with things the way they are now.

[iumgrad]

Slugger, I did as you suggested (download the sdepg.jar link you sent) and made sure that the sdjson-api-0.20131021.1.jar was present. Two things happened, I stop sage and added the file. When I restarted the sdepg.jar file was removed. I added back.
BTW each time I restart sage removes the .jar file
Second I checked status of schedules direct using web interface. I was given the following error message:
HTTP ERROR 500

Problem accessing /sage/sdjson/index.gsp. Reason:
Failed to parse template script (your template may contain an error or be trying to use expressions not currently supported): startup failed:
SimpleTemplateScript4.groovy: 94: unable to resolve class sagex.epg.schedulesdirect.io.EpgDownloader
@ line 94, column 2.
import sagex.epg.schedulesdirect.io.EpgDownloader
^

SimpleTemplateScript4.groovy: 95: unable to resolve class sagex.epg.schedulesdirect.plugin.Plugin
@ line 95, column 2.
import sagex.epg.schedulesdirect.plugin.Plugin
^

2 errors



Caused by:
groovy.lang.GroovyRuntimeException: Failed to parse template script (your template may contain an error or be trying to use expressions not currently supported): startup failed:
SimpleTemplateScript4.groovy: 94: unable to resolve class sagex.epg.schedulesdirect.io.EpgDownloader
@ line 94, column 2.
import sagex.epg.schedulesdirect.io.EpgDownloader
^

SimpleTemplateScript4.groovy: 95: unable to resolve class sagex.epg.schedulesdirect.plugin.Plugin
@ line 95, column 2.
import sagex.epg.schedulesdirect.plugin.Plugin
^

2 errors

at net.sf.sageplugins.webserver.groovy.templates.IncludeTemplateEngine.createTemplate(IncludeTemplateEngine.java:76)
at groovy.servlet.TemplateServlet.createAndStoreTemplate(TemplateServlet.java:265)
at groovy.servlet.TemplateServlet.getTemplate(TemplateServlet.java:320)
at net.sf.sageplugins.webserver.groovy.servlets.SageTemplateServlet.service(SageTemplateServlet.java:66)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)

[Slugger]

@JerryB: Actually, looking at the Sage APIs for this, it seems that I may be able to do this. But it requires that I can serve the files directly from SD's servers, which I don't think they will appreciate. Seems the API takes a URL for the image so in theory this should work and it should work on demand just like the core EPG service does. I'll talk to SD in the next day or two and see what they say.

[Slugger]

Originally Posted by iumgrad

Slugger, I did as you suggested (download the sdepg.jar link you sent) and made sure that the sdjson-api-0.20131021.1.jar was present. Two things happened, I stop sage and added the file. When I restarted the sdepg.jar file was removed. I added back. BTW each time I restart sage removes the .jar file

Something is wrong with the Sage plugin manager. I suggest uninstalling the plugin then reinstalling it, hopefully that prevents Sage from deleting the jar at startup.

Copying the jar back in while Sage is running won't work; Sage won't recognize it until it's restarted.

Second I checked status of schedules direct using web interface. I was given the following error message:
HTTP ERROR 500

Problem accessing /sage/sdjson/index.gsp. Reason:
<snip>

Yeah, so this is expected as long as Sage keeps deleting the jar file. All the classes those web pages are looking for are inside that jar. Fix the first problem, you fix this one as well.

[JerryB]

Originally Posted by Slugger

@JerryB: Actually, looking at the Sage APIs for this, it seems that I may be able to do this. But it requires that I can serve the files directly from SD's servers, which I don't think they will appreciate. Seems the API takes a URL for the image so in theory this should work and it should work on demand just like the core EPG service does. I'll talk to SD in the next day or two and see what they say.

Thank you. It'll be interesting to see what SD says to this but I certainly understand if they think it might overload their servers. As I understand the comments when I reported the issue to the Gemstone developers, the default SageTV epg provides a link to the fanart image on a zap2it server ("in theory it should be returning a zap2it path to an image [at least that is what returns in the default EPG provider]"). If SD doesn't permits the fanart images to be served from their services, do you think it would it be possible for your plugin to use the SD data to create and provide the same URL to the fanart image on a zap2it server that is provided by the default SageTV epg?

[iumgrad]

Originally Posted by Slugger

Something is wrong with the Sage plugin manager. I suggest uninstalling the plugin then reinstalling it, hopefully that prevents Sage from deleting the jar at startup.

Copying the jar back in while Sage is running won't work; Sage won't recognize it until it's restarted.




Yeah, so this is expected as long as Sage keeps deleting the jar file. All the classes those web pages are looking for are inside that jar. Fix the first problem, you fix this one as well.

Ok, ill try that when I get home from work, hopefully it works

[iumgrad]

Originally Posted by Slugger

Something is wrong with the Sage plugin manager. I suggest uninstalling the plugin then reinstalling it, hopefully that prevents Sage from deleting the jar at startup.

Copying the jar back in while Sage is running won't work; Sage won't recognize it until it's restarted.




Yeah, so this is expected as long as Sage keeps deleting the jar file. All the classes those web pages are looking for are inside that jar. Fix the first problem, you fix this one as well.

Sage is not showing an "uninstall" option for this plugin. Do I need to remove video sources first, for option to become available?

[Slugger]

Originally Posted by iumgrad

Sage is not showing an "uninstall" option for this plugin. Do I need to remove video sources first, for option to become available?

Nope. There is something really messed up with the state of the Sage plugin manager on your system. It's really weird because apparently Sage knows it's still installed because it's listed as installed, but won't give you the option to uninstall? Are you sure you clicked on it from the Installed Plugins list and not the All Available list? I believe the latter won't give the option to uninstall, but the former will. If you were on the Installed list then I can't offer much more than submitting a support request to Sage and crossing your fingers that they still respond to those requests (or restoring your SageTV directory from a backup, if you have one).

[iumgrad]

Originally Posted by Slugger

Nope. There is something really messed up with the state of the Sage plugin manager on your system. It's really weird because apparently Sage knows it's still installed because it's listed as installed, but won't give you the option to uninstall? Are you sure you clicked on it from the Installed Plugins list and not the All Available list? I believe the latter won't give the option to uninstall, but the former will. If you were on the Installed list then I can't offer much more than submitting a support request to Sage and crossing your fingers that they still respond to those requests (or restoring your SageTV directory from a backup, if you have one).

Ok, i'll do a restore from 5:00 AM this morning (before I did update). Should I risk trying upgrade again? Can my system continue functioning without upgrades?

[Slugger]

Using my epg plugin? Yes, but only for a limited time. It will continue to work until Schedules Direct decides to shutdown the old web service that older versions of the plugin talk to. Once SD does that, you will require 0.0.6.x in order to talk to the new SD web service servers. I do not know when they will do that, but I would assume it will be sooner rather than later (like less than a month for sure).

I would go ahead and reattempt the upgrade, if the same thing happens with the jar files then submit a service request with Sage and hope for the best. It's been my experience that sometimes the plugin manager just does the wrong/weird things and if you rerun it again, it works right. YMMV. I think you may want to ping stuckless, too. I think he has some kind of tool that can fix plugin manager screw ups. Not sure, never used it. Again, YMMV.

[gdippel]

Originally Posted by gdippel

There are several Broadcast Network sub-channels that are not populating with guide data, e.g. 2-2, 5-2, 9-2. They did have data before the epg upgrade. Should I send a email to Schedules Direct or is this a plugin issue?

I received a reply from SD that I should refresh the epg and after refreshing my epg, I now have data for the sub channels. SD sent the following:

You may want to talk to slugger about that so that the grabber either auto-refreshes the lineup, or at least tells you that the lineup that you have and the one on the server have different "last modified" dates (because that's in the status that's sent by the server each time you
connect)

I'm not sure I'm understanding the issue. BTW, I once again had to delete the epg and a tmp file before the final successful epg refresh.

[Slugger]

Originally Posted by gdippel

I received a reply from SD that I should refresh the epg and after refreshing my epg, I now have data for the sub channels. SD sent the following:



I'm not sure I'm understanding the issue. BTW, I once again had to delete the epg and a tmp file before the final successful epg refresh.

With the same access denied exception? If so, that's the problem. I can't load the epg data into wiz.bin (or save the epg data from SD to disk) if I'm unable to write/read to the epg file.

We're ok on the schedule/channel refreshes, I always pull the latest data from SD on every epg pull. The issue is that sometimes the sdjson.epg file is not accessible, this is what you need to address. Next time this happens, look at the details of the file (owner/ACL/etc.) before deleting it and try to figure out why the grabber process (executed as the same user as SageTV is running as) can't access the file. This is the underlying problem.

[Slugger]

Another possibility is that something else has the file locked. Do you have a zip viewer of the file open still from before? If not, you might want to grab process explorer and see if anything is keeping the file open. After an epg update completes, nothing should have that file open, if SageTV.exe does then that's my plugin and I need to go searching (which will be tough especially since no one else is seeing this). If it's another program (like a zip viewer, etc.) then that's easy to fix. Just close the zip viewer.

[gdippel]

I checked the log file and it does seem to be a permission issue. I log into my windows 7 sage server as a user as I run sage as a service under that user in order to connect to my media server via unc. Could that be the issue? I had no problem before the upgrade. I've attached the portion of the log from today.

[iumgrad]

Originally Posted by Slugger

Using my epg plugin? Yes, but only for a limited time. It will continue to work until Schedules Direct decides to shutdown the old web service that older versions of the plugin talk to. Once SD does that, you will require 0.0.6.x in order to talk to the new SD web service servers. I do not know when they will do that, but I would assume it will be sooner rather than later (like less than a month for sure).

I would go ahead and reattempt the upgrade, if the same thing happens with the jar files then submit a service request with Sage and hope for the best. It's been my experience that sometimes the plugin manager just does the wrong/weird things and if you rerun it again, it works right. YMMV. I think you may want to ping stuckless, too. I think he has some kind of tool that can fix plugin manager screw ups. Not sure, never used it. Again, YMMV.

ok so I bit the bullet and reinstalled sage and copied over wiz.bin. Then reinstalled plugins. When I use the web interface, I get the following error.?? I did a complete reinstall (uninstalled and then deleted sage directory)


Problem accessing /sage/sdjson/index.gsp. Reason:
Token: A JSONObject text must begin with '{' at 1 [character 2 line 1]


Caused by:
org.schedulesdirect.api.exception.JsonEncodingException: Token: A JSONObject text must begin with '{' at 1 [character 2 line 1]
at org.schedulesdirect.api.NetworkEpgClient.authorize(NetworkEpgClient.java:232)
at org.schedulesdirect.api.NetworkEpgClient.<init>(NetworkEpgClient.java:202)
at org.schedulesdirect.api.NetworkEpgClient.<init>(NetworkEpgClient.java:174)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
at SimpleTemplateScript1.run(SimpleTemplateScript1.groovy:100)
at net.sf.sageplugins.webserver.groovy.templates.IncludeTemplateEngine$SimpleTemplate$1.writeTo(IncludeTemplateEngine.java:120)
at net.sf.sageplugins.webserver.groovy.servlets.SageTemplateServlet.service(SageTemplateServlet.java:99)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
Caused by: org.json.JSONException: A JSONObject text must begin with '{' at 1 [character 2 line 1]
at org.json.JSONTokener.syntaxError(JSONTokener.java:433)
at org.json.JSONObject.<init>(JSONObject.java:194)
at org.json.JSONObject.<init>(JSONObject.java:321)
at org.schedulesdirect.api.NetworkEpgClient.authorize(NetworkEpgClient.java:230)
... 31 more


Caused by:
org.json.JSONException: A JSONObject text must begin with '{' at 1 [character 2 line 1]
at org.json.JSONTokener.syntaxError(JSONTokener.java:433)
at org.json.JSONObject.<init>(JSONObject.java:194)
at org.json.JSONObject.<init>(JSONObject.java:321)
at org.schedulesdirect.api.NetworkEpgClient.authorize(NetworkEpgClient.java:230)
at org.schedulesdirect.api.NetworkEpgClient.<init>(NetworkEpgClient.java:202)
at org.schedulesdirect.api.NetworkEpgClient.<init>(NetworkEpgClient.java:174)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
at SimpleTemplateScript1.run(SimpleTemplateScript1.groovy:100)
at net.sf.sageplugins.webserver.groovy.templates.IncludeTemplateEngine$SimpleTemplate$1.writeTo(IncludeTemplateEngine.java:120)
at net.sf.sageplugins.webserver.groovy.servlets.SageTemplateServlet.service(SageTemplateServlet.java:99)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)