Plugin: Schedules Direct EPG (not for v9)

[Slugger]

There's either a race condition between the watchdog thread and the main plugin thread or the cache file is corrupted.

Regardless of what that watchdog is doing, the core should be requesting an epg update once every 24 hours. Is there a record of the grabber being called within the last 24 hours in sdepg.log? The webui's home page would tell you the last time the core called for an epg update. Then check the sdepg.log file for entries around the same time. The grabber should have been called at that time. Was it? If not, errors?

If the grabber was not, called at the expected time then have a look at sagetv_0.txt, it should tell us why the core's request for an epg update is failing.

Let's see those logs first. Ultimately, deleting the sdjson.epg file and forcing an epg update should fix everything, but the underlying cause is probably a bug worth fixing if we can find some logs to tell me what is causing the issue.

Another interesting data point right now would be: are you able to actually delete the sdjson.epg file while Sage is still running? You should be able to, if you can't (file is locked error while trying) then that tells me something is holding that file open that shouldn't be, which would explain the watchdog error. First, check the logs and let's see if that tells us anything before deleting the cache file.

[Phydeaux]

When I look through the sdepg.log all I see is iterations of the same types of entries previously posted. They seem to kick off every hour.


From the web UI:

Schedules Direct Status

User: Phydeaux
Account Expires: Tue Mar 15 12:52:33 CST 2016
Last server EPG update: Mon Sep 07 01:00:54 CST 2015
Server status: Online as of Mon Mar 23 12:47:00 CST 2015 [msg: Lineup responses updated. See http://forums.schedulesdirect.org/vi...&t=2645&p=8379 for details.]
Maximum lineups: 4
Configured lineups: 1
Schedules Direct Messages

No messages.



I don't find a sagetv_0.txt file, where should it be located? The attached log only includes the last two days, I had to remove quite a bit older stuff to get it down to size for upload.

RLW

[Slugger]

The sagetv_0.txt file is in the SageTV\ folder. If it's not there then you need to turn on Sage's debug logging. I think you can turn it on without stopping sage from the Sage settings screens, but don't remember. After you've enabled it, force an epg update and let's see what that log has to say.

There are no signs of the grabber being executed by the plugin, which is why you've run out of data. So the core is failing to trigger it for some reason and it's probably related to why the watchdog thread is also failing.

Again, deleting sdjson.epg and forcing an epg update should fix everything up, but if you indulge me, I think we can maybe get an answer as to the underlying issue, which I can then hopefully fix. So before deleting the .epg file, please try to enable Sage debug logging and force the epg update.

[Phydeaux]

I'm happy to play along if it's of value to you Slugger. So I toggled SageTV debugging on but when I did it prompted me saying it would need to be restarted. I chose OK instead of Restart Now and then checked to see if there was a sagetv_0.txt created - nope. So I had to start and stop the SageTV service and the file was immediately created.

I added a single channel via Setup Video Sources, changing nothing else. I see the following in the sdepg.log so far (first information in the log other than lots of the stuff I have already posted):

2015-09-07 14:21:04,854 INFO  [Plugin]: EPG watchdog started!
2015-09-07 14:22:28,056 INFO  [ProgramGenerator]: Processed show generators in 165ms
2015-09-07 14:22:28,376 INFO  [EpgDownloader]: Backed up local cache file [plugins\sdepg\sdjson.epg]
2015-09-07 14:22:28,587 INFO  [EpgDownloader]: Requesting 20 channels
2015-09-07 14:22:28,638 INFO  [EpgDownloader]: JVM options: [-Xmx512m, -Dsdjson.fs.capture="C:\Program Files (x86)\SageTV\SageTV\plugins\sdepg\capture\grabber"]
2015-09-07 14:22:28,646 INFO  [EpgDownloader]: [C:\Program Files (x86)\SageTV\SageTV\plugins\sdepg\tools\grabber\bin\sdjson-grabber.bat, --username, Phydeaux, --password, *****, --user-agent, sagetv-sdepg/0.0.8.1 (Windows 7 x86 6.1), --max-threads, 200, --url, https://json.schedulesdirect.org, --grabber-log-level, INFO, grab, --target, C:\Program Files (x86)\SageTV\SageTV\plugins\sdepg\sdjson.epg, --max-prog-chunk, 5000, --max-sched-chunk, 250, --stations, C:\Program Files (x86)\SageTV\SageTV\plugins\sdepg\stations.txt]
2015-09-07 14:22:36,254 INFO  [EPGImportPluginSchedulesDirect]: Performed EPG download in 8177ms
2015-09-07 14:22:41,666 INFO  [ChannelGenerator]: Processed channel generators in 15ms
2015-09-07 14:22:41,673 INFO  [EPGImportPluginSchedulesDirect]: Performed channel processing in 5418ms
2015-09-07 14:22:47,512 INFO  [EPGImportPluginSchedulesDirect]: Performed lineup map configuration in 5839ms
2015-09-07 14:22:54,876 WARN  [SageProgram]: Unhandled credit type encountered: COSTUME_DESIGNER
2015-09-07 14:22:54,878 WARN  [SageProgram]: Unhandled credit type encountered: CASTING
2015-09-07 14:22:54,878 WARN  [SageProgram]: Unhandled credit type encountered: PRODUCTION_DESIGNER
2015-09-07 14:22:54,878 WARN  [SageProgram]: Unhandled credit type encountered: SET_DECORATION
2015-09-07 14:22:54,879 WARN  [SageProgram]: Unhandled credit type encountered: ORIGINAL_MUSIC
2015-09-07 14:22:54,879 WARN  [SageProgram]: Unhandled credit type encountered: FILM_EDITOR
2015-09-07 14:22:54,879 WARN  [SageProgram]: Unhandled credit type encountered: ART_DIRECTION
2015-09-07 14:22:54,879 WARN  [SageProgram]: Unhandled credit type encountered: CINEMATOGRAPHER
2015-09-07 14:22:55,226 WARN  [SageProgram]: Unhandled credit type encountered: UNKNOWN
2015-09-07 14:22:58,180 WARN  [SageProgram]: Unhandled credit type encountered: PRODUCTION_MANAGER
2015-09-07 14:23:09,938 WARN  [SageProgram]: Unhandled credit type encountered: ASSISTANT_DIRECTOR
2015-09-07 14:23:12,839 INFO  [EPGImportPluginSchedulesDirect]: Performed EPG data load in 25327ms
2015-09-07 14:23:12,861 INFO  [AiringGenerator]: Processed airing generators in 12ms
2015-09-07 14:23:12,910 INFO  [SeriesInfoLoader]: Generated series info entries for 0 objects in 17ms!

I'll do nothing further until I hear back from you.

RLW

[Slugger]

Yeah, so if you check your epg, it should have data. That log is the log of a clean epg run. Whatever was causing the problem went away with the Sage restart. So that kind of leads me to believe it was a race condition within my plugin threads, but that's only a guess. The best we can do -- assuming you now have epg data -- is leave the logging on as is and if it happens again hopefully we can get more answers.

[jamesdisco]

Does using this plugin get the channel tuning data into sage so that you dont have to do a channel scan.
Also does the guide data for season and episodes get imported in. I use lm gestions xmltv importer at the moment and the season/episode data can be put into the program title but not into the proper season/episode header.

[Slugger]

Originally Posted by jamesdisco

Does using this plugin get the channel tuning data into sage so that you dont have to do a channel scan.

Nope.

Also does the guide data for season and episodes get imported in. I use lm gestions xmltv importer at the moment and the season/episode data can be put into the program title but not into the proper season/episode header.

Yes, the SD data include S/E info and uses the proper fields within wiz.bin.

[rjleurck]

This line is triggering an error after the json epg file is downloaded

clnt = new ZipEpgClient(EPG_SRC)

in addChannel call .

See stacktrace

INFO - Created '/home/bob/sage/sagetv-server_9.0.0_amd64/plugins/sdepg/sdjson.epg' successfully! [27742ms]

2015-09-27 12:50:36,197 DEBUG [EpgDownloader]: stderr:

2015-09-27 12:50:36,197 INFO [EPGImportPluginSchedulesDirect]: Performed EPG download in 31860ms
2015-09-27 12:50:36,198 ERROR [EPGImportPluginSchedulesDirect]: sdjson error
java.nio.file.ClosedFileSystemException
at com.sun.nio.zipfs.ZipFileSystem.ensureOpen(ZipFileSystem.java:1084)
at com.sun.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:316)
at com.sun.nio.zipfs.ZipPath.checkAccess(ZipPath.java:767)
at com.sun.nio.zipfs.ZipFileSystemProvider.checkAccess(ZipFileSystemProvider.java:187)
at java.nio.file.Files.exists(Files.java:2385)
at org.schedulesdirect.api.ZipEpgClient.<init>(ZipEpgClient.java:153)
at org.schedulesdirect.api.ZipEpgClient.<init>(ZipEpgClient.java:212)
at sun.reflect.GeneratedConstructorAccessor53.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
at sagex.epg.schedulesdirect.EPGImportPluginSchedulesDirect.addChannels(EPGImportPluginSchedulesDirect.groovy:559)
at sagex.epg.schedulesdirect.EPGImportPluginSchedulesDirect.doUpdate(EPGImportPluginSchedulesDirect.groovy:358)
at sagex.epg.schedulesdirect.EPGImportPluginSchedulesDirect.updateGuide(EPGImportPluginSchedulesDirect.groovy:207)
at sage.EPG.pluginExtractGuide(EPG.java:1597)
at sage.WarlockRipper.extractGuide(WarlockRipper.java:316)
at sage.EPGDataSource.expand(EPGDataSource.java:707)
at sage.EPG.run(EPG.java:766)
at java.lang.Thread.run(Thread.java:745)
2015-09-27 12:50:36,199 DEBUG [ChannelGenerator]: Channel generator processing has started!

[Slugger]

I'll need to see the full sdepg.log file. Also, in the plugin settings, do you have the "update epg all day" setting on or off? A v7 user reported a similar error that was never tracked down. I had a hunch that there may be a race condition and this result tends to suggest I'm right, depending on the value of that setting.

[remmuh1]

I built a new sagetv 7.1.9 box.
At first I tried using the mc2xml epg but couldn't get consistent listings. Then I found information about the SD epg.
I have removed mc2xml epg & restarted computer.
Created a trial account on SD (2 days ago). Created a lineup for OTA in my area.
I've installed the SD EPG Plugin from available plugins & followed this guide to configure. http://sourceforge.net/p/sagetv-addo...pg/UsersGuide/
Unable to get consistent lineups.

Followed through this thread and saw a post about problems when using mixed case letters in the SD password. I had done that, so I went back & changed my SD password, then also changed it in the configuration of the plugin in sagetv. I saw another post about stopping sagetv, deleting files in the sdepg folder, the starting sagetv again to try & clear a possible config corruption. No change with that.

Looking through my sdepg.log file I get a Watchdog error which is similar to that discussed in post #778 of the thread, but not exactly the same. My sdepg.log file leads me to believe that I am missing a component "sdjson.epg" based on the part of the error line that points me to that withing my directory structure. I've looked all around & I can't find anything about that as a piece that I need to get & install. So now I'm sort of confused & stuck, with the SD trial timer clicking.

I have attached the latest version of my sdepg log, after the last time I'd removed it & restarted sagetv.

Any guidance would be greatly appreciated and sorry for the long winded post!!

[Slugger]

Looks like the grabber has never run successfully to create the epg file. Based on the contents of sdjson-api.log, I'd guess your user/password are wrong in the plugin settings. Posts to the web service are returning 400 so that usually means bad id/pwd. If you've confirmed that then you'll need to open a ticket with Schedules Direct to find out why you can't login to the web service.

[remmuh1]

Slugger, I opened a ticket with SD & if/when they get back to me I'll try it & update with any information.

Thank you again for the suggestions!

[remmuh1]

Here is what I got from SD regarding the ticket that I opened & my response to them.

SD to Me:
You haven't added any lineups to your JSON account; in the Sage grabber there is a function that allows you to add lineups.

The lineup that you added on the website is for our XML service, which Sage doesn't use.


Me back to SD:
Is there documentation on what needs to be done?

SD back to Me:
I don't run Sage, so you'll need to ask in the Sage forums.

Me to SageTV Forum:
So that makes it sound like configs that I'd make within SageTV & not within SD, even though I am getting the epg lineups from SD (bit of a head scratcher but I'll go with it).

So does any of this make since & if so, is there any documentation that I can use to do what SD support suggests?

Thank you!

[Slugger]

What they say is true, but I'm also seeing login failures in the logs. Regardless, you have to register your lineups with the json service anyway. If you have the webui plugin installed, use it -- there will be a Schedules Direct menu.

If you don't have the webui plugin, you'll have to do it with the command line tool:

https://github.com/Slugger/sdepg/wik...and-Line-Usage

That wiki is slightly out of date:

Instead of the command: java -jar sdjson.jar ...

When you go to the tools dir, there should be a bin dir. Go into bin and just run the batch file. All the command line options, etc. are the exact same as documented, but just run the batch file instead of the java command line.

[rnadeau]

Sage 9 web UI has given some users problems when used with too new a version of Java...ie java 8. Try removing java 8 and install java 7.

[rdefino]

Hi there,

I'm seeing this for some reason, looks like my SD login is not working. So I went the SD site and changed my password. But the plugin keeps telling me it and incorrect login. Any thoguhts? thanks


2016-08-31 16:27:46,046 INFO [Plugin]: EPG watchdog started!
2016-08-31 16:27:47,419 ERROR [Plugin]: Error in Watchdog
org.schedulesdirect.api.exception.InvalidCredentialsException: Invalid username or token has expired.
at org.schedulesdirect.api.NetworkEpgClient.authorize(NetworkEpgClient.java:242)
at org.schedulesdirect.api.NetworkEpgClient.<init>(NetworkEpgClient.java:201)
at org.schedulesdirect.api.NetworkEpgClient.<init>(NetworkEpgClient.java:174)

[Fuzzy]

Originally Posted by rdefino

Hi there,

I'm seeing this for some reason, looks like my SD login is not working. So I went the SD site and changed my password. But the plugin keeps telling me it and incorrect login. Any thoguhts? thanks


2016-08-31 16:27:46,046 INFO [Plugin]: EPG watchdog started!
2016-08-31 16:27:47,419 ERROR [Plugin]: Error in Watchdog
org.schedulesdirect.api.exception.InvalidCredentialsException: Invalid username or token has expired.
at org.schedulesdirect.api.NetworkEpgClient.authorize(NetworkEpgClient.java:242)
at org.schedulesdirect.api.NetworkEpgClient.<init>(NetworkEpgClient.java:201)
at org.schedulesdirect.api.NetworkEpgClient.<init>(NetworkEpgClient.java:174)

If you have numbers in your password, it may not have entered correctly (due to the 'phone dialer' input method sage uses). To verify, check in your sage.Properties file. the password should be in sdepg/password (or something like that). If you want, you should be able to just set it there (make sure the service is stopped before trying to open and edit the .properties file, or your changes won't stick.

[rdefino]

Originally Posted by Fuzzy

If you have numbers in your password, it may not have entered correctly (due to the 'phone dialer' input method sage uses). To verify, check in your sage.Properties file. the password should be in sdepg/password (or something like that). If you want, you should be able to just set it there (make sure the service is stopped before trying to open and edit the .properties file, or your changes won't stick.

thank you!!! that seemed to work, but I see this in the log file now. Not sure if it's from trying a few time to refresh the epg or what.


2016-08-31 18:35:22,389 INFO [Plugin]: EPG refresh started for Comcast Georgetown - Digital Haverhill (Cable)
2016-08-31 18:35:22,389 INFO [Plugin]: EPG refresh forced by user!
2016-08-31 18:54:31,595 ERROR [EPGImportPluginSchedulesDirect]: sdjson error!
org.schedulesdirect.api.exception.InvalidJsonObjectException: Program[EP016613400002]: null
at org.schedulesdirect.api.Program.<init>(Program.java:749)
at org.schedulesdirect.api.ZipEpgClient.fetchProgram(ZipEpgClient.java:301)
at org.schedulesdirect.api.ZipEpgClient.fetchSchedule(ZipEpgClient.java:270)
at org.schedulesdirect.api.ZipEpgClient.fetchSchedules(ZipEpgClient.java:319)
at org.schedulesdirect.api.Lineup.fetchDetails(Lineup.java:139)
at org.schedulesdirect.api.ZipEpgClient.getLineups(ZipEpgClient.java:360)
at org.schedulesdirect.api.ZipEpgClient.getLineupByUriPath(ZipEpgClient.java:460)
at org.schedulesdirect.api.ZipEpgClient$getLineupByUriPath.call(Unknown Source)
at sagex.epg.schedulesdirect.EPGImportPluginSchedulesDirect.setLineupMap(EPGImportPluginSchedulesDirect.groovy:494)
at sagex.epg.schedulesdirect.EPGImportPluginSchedulesDirect.doUpdate(EPGImportPluginSchedulesDirect.groovy:365)
at sagex.epg.schedulesdirect.EPGImportPluginSchedulesDirect.updateGuide(EPGImportPluginSchedulesDirect.groovy:210)
at sage.ae.y(Unknown Source)
at sage.a2.e(Unknown Source)
at sage.a9.dK(Unknown Source)
at sage.ae.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.ClosedFileSystemException
at com.sun.nio.zipfs.ZipFileSystem.ensureOpen(ZipFileSystem.java:1084)
at com.sun.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:316)
at com.sun.nio.zipfs.ZipPath.checkAccess(ZipPath.java:767)
at com.sun.nio.zipfs.ZipFileSystemProvider.checkAccess(ZipFileSystemProvider.java:187)
at java.nio.file.Files.exists(Unknown Source)
at org.schedulesdirect.api.ZipEpgClient.fetchProgram(ZipEpgClient.java:289)
at org.schedulesdirect.api.Program.<init>(Program.java:747)
... 15 more
2016-08-31 18:54:31,597 ERROR [EPGImportPluginSchedulesDirect]: Cannot process an empty lineup map!

[trk2]

Originally Posted by Fuzzy

If you have numbers in your password, it may not have entered correctly (due to the 'phone dialer' input method sage uses).

That's very interesting. Will this still be an issue when Schedules Direct support is integrated into the core?

[UgaData]

@rdfino - have you "told" Schedules Direct what line-up you want to use? It looks like it can't find a line-up or that no line-up exists.

I never used SD with SageTV7, only with SageTV9, I am presuming that both are using the newer JSON method. If so, you need to let SD know what line-up(s) you want use. There is more than one way to do this but it is not done on the Schedules Direct website.
It can be done thru the Sage webserver (after the SDEPG plugin is installed) or if you haven't loaded the webserver you can use EPG123 (it can be found at TheGreenButton website).

There seems to be some confusion on this portion of the SDEPG setup. Selecting the line-ups only needs to be done once. If you use or play with other "Media Center" programs (WMC, MythTV, NextPvr, etc) and have already setup SD you don't or won't need to do it again. Unless you move to a different zipcode or change providers, etc.