Plugin: Schedules Direct EPG (not for v9)

[Slugger]

Go to the web ui, go to the status menu in the Schedules Direct options. Does it show that you have at least one headend registered?

[mzup]

It says Configured Headends.
a checkbox and ID Description written next to it.

Edit: Under Status it says I have no lineups configured but I have been using this for a long time.

[Slugger]

Right, as was mentioned in the 0.5.0 notes, a data format change at SD requires uses to register their headends again.

Do that then the epg pull should work fine.

[JerryB]

I don't know whether it's a bug in 0.5, a bug in SD, or some corruption that's entered my system but, since upgrading to latest beta, my system is no longer picking up the programLanguage field for my foreign channels from the SD data download and my Airing filter, that previously automatically detected and corrects the program language on channels when it wasn't present and correct in the SD data is not being called at all. I did check the SD epg download, and the programLanguage field is present and correct in this download.

[Slugger]

Hmm... assuming the language data is in the .epg file then it's most likely something I need to fix in the plugin. There were alot of data format changes internally in this new SD API release so it's likely that even though the data is still in the .epg file, it might be a different location within an object such that my plugin code isn't finding it.

I will investigate this over the weekend.

[JerryB]

If it help, here's a copy of the lineup listing from the epg for one of the channels:

{
"affiliate": "Satellite",
"broadcaster": {
"city": "Beijing",
"country": "China",
"logo": {
"URL": "https://s3.amazonaws.com/schedulesdirect/images/9cc4d63256a4a8534be108a7b05cfc95.png",
"dimension": "180x135",
"md5": "n9/iLWEVcTo02Vs1PA4wJw",
"modified": "2013-10-03T02:57:30Z"
},
"state": "XX",
"zipcode": "80122"
},
"callsign": "CCTV4",
"language": "Mandarin",
"name": "CCTV4--China Central TV",
"stationID": 24248
},

[Slugger]

So I'll need to see a scheduled airing for that channel plus the associated program entry. It needs to be a sample where you're expecting the language to be set, but it hasn't been.

[JerryB]

Here's the program entry:

{
"colorCode": "Color",
"descriptionLanguage": "English",
"genres": ["Variety"],
"md5": "8ozJJy0uBJ9W1qxLbu8K4g",
"originalAirDate": "2009-08-02",
"programID": "SH011756660000",
"showType": "Series",
"sourceType": "Local",
"titles": {
"title10": "Happy...",
"title120": "Happy Chinese"
}

and the airing entry:

{
"airDateTime": "2013-10-25T09:15:00Z",
"cc": true,
"duration": 900,
"md5": "8ozJJy0uBJ9W1qxLbu8K4g",
"programID": "SH011756660000",
"programLanguage": "Mandarin",
"stereo": true
},
}

[Slugger]

That's what I thought... the descriptionLanguage="English", which I didn't expect in these cases (I made some changes to optimize the language lookup stuff).

So that's why the internal language isn't being set correctly. I'm stumped still as to why your filter script isn't working. Can you paste that in?

I'll have to think about how to handle the situation where a program description is in English, but the program doesn't air in English. I'll probably have to just ignore the "trick" I implemented and undo it.

[wayner]

Hi Slugger - I haven't tried this yet since everything is still go with the main Sage EPG but I noticed this in post 1:

In order to use more than one unique lineup within SageTV via this plugin, you must also have a sagetv-addons license. This is NOT the same as your SageTV server, client, or placeshifter license. See the link for details. If you only need one unique lineup then you may not need a license. A single lineup can be assigned to multiple input sources without the need for a plugin license. There are also other ways to remove the need for a license.

Is that license requirement still the case as didn't you nuke the sagetv-addons license recently?

[Slugger]

The license requirement still exists in the code, but as long as you install the latest version of the license server plugin, then you automatically have the full license.

Really, any plugin (of mine) that says it's restricted by licensing can be ignored as long as you have the latest license server plugin installed. That license server update is what bypasses the licensing requirement (by just "saying yes" to anything that asks it about its licensing state).

[JerryB]

Originally Posted by Slugger

That's what I thought... the descriptionLanguage="English", which I didn't expect in these cases (I made some changes to optimize the language lookup stuff). So that's why the internal language isn't being set correctly.

In general, I've found that the descriptionLanguage always uses a Roman-style alphabet. It will match the programLanguage when the programLanguage also uses a Roman-style alphabet but not when the program language uses some other system such as character-based Asian languages.

I'm stumped still as to why your filter script isn't working. Can you paste that in?

Here's the Airing Filter

// Required imports
import sagex.api.*
import org.apache.log4j.Level

// Set log level for the script
_log_.setLevel((Level) Level.INFO)

// Retrieve channel number for the Airing being processed
def chan = ChannelAPI.GetChannelForStationID(_src_.station.id.toInteger())
def channelNumber = ChannelAPI.GetChannelNumber(chan)

/*
* Determine if the Airing is on a French or Chinese channel
* French channel(s): 1771
* Chinese channel(s): 1795, 1796
* If the Airing is on one of these channels, check the broadcastLanguage and correct if necessary.
*/
if (channelNumber ==~ /1771|1795|1796/) {
switch (channelNumber) {
case ~/1771/: // French Channel
if(_src_.broadcastLanguage ==~ /French/) {
_src_.broadcastLanguage = "French"
_log_.error("Note: $channelNumber language changed to French")
}
break
case ~/1795|1796/: // Chinese Channel
if(_src_.broadcastLanguage ==~ /Mandarin|Chinese/) {
_src_.broadcastLanguage = "Mandarin"
_log_.error("Note: $channelNumber language changed to Mandarin")
}
break
default: // Should never get here
_log_.error("Error: $channelNumber being erroneously processed")
break
}
}

[Slugger]

Your filter basically says if the language is already set then set it again, so now I see why that's not working.

And I know why the internal code isn't working (for you, on the Chinese stations).

Am I right in assuming you want the language of every airing on those channels to be set? If so, then just remove the if conditions inside the switch statement, they shouldn't be necessary. By doing that, your filter basically says to set the language on all airings on those channels that match the outer regex, which is what I think you want.

Obviously, that filter shouldn't be necessary after I fix my bug, which I will try to fix tomorrow morning before football kicks off and then try to get the fix out early next week.

[JerryB]

You're right. I revised my airing filter when the program language was added to the SD download and and made an error when I revised it. It is supposed to say if the language is "not" set then set it but I apparently left on the "not" when I made the change. I didn't detect this before since the filter was not necessary as long as the correct language was being picked up from the SD download.

Edit: I believe I corrected the problem (I defined externalID and revised the case statement [see below]) and the Airing filter is still not being processed.

case ~/1771/: // French Channel
if (_src_.broadcastLanguage ==~ /French/) { // Already correct; print debug message and exit
_log_.debug("Skipping ExternalID: $externalID - Correct language already set")
return
}
// Set language to French
_src_.broadcastLanguage = "French"
break

[rkulagow]

If this is better suited for the server side (so that anyone who also has the same channel will get the same benefit), then please send me the stationID and the override language (so that all programs will get the language set)

[Slugger]

I believe the language is already set on all the airings on these channels. The issue is that the Sage plugin isn't processing the language value properly in this new release. I thought I was being smart with some fancy optimizations to the routine, instead I broke it.

[rkulagow]

Understood. At this time I only have a few examples of stations which need overrides, so if there are more then people should let me know so that I can make the adjustments server-side.

Thanks!

[JerryB]

I think I know why my Airing filter is not being called. Unfortunately, it raises another issue. It appears that while updated SDEPG data are being downloaded, these data are not being processed and incorporated in SageTV. I noticed this when I realized that my SageTV listing only had a few days of data left even though SD had reported more recent data being available and an examination of the sdepg download file confirmed that this file contained the more recent data. I tried deleting my headends and then adding them back just in case something got messed up when I added them back following the upgrade and that didn't fix the problem. Do you have any idea what could be causing this or what I should try next? Thanks.

[Slugger]

Any errors in the sdepg.log file? That's always the place to start when things aren't working as expected.

[JerryB]

Originally Posted by Slugger

Any errors in the sdepg.log file? That's always the place to start when things aren't working as expected.

That was the first thing I checked but I there were no errors, only warnings, and nothing that I could tell that indicated what is going on, see below.

2013-10-28 08:33:54,124 INFO [ProgramGenerator]: Processed show generators in 11ms
2013-10-28 08:33:59,283 WARN [Plugin]: Missing PluginProperty: sdepg/grabberHeap
2013-10-28 08:33:59,313 WARN [Plugin]: Missing PluginProperty: sdepg/sdjsonThreads
2013-10-28 08:33:59,313 WARN [Plugin]: Missing PluginProperty: sdepg/sdjsonUrl
2013-10-28 08:33:59,313 WARN [Plugin]: Missing PluginProperty: sdepg/grabberLogLvl
2013-10-28 08:33:59,313 WARN [Plugin]: Missing PluginProperty: sdepg/sdjsonProgChunk
2013-10-28 08:33:59,313 WARN [Plugin]: Missing PluginProperty: sdepg/sdjsonSchedChunk
2013-10-28 08:33:59,403 WARN [Plugin]: Missing PluginProperty: sdepg/cacheTTL
2013-10-28 08:35:09,515 INFO [ProgramGenerator]: Processed show generators in 1ms
2013-10-28 08:35:09,553 INFO [EpgDownloader]: [C:\Program Files (x86)\Java\jre7\bin\java, -Xmx512m, -jar, P:\SageTV\SageTV\plugins\sdepg\tools\sdjson.jar, --username, *****, --password, *****, --user-agent, sagetv-sdepg/0.0.5.0 (Windows 8 x86 6.2), --max-threads, 200, --url, https://data2.schedulesdirect.org, --grabber-log-level, INFO, grab, --target, P:\SageTV\SageTV\plugins\sdepg\sdjson.epg, --max-prog-chunk, 50000, --max-sched-chunk, 1000, --ignore-stations, P:\SageTV\SageTV\plugins\sdepg\ignore.txt]
2013-10-28 08:36:32,030 INFO [EPGImportPluginSchedulesDirect]: Performed EPG download in 82515ms
2013-10-28 08:36:32,585 INFO [ChannelGenerator]: Processed channel generators in 3ms
2013-10-28 08:36:32,586 INFO [EPGImportPluginSchedulesDirect]: Performed channel processing in 556ms
2013-10-28 08:36:32,971 INFO [EPGImportPluginSchedulesDirect]: Performed lineup map configuration in 385ms
2013-10-28 08:37:34,396 INFO [ProgramGenerator]: Processed show generators in 1ms
2013-10-28 08:37:34,405 INFO [EpgDownloader]: [C:\Program Files (x86)\Java\jre7\bin\java, -Xmx512m, -jar, P:\SageTV\SageTV\plugins\sdepg\tools\sdjson.jar, --username, *****, --password, *****, --user-agent, sagetv-sdepg/0.0.5.0 (Windows 8 x86 6.2), --max-threads, 200, --url, https://data2.schedulesdirect.org, --grabber-log-level, INFO, grab, --target, P:\SageTV\SageTV\plugins\sdepg\sdjson.epg, --max-prog-chunk, 50000, --max-sched-chunk, 1000, --ignore-stations, P:\SageTV\SageTV\plugins\sdepg\ignore.txt]
2013-10-28 08:37:35,947 INFO [EPGImportPluginSchedulesDirect]: Performed EPG download in 1551ms
2013-10-28 08:37:36,100 INFO [ChannelGenerator]: Processed channel generators in 1ms
2013-10-28 08:37:36,100 INFO [EPGImportPluginSchedulesDirect]: Performed channel processing in 153ms
2013-10-28 08:37:36,355 INFO [EPGImportPluginSchedulesDirect]: Performed lineup map configuration in 255ms
2013-10-28 08:38:46,798 INFO [ProgramGenerator]: Processed show generators in 0ms
2013-10-28 08:38:46,807 INFO [EpgDownloader]: [C:\Program Files (x86)\Java\jre7\bin\java, -Xmx512m, -jar, P:\SageTV\SageTV\plugins\sdepg\tools\sdjson.jar, --username, *****, --password, *****, --user-agent, sagetv-sdepg/0.0.5.0 (Windows 8 x86 6.2), --max-threads, 200, --url, https://data2.schedulesdirect.org, --grabber-log-level, INFO, grab, --target, P:\SageTV\SageTV\plugins\sdepg\sdjson.epg, --max-prog-chunk, 50000, --max-sched-chunk, 1000, --ignore-stations, P:\SageTV\SageTV\plugins\sdepg\ignore.txt]
2013-10-28 08:38:48,403 INFO [EPGImportPluginSchedulesDirect]: Performed EPG download in 1605ms
2013-10-28 08:38:48,605 INFO [ChannelGenerator]: Processed channel generators in 0ms
2013-10-28 08:38:48,605 INFO [EPGImportPluginSchedulesDirect]: Performed channel processing in 201ms
2013-10-28 08:38:48,905 INFO [EPGImportPluginSchedulesDirect]: Performed lineup map configuration in 300ms
2013-10-28 08:39:49,918 INFO [ProgramGenerator]: Processed show generators in 0ms
2013-10-28 08:39:49,926 INFO [EpgDownloader]: [C:\Program Files (x86)\Java\jre7\bin\java, -Xmx512m, -jar, P:\SageTV\SageTV\plugins\sdepg\tools\sdjson.jar, --username, *****, --password, *****, --user-agent, sagetv-sdepg/0.0.5.0 (Windows 8 x86 6.2), --max-threads, 200, --url, https://data2.schedulesdirect.org, --grabber-log-level, INFO, grab, --target, P:\SageTV\SageTV\plugins\sdepg\sdjson.epg, --max-prog-chunk, 50000, --max-sched-chunk, 1000, --ignore-stations, P:\SageTV\SageTV\plugins\sdepg\ignore.txt]
2013-10-28 08:39:51,504 INFO [EPGImportPluginSchedulesDirect]: Performed EPG download in 1586ms
2013-10-28 08:39:51,929 INFO [ChannelGenerator]: Processed channel generators in 0ms
2013-10-28 08:39:51,929 INFO [EPGImportPluginSchedulesDirect]: Performed channel processing in 424ms
2013-10-28 08:39:52,165 INFO [EPGImportPluginSchedulesDirect]: Performed lineup map configuration in 236ms