Schedules Direct EPG (integrated BETA)

[graywolf]

Originally Posted by ranger

Perhaps. But it does this EVERY time I look at the page. I can come back an hour later and it will again show the current time and 0 minutes to next update. Either it is continually updating the EPG (24x7), or something is telling this screen (and the web interface) that the EPG is updating. A problem no matter which way you look at it.

SageTV version 9.1.2.662 is running on both my servers and only one of them behaves in this manner.

I have this same on one of my Sage boxes but not another.
Both SD EPG.

Turned on JSON logging for sdepg and I see the following in the sd_epg.log on the pc having issues

#### Exception Start ####-1
java.lang.ArrayIndexOutOfBoundsException: -1
	at sage.epg.sd.SDRecommended.getRecommendations(SDRecommended.java:365)
	at sage.epg.sd.SDRipper.extractGuide(SDRipper.java:2375)
	at sage.EPGDataSource.expand(EPGDataSource.java:709)
	at sage.EPG.run(EPG.java:783)
	at java.lang.Thread.run(Unknown Source)

#### Exception End ####

[Tiki]

Originally Posted by ranger

Perhaps. But it does this EVERY time I look at the page. I can come back an hour later and it will again show the current time and 0 minutes to next update. Either it is continually updating the EPG (24x7), or something is telling this screen (and the web interface) that the EPG is updating. A problem no matter which way you look at it.

SageTV version 9.1.2.662 is running on both my servers and only one of them behaves in this manner.

It normally takes more than an hour to complete the update.

[ranger]

Originally Posted by Tiki

It normally takes more than an hour to complete the update.

Thanks, but it doesn't matter WHEN I look at this. It always says the same thing. Two, three, four hours later. No difference.

[wnjj]

Originally Posted by graywolf

I have this same on one of my Sage boxes but not another.
Both SD EPG.

Turned on JSON logging for sdepg and I see the following in the sd_epg.log on the pc having issues

#### Exception Start ####-1
java.lang.ArrayIndexOutOfBoundsException: -1
	at sage.epg.sd.SDRecommended.getRecommendations(SDRecommended.java:365)
	at sage.epg.sd.SDRipper.extractGuide(SDRipper.java:2375)
	at sage.EPGDataSource.expand(EPGDataSource.java:709)
	at sage.EPG.run(EPG.java:783)
	at java.lang.Thread.run(Unknown Source)

#### Exception End ####

Looks like that bug was fixed 7 days ago. Should be fixed in 9.1.4.

https://github.com/google/sagetv/com...c8aa9a681df5b0

[graywolf]

Originally Posted by wnjj

Looks like that bug was fixed 7 days ago. Should be fixed in 9.1.4.

https://github.com/google/sagetv/com...c8aa9a681df5b0

Thanks
I'll keep an eye out on the Windows Installer thread for when 9.1.4 is available

[EnterNoEscape]

Originally Posted by graywolf

I have this same on one of my Sage boxes but not another.
Both SD EPG.

Turned on JSON logging for sdepg and I see the following in the sd_epg.log on the pc having issues

#### Exception Start ####-1
java.lang.ArrayIndexOutOfBoundsException: -1
	at sage.epg.sd.SDRecommended.getRecommendations(SDRecommended.java:365)
	at sage.epg.sd.SDRipper.extractGuide(SDRipper.java:2375)
	at sage.EPGDataSource.expand(EPGDataSource.java:709)
	at sage.EPG.run(EPG.java:783)
	at java.lang.Thread.run(Unknown Source)

#### Exception End ####

That situation is cleaned up in version 9.1.4.

[EnterNoEscape]

On the same note about exceptions in EPG data processing, I think SageTV should tone down the retry times by default. I think at the moment, it retries every few minutes. Something like once an hour would make the most sense to me. That might be something to be revisited after 7/1.

[wnjj]

Originally Posted by EnterNoEscape

On the same note about exceptions in EPG data processing, I think SageTV should tone down the retry times by default. I think at the moment, it retries every few minutes. Something like once an hour would make the most sense to me. That might be something to be revisited after 7/1.

Agree. While we're at it, why not make a 'manual refresh' option like we have for imported media? While not often, there's just enough times were a refresh is the right answer. Turning off and on a channel in the lineup isn't intuitive for new users.

[Taddeusz]

Originally Posted by wnjj

Agree. While we're at it, why not make a 'manual refresh' option like we have for imported media? While not often, there's just enough times were a refresh is the right answer. Turning off and on a channel in the lineup isn't intuitive for new users.

There is already an option in the web interface to force an EPG update. I'll second that there should be one in the STV. Removing and adding back a channel isn't exactly an intuitive way to force an update.

[graywolf]

[0002209] [SageTV log     ] Sun 6/18 10:45:12.855 [pool-4-thread-19@abbedd] SDEPG Unable to get head-shots for: SDPerson{personId='369894', nameId='378769', name='Michael S. Glick', role='Executive Producer', characterName='null', billingOrder='03'}
[0002210] [SageTV log     ] Sun 6/18 10:45:12.862 [pool-4-thread-19@abbedd] java.net.ConnectException: Connection timed out: connect

Is that normal/expected?
If so, might want to add it to the list of "messages" to suppress from the SageTV logs.

[EnterNoEscape]

Originally Posted by graywolf

[0002209] [SageTV log     ] Sun 6/18 10:45:12.855 [pool-4-thread-19@abbedd] SDEPG Unable to get head-shots for: SDPerson{personId='369894', nameId='378769', name='Michael S. Glick', role='Executive Producer', characterName='null', billingOrder='03'}
[0002210] [SageTV log     ] Sun 6/18 10:45:12.862 [pool-4-thread-19@abbedd] java.net.ConnectException: Connection timed out: connect

Is that normal/expected?
If so, might want to add it to the list of "messages" to suppress from the SageTV logs.

Yes, this is normal and will not cause EPG retrieval to be considered a failure. I don't believe in suppressing most exception messages if you're not going to handle them in any special way when the exception happens because then you're creating a gap in logging that might be significant in future diagnosis of a problem. In this case each entry you see is for a different person, so it's not really possible to reduce this down to something like a note of how many times it happened.

If you're seeing enough of these that your logs are actually rolling over from them alone, that might be an indication that we should adjust the thread count for everyone. I did tune this feature on a 75/75 FiOS connection, so I might have over-estimated what most setups will be able to handle. Either way, you can see how this has already lead to a discussion we wouldn't even be having without the exceptions being logged. Alternatively we'd be having a discussion where someone is asking why does this person have a headshot sometimes and other times no headshot and we wouldn't have anything in the logs to help us.

Basically there was a whole discussion about getting these headshots on GitHub whereby I reported that checking all possible headshots takes hours. We decided on multi-threading it since it's more of a latency problem because each check is it's own connection/handshake. That drove the time spent down a lot.

I'll just namedrop here. Narflex asked if we could have headshots since SageTV actually already had the support for it in the STV. He helped me work out some of the details about the Person objects and requested that we not perform lookups for celebrity images on demand and that instead the Person objects already have the URL that will take us directly to the image.

Schedules Direct doesn't provide any way to know if a celebrity image exists without just checking and they don't provide a way to batch check if the images will exist. We have to check each one individually and that's always thousands. This blast of checking happens after all important guide data has been imported and is limited to people that were in the shows we just update/added. It is required because otherwise I would not be able to meet the requirement to have the image URL already resolved.

[jpwegas]

Switched one of my 4 tuners to use SD. After updating, while scrolling through the "Parallel Recording Schedule", the Sage process locked up. Twice. Had to kill -9 to force stop and then restarted the process. Attached the portion of the sagetv_0.txt when it happened.

Maybe I was too quick and the guide isn't finished updating and it's referencing something that hasn't loaded from the SD site yet?

Let me know what other info I can provide to help debug.

--John

[EnterNoEscape]

Originally Posted by jpwegas

Switched one of my 4 tuners to use SD. After updating, while scrolling through the "Parallel Recording Schedule", the Sage process locked up. Twice. Had to kill -9 to force stop and then restarted the process. Attached the portion of the sagetv_0.txt when it happened.

Maybe I was too quick and the guide isn't finished updating and it's referencing something that hasn't loaded from the SD site yet?

Let me know what other info I can provide to help debug.

--John

The logs do not appear to indicate this is a Schedules Direct issue. It appears to be an issue loading an image.

[jpwegas]

Originally Posted by EnterNoEscape

The logs do not appear to indicate this is a Schedules Direct issue. It appears to be an issue loading an image.

Agreed, it does appear to be due to an image load issue, but it seems related to setting up the new lineup, even if the SD plugin wasn't the direct cause.

Perhaps something in the new lineup referenced an image that hadn't been downloaded yet?

I haven't had a hard hang like that, especially twice in 10 minutes, in years. It happened right as I scrolled into view the only upcoming show on the tuner using the new lineup.

I have a core dump too I can send if someone would be able to analyze it.

--John

[jpwegas]

Originally Posted by jpwegas

I have a core dump too I can send if someone would be able to analyze it.

Looking closer, the core dump was truncated (possibly due to me killing the process) so I'm not sure it's useful.

[EnterNoEscape]

Originally Posted by jpwegas

Agreed, it does appear to be due to an image load issue, but it seems related to setting up the new lineup, even if the SD plugin wasn't the direct cause.

Perhaps something in the new lineup referenced an image that hadn't been downloaded yet?

I haven't had a hard hang like that, especially twice in 10 minutes, in years. It happened right as I scrolled into view the only upcoming show on the tuner using the new lineup.

I have a core dump too I can send if someone would be able to analyze it.

--John

Knowing a little about how images are loaded, this isn't possible. A show is added with the image URLs. SageTV downloads the image via the URL when the STV wants the image. There isn't a timing issue with the Schedules Direct EPG data. There is a problem more likely with loading the image after it was downloaded. The problem could have been because SageTV can't load the image correctly as it was downloaded.

[jpwegas]

I avoided the Scheduled Recordings screen for a while and whatever was causing it I can't reproduce it any longer.

Thanks,

--John

[Tiki]

If this was for a recording that was in progress, on some screens Sage will try to create and display a thumbnail from a few second into the actual recording. It uses the Sage Transcoder to generate the thumbnail. I found in the past that some badly formatted Comcast recordings would trip-up the transcoder and cause the Sage UI to hang if it was trying to display that thumbnail. I saw this in the Malore Recording Scedule screen and the Malore current Recordings screen with Sage v7. There may be other screens where this could happen.

[MickBurke]

So I'm running thought the change for this now, and I'm wondering; is there no way to change the lineup without redoing all the channels?

When i changed the lineup, it resets all the channels back to 'default'.

I already started the one, and I see no 'cancel' so I think I'm stuck with this one anyway, but figured I'd check in...?

[pendragonsound]

I moved from Sage 7 to 9 a couple of days ago, and partially converted to Schedules Direct today. There's still one batch of channels to convert where no good solution appears to exist, and in some ways I am amplifying MickBurke's post above. I have 4 pairs of tuners with the following sources:

OTA broadcast
Dish Network satellite subscription
Shaw (Canada) satellite subscription
Free-To-Air (FTA) satellites

I converted the first three from line-ups with the old licensed EPG service to the equivalent Schedules Direct line-ups. That was straight-forward, but very tedious. In particular, Dish and Shaw each have close to 1000 physical and virtual channels, and their ranges completely overlap numerically. To solve this in the past, I had remapped all of the Shaw channels so the logical channel numbers are always 1000 larger than the physical channel numbers. Because converting to Schedules Direct requires reimporting the entire channel line-up, I had to manually remap each of the Shaw channels again. Furthermore I don't subscribe to all of the Dish and Shaw channels, so I had to go through each list and manually disable the channels I do not pay for. Fortunately I had saved a channel list from before, and thus I had a guide. But it took me nearly four hours to readjust everything back to the way it was. As MickBurke pointed out, it would have saved a lot of time to keep the channels unaltered when switching over. Anyway that's done and it appears to be working.

Now I am expecting the above problems in spades for the FTA channel conversion. I have dishes pointed at dozens of satellites, with several thousand total channels. Most of these don't connect to an EPG, but there are still a lot that do. There is no 'line-up' that covers any of these channels. With the old EPG service, I found I did not have to specify a line-up to get EPG information for the FTA channels, and thus I did not. Whenever I found the 'Station ID' (5 digit number) for a FTA channel on Zap2It, I simply entered that manually into my FTA channel list. The EPG was populated for these channels even without a formal line-up. I had hoped this might continue working with Schedules Direct, but even though the Station IDs translate over perfectly, it appears to be continuing to pull these through the old, licensed path.

Ideally I would like to leave my FTA channel list alone, but pull the EPG for those channels that can get it from Schedules Direct. I cannot find any way to do this, and would be grateful to anyone who could point me in the right direction. I'm assuming that means my only choice is to (1) import an irrelevant channel line-up for my FTA tuners, (2) disable all of the channels in that line-up, (3) reenter thousands of channels that will never have an EPG, and (4) reenter the Station IDs for the channels where I was able to find them. This will take days, and if MickBurke's concept were to work, it would take minutes.