Plugin: Sage Recording Extender 4 (SREv4)

[Slugger]

Just received a correction from Google. Appengine apps such as this one will not be affected so it seems the service isn't going anywhere.

So no rush to port/move/etc, but since I can't open source this entire app, continuing users of this web service should start planning to come up with some other backend.

I will get my monitors up in github sometime soon, feel free to nudge me in a few weeks if I happen to forget.

[KarylFStein]

My hosting package (Namecheap) doesn't support JAVA, but does Python and PHP and MySQL. I don't use the web/DB parts of it and you'd be welcome to it. I'm not sending a PM because this got me thinking about another approach.

I have servers at home that support JAVA. What if there was some front-end available, (PHP/MySQL since it's pretty available in hosting packages or otherwise), along with the JAVA server rewrite you mentioned? The front-end could keep tabs on what servers are available, (servers register themselves--IP/port--on a regular basis so no dynamic DNS needed), proxy requests to them, (with short time-outs and retire/move to the next if no response), and maybe maintain some sort of cache. The client would have a configuration option like now, but to select a front-end instead of a server. The server code would have a similar configuration option for a front-end URL for registration.

The ideal would be that someone hosts the front-end in a stable environment and multiple people connect to it as available servers. Clients would hit the front-end and it would possibly send a cached response or reach out to a server. If that front-end goes away then someone else with a hosting package could take up the torch. If you were the last person using SRE, or want to do everything yourself, then you could run your own front-end and server.

[Slugger]

Are these servers running on people's sage boxes? If so, you'd have a lot of support questions as people struggle with firewall issues, etc. The architecture of such an app isn't exactly simple/low maintenance either. But the real big problem is getting users properly configured so you can talk to these servers behind people's routers/firewalls.

Since Google isn't kicking the web service off after all, there's no rush so maybe I'll make a quick and dirty python or php port of the web service a rainy Sunday project over the next little while. Whatever I come up with I'll throw in github and let the community here do with it what you will. My port would be bare bones/PoC/alpha type stuff -- a starting point for an eventual replacement of the current web service.

However, if someone here wants to run with this completely on their own then please do speak up. I don't mind doing a quick and dirty port, but it's also not something I'm dying to do either.

[pvr599]

Anyone having trouble with this now?

It has not worked for me for a couple weeks now. I de-installed the previous version and re-installed 4.0.1. On the surface everything looks great. In Sage Webserver it says the event isn't being monitored until the day of and then it switches to event is being monitored. Then on the recording, in options it is set up to be monitored but, it cuts off right at the guide time. Getting super frustrating and I'm not sure what's wrong.

Any help would be greatly appreciated. This is one of my favorite features in SageTV.

[cat6man]

Originally Posted by Slugger

Just received a correction from Google. Appengine apps such as this one will not be affected so it seems the service isn't going anywhere.

So no rush to port/move/etc, but since I can't open source this entire app, continuing users of this web service should start planning to come up with some other backend.

I will get my monitors up in github sometime soon, feel free to nudge me in a few weeks if I happen to forget.

excellent news!
i'd hate to lose my favorite plug-in of all time!

[Slugger]

Originally Posted by pvr599

Anyone having trouble with this now?

It has not worked for me for a couple weeks now. I de-installed the previous version and re-installed 4.0.1. On the surface everything looks great. In Sage Webserver it says the event isn't being monitored until the day of and then it switches to event is being monitored. Then on the recording, in options it is set up to be monitored but, it cuts off right at the guide time. Getting super frustrating and I'm not sure what's wrong.

Any help would be greatly appreciated. This is one of my favorite features in SageTV.

Events saying they aren't monitored until the day of the game is normal behaviour. I had to switch to monitors that no longer support future schedules so the monitors don't know about games until the day of the game (the only exception is MLB, which last time I checked still supports full future look ahead scheduling).

If the monitors don't actually monitor once the game starts then have a look at the sre.log file that the plugin generates. If there are problems with the monitor, they should be logged there.

Unfortunately, I don't use this plugin (nor Sage, for that matter) anymore so I don't have any idea if/when monitors may break, etc. This is another reason I've recommended a port of the web service to be administered and maintained by active users.

[pvr599]

Originally Posted by Slugger

Events saying they aren't monitored until the day of the game is normal behaviour. I had to switch to monitors that no longer support future schedules so the monitors don't know about games until the day of the game (the only exception is MLB, which last time I checked still supports full future look ahead scheduling).

If the monitors don't actually monitor once the game starts then have a look at the sre.log file that the plugin generates. If there are problems with the monitor, they should be logged there.

Unfortunately, I don't use this plugin (nor Sage, for that matter) anymore so I don't have any idea if/when monitors may break, etc. This is another reason I've recommended a port of the web service to be administered and maintained by active users.

Thank Slugger for responding.
On SageWeb I see: This recording is currently being monitored

On the log, I see a lot of this:
2016-03-12 16:36:00,001 INFO : 14083461/NHL Hockey: Fetching status with data: [NHL Hockey, New York Rangers at Detroit Red Wings, 1457809200000]
2016-03-12 16:36:00,226 WARN : 14083461/NHL Hockey: Error from web service; safety measures already applied

I have no idea what this means.

[skiingwiz]

Originally Posted by pvr599

Thank Slugger for responding.
On SageWeb I see: This recording is currently being monitored

On the log, I see a lot of this:
2016-03-12 16:36:00,001 INFO : 14083461/NHL Hockey: Fetching status with data: [NHL Hockey, New York Rangers at Detroit Red Wings, 1457809200000]
2016-03-12 16:36:00,226 WARN : 14083461/NHL Hockey: Error from web service; safety measures already applied

I have no idea what this means.

The client is getting an error response from the web service. Unfortunately, looking at the code, I don't think there is going to be any logging showing what that error is.

Can you hit the web service from your browser? http://livepvrdata-hrd.appspot.com/events.jsp

[Slugger]

Originally Posted by skiingwiz

Can you hit the web service from your browser? http://livepvrdata-hrd.appspot.com/events.jsp

... on the server running SageTV. Unless everyone is having problems, my guess is it's a firewall issue with the Sage server unable to hit the web service url.

[pvr599]

Originally Posted by Slugger

... on the server running SageTV. Unless everyone is having problems, my guess is it's a firewall issue with the Sage server unable to hit the web service url.

Thanks for the suggestion from both of you.

I can hit that website through a browser. I haven't changed anything in my firewall, which is either Windows firewall or my router. I'm not sure how to add that link to the firewall unless there is a part of sage that livepvrdata needs that I'm missing that is being blocked that I don't know about.

I am completely stumped on this one. Not sure which direction to go.

[Slugger]

I took a look at the server logs for that game yesterday. There were 3 people monitoring the game and every request for that game was responded to with an HTTP 200 response. That means, as far as I can tell, the web service was responding with proper info -- at the very least it recognized the request as valid and responded with something.

Assuming other people can confirm that things are being monitored as expected, I'd still say it's a firewall issue. Unfortunately, to get any more detailed info logged from the server side would require me to standup a dev environment, which is buried in a vm snapshot that is out of date and would require some effort to get up and running. Any such effort I'd put into this now would be to rewrite a fully open source version of the server to pass back to the community, and that's a "rainy day" project to be completed over the next number of months (or longer). Your best bet is to get an updated version of the plugin that logs the details of the responses received. The mystery that needs to be solved is what was the exact response being received from the web service (and why was it recognized as an error response)? To answer that question will require skiingwiz to provide a new version of the plugin with the ability to enable such detailed logging.

[pvr599]

Originally Posted by Slugger

I took a look at the server logs for that game yesterday. There were 3 people monitoring the game and every request for that game was responded to with an HTTP 200 response. That means, as far as I can tell, the web service was responding with proper info -- at the very least it recognized the request as valid and responded with something.

Assuming other people can confirm that things are being monitored as expected, I'd still say it's a firewall issue. Unfortunately, to get any more detailed info logged from the server side would require me to standup a dev environment, which is buried in a vm snapshot that is out of date and would require some effort to get up and running. Any such effort I'd put into this now would be to rewrite a fully open source version of the server to pass back to the community, and that's a "rainy day" project to be completed over the next number of months (or longer). Your best bet is to get an updated version of the plugin that logs the details of the responses received. The mystery that needs to be solved is what was the exact response being received from the web service (and why was it recognized as an error response)? To answer that question will require skiingwiz to provide a new version of the plugin with the ability to enable such detailed logging.

The funny thing is that this plugin has been working flawlessly for years. Then about a month or so ago it just stopped. I upgraded the plugin with no luck. Deinstalled it completely and reinstalled v4.1 (or whatever the latest is) and still no luck.

What would the firewall be blocking? I mean, do you have an idea of what to look for that pvrdata would use to get data? At least I can rule that out. I am hoping others would come forward so I can know if it's just me or not.

I really appreciate you taking the time to respond. I wish Sage was the way it used to be before the Google thing. I am a die hard and do not want to give it up. There are so many things that I would miss if I went to a DVR.

[Slugger]

Port 80 at livepvrdata-hrd.appspot.com is all the service uses.

The quickest way to test the firewall theory would be to just completely disable your firewall temporarily a minute or so before the start of the next recording that is to be monitored. If the game is properly monitored without the firewall enabled then at least you know it's a "you" problem. If the errors continue even with the firewall disabled then it's most likely a plugin and/or web service problem. If you eliminate the firewall as the problem then the next step is to ask skiingwiz for a debug build that fully logs the web responses being received from the service to determine why they're error responses.

[pvr599]

Originally Posted by Slugger

Port 80 at livepvrdata-hrd.appspot.com is all the service uses.

The quickest way to test the firewall theory would be to just completely disable your firewall temporarily a minute or so before the start of the next recording that is to be monitored. If the game is properly monitored without the firewall enabled then at least you know it's a "you" problem. If the errors continue even with the firewall disabled then it's most likely a plugin and/or web service problem. If you eliminate the firewall as the problem then the next step is to ask skiingwiz for a debug build that fully logs the web responses being received from the service to determine why they're error responses.

Ok, I am testing this tonight. I have turned off Windows firewall completely now. Around 2.5 hours before game time. In Webserver, I have SRE Status: Monitor status is invalid. But, that seems normal as I've seen and you said.
Do you also suggest I turn off my router firewall?
Oh, I hope this is it.

[Slugger]

Originally Posted by pvr599

Do you also suggest I turn off my router firewall?

Definitely not. What game are you planning to monitor?

[pvr599]

Originally Posted by Slugger

Definitely not. What game are you planning to monitor?

NHL Hockey

Episode: Detroit Red Wings at Columbus Blue Jackets

1 hour before game time. Monitor status invalid But, I know I've seen that change once the game starts.

[Slugger]

Apparently no one else is monitoring NHL or at least isn't noticing that it hasn't been working. I happen to have a test driver on my server that I was able to use to query the web service. The problem is that when I had to switch sources for the NHL, I ended up breaking the NHL monitor because I was missing some mappings for the team names.

You can turn back on your firewall as it was, the game should work fine as I've fixed the data on the server. Let me know if things are working fine.

[pvr599]

Originally Posted by Slugger

Apparently no one else is monitoring NHL or at least isn't noticing that it hasn't been working. I happen to have a test driver on my server that I was able to use to query the web service. The problem is that when I had to switch sources for the NHL, I ended up breaking the NHL monitor because I was missing some mappings for the team names.

You can turn back on your firewall as it was, the game should work fine as I've fixed the data on the server. Let me know if things are working fine.

Oh wow, well I have that favorite setup as having :30 minute padding since it wasn't working. The game started, could I change that padding in the middle to see if it works or should I wait for the next game? I should have changed that but, I forgot all about it.
This is great news if it starts working. Especially for playoffs. That plugin is a must

EDIT: I may have to leave it like it is for tonight since I have to run out tonight and I don't even think I can change the padding now.

[Slugger]

Just check the sre.log file, the errors that were there should now be gone. If so, then it's monitoring fine.

[pvr599]

Originally Posted by Slugger

Just check the sre.log file, the errors that were there should now be gone. If so, then it's monitoring fine.

I don't see the error I had at all anymore. The last line says 2016-03-17 22:14:00,001 WARN : 14121846/NHL Hockey: Terminating monitor because this airing has already had a completed monitor!

I can also tell, even with the padding, that it extended. There is a line in the timeline between the time the game ended and where my padding is. I'm willing to bet it's working like you said. (but you already knew that )

Thank you SO, SO much for looking into this for me and fixing it!!! This is my favorite plugin with comskip close behind. They make Sagetv invaluable.