XMLTV TV_GRAB_UK_RT - Has something changed these days?

[bmadgett]

It really _does_ stretch their servers too, I suspect - for example my channel lineup with --slow --categories says it downloads ~80Mb per day! Which is ridiculous when the resulting XMLTV output is ~1Mb.

What would help a lot is if there was a single plaintext (or marked-up somehow) page which contained all the program details for a channel for a day. You'd only then need to have a limited number of fetches. It's the fetch-per-program to get the details that really causes problems from what I've seen.

Whether they'll be prepared to put that sort of easily accessible info up is another matter. I'm sure it would be very easy technically for them (they've got all the info there, after all) - it's more a case of whether they really want to allow end users to grab all the detail. After all they're charging for the same sort of thing for their PDA app.

The noises coming from this RT technical contact do sound positive, though...

It'll all take time to arrange, if at all - hence we may need to switch to Bleb in the meantime.

Cheers

Barnaby

[eldingo]

The bleb listings work ok for me, however I had to set up all my tv channels again when I first put them on

[ron.smythe]

I've just come across the RT problem now my EPG data has run out. I like the idea that the Tech at the RT wants to enable us to carry on working, I can understand there concerns, with all of us hitting there links there would have been a few problems arising. I wonder if we can get Sage involved at this stage, as the RT may want there epg data protected, and if Sage can make assurances to them that this is what is going on at the moment in the US, the I'm sure the RT would be happy to supply data for a cost, like there PDA stuff.

And yes I know there is opposition to epg data being a cost for item, but wouldn't it be better to have a data service then not, remember every time Digiguide are approached they tell us it's not possible because of licensing, at the moment the Tech at RT wants to help, if we offer an olive branch maybe, just maybe we can get TV recorded in an organised manor so that all of us can start recommending Sage to anyone and everyone in the UK, not just the people that have time to fix XMLTV issues. And I am not saying the problems are normally caused by the XMLTV, but with bad data, slow links, and a large amount of data downloaded each day it can takes it's toll.

I would also like to take this opportunity, whilst I stand on my virtual soap box, to thanking everyone in the XMLTV project, as although not perfect because of data restrictions, it is a brilliant tool.

[nielm]

Originally posted by Kryton
bleb.org can actually outputs the info as an xmltv file rather than having to parse it using xmltv.exe. Just specify &format=XMLTV in the URL
See here for Syntax
http://www.bleb.org/tv/data/listings?

You can download your weeks listings as a zip file in under a minute like this, I use it as such:
my example (copy and edit the url here, this forum added <BR />'s to it)

The bleb grabber uses this interface... and appears cleans up the resulting XML, addign TZ, removing overlaps etc.

[Lucas]

Yes, I also tried to use the "XMLTV" data straight from bleb but the sage plug in did not like it very much.

Only by doing processing using grep before the XMLTV bleb grabber appeared did I manage to get it to work.

The alpha XMLTV uk_bleb grabber seems to format the data correctly and works relatively well.

[ukmgranger]

Originally posted by ron.smythe
And yes I know there is opposition to epg data being a cost for item, but wouldn't it be better to have a data service then not, remember every time Digiguide are approached they tell us it's not possible because of licensing, at the moment the Tech at RT wants to help, if we offer an olive branch maybe, just maybe we can get TV recorded in an organised manor so that all of us can start recommending Sage to anyone and everyone in the UK, not just the people that have time to fix XMLTV issues.

I am in total agreement with this!

The one and only reason that I DO NOT recommend Sage to my friends/colleagues is the whole EPG thing in the UK. Although XMLTV is great and I am now starting to get the hang of it, most people I know just wouldn't get on with it. If there was a solid, reliable way to get the data I feel Sage would be so much more accessible to people in the UK.

[bmadgett]

Absolutely. The only way I got XMLTV and the RT grabber even halfway stable for me was by running it on my linux server with a shellscript that does various things like retrieving daily portions, detecting "failed" downloads (if the filesize was less than expected), merging portions together, declumping etc. I'm happy hacking about doing this sort of thing myself (I hack code for a living after all), but e.g. there's no chance I could recommend it to my parents.

Until a Sage user can just select "Telewest Supreme Package" or "Sky" or whatever and get the EPG downloads automatically every day for the appropriate channels (or at least manually select the channels), it's going to remain a tech-savvy-only thing in the UK.

The data also needs to be a lot more reliable than the stuff we get from XMLTV - it's not "bad" as such, just it has inconsistencies where some episodes have good details, others have none so you end up with duplicate recordings, etc. We need proper episode tagging and the like in order for Sage to do its thing to best effect, which you can't get from webpage scraping.

Barnaby

[ukmgranger]

Anyone heard any news on the UK_RT thang?

[robertpoll]

Good news...

Date: 2004-09-20 11:43
Sender: alanbirtles
Logged In: YES
user_id=1120910

my grabber is now working again
http://www.alanbirtles.dsl.pipex.com/xmltv/xmltv.htm and ed
has said that xmltv will include the required changes some
time in the next week

This was from http://sourceforge.net/tracker/index...46&atid=424136

I've tried the grabber and it is very fast and has more detail than the old one (actors, categories etc.)..

..only problem is that xmltv seems not to work with the xmltv plugin. I haven't looked in detail yet and have had problems in the past (I could never get it to work with --slow) so would be keen to hear if anyone else can make it work (and if so which version of the plugin they are using).

[nielm]

Originally posted by robertpoll
..only problem is that xmltv seems not to work with the xmltv plugin. I haven't looked in detail yet and have had problems in the past (I could never get it to work with --slow) so would be keen to hear if anyone else can make it work (and if so which version of the plugin they are using).

It does not generate the correct format of xml (the xmltv tools raise a ton of errors on the generated data), and the timestamps are missing seconds.
On the plus side, someone is writing a new RT grabber for the xmltv project that will use the new offical RT data source (which gives access to the pure data and is damn fast)...

[edit]the xmltv grabber has already been written, and a new release will be available soon --- see http://article.gmane.org/gmane.comp.tv.xmltv.devel/4117

[Lucas]

Tried the alanbirtles grabber.
Yes the XMLTV tools generate a lot of errors but so does the bleb_grabber even if it is to a lesser extent.

The new UK_RT data looks good though.

It has some fields like subtitles, aspect, rating, star-rating.....

Any idea how these would be translated in the SageTV EPG?

Also under what conditions RT made this data available?

It seems too good to be true.

[nielm]

...subtitles, aspect, rating, star-rating.....
They were always there in --slow mode.. and were ignored by the XMLTV plugin...

I had the intention a long time ago to update the xmltv plugin to add this extra data to the description, but got distracted

The RT conditions seem to be the same as the webpage--personal use only, no redist, and the alanbirtles grabber webpage suggest that it may become a subscription service in the future (I guess RT are going to evaluate the number of users and decide whether it would be worth setting up a subscription scheme).

[birty]

my grabber should no longer produce errors in the xmltv tools, i didn't know that the attributes in the xml had to be in the exact order as in the xmltv specification

[Lucas]

Hmmm, yes. I know about distractions!

Maybe its time to do my bit of sharing the burden....
Nielm, can you please point me to or provide the plugin's(the multi-input source) source code?

[nielm]

birty; there are several XML validators around that check XML against the corresponding DTD... one is nsgmls, mentioned here

Lucas... There are several that need to be merged... My multi-source one is here: http://forums.freytechnologies.com/f...9299#post39299 (apologies for the code -- but at least I can say hardly any was mine!)

My plan was to combine the debugging features of TallyHo's grabber (http://forums.freytechnologies.com/f...1607#post51607) -- which otherwise is less fully featured, and Demideus' category translation (which can translate Film->Movie and get EPG highlighting working), and find out from Sage the difference between EPXX episode ID's and SHXXX show Id's...

Good luck!

[ukmgranger]

Sorry for being a dumb-ass, but is this new RT grabber available yet?
Also, has RT now given permission for people to grab the data? Thats a pretty cool way of thinking from them if they have - letting loads of people suck data from their site!!

[nielm]

Originally posted by ukmgranger
Sorry for being a dumb-ass, but is this ne grabber available yet?
Also, has RT now given permission for people to grab the data? Thats a pretty cool way of thinking from them if they have - letting loads of people suck data from their site!!

The new XMLTV project grabber has been written and is currently under test...

RT have been very cool and have actually set up special pages to download the raw data from. eg for BBC1: http://xmltv.radiotimes.com/xmltv/92.dat

[Lucas]

Alan's grabber - rel 2.2.2 works very well with SageTV now as well.

[ukmgranger]

Whats this 'Alan's grabber' and where can I get it to check it out?

[birty]

http://www.birtles.org.uk/xmltv/xmltv.htm