XMLTV TV_GRAB_UK_RT - Has something changed these days?

[Lucas]

I noticed that last night and this morning the tv_grab_uk_rt grabber was returning no programme info.
Each page request was returning no data....

Has something changed on the RT site?

[bmadgett]

Yep, the grabber's broken at present. Take a look at the threads on the XMLTV-devel list at :

http://news.gmane.org/gmane.comp.tv.xmltv.devel

Looks like the RT site is blocking requests from grabbers, whether intentionally or not noone is quite sure yet. Whether it'll be fixed or not, again, noone seems sure.

I'm attempting to switch to getting XMLTV from bleb.org, but that's less than ideal as there's loads of channels missing.

I wish there was somewhere we could PAY to get decent reliable TV listings in the UK (other than DigiGuide, which is no use to us Sage users)

Barnaby

[srcurtis]

If anyone comes up with a resolution for this, please let me know.

Steve

[Lucas]

Looks like Neil has found a workaround.

Has he posted a link to the recompiled win32 executables?

PS. I wouldn't mind paying for the feed either!!!

Andrew

[bmadgett]

Neil's "workaround" (getting the User Agent set correctly) doesn't actually fix the problem for me - the grabber gets further but still fails to download any program details. So there's something else still going on. Unless I patched my local perl wrongly - I'm no perl expert! (or even a novice...)

Fingers crossed someone gets a solution working before the 21st - that's the day my currently downloaded schedules run out, after that I'll be relying on Bleb and there's lots of channels missing there.

I wish there was a Digiguide->XMLTV converter, that's the best source of UK schedule info, I can see why they don't want to provide one though (the copyright laws, their business model, etc). Even so, a shame...

Barnaby

[bmadgett]

Actually, seems like I was wrong - Neil's fix DOES allow the grabber to work. What was looking like failures to me was just that the RT site doesn't seem to have listings for the 22nd & onwards, which is where my grabber is trying to fetch from (I already have up to the 21st cached on disk).

If you try thro' a browser to search for progs on the 22nd on the RT site it fails, too, so it's a problem at their end, not with the grabber.

If I reset it so that it tries to fetch from tomorrow instead, it seems like it's correctly getting programmes again.

Wonder why listings aren't available for next week?

Barnaby

[nielm]

my workaround was not a workaround -- it was a fix for a bug (tv_grab_uk_rt was not using the correct HTTP user agent).

It is unlikely to make it to a released XMLTV version because the reasons for the RT web site to start blocking the grabber are equally valid whatever user-agent string it uses, and constantly changing the user-agent to get around RT's blocking will only upset them and make them start to use more aggressive (possibly legal) measures against the grabber, or worse the publisher and users of the grabber.

Our only hope is though contacting the RT folks and hoping that they are nice...

[bmadgett]

Yeah - fingers crossed they're reasonable. Looking at the T&Cs on the RT site, this is the relevant portion :

...quote...
Personal Non-commercial Use Limitation.
Any material found within the pages of www.radiotimes.com, including text or images may not be copied, reproduced, republished, downloaded, posted, broadcast or transmitted in any way except for your own personal non-commercial use. You will not edit, adapt, alter or create any derivative work from any of the material contained in this site.

Permission for any other use must be obtained by contacting BBC Worldwide Limited. For further information please email us at radio.times@bbc.co.uk.
...end quote...

So I would say that running a grabber to produce XMLTV files for personal PVR use is within those conditions (ie we wouldn't be republishing the data anywhere). However some of the plans mentioned for providing mirrors etc would definitely seem to infringe on these conditions.

It all depends what their REAL objection to the grabber is. If it's purely a technical too-many-hits problem, then maybe they could be convinced to provide a location where the raw text could be downloaded in one big lump for a day (ala Bleb) - thus eliminating most of the thrashing on their webserver due to the many hits the grabber has to do.

If it's more of a "well we might possibly want to start charging for this sort of thing sometime in the future so we don't want you to be able to get it for free", then we'd be stuck.

My suspicion is that it'll be being blocked for technical reasons (the webmaster has spotted 1000s of hits from the grabber agent), and that inertia / lack of motivation / lack of resources will mean the route of least resistance ie blocking the grabber will be taken. It's less work for a sysadmin to block a grabber than to provide an alternative, more-efficient mechanism. Plus the sysadmin would have to get it cleared with the legal dept etc etc...

Hey ho! It's Sunday evening, tomorrow is Monday & work again, never puts me in a very positive frame of mind :-)

Barnaby

[Lucas]

Hi all,

unfortunately my UK_RT data run out yesterday so ..........
Tried using the o2grabber and got a "Connection with webserver failed" msg.

Is this server only available at certain times during the day or night?

Any other o2grabber users out there with a working setup?

Any help will be much appreciated......

[nielm]

Just to let you know that there is now an alternative UK grabber in XMLTV... Nothing to do with me!

It does not grab as much detail as uk_rt, nor as many channels, but it is a start.

Data comes from www.bleb.org

Installation:
Download the latest alpha xmltv executable
Save it in the same directory as the old XMLTV exe.

Download the icon_urls file (right-click the link and select save-as
Save this as: <xmltv-dir>\share\xmltv\tv_grab_uk_bleb\icon_urls
AFTER SAVING MAKE SURE THAT IT HAS NO .TXT EXTENSION

start a command prompt and cd to the XMLTV dir
run:
xmltv tv_grab_uk_bleb --configure
and set up the channels as you did with uk_rt
then run
xmltv tv_grab_uk_bleb --days n > epgdata.xml

You may have different channel IDs in the XML to the channel IDs produced by the RT grabber. If this is the case you may need to remap the new channels in Sage

Good luck.

[bmadgett]

Not only does the RT site block XMLTV now, but they seem to have changed what days are available on their website. I can't get it to give me program listings any further ahead than the 19th, even though before the grabber stopped working I'd downloaded up to the 21st...

Can anyone else (using a browser) get the RT site to give them details past the 19th, for any channel?

Looks like more has changed than just blocking XMLTV.

I'll be investigating the bleb thing come the weekend, I suspect. It's a real pain bleb doesn't have the various extra Discovery channels, and doesn't have the +1 channels, as if it had those then I wouldn't be missing much from the lack of the RT one. Perhaps it'll be possible to cobble together a script that could take e.g. a "Discovery" listing and produce a "Discovery +1" listing just by going through adding 1hr to all the times? Anyone here whizzy enough with perl?

Alternatively, has anyone investigated the format for the program listing downloads that's used for the RT PDA application? It struck me that since the listings get downloaded via your PC & then get synced onto your PDA, it should be possible to grab the downloaded listings & parse into XMLTV format. Are they encrypted? I can't see this could annoy them since a) it'd still be for personal use with no redistribution, b) you'd still be paying your subscription of £15/yr to them, and c) you would be downloading in just the same way as any other PDA user, thus not generating any extra strain on their servers. Thoughts?

Barnaby

[Beefcake]

Originally posted by nielm
Download the latest alpha xmltv executable

http://alpha-exe.xmltv.org/

Site seems to be down, amyone have a copy of the file they can host please?

Scrap that site is back up

[Lucas]

The tv_grab_uk_bleb grabber works very well. It is also much faster.

Thanks for the tip Niel!!!!

PS. OK so the program info is less than RT and some channels are missing, the important ones are there BTW.

Credit should go to all those who put their time in developing something that we can also use. I personnally don't have the time to write something like this so I use whatever people are kind enough to make available and since I can't pay for an EPG service right now, I don't think I can complain.

I think some people complained that the +1 channels were not there....... Well who needs that anyway? That's why we have invested in a PVR... We record shows whenever they are on and watch them whenever we want. The +1 channels are for people who don't have this setup. Or maybe there is some other use that I haven't thought about.

[bmadgett]

The +1 channels are handy to have available to Sage so that it can resolve conflicts more easily e.g. say you have something on BBC1 and something on Discovery, both running 8-9pm. If you also have Discovery+1 then Sage can just record from that at 9-10pm, thus avoiding the clash with the program on BBC1. A minor point, but helpful nonetheless.

I ought to be able to cobble something together to generate +1 channels from the "base" channel coming out of bleb, if a tool doesn't already exist. Won't be for a week or two though - very busy just now.

I agree with the sentiment about thanking people who put time & effort into all this - without them, us PVR users in the UK would be well & truly stuffed. I've still got my fingers crossed that one day we'll be able to get decent EPG in the UK, even on subscription. If there was one thing that might cause me to think about switching to another PVR software app, it'd be them having an agreement with UK EPG suppliers...

Barnaby

[nielm]

As the XMLTV exe has all the code for parsing XML and adding dates in it, how about this:

create an epgdata.xml file from bleb as normal

Create a discplus1channel.xml file to define the new channel containing

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE tv SYSTEM "xmltv.dtd">
<tv source-info-url="http://www.bleb.org/tv/data/listings" generator-info-name="bleb2xmltv">
  <channel id="plus1.discoveryeurope.com">
    <display-name>Discovery+1</display-name>
    <icon src="http://discoverychannel.co.uk/_includes/sitewide_images/logo.jpg" />
  </channel>
</tv>

Run the following commands:

xmltv tv_grep --channel-id discoveryeurope.com  epgdata.xml > discovery.xml
xmltv tv_grep --eval "use XMLTV::DST qw(utc_offset); ${$_}{start}=utc_offset(DateCalc(${$_}{start}, '+1 hour') . ' UTC', '+0000'); if ( ${$_}{stop} ) { ${$_}{stop}=utc_offset(DateCalc(${$_}{stop}, '+1 hour') . ' UTC', '+0000'); } ${$_}{channel}='plus1.discoveryeurope.com'; 1 " discovery.xml > discplus1.xml
xmltv tv_sort discplus1.xml  discplus1channel.xml epgdata.xml > epgdata_new.xml

The 3 command lines do:

• extract programs from discovery channel into a separate file
• Add 1hr to start and stop times, and update the channel ID for these program
• Combine the new programs with the new channel definition and the old epgdata to get a new epgdata file with all programs.

(output untested in Sage, but it looks OK...)

(zip file attached containing above command lines as a bat file and the discchannel.xml)

[bmadgett]

Cripes! I should've guessed that XMLTV would have something builtin. And I've just spent a happy couple of hours this morning cobbling together a bit of Java code to do the same thing. D'oh!

My Java thing was going to parse an input XML file (which you would have grabbed to only contain channels you want to make +1), let you add a number of hours to every programme start & stop, prefix something you want to the channel URL (eg 'plus-1.', suffix something to the channel description (eg 'Plus-1'), and put a timezone specifier after every time (eg '+0100'). This would all be output to a new file, which you could then merge with all the rest of the non-plus-1 channels from a separately fetched XML file.

The timezone specifier thing was just because the bleb data doesn't have those in it, whereas the RT XMLTV did, so I was trying to make it look the same to avoid any possible hiccups.

Seems to work ok, but obviously if the functionality is already there in XMLTV then it's all a bit redundant!

Cheers

Barnaby

[ukmgranger]

Is this really the only way to get EPG data now? Bleb just doesn't have the freeview channels that I am using (well about 4 or 5 anyway).

I am with everyone else that is saying that they would pay for a reliable EPG service.

[Kryton]

Originally posted by nielm
[B]Just to let you know that there is now an alternative UK grabber in XMLTV... Nothing to do with me!

It does not grab as much detail as uk_rt, nor as many channels, but it is a start.

Data comes from www.bleb.org

bleb.org can actually outputs the info as an xmltv file rather than having to parse it using xmltv.exe. Just specify &format=XMLTV in the URL
See here for Syntax
http://www.bleb.org/tv/data/listings?

You can download your weeks listings as a zip file in under a minute like this, I use it as such:
my example (copy and edit the url here, this forum added <BR />'s to it)

[nielm]

Latest news on the RT grabber front...

The RT technical guys have got into contact with the XMLTV project admin...

Hi Ed,

I run the technical side of the Radio Times website for BBC
Worldwide. Could you please contact me to discuss XMLTV. As you know
we have blocked the use of XMLTV, but we are willing to work
with you to find a more efficient and less harmfull way for you
to keep this product working.

Looks promising...
(Xmltv development mailing list:
http://news.gmane.org/gmane.comp.tv.xmltv.devel
news://news.gmane.org/gmane.comp.tv.xmltv.devel )

[eldingo]

Thats good news. I wonder how they could solve the problem of it stretching the RT's servers though ?

Unless they are going to look at providing an XML formatted output for XMLTV to pick up, which would save all those requests to the different pages.