XMLTV 5.28 Problems

[ron.smythe]

Is anyone having any problems with XMLTV 5.28. I had it workin two days ago downloading one day of info at a time. The command I type is: C:\XMLTV528>xmltv tv_grab_uk_rt --days 2 --slow --share="c:/XMLTV528/share/XMLTV" --output epginfo.xml

the download takes an age with a smilimar message to this:

time 20040207000000, channel Sky Cinema 2: #http://www.radiotimes.com/Listi
ngsServlet?event=10&channelId=250&programmeId=16050082&jspLocation=/jsp/prog_det
ails.jsp: discarding information about the parts played by each actor

and then the epgdata.xml file only has the following in it:

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE tv SYSTEM "xmltv.dtd">

<tv source-info-url="http://www.radiotimes.com/" source-info-name="Radio Times" generator-info-name="XMLTV" generator-info-url="http://membled.com/work/apps/xmltv/">

[danward79]

Hi Ron

I had loads of probs with it not working, had to switch back to .27 xmltv, to get it to work and use the hosts hack.

My files were not recognised by sage and they also ended up with --share something or other in them, there is a few posts about it around

Dan

[jimbobuk]

I just suffered this.. bloody hell just as Sage had managed to catch up with a semi-decent EPG enough to start copying again.. this situation is really annoying... is everyone getting this problem?! does it go away... will EPG grabs start working again..

I sadly can't go back to the .27 pack unless someone has that patched and working in a windows exe.. i've had other computer woes recently and until they are sorted i aint going to install cygwin just yet.

I wish there were more of us so Sage would push for getting some UK listings.. sadly as it stands no one is going to be attracted to Sage in the UK with it being crippled in its EPG...

[jimbobuk]

I'm getting these errors as well

http://www.radiotimes.com/ListingsSe...rogrammeId=161
44856&jspLocation=/jsp/prog_details.jsp: unknown remnant bit (adding to descript
ion): Nia Long at /PerlApp/grab/uk_rt/tv_grab_uk_rt line 1027.

[nielm]

I'm getting these errors as well

Radiotimes has changed the format of it's cast list... you will probably see the entire cast appended to the description...

I already have a possible fix, and it will be in the next version along with a fixed (and faster) fast mode a fixed 'adding --share=' message, and possibly (if I get it approved by XMLTV maintainer) a way of only getting detailed info for programs that start within a certain time range (thus reducing page grabs, and increasing speed)

This error shouldn't corrupt the actual XML file, nor prevent the download from completing though...

[jimbobuk]

Thanks Neilm...

Yet more fixes sound interesting.. My rip today went ok so i decided to let the one that seemed to fail try again (with smaller offset obviously, being one day later ) .. that seems to have worked ok as well..

ANyways you hint at getting faster rips anyways by selecting times to get detailed info for.. I've just noticed something very bizarre.. I didn't time this but i'm 80% certain that my RT rip today spent about 80% of its time doing 0:00 -> 06:00 ... I started it running.. and left it for a long time.. i'd say an hour perhaps.. coming back expecting it to be nearly finished to see its still at 06:00 ... gutted at the idea of it taking 3 hours instead of 1.5 i sat back and left it running... then only 10-15 mins later if that i looked and saw it was already at 12:00 ... the same again or less and it had practically done... why do these times take so long to grab!? its almost the most dead period of the day early mornings.. its certainly something that i'll very rarely record, especially as when i sleep i generally power down the machine.. so as far as being able to speed things up i think there is a lot of scope here.. but i'm just not sure why the time difference is so big

Any thoughts?

[nielm]

I can't explain why certain periods takes so long it could be any number of factors -- I have noticed that sometimes the RT website is VERY slow (probably all these people doing XMLTV rips!)...

This is compounded by the 0.5.28 (and lower) grabber doing a very large number of web page 'gets': it retrieves a page from the RT website for each 2-hour timeslot, for each channel, and then gets another page with detailed info for each programme found..
On average this means approx 50 web pages/1.5Mb per channel per day!

0.5.29 will be optimised -- 2 pages/200Kb per channel per day in fast mode, ~ 40 pages/800Mb of data per channel per day in slow mode, or ~20 pages/500kb per channel per day if you limit getting detailed info to programmes starting between 16:00 and 01:00.

And at some point in the near future there will be a new tv_grab_uk... But I have no idea when!

[jimbobuk]

0.5.29 will be optimised -- 2 pages/200Kb per channel per day in fast mode, ~ 40 pages/800Mb of data per channel per day in slow mode, or ~20 pages/500kb per channel per day if you limit getting detailed info to programmes starting between 16:00 and 01:00.

Is this a typo Nielm?

40 pages.. fine.. but 800Mb of data??? you just said unoptimised its about 1.5meg a day a channel..

sounds good tho

[nielm]

> but 800Mb of data???

Oops! yep, that should be Kb!

[jimbobuk]

Ok.. I've had the problem with it not grabbing EPG properly... I've looked more closely this time.. its definately midway through i think (6am bizarrely i'm certain its more than load on their server.. 0-6 seems to take an age) .. anyways here's the output as it failed..

http://www.radiotimes.com/ListingsSe...rogrammeId=162
31628&jspLocation=/jsp/prog_details.jsp: unknown remnant bit (adding to descript
ion): Anna Massey at /PerlApp/grab/uk_rt/tv_grab_uk_rt line 1027.
#http://www.radiotimes.com/ListingsSe...=%2Fjsp%2Ftv_l
istings_grid.jsp&jspListLocation=%2Fjsp%2Ftv_listings_single.jsp&jspError=%2Fjsp
%2Ferror.jsp&searchDate=15/02/2004&searchTime=06:00&channels=249: cannot parse d
ate '20040215 French Twist)  (10:00pm' at /PerlApp/Memoize.pm line 247

which is the usual line 1027 error of which there were 1000s before this break.. but then there is a new one at the end.. cannot parse... pretty bad.. real bad that it stops the grab, rather than continuing.. dunno if i had a net connection issue.. any idea on what this fatal error is!? and why its so fatal.. I'd accept programmes being unknown in the final data rather than the whole epg being stopped..

The one thing i do hate is windows redirecting to a file command.. why the hell can't it write to the file as it goes... after all this work you are left with nothing!!! i'm assuming thats not xmltv's fault (with it not outputting till the end?)

ANyways.. dont know what to do.. if you're there nielm and you have a clue could you chime in?! is it worth trying to grab again?! i've had this error before and a later grab seemed to work..

[nielm]

Just realised why grabs at 6am take so long: the data generated by the 6am page get is actually for the next 24 hours, so it gets detailed info for all of these programmes.. This is why it prints so many '#' characters (if you can see them between the warnings!) for the 6am grab (1 # per programme)

The next 12 grabs for 08:00-22:00 and 00:00-04:00 are redundant -- it sees that it already has details for the found programmes and ignores them. This is one of the things that is optimised in the next version...

The error: The reason why it stops at that point is because it does not handle the brackets in the title of the film:
Gazon Maudit (French Twist)
because it expects start and stop times to be in brackets... if you view the source at the URL listed, the offending bit of HTML is:
title="Gazon Maudit (French Twist) (10:00pm-11:50pm)"

This should already have been fixed in the v0.5.28 -- which version are you running (xmltv.exe --version)

In general, the grabber dies whenever it cannot find something in the HTML that it is expecting to find... This is brutal, but possibly safer then producing corrupt data, and the XML is only output at the very end of the grabbing process...

[jimbobuk]

Thanks Nielm.. I thought something was going off.. so the 6am page is the entire contents of that day!? isn't that then 1 page hit per day!?

Fair enough on the brackets although bad news.. I am indeed running 0.5.28 so in this case it doesn't appear to be fixed

its a days offset 4 grab today if you want to try and replicate it..

In short this isn't going to change is it so there is no point in attempting to grab again?

I use the small output as a check for whether the process has worked ok... this never needed checking before but what the hell.. its gone wrong twice now... the last one repaired itself..

Cheers

[nielm]

> I am indeed running 0.5.28

My bad -- I tested .28 on the wrong day!
I have found and fixed the problem, and I have submitted a fix for this to Ed... I will let you know when there is a new alpha 0.5.29 exe...

[nielm]

an alpha EXE of xmltv has been built which *should* contain my fixes for the problems found in 0.5.28: alpha-exe.xmltv.org.

I have not yet downloaded and tested it (I am not at home this weekend) but it may be useful for those who are having problems!

The tv_grab_uk_rt now has a --limit-details option so that you only need to get details for programs that happen at a time you are likely to be interested in... eg 17:00-01:00. The outut for each timeslot now shows a '#' for each programme found, and a '@' when it is getting detailed info for a programme.

You can of course now create 3 config files

• one for channels where you only want title info (fast mode),
• the second for channels with partial details (--slow --limit-details 17:00-01:00)
• the third for channels where you want full details for all programmes (--slow)

and then use tv_cat to combine the output files into a single file:
tv_cat 1.xml 2.xml 3.xml --output epgdata.xml

[nielm]

XMLTV 0.5.29 has now been released at files.xmltv.org

[jimbobuk]

thanks

[danward79]

Neilm, the uk rt rippe is not in it!!

Dan

[nielm]

f:\Downloads\xmltv\xmltv-0.5.29-win32>xmltv tv_grab_uk_rt --days 1
Timezone is CET
adding '--share=f:/Downloads/xmltv/xmltv-0.5.29-win32/share/xmltv'
cannot find f:/Downloads/xmltv/xmltv-0.5.29-win32/share/xmltv/tv_grab_uk_rt/channel_ids, 
xmltv was not installed correctly

The grabber is included, but it seems a file is missing... you can copy that file from 0.5.28's zipfile, or get it directly from the sources on Sourceforge -- just save it in the directory that your error message warns you about.

[danward79]

Ah I got it now, cheers Neil. I tested it for one day so far using limit and fast then tv_cat, it worked well, I am now doing a full download, so in a long while i will beable to test that.

Cheers

Dan