WebGrab+Plus , an advanced xmltv epg grabber

[WG++Maker]

I have made an advanced /multisite/incremental EPG grabber and like to share it with SageTV users

Some background:
* I live in a rural area on the Canary Islands where the internet connections are just dialup speed (no ADSL or 3G). On occasion I can use a satellite downlink . Lets say the speed varies between 2-50kB/s.
* Under these conditions I have been using several epg grabbers, but it always took 'hours' to just the daily update.
* The above was the reason I decided to make my own epg grabber which limits downloading to an absolute minimum without loosing richness of data. With it, the daily download for 25 channels in full detail takes about 10 minutes (sat downlink) to 1 hour (in dial-up)

Some features:
* Incremental grabbing. It only grabs what is new or what needs updated because it is changed or missing.
* Can grab from multiple tvguide sites in one run. Each site is programmable through a SiteIni file (I have made a few Dutch, Belgium, Spanish, English Swedish and Brasilian siteini files so far). These siteinifiles can be made by the user with the help of debugging modes and the documentation. Of course I am prepared to help or make one if requested
* The output follows the xmltv standard.
* Most xmltv elements are supported, like start and stop times, title (+ original title), sub-title, category, rating, star-rating, credits (director, actor, presenter), (production)date and episode. All of which are just a choice of the user and programmable per channel.
* All kinds of other options, like, days to grab, one-show-update-only and site-index-page-only (less detail but super fast) and xmltv-element-value filtering and modification.
* A special option is the ability to compose the epg for combi-channels (the epg listings of different channels for different day periods or different title filters in one listing)
* Written in C# , executable less that 1MB. Tested for XP, Vista and W7. It runs in command mode, no GUI , no drivers to install. FreeWare, Absolutely non commercial, no malware or virus (I wouldn't even know how to get them in it !?)

In function it comes close to tvxb but tests show it is about 2 to 3 times faster while providing a more detailed epg.

The program is available for download on
http://www.servercare.nl/Lists/Posts/Post.aspx?ID=98
Further there is an active user group in Spain on
http://www.todohtpc.com/foro/softwar...-guia-epg.html

WG++Maker -- Jan

[WG++Maker]

Latest version 1.0.2 went public today @
http://www.servercare.nl/Lists/Posts/Post.aspx?ID=98

Supported tvguide sites:

Belgium: tv.be, mijn-tv-gids.be
Brazil: hagah.com.br
Netherlands: tvgids.nl, gids.publiekeomroep.nl, tvgids.upc.nl tvtv.nl
Spain: plus.es, laguiatv.com, teleprogramma.tv
Sweden: dagenstv.com, svt.se
UK: tvguide.co.uk, radiotimes.com, tvtv.co.uk

[jamesdisco]

This looks interesting, I got an email from digiguide.com saying they have just been bought so who knows what might happen, they might stop us from taking xmltv from them.

[WG++Maker]

Hi JamesDisco,

I'm innocent (I didn't buy them to get you to use my grabber)!
But .. it's an idea, buy all the commercial guide sellers ... and ..

Have you tried WG++ ? Which tvguide site lists your channels? Any site you like to be added?

[jamesdisco]

At the moment i use Birtles grabber to get xmltv from www.digiguide.com . Its the only site that lists all channels and radio stations in the UK, it does require a yearly subscription though. I would definately like to try a free provider but I cant see radio on the 3 sites WG++ supports. Post down in the UK section i'm sure there will be people there would like to try this out.

[WG++Maker]

Jamesdisco

do I understand you correct that you want 'radio guide listings' (next to tv guide listings)?
If so, radiotimes.com list radiostations . To support them with WG++ a separate siteini file is needed which I can make if you want. Please tell me if that is what you want.

[WG++Maker]

Originally Posted by jamesdisco

Post down in the UK section i'm sure there will be people there would like to try this out.

Can you give me the link to that uk section? Thanks

[brewston]

Originally Posted by WG++Maker

Can you give me the link to that uk section? Thanks

Its here :

http://forums.sagetv.com/forums/forumdisplay.php?f=25

[WG++Maker]

There is a new version of the program 1.0.4 that went public a couple of days ago.
http://www.servercare.nl/Lists/Posts/Post.aspx?ID=98

Supported tvguide sites:

Belgium: tv.be, mijn-tv-gids.be, yelo.be (new)
Brazil: hagah.com.br
Netherlands: tvgids.nl, gids.publiekeomroep.nl, tvtv.nl
Spain: plus.es, laguiatv.com, teleprogramma.tv, sincroguia.tv (soon)
Sweden: dagenstv.com, svt.se
UK: tvguide.co.uk, radiotimes.com, tvtv.co.uk
Denmark: tvguide.dk (new)
USA and N/S America's : directv.com (new)

[ttriplett]

I am a subscriber to OSNetwork (orbit showtime network, formerly showtime arabia) in Jordan. Any way to get epg data for their HD channels (only 12 channels). I am unaware of a XMLTV feed from osnetwork but most if not all are US channels that might be available from another XMLTV listing... of course the times might be screwed up. Maybe a screen scrape from their online guide would be possible. Any thoughts?

[WG++Maker]

Currently it seems that the online guide of OSNetwork is having some problems on the server side. So I cannot check its 'grabbability' . I try again tomorrow. Can you in the mean time check if any of the sites listed above carries your channels with the same program schedules (possibly timeshifted)? Directv has a lot of US channels.

WG++Maker

[ttriplett]

On closer inspection, it looks like only a few of the channels might be available from a US XMLTV listing. However, their server seems to be working again. Maybe you could check the "grabability"? If you go to osnetwork.com and hover over "TV Guide", then click schedules, then click HD you will get a program listing.

[WG++Maker]

ttriplett,

sorry to have kept you waiting, I needed some time to think about whether osnetwork is grabbable or not. The epg data is in html format, that is good. But it has a very different structure than 'normal' . It cannot be grabbed by WG++ as it is now. The main reason is that the site doesn't provide 'direct' start and stop times but uses grid coordinates instead. These have to be converted into time values which is not supported right now.
As soon as I have some time I will do some experiments and let you know .
Do you think there will be more users of osnetwork that need an xmltv grabber for their epg?
WG++Maker

[ttriplett]

Originally Posted by WG++Maker

ttriplett,

-
Do you think there will be more users of osnetwork that need an xmltv grabber for their epg?
WG++Maker

I would think so since osnetwork covers north africa, mideast, south asia, etc. But who knows.

[WG++Maker]

An update: I am still working on it, but it's a rather complicated site to connect! It needs a few new features of the program to be implemented of which I have a working test version. The first results indicate that epg grabbing can be done, but may be only on sundays!! (This is due to the fact that the site only has weekly schedules which only have a grabable structure on sunday!)

[WG++Maker]

I have completed a first version of a SiteIni file (= site connection file for WebGrab+Plus) for Osnetwork. It grabs 7 days of epg data. There is a limitation though that it will only work on sunday! (See previous post)
It is not yet public, so for anyone interested , send me a private message and I will send you a test version of the setup.

[WG++Maker]

Another update: The setup for OSNETWORK with WebGrab+Plus now works every day (the 'Sunday Only' limitation is gone). It grab 7 days of EPG with title, description, category, producer, actor, (production)date and rating as details. It will be public in a week or so, together with the next program update http://www.servercare.nl/Lists/Posts/Post.aspx?ID=98
People interested before that moment please let me know, as private message or here or at the link above.

[WG++Maker]

There is a new version of the program 1.0.6 http://www.servercare.nl/Lists/Posts/Post.aspx?ID=98

Supported tvguide sites:

Belgium: tv.be, mijn-tv-gids.be, yelo.be
Brazil: hagah.com.br
Netherlands: tvgids.nl, gids.publiekeomroep.nl, tvtv.nl
Spain: plus.es, laguiatv.com, teleprogramma.tv, sincroguia.tv (improved)
Sweden: dagenstv.com, svt.se
Norway : sol.no (new)
UK: tvguide.co.uk, radiotimes.com, tvtv.co.uk, bfbs.com (new British Forces Broadcasting Services)
Denmark: tvguide.dk
USA and N/S America's : directv.com
Far East : osnetwork.com (new)

Just drop a message if you want support for a site that is not in the list ...

[perf]

I will start using WG++ for Sweden, with data from Dagenstv.com. However, I need the sub-title element to be equal to the episode element. I have not been able to modify the .ini file to achieve that. How should I do?

(Part of the problem is that all information is on the index page - there are no show details pages - and sub-title can not be scrubbed from the index page.)

/Per

[WG++Maker]

Originally Posted by perf

I will start using WG++ for Sweden, with data from Dagenstv.com. However, I need the sub-title element to be equal to the episode element. I have not been able to modify the .ini file to achieve that. How should I do?

(Part of the problem is that all information is on the index page - there are no show details pages - and sub-title can not be scrubbed from the index page.)

/Per

Hopefully I understand you correct that you want what is now in the episode-num element in the sub-title element. You can do that by adding at the end of your dagenstv.com.ini the following :
index_subtitle.modify {addstart(null)|'index_episode'}
This will leave you with the same data in both sub-title and episode-num.

If however, in fact, you don't use the episode-num element at all, you can also just change the line:
index_episode.scrub {single(separator="." include="Säsong ""Del ")|<div class="tbl_lrg_event_description|>|<|</div>}
into:
index_subtitle.scrub {single(separator="." include="Säsong ""Del ")|<div class="tbl_lrg_event_description|>|<|</div>}

After you are happy with the changes it is best to change the name of the dagenstv.com.ini into dagenstv.com.perf.ini and also change the site attribute in the config file from dagenstv.com into dagenstv.com.perf .This to avoid problems with possible future updates of the program or the siteinipack .

hopes this helps WG++Maker ... Jan