WebGrab+Plus , an advanced xmltv epg grabber

[WG++Maker]

I made a fast, incremental, multi-site, full detail epg grabber. Output is standard xmltv.
Some test show it's about 2-3 times faster than tvxb while grabbing more details.
See : http://forums.sage.tv/forums/showthread.php?t=52849 and
http://www.servercare.nl/Lists/Posts/Post.aspx?ID=98

Supported tvguide sites for the UK :
tvguide.co.uk, radiotimes.com and tvtv.co.uk

other countries:
Belgium: tv.be, mijn-tv-gids.be
Brazil: hagah.com.br
Netherlands: tvgids.nl, gids.publiekeomroep.nl, tvgids.upc.nl and tvtv.nl
Spain: plus.es, laguiatv.com, teleprogramma.tv
Sweden: dagenstv.com, svt.se

Other sites coming and on request.

[birty]

if you want fast uk listings please don't scrape listings from the radio times web pages use the official feed at xmltv.radiotimes.com instead. radio times have been aggressive in blocking listings scrapers in the past and I assume that your grabber will suffer the same fate

[WG++Maker]

There is a new version of the program 1.0.4 that went public a couple of days ago.
http://www.servercare.nl/Lists/Posts/Post.aspx?ID=98

Supported tvguide sites:

Belgium: tv.be, mijn-tv-gids.be, yelo.be (new)
Brazil: hagah.com.br
Netherlands: tvgids.nl, gids.publiekeomroep.nl, tvtv.nl
Spain: plus.es, laguiatv.com, teleprogramma.tv, sincroguia.tv (soon)
Sweden: dagenstv.com, svt.se
UK: tvguide.co.uk, radiotimes.com, tvtv.co.uk
Denmark: tvguide.dk (new)
USA and N/S America's : directv.com (new)

[WG++Maker]

Originally Posted by birty

if you want fast uk listings please don't scrape listings from the radio times web pages use the official feed at xmltv.radiotimes.com instead. radio times have been aggressive in blocking listings scrapers in the past and I assume that your grabber will suffer the same fate

Hi Birty,

thanks for the advice. What actually happens when they block the grabber?
Another question: Why can't I find any xmltv feed when I go to xmltv.radiotimes.com ? I think I am stupid or blind! Can you give me a link for it?

[birty]

i seem to remember you just get blank pages
the list of channels is at http://xmltv.radiotimes.com/xmltv/channels.dat
data for each channel is at http://xmltv.radiotimes.com/xmltv/<id>.dat, eg: http://xmltv.radiotimes.com/xmltv/92.dat

[WG++Maker]

There is a new version of the program 1.0.6 http://www.servercare.nl/Lists/Posts/Post.aspx?ID=98

Supported tvguide sites:

Belgium: tv.be, mijn-tv-gids.be, yelo.be
Brazil: hagah.com.br
Netherlands: tvgids.nl, gids.publiekeomroep.nl, tvtv.nl
Spain: plus.es, laguiatv.com, teleprogramma.tv, sincroguia.tv (improved)
Sweden: dagenstv.com, svt.se
Norway : sol.no (new)
UK: tvguide.co.uk, radiotimes.com, tvtv.co.uk, bfbs.com (new British Forces Broadcasting Services)
Denmark: tvguide.dk
USA and N/S America's : directv.com
Far East : osnetwork.com (new)

Just drop a message if you want support for a site that is not in the list ...

[PKondrat]

Hi,
I'll appreciate if you could add support for Polish channels using one of the following websites www.telemagazyn.pl or http://tv.wp.pl/ or www.teleman.pl.

Regards,
PKondrat

[WG++Maker]

Originally Posted by PKondrat

Hi,
I'll appreciate if you could add support for Polish channels using one of the following websites www.telemagazyn.pl or http://tv.wp.pl/ or www.teleman.pl.

Regards,
PKondrat

I am working , just now, on http://www.tv.wp.pl and http://www.cyfrowypolsat.pl/program-tv/

If I have time I will have a look at the others you mention.

Within a week or two there will be a new version of the program which will include support for at least one Polish site.

WG++Maker ... Jan

[WG++Maker]

A new version of WebGrab+Plus (V1.0.8 Beta) is available:

It now supports 66 tvguide sites and comes in two installation variants:

-- The standard .msi variant is available here: (http://www.servercare.nl/Docs/WebGra...0.8BetaMsi.rar)
-- and the ClickOnce variant here: (http://www.servercare.nl/Docs/WebGra...aClickOnce.rar)
(the ClickOnce installation was use for all the previous versions and is kept available for those user that want to continue using utilities or batch scripts that rely on the file structure that is typical for this type of installation)

Supported sites for the following countries :

Belgium(2), Brazil, Croatia, Czech Republic, Denmark(2), Finland, France(3), Greece, Germany(2), HongKong, Hungary, Israel, Italy(5), Litauen, Netherlands(5), Norway(2), Poland(2), Portugal(2), Romania(3), Serbia, Singapore, Slovakia, Republic South Africa(2), Spain(4), Sweden(3), Switserland(2), Turkey, UK(6) and
International Networks like: DirectTV(North America), OSN (Libanon, Saudi Arabia, North African Countries etc) Discovery, TvGuide.com (N/M/S-America and Caribbean), BBC, Australia Network and RadioAustralia.

A complete list of supported tvguide sites can be had at: (http://www.servercare.nl/Docs/SiteIni-List-Pack8.pdf)

For detailed information for users and siteini developers read the documentation: (http://www.servercare.nl/Docs/Docume...V1.0.8Beta.pdf)

For support, siteini request and more information (http://www.servercare.nl/Lists/Posts...a97ea371&ID=98)
or just mail (my mail address is on the cover of the documentation)

Wg++Maker .... Jan

[WG++Maker]

Sorry but there is a wrong setting in the configfile, a forgotten leftover from an experiment !
<retry show-delay=15>4</retry> should be <retry>4</retry>

(remove the show-delay, it pauses for 15 seconds between the grab of any show!!)

The error is corrected in the mean time.

again sorry .... wg++maker .. Jan

[tvmaster2]

Originally Posted by WG++Maker

Sorry but there is a wrong setting in the configfile, a forgotten leftover from an experiment !
<retry show-delay=15>4</retry> should be <retry>4</retry>

(remove the show-delay, it pauses for 15 seconds between the grab of any show!!)

The error is corrected in the mean time.

again sorry .... wg++maker .. Jan

So has anyone used this with Sage?

[WG++Maker]

Originally Posted by tvmaster2

So has anyone used this with Sage?

I get support requests from Sage users so I assume they use it. See also
http://forums.sagetv.com/forums/show...t=52849&page=3

WG++Maker .. Jan

[WG++Maker]

A new version update (1.0.9) is available @ServerCare.nl

It offers for the user:
- Support for 80 TvGuide sites World Wide
- Introduction of the option to load a cookie file for sites that require this.

And for the SiteIni developer:
- A lot effort has gone into improving, extending and bug fixing of the scrub engine, allowing to capture a wider range of TvGuide websites.
- Further completion of the operation calculate, adding time and date calculations.

Supported sites for the following countries (see the full list of supported sites):
Australia, Belgium(2), Brazil, Croatia(2), Czech Republic, Denmark(2), Finland, France(3), Germany(2), Greece, HongKong, Hungary, Iceland(3), Israel, Italy(5), Litauen, Netherlands(6), New Zealand, Norway(2), Poland(3), Portugal(3), Romania(3), Serbia(2), Singapore, Slovakia, Slovenia(3), Republic South Africa(2), Spain(6), Sweden(3), Switserland(2), Turkey, UK(6) and International Networks: DirectTV(North America), OSN (Libanon, Saudi Arabia, North African Countries etc), Discovery, TvGuide.com(N/M/S America and Caribbean), BBC, Australia Network and RadioAustralia.

Download links:
MSI installer package (installs the whole lot, program, all the siteini files, documentation)
SiteIni.Pack (apart from the siteini files it gets you the channellist files for nearly all the available sites)
Full list of supported tvguide sites.
Documentation only.

WG++Maker ... Jan

[WG++Maker]

[12-April-2012] WebGrab+Plus Version update V1.1.0
This version adds an optional postprocessor that will add IMDb data (from IMDb.com) automatically to the xmltv output file. It does that in 4 steps:
1. Select shows from the xmltv input as candidate for postprocessing based on configurable selection settings
2. Tries to match the selected shows with IMDb entries, based on configurable matching settings.
3. Collect IMDb data for matched shows (Data collected : title, starrating, plot, description, review, viewer comments)
4. Merge of the collected IMDb data with the source xmltv data based on configurable allocation and presentation settings.

It also maintains a local mdb datafile that enables to re-use the data without timeconsuming reconnection with IMDb.com for the same show.

Many thanks to Jacon Berggren and Karsten Kugel for the apha testing and all the usefull suggestions for improvements.

Apart from this new postprocessor, the functionality of the EPG is improved with the option to grab from a subdetail html page. This can be used if certain epg data is on a available on a different html page than the two already available (index - and detail page)

[12 April 2012] Site.Ini.Pack10.0
Now containing siteini files for more than 100 Tvguide sites.
New siteini files for : Austria, Denmark, Germany, Hungary, Ireland, Poland, Sweden, UK and Spain (Basque) and updates for 22 siteini's.
Many thanks again to Willy de Wilde for contributing a lot of the new sites and updates.

Visit:
http://www.servercare.nl/Lists/Posts...ost.aspx?ID=98

Download links:
The program:
http://www.servercare.nl/Docs/WebGrab+PlusV1.1.0MSI.rar
The siteini pack:
http://www.servercare.nl/Docs/SiteIni.Pack.10.rar
The list of all supported tvguide sites:
http://www.servercare.nl/Docs/SiteIni-List-Pack.10.pdf
The documentation:
http://www.servercare.nl/Docs/Documentation.V1.0.9.pdf
A small utility that automatically installs the required .net versions for the msi installer version:
http://www.servercare.nl/Docs/Instal...pendencies.rar

WG++Maker ... Jan

[WG++Maker]

New Version of WebGrab+Plus V1.1.1

This version adds a few major features:

1. Another postprocessor REX (Reallocate and Edit Xmltv) that can
- Move the content of xmltv elements to other xmltv elements
- Merge the content of several xmltv elements
- Add comments/prefi/postfix text
- Remove or create xmltv elements
E.g. If the PVR software do not recognize some of the xmltv elements grabbed (like actors) it can add
these to the description and remove the now useless actor xmltv element.

2. Improved MDB postprocessor (automatically adds IMDB data to your xmltv file)

3. Added support for LINUX (in combination with MONO) and tested in W8

4. A command line parameter that specifies and overrules the default homefolder-path of the program for
user that prefer another location of it.

5. Support for sites that use the POST HttpWebRequest method, further extending the range of websites.

6. The installation packages contain ini files for 126 TvGuide websites in about 60 countries worldwide!!

... and , on top of that, many small improvements and bugs fixed as usual.

Download links:
For LINUX:
http://www.servercare.nl/Docs/WebGra...1.1.1LINUX.rar
For Windows:
http://www.servercare.nl/Docs/WebGrab+PlusV1.1.1MSI.rar
The documantation:
http://www.servercare.nl/Docs/Documentation.V1.1.1.pdf
All the available siteini files:
http://www.servercare.nl/Docs/SiteIni.Pack.11.rar
The list of all available supported TvGuide sites:
http://www.servercare.nl/Docs/SiteIni-List-Pack.11.pdf

For more info and support:
http://www.servercare.nl/Lists/Posts...ost.aspx?ID=98
WebSite in preparation:
http://www.webgrabplus.com

Many thanks to everyone contributing to the development, especially Alberto and Willy.

WG++Maker ... Jan

[starfire]

Just given this a go but keep getting an error after a short period:

WebGrab+Plus/w MDB & REX Postprocess -- version 1.1.1/18 -- Jan van Straaten


many thanks to Paul Weterings, whose WebGrab was a great inspiration
----------------------------------------------------------------------

processing C:\ProgramData\ServerCare\WebGrab\guide.xml ......
Update requested for - 1 - out of - 1 - channels for 7 day(s)
Update mode - set per individual channel


      i=index  .=same  c=change  g=gab  r=replace  n=new

Channel 4 updating, using site UK-SKY.COM, mode incremental
iiiiiiinnnnnnnnnnnnnnnnnnnnnnnnnnnn
Error downloading page: The remote server returned an error: (500) Internal Serv
er Error.
Pausing 1 of 4 times for 5 seconds before re-try.

Error downloading page: The remote server returned an error: (500) Internal Serv
er Error.
Pausing 2 of 4 times for 10 seconds before re-try.

it keeps re-trying but keeps timing out, any ideas?

[WG++Maker]

Originally Posted by starfire

Just given this a go but keep getting an error after a short period:

WebGrab+Plus/w MDB & REX Postprocess -- version 1.1.1/18 -- Jan van Straaten


many thanks to Paul Weterings, whose WebGrab was a great inspiration
----------------------------------------------------------------------

processing C:\ProgramData\ServerCare\WebGrab\guide.xml ......
Update requested for - 1 - out of - 1 - channels for 7 day(s)
Update mode - set per individual channel


      i=index  .=same  c=change  g=gab  r=replace  n=new

Channel 4 updating, using site UK-SKY.COM, mode incremental
iiiiiiinnnnnnnnnnnnnnnnnnnnnnnnnnnn
Error downloading page: The remote server returned an error: (500) Internal Serv
er Error.
Pausing 1 of 4 times for 5 seconds before re-try.

Error downloading page: The remote server returned an error: (500) Internal Serv
er Error.
Pausing 2 of 4 times for 10 seconds before re-try.

it keeps re-trying but keeps timing out, any ideas?

I am puzzled why this happens , but fortunately it works with grabengine wget.
Add :
site {grabengine=wget}
to the siteinifile uk-sky.com.ini

WG++Maker .. Jan

[silkshadow]

Jan, I just wanted to thank you so much for working on the Philippines guide data!!

I'm still testing it and passed it on to 3 others here who are using MCE to test. So far so good, though the website itself has stopped publishing info for some channels that it had just a couple weeks ago. I am looking into that to see if its coming back or what happened.

I'll be emailing you a by channel report when I find out if those channels will come back. Thanks again!

[WG++Maker]

As of today there is also another place to get support for WebGrab+Plus .

Visit its new website http://www.webgrabplus.com/

See you there --- WG++Maker --- Jan

[johndb]

is there any way you could post the files i need to run with webgrabplus which will just grab all uk channels from radio times and guardian.

then i can just choose which to import into stephanems importer.

i just cant get my head around how to set it up.

thanks

john