WebGrab+Plus , an advanced xmltv epg grabber

[perf]

Originally Posted by WG++Maker

Hopefully I understand you correct that you want what is now in the episode-num element in the sub-title element. You can do that by adding at the end of your dagenstv.com.ini the following :
index_subtitle.modify {addstart(null)|'index_episode'}
This will leave you with the same data in both sub-title and episode-num.

If however, in fact, you don't use the episode-num element at all, you can also just change the line:
index_episode.scrub {single(separator="." include="Säsong ""Del ")|<div class="tbl_lrg_event_description|>|<|</div>}
into:
index_subtitle.scrub {single(separator="." include="Säsong ""Del ")|<div class="tbl_lrg_event_description|>|<|</div>}

After you are happy with the changes it is best to change the name of the dagenstv.com.ini into dagenstv.com.perf.ini and also change the site attribute in the config file from dagenstv.com into dagenstv.com.perf .This to avoid problems with possible future updates of the program or the siteinipack .

hopes this helps WG++Maker ... Jan

Great! Thanks! I did not find the index_subtitle element in the documentation.

I will report back if I have any further problems.

/Per

[WG++Maker]

Perf,

If you go for the second solution you also need to change:
index_episode.modify {cleanup}
into:
index_subtitle.modify {cleanup}

Sorry , I noticed too late that other episode entry ...

WG++Maker ... Jan

[perf]

Jan,

I have started to understand the xmltv-ns episode numbering system, and I am trying to figure out whether I can get WebGrab++ to calculate numbers and format the element based on the information included in my source. I am a bit lost in the modify and calculate operations of WebGrab++.

The source I have contains episode and season information in the formats:

"Säsong 3. Del 5."
or
"Del 5."
or
"Säsong 3. Del 5 av 10."
or
"Del 5 av 10."

As far as I understand xmltv-ns, this should be converted into

"Säsong 3. Del 5." - "2.4.0"
"Del 5." - ".4.0"
"Säsong 3. Del 5 av 10." - "2.4/10.0"
"Del 5 av 10." - ".4/10.0"

Can this be done?

/Per

[WG++Maker]

The following should help you further. Its the most straitforward approach. I haven't tried it so I hope I didn't forget something. This is for dagenstv , right?

input :							
	Säsong 3. Del 5.	Del 5.	 	Säsong 3. Del 5 av 10.	Del 5 av 10.
target result:
	2.4.0			.4.0		2.4/10.0		.4/10.0
	
* scrub the Säsong component from the episode in the html page:
episode.scrub{single(separator=". " include="Säsong")|scrubstring to get the episode}
result episode:
	Säsong 3		empty		Säsong 3		empty

* substract 1 from the numeric component if not "" (empty), result in whole numbers
episode.modify{calculate(not "" format=F0)|1 -} 
result:
	2			empty		2			empty

* scrub the Del component from the episode in the html page in a temp element:	
temp_1.scrub{single(separator=". ""av " include="Del")|scrubstring to get the episode}
result temp_1:
	Del 5			Del 5		Del 5			Del 5

* substract 1 from the numeric component, , result in whole numbers
temp_1.modify{calculate(not "" format=F0)|1 -} 
result temp_1:
	4			4		4			4

* add a period (.) followed by temp_1 to the episode
episode.modify{addend|.'temp_1'}
result episode:
	2.4			.4		2.4			.4
	
* scrub the av component from the episode in the html page in a temp element:
temp_2.scrub{single(separator=". ""av " include="av")|scrubstring to get the episode}
result temp_2:
	empty			empty		av 10			av 10
	
* replace av by /
temp_2.modify{replace|av |/}
result temp_2:
	empty			empty		/10			/10
	
* add temp_2 followed by .0 to the episode
episode.modify{addend|'temp_2'.0}
final result episode:
	2.4.0			.4.0		2.4/10.0		.4/10.0

WG++Maker ... Jan

[perf]

Thanks!

I had some problems using your suggestions. (The calculate operation did not work unless the number was at the start of the string, and using "av" as separator and "include" marker ended up not working - the right separated element was not being picked.)

By trial and error I ended up with this:

index_episode.scrub{single(separator=".""(" include="Säsong")|<div class="tbl_lrg_event_description|>|<|</div>}
index_episode.modify {remove|Säsong}
index_episode.modify{calculate(not "" format=F0)|1 -}
index_episode.modify{addstart|+}
index_temp_1.scrub{single(separator=".""av ""(" include="Del")|<div class="tbl_lrg_event_description|>|<|</div>}
index_temp_1.modify {remove|Del}
index_temp_1.modify{calculate(not "" format=F0)|1 -}
index_episode.modify{addend|.}
index_episode.modify{addend|'index_temp_1'}
index_temp_2.scrub{single(separator="av " include=2)|<div class="tbl_lrg_event_description|Del |.|</div>}
index_temp_2.modify{addstart(notnull)|/}
index_episode.modify{addend|'index_temp_2'}
index_episode.modify{addend|.0}
index_episode.modify{remove|+..0}
index_episode.modify{remove|+}

Now I just need to make sure the numbers are picked up by Sage...

/Per

[perf]

To get StephaneMs xmltv importer to use the season and episode numbers, the episodesystem needs to be:

episodesystem=xmltv_ns

(not xmltv-ns, as the WebGrab++ documentation states).

/Per

[perf]

I am trying to do some more manipulations:

I wanted to create a new, more sortable, subtitle, where I want two-digit numbers for Season and Episode. I tried to force the number format D2 from

http://msdn.microsoft.com/en-us/library/dwhawy9k.aspx

But that does not work for me.

index_temp_3.modify{calculate(not "" format=D2)|0 -}
index_temp_1.modify{calculate(not "" format=D2)|1 +}
index_subtitle.modify{addstart|S'index_temp_3'E'index_temp_1}

the log says:

Channel SVT1.se site -- DAGENSTV.COM.PERF -- update mode full
Unable to update Channel SVT1.se

System.FormatException: Formatspecificeraren var ogiltig.
   vid System.Number.FormatDouble(Double value, String format, NumberFormatInfo info)
   vid WebGrab.Scrub.Format(Double element, String format)
   vid WebGrab.Scrub.Calculate(String element, String condition, String format, Boolean debug, Utils u)
   vid WebGrab.Scrub.Edit(String element, String[] operation, String condition)
   vid WebGrab.Scrub.ScrubOperations(Dictionary`2 operations, Boolean index, Boolean channelfile)
   vid WebGrab.Scrub.IndexShowDetails(String indexshow, SiteIni scrubstrings)
   vid WebGrab.Program.UpdateChannel(String strIndex, ChannelToUpdate Chan, XmlTarget xTarget)
   vid WebGrab.Program.Main(String[] args)

[WG++Maker]

Hi Perf,

sorry that my proposal didn't work. Next time I will check it first.
The things that went wrong:
1. I forgot to add the prefix index_ , you figured that out
2. element.modify {calculate|1 -}
doesn't work when element is not a number because element is not expanded to a number (which is a bug) . To expand it it must be entered like this:
element.modify {calculate|'element' 1 -}
I will fix this bug in the next version.
3. element.scrub {single (separator=". ""av " include="av")|....}
Using the same string in separator and include is rubbish. The separator string is removed from the result, so there is nothing to include.
4. I didn't realise that "av" is a much to common word to use as include string. Asks for troubles.

You did some excellent work!!

Here is my solution that works (at least with the little testing I did)

* the episode scrubs:
index_episode.scrub {single(separator=". " include="Säsong")|<div class="tbl_lrg_event_description|>|<|</div>} * the Säsong part
index_temp_2.scrub {single(separator=". ""av " include="Del ")|<div class="tbl_lrg_event_description|>|<|</div>} * the Del part
index_temp_3.scrub {single(separator="." include="Del ")|<div class="tbl_lrg_event_description|>|<|</div>} * Del and av
*
*disable this:
*index_episode.modify {replace|\||.}
*
* use this to remove the episode data from the description
index_description.modify {remove|'index_episode'}
index_description.modify {remove|.'index_temp_3'}
*
* compose the episode in xmltv_ns  format
index_episode.modify{calculate(not "" format=F0)|'index_episode' 1 -} * convert Säsong s+1 in s
index_temp_4.modify{calculate(format=F0)|'index_temp_2' 1 -} * convert Del d+1 in d, store in index_temp_4
index_temp_3.modify{replace|av |/} * convert Del d+1 av m in Del d+1/m
index_episode.modify{addend|.'index_temp_3'.0} * add together s.Del d+1/m.0 
index_episode.modify{replace| 'index_temp_2'|'index_temp_4'} *replace Del d+1 in d .. s.d/m.0

Further ... :
Thanks for pointing to the correct value episodesystem=xmltv_ns

format=Dn. This was not yet implemented in V1.0.6
It is in the next release V1.0.7
If you want I can send you a pre-release test build of it.
Drop me a mail if you do.

WG++Maker ... Jan

[perf]

Jan,

I have not had time to work further on this for a while. But now I am looking at another thing I would like to include: HDTV information.

Can I add the "quality" tag to the programme information:?

<video><quality>HDTV</quality></video>

/Per

[WG++Maker]

Originally Posted by perf

Jan,

I have not had time to work further on this for a while. But now I am looking at another thing I would like to include: HDTV information.

Can I add the "quality" tag to the programme information:?

<video><quality>HDTV</quality></video>

/Per

The current version doesn´t support the video element. I will have a look if I can include it in the next one.
Q: Does Sage something special with this element in the EPG?

[perf]

Yes, StephaneM's xmltv importer imports the element:

http://forums.sagetv.com/forums/show...&postcount=570

And Sage then prioritises recordings that are in HD.

/Per

[WG++Maker]

According to the xmltv specification the video element has 4 possible subelements:
*present (values yes or no)
*colour (values yes or no)
*aspect (no predifined values but like 4:3 16:9)
*quality (no predifined values but like HDTV or 800*600)

The first two, present and colour, are not imported I assume and maybe not very usefull. Agree?
The other two then:
I have the impression from your link http://forums.sagetv.com/forums/show...&postcount=570 that the importer expects certain values to react upon, as it states :

* video (aspect : 4:3 or 16:9 / quality: SDTV or HDTV)

Does that mean that other values are ignored? Imagine that the epg site list "HDTV" as simply "HD" and aspect as "Wide Screen" or in another language "Breit Bild". If so, the scrubbed value has to be modified into these "importer" standards. (WG++ has that option with the modify command, so its no problem). My question : Am I right that the importer expects certain standard values?

What about audio?
* audio (stereo or dolby digital)

May be of interest : the new version will support the ´boolean´ type elements premiere, subtitles and previously-shown on request of another user.
So if a show is subtitled you it can add
<subtitiles /> to the xmltv

WG++Maker ... Jan

[perf]

I actually do not know, but I agree with your interpretations. I think the importer expects the standard values only, see:

http://forums.sagetv.com/forums/show...&postcount=573

where StephaneM says that the value "HD" will not work - only "HDTV"

/Per

[WG++Maker]

I have added the video elements quality and aspect to the coming next version.

If you don´t want to wait for its release I can send you a test version. Drop me an email in that case.

WG++Maker .... Jan

[WG++Maker]

The new Version 1.0.7Beta is available @ http://www.servercare.nl/Docs/WebGrab+PlusV1.0.7Beta
for the user it adds:
*support for about 25 new sites(see below)! Thanks to Willy de Wilde and Quacke505!! It now covers most of Europe and some countries beyond that.
*New xmltv elements : subtitles, premiere, previously shown, video aspect and quality.
*Timeoffset channels : channels that only differ in start and stop time from another channel.
*new parameters in the retry setting: timeout, channel-delay, index-delay, show-delay
*possibility to disable the skip function of very long and very short shows

and for the siteini developer:
*new preconditionals > < >= and <=
*support for fragmented multi-day index pages
*new attribute 'force' for index_date
*operations addstart, addend, remove and replace will now support multi value elements
*site value retry, overruling the config retry for that channel

Supported sites for the following countries (see the full list in Siteini.Pack.7 @ http://www.servercare.nl/Docs/SiteIni-List-Pack7.pdf):
Belgium(2), Brazil, Croatia, Czech Republic, Denmark(2), Finland, France(2), Germany(2), Hungary, Italy(3), Netherlands(3), Norway, Poland(2), Portugal, Romania(3), Serbia, Slovakia, Spain(4), Sweden(2), Switserland(2), UK(6) and Networks like DirectTV(North America), OSN (Libanon, Saudi Arabia, North African Countries etc), Discovery.

More information @WebGrab+Plus

[milll36]

Hi could you try to add panama Listing, my operator name is cableonda, is the bigger one here i see tvguide.com has it on their listing.

Tks
Daniel

[WG++Maker]

Originally Posted by milll36

Hi could you try to add panama Listing, my operator name is cableonda, is the bigger one here i see tvguide.com has it on their listing.

Tks
Daniel

Hi Daniel,

I have been looking at tvguide.com. It has a very special structure which requires a small program change to grab the xmltv guide. The first results are promissing. I need a couple of days more to get it ready. If you want I can send you a test version at that time. Please send me an Email if you want that. My mail adress is on the cover of the documentation.

WG++Maker .... Jan

[WG++Maker]

The TVGuide.com ini file for WG++ is ready. Since it´s a network site each supported provider needs a separate one. I have made one for Panama Cable Onda as you requested and a few more for US timezones.
Also incluided is a guide how to customize a 'general' TVGuide.com ini for a provider of your choice.
As mentioned, there were some program changes involved, so it needs a pre-public new version of the program. If you want I will send you the whole lot.
In that case , send me an Email.

WG++Maker ... Jan

[karhill]

Originally Posted by WG++Maker

The TVGuide.com ini file for WG++ is ready. Since it´s a network site each supported provider needs a separate one. I have made one for Panama Cable Onda as you requested and a few more for US timezones.
Also incluided is a guide how to customize a 'general' TVGuide.com ini for a provider of your choice.
As mentioned, there were some program changes involved, so it needs a pre-public new version of the program. If you want I will send you the whole lot.
In that case , send me an Email.

WG++Maker ... Jan

Interesting. I have a Panama Cable Onda scapper that generates xmltv format data from the tvguide.com website. I've been using it for my own personal needs (it's a .NET executable), but I'll have to look at WG++, which I'm sure is much less "alpha" than my quick hack. Thanks for this.

[xDude]

If it is right, you´re a blessing.

I'm in Costa Rica and I was trying to figure out how to get this done.
My provider id is "65400" wich is called CableTica

Is this the same structure you´re parsing?
http://online.tvguide.com/listings/d...=0&srvid=65400

Thanks in advance, please keep us updated.