MyTheatre and New XMLTV Import Tool

[rsagetv99]

Anyone using the new XMLTV Import tool by StephaneM importing MyTheatre guide data?

I am interested to find out how well it works in identifying repeat airings of the same show.

[StephaneM]

Originally Posted by rsagetv99

how it works in identifying repeat airings of the same show.

Identification of shows are based on:

* title
* episode title or description or air date (your choice)

The Importer will create an ID for a series of shows, that is all shows that share the same title will be part of the same serie. Then it will create a # ID for a show in the serie if the ep. title / description / air date is not the same as another one.

It will also "remembers" all assigned IDs, so if Ep 1 of a series was aired last year and is being imported again, the Importer will assign it the same ID (this of course doesn't work if you chosse the air date as a distinction method and no episode title is given in your XMLTV file). Also it will tell you that the show is a repeat.

The whole process works if you have more than one XMLTV file.

Regards,
Stéphane.

[rsagetv99]

Thanks for taking the time to explain. I had a pretty good idea of how it worked from your thread. What I really wanted to know was how "well" it worked.

Please do me a favor and don't mis-quote me. Not a big deal, but you could have just said, "I know you wanted to know how "well" it worked, but in case you were wondering, this is "how" it works."

Instead you changed my wording which aint cool.

[StephaneM]

Originally Posted by rsagetv99

Please do me a favor and don't mis-quote me. Not a big deal, but you could have just said, "I know you wanted to know how "well" it worked, but in case you were wondering, this is "how" it works."

This is exactly why I changed your wording.

Now, I really don't think anybody will tell you "how well it works", that's why I replied to your post with "how it works".

You should be able with this explanation to respond to your first question :

* you know how are your MyTheater XMLTV data
* you know how works the Importer

you can now deduce how well it will work.

You can also try for yourself. What I would do is : make a backup of your SageTV installation and try the new Importer and see if it handle the show as you want it to (depending on your data you'll have to choose between the "air date" or "description" identity detection method, usually I would recommend the "description" method).

And if it doesn't work as you want it you can safely go back to your backup (the backup guarantee that the operation will not mess your historical data).

Then you can always come to me to see if there is a solution or not.

Currently one thing I can try to improve is how to reduce the use of the "EPxxxxxxx" IDs. Those are dedicated to series usually, and the Importer will assign this type of IDs for any show that has more than one episode (wether it's right or not, and except for movies if your data does have a "movie" category). It would be better if the Importer use only "SHxxxxxx" IDs for shows that occasionally do have more that one episode (for instance a "news" show with one special edition per week). Handling this kind of situation would require extensive use of categories for shows and exception list and I'm considering if it's not too difficult to setup for users (my requirements for the Importer is not too have billions of options, it should work with no questionning / difficult set up).

I will surely begin with a way to assign new categories for shows and I will see then.

Regards,
Stéphane.

[rsagetv99]

OK well since you sound like you are willing to consider some changes I am willing to give your software a shot.

One thing to consider right off the bat is that both Zap2it and MyTheatre data provide star-ratings for movies so if you are looking for an easy way to identify a movie and assign it the appropriate category as well as the appropriate EP/SH ID classification you might have your importer check for the presence of the star-rating element.

Also, is it possible to have your importer derive the channel number from the xml file? The data is there, it's just a matter of extracting it. Alternatively I can have a script do it, but there have been a couple of other importers that could do it without my script.

[StephaneM]

Originally Posted by rsagetv99

One thing to consider right off the bat is that both Zap2it and MyTheatre data provide star-ratings for movies so if you are looking for an easy way to identify a movie and assign it the appropriate category as well as the appropriate EP/SH ID classification you might have your importer check for the presence of the star-rating element.

Movies are not the problem (with Zap2it at least), this is for other shows : I do not know for sure if category A needs "EP" (except for obvious ones) or category B needs "SH".

So I must think about it some time (I was a BTV user and still need more than one year of SageTV usage to fully tweak the beast with data that are available to me : that is French TV data)

To make it short, the general issue is : I know (as a human being) some facts about TV shows but I can't expect an algorithm to know it about unless I tell him about it (manually).

I was thinking about a simple way to do this : be able to go in the Importer and lookup up for a particular show and then choosing to "correct" what the Importer think about the show in the first place. Something like : this show is a series with known episodes / this show is a series but no episode is known / All "Friday" airing are "New" (all following are repeats of this one) etc...

Also, is it possible to have your importer derive the channel number from the xml file? The data is there, it's just a matter of extracting it. Alternatively I can have a script do it, but there have been a couple of other importers that could do it without my script.

As I already told you I do prefer that you use a script to edit the XML files (I you want some hints about quick and dirty XML handling I can help you, I already submitted some script to edit xml / xmltv file directly). I want to adhere strictly to the XMLTV DTD : this is the best way to ensure that something you do for XMLTV produced by grabber X will not interfere with other XMLTV sources (and thus not having to add much user options in the Importer).


Regards,
Stéphane.

[xred]

Originally Posted by rsagetv99

Also, is it possible to have your importer derive the channel number from the xml file? The data is there, it's just a matter of extracting it. Alternatively I can have a script do it, but there have been a couple of other importers that could do it without my script.

I've been out of the forums for a while, but has there be any progress on this? I just reactivated my XMLTV lineup after not using it for a couple of months and promptly had to re-edit about half my channels... so that means the manual process of adding the channel # in, then stripping the channel # out of the callsign (so the channel icons work again) then 0 out all the duplicate channels and assign all the 0 channels to a "garbage" group

There has to be a better way to do this.... I've previously asked if the channel editor could just automatically "zero out" blank channels or have the importer ignore any channel with no number assigned, but I don't think either of those are implemented.

[StephaneM]

Originally Posted by xred

There has to be a better way to do this.... I've previously asked if the channel editor could just automatically "zero out" blank channels or have the importer ignore any channel with no number assigned, but I don't think either of those are implemented.

Well this is a great idea, I'll introduce it in an upcoming version (ignoring channels with no channels number).

Regards,
Stéphane.

[xred]

Originally Posted by StephaneM

Well this is a great idea, I'll introduce it in an upcoming version (ignoring channels with no channels number).

Regards,
Stéphane.

Just went through another 2 hour editing session here after updating my channels in Mytheatre..... If that wasn't bad enough, I ran the importer and had to watch it run the import on both my "satellite" lineup (#1) and "garbage" lineup (#2).

Stephane, is there any way you would consider taking all the channels that are automatically "zeroed out" (channels with no number as suggested above) and automatically assigning them to their own default lineup.. lets say lineup #9 and and name it something like "unused channels". Then when the importer runs it would know NOT to process lineup #9? This would make junk collection after editing a LOT easier and the import itself run a lot faster for users with lots of junk channels.

I'm willing to pay another registration fee to see this feature incorporated (yes I hate the editing sessions that much!).

[StephaneM]

Originally Posted by xred

Stephane, is there any way you would consider taking all the channels that are automatically "zeroed out" (channels with no number as suggested above) and automatically assigning them to their own default lineup..

Yes, this will be in the next version (with other things too), should be out in one or two weeks from now. Any channel with blank channel number will be ignored (so you will not need any garbage lineup).

Regards,
Stéphane.

[grooves12]

Originally Posted by StephaneM

As I already told you I do prefer that you use a script to edit the XML files (I you want some hints about quick and dirty XML handling I can help you, I already submitted some script to edit xml / xmltv file directly). I want to adhere strictly to the XMLTV DTD : this is the best way to ensure that something you do for XMLTV produced by grabber X will not interfere with other XMLTV sources (and thus not having to add much user options in the Importer).


Regards,
Stéphane.

I am looking for this info... can you give a quick rundown on some basic scripting applications and techniques for editing the xml data??

[StephaneM]

Hello,

Originally Posted by grooves12

I am looking for this info... can you give a quick rundown on some basic scripting applications and techniques for editing the xml data??

Usually I use VBScript with the MSXML parser. It's pretty straightforward if you know VBScript. Then you only have to get used to XSL Path to query the XML DOM.

The main advantage is that it's the quickest DOM parser I know of.

Tell me what you exactly want to do and I'll give you a basic workable script (you'll be able to customize it).

Regards,
Stéphane.

[xred]

Doesn't look like theres any scripts imminent so I'm starting up a bounty!

I'll put up $10 USD for anyone that can create a North American user script for Stephanes XMLTV importer that automatically;

- Extracts the channel number (first set of digits) from the "Name" column and automatically puts it in the "Number" column.

- Extracts the call sign (second set of characters) from the "Name" column and automatically inserts it in the "Call Sign" column.

- Lets you select which set of "XMLTV ID" you would like perform the above two operations on. To clarify, if I have two source XML files (in my case one from Zap2it, one from MyTheatre) I would like to be able select which set of XMLTV ID's to run the operations against.

I'm sure I can't be the only user who has had to endure marathon sessions in the channel editor after I rescan the satellite transponders, or channels are added / deleted from the lineups etc etc

[StephaneM]

- Extracts the channel number (first set of digits) from the "Name" column and automatically puts it in the "Number" column.

- Extracts the call sign (second set of characters) from the "Name" column and automatically inserts it in the "Call Sign" column.

Ok, that is understandable

- Lets you select which set of "XMLTV ID" you would like perform the above two operations on. To clarify, if I have two source XML files (in my case one from Zap2it, one from MyTheatre) I would like to be able select which set of XMLTV ID's to run the operations against.

I don't understand this one. If you have to XMLTV files, wouldn't it be simpler to handle the XMLTV files separately (that is applying one transformation rule on Zap2it XMLTV and another one on MyTheatre XMLTV). What should be handled differently ?

I will create your script without problems : I only need to understant what you want to do. For now I see your need as :

* load channels from the Zap2it XMLTV and create corresponding channels for the Importer assigned to a specific lineup (all channels?)
* load channels from the MyTheatre XMLTV and create corresponding channels for the Importer assigned to another lineup. Ignoring some channels (which ones?)

Also do you create duplicate channels in the Importer or not ?

I suppose you will not need to run this script for every Import, just once in a while when your lineup is changing?

Regards,
Stéphane.

[xred]

Originally Posted by StephaneM

I don't understand this one. If you have to XMLTV files, wouldn't it be simpler to handle the XMLTV files separately (that is applying one transformation rule on Zap2it XMLTV and another one on MyTheatre XMLTV). What should be handled differently ?

I think we are in agreement, since thats exactly what I would like... for example I would like to be able to run the channel number / call sign extraction against the Zap2it sourced channels independant of the current MyTheatre channel setup. This is handy for most Mytheatre users since the Zap2it sources are usually a subset of the total channels Mytheatre provides.

You do have a point that running it against the MyTheatre source would generate a large amount of duplicates, but I am OK with this as going through the channel editor and simply deleting the whole channel number field takes far far less time than it does to manually type in the channel number and delete the channel number digits from the call sign field.

It would be nice if it was possible to have a method of easily, graphically selecting which channels to run against within each source file, but for the sake of simplicity (I only put up $10 after all!) I didn't request it.

Originally Posted by StephaneM

I will create your script without problems : I only need to understand what you want to do. For now I see your need as :

* load channels from the Zap2it XMLTV and create corresponding channels for the Importer assigned to a specific lineup (all channels?)
* load channels from the MyTheatre XMLTV and create corresponding channels for the Importer assigned to another lineup. Ignoring some channels (which ones?)

As mentioned above if it was possible to specify ranges it might be possible to get around this. I was thinking about this a little more while typing this out, and ideally there would be a way to prioritize one lineup over the other or allow overwriting to avoid duplicates. So for example:

Run extraction against MyTheatre.xml for all channels it contains. (e.g 1-1000)
Run extraction against Zap2it.xml for all channels it contains. (e.g. 1-400, 700-800).

Since Zap2it was run second (or has a higher priority) any channels it has that overlap Mytheatre would overwrite the MyTheatre channel #.

Originally Posted by StephaneM

I suppose you will not need to run this script for every Import, just once in a while when your lineup is changing?

Correct. This is not something to be run with every import, it is a way to avoid the large amount of editing required during initial setup or when lineup changes occur.

[StephaneM]

Hello,

Originally Posted by xred

North American user script for Stephanes XMLTV importer that automatically...

Try the attached script, it should do what you want. Download the zip file and extract the wsf and inc files in a folder of your choice.

Now open the constantes.inc file in Notepad and customize it :

* XMLTV files / Const sEPGFiles = "epg.xml;epg.xml" : here you'll have to type the full path to your XMLTV files. Separate each of your XMLTV files by a semi colon

* Lineup Numbers / Const sLineupNumbers = "1;2" : here you'll have to type the lineup where to put the channels found in each of your XMLTV files. You must have a lineup number for each XMLTV file, even if it is the same (so for instance "1;1" is Ok)

* Destination Lineup / Const sFinalLineup = "channels.xml" : here you'll have to type the full path to the channels.xml file that Importer is using (at first leave it without a path : it will be saved in the folder where you put the wsf and inc files, this way you'll be able to verify that the lineup is Ok).

When all is customized, just double click on the wsf file and it will run the script and the channels.xml file will be created.

What it should do is :

* Create a channel for each channel found in the XMLTV files (extracting name / number) and avoiding duplicate channel number per lineup.

* So in your case, you should list Zap2it first (in the sEPGFiles), as there is no overwrite if a duplicate channel is found (the script simply ignore the duplicate)

NOTE : Do not open the XMLTV Importer channel editor as it will modify the created lineup by adding the channels that were skipped by the script (after all no at big deal => with no channel number they will be skipped during the Import, but it will be cleaner that way).


Regards,
Stéphane.

[xred]

Originally Posted by StephaneM

Hello,


Try the attached script, it should do what you want. Download the zip file and extract the wsf and inc files in a folder of your choice.

Finally got a chance to try this. Either I am doing something wrong (likely) or the script is not able to read the MTEPG.XML file I am specifying. I tried reversing the order of the xml files, but it still only reads the dishepg.xml file and not the mtepg file. Here's the contents of my constantes.inc file.

'XMLTV Files
Const sEPGFiles = "dishepg.xml;mtepg.xml"
'Lineup Numbers
Const sLineupNumbers = "1;1"
'Destination lineup
Const sFinalLineup = "channels.xml"

Heres a sample of the channel data from the DISHEPG xml file that IS working:

<channel id="I12553.labs.zap2it.com">
<display-name>213 NASA</display-name>
<display-name>213 NASA ECHOST:-</display-name>
<display-name>213</display-name>
<display-name>NASA</display-name>
<display-name>NASA TV</display-name>
<display-name>Satellite</display-name>
</channel>

and heres a sample of the MTEPG file that is NOT working

<channel id="C6352.www.mtepgdplu.com">
<display-name>213 NASA</display-name>
<display-name>213</display-name>
<display-name>NASA</display-name>
</channel>


Any clues as to what could be wrong? The channels.xml that is coming out now looks great, I just need it to work for both my XML source files.

[StephaneM]

Originally Posted by xred

Any clues as to what could be wrong? The channels.xml that is coming out now looks great, I just need it to work for both my XML source files.

Do the xml file that is not handled makes use of the DTD, if so, can you try to remove the DTD line of the XML just to see if it works.

I'll send post here an updated script that will at least display error message when it goes wrong.

Regards,
Stéphane.

[xred]

Bon travail Monsieur!

Removing the "<!DOCTYPE tv SYSTEM "xmltv.dtd">" line from the MTEPG did the trick.

Is there a way to get the script to ignore it? Its not a big deal if I have to manually delete it when running the lineup creator script.

The importer itself has run fine for many months with the MTEPG xml.

[StephaneM]

Originally Posted by xred

Removing the "<!DOCTYPE tv SYSTEM "xmltv.dtd">" line from the MTEPG did the trick. Is there a way to get the script to ignore it? Its not a big deal if I have to manually delete it when running the lineup creator script.

Not that I know of, if the DTD is specified then it must be verified. But simply copy the dtd file in the script folder and it should be fine.

I updated the script to cry if something is wrong with the XMLTV file.

Regards,
Stéphane.