MyTheatre and New XMLTV Import Tool

[xred]

Just ran the script to create a new channel lineup and added it to the importer. Took a look in the channel editor and sure enough all my channels are numbered and callsigns are in place.... I could have wept with joy!

Just finished running an import and everything looks perfect. This script literally saves me hours of editing I had to do on initial setup or whenever I add / change the lineup.

There should be a second registration from me for 10 euros waiting for you in PayPal Stephane, so walk down to your local bar and order a glass of something good for xmas =)

[technazz]

Originally Posted by xred

Just ran the script to create a new channel lineup and added it to the importer. Took a look in the channel editor and sure enough all my channels are numbered and callsigns are in place.... I could have wept with joy!

Just finished running an import and everything looks perfect. This script literally saves me hours of editing I had to do on initial setup or whenever I add / change the lineup.

There should be a second registration from me for 10 euros waiting for you in PayPal Stephane, so walk down to your local bar and order a glass of something good for xmas =)

Do you have to run this everyting it imports or just when you set up the tuner?

Technazz

[xred]

Originally Posted by technazz

Do you have to run this everyting it imports or just when you set up the tuner?

Technazz

Only when you doing setup. It lets you avoid the huge amount of editing that was required to cut / paste each of the channel numbers in to the correct cell. Between this script and the importers new ability to automatically skip unused channels, my setup time has gone from almost an hour to literally a minute or two.

[grooves12]

What about when you do a re-scan of channels in Mytheatre??

I typically do a complete delete and re-scan because otherwise I end up with some duplicate channels that take quite a while to go through and find manually. If I do that will I have to run the script and import again??

[xred]

Originally Posted by grooves12

What about when you do a re-scan of channels in Mytheatre??

I typically do a complete delete and re-scan because otherwise I end up with some duplicate channels that take quite a while to go through and find manually. If I do that will I have to run the script and import again??

Re running the scan is like importing a new lineup, so yes you will need to run the script. The good news is it handles duplicates, so if you have a Zap2it based xml file and an MT based XML file, just put the Zap2it first and MT Second. The script will not overwrite a channel number, so if Zap2it "takes" ownership first, the MT xml file simply fills in all the blanks for channels that Zap2it doesn't have data for (PPV, Sports etc). You will have to configure your Zap2it account to check only channels you want data for, but thats a one time deal.

[grooves12]

Originally Posted by StephaneM

Hello,

Usually I use VBScript with the MSXML parser. It's pretty straightforward if you know VBScript. Then you only have to get used to XSL Path to query the XML DOM.

The main advantage is that it's the quickest DOM parser I know of.

Tell me what you exactly want to do and I'll give you a basic workable script (you'll be able to customize it).

Regards,
Stéphane.

Here is my post from another thread I created on the topic, any help would be greatly appreciated....

Originally Posted by grooves12

Ok, I want to make a couple of changes to some of my xmltv data... for the program below, special attention to the item in bold:

First all of my data has the " - " which is useless right at the beginning of the <desc> field, and I would like to remove that out.

For series that have episode titles, like this one does, there is a carriage return right after the episode name. I looked through the data and there are not any carriage returns in any other <desc> data unless it is following an episode name, and the episode names are alwats rights at the beginning. I would like to parse the data so that the info before a carriage return is removed from the <desc> field and inserted into a new field within the listing of <subtitle> like this: <sub-title lang="en">Descent Into the Ice</sub-title>

Can anyone give me a little direction on creating a script to do this??

Also, another item is... my provider adds data to some of its PPV listing... some of the titles are preceded by "Family:" or one of the following is added to the end of the title "(All Day)","(All Day, DD)", or "(All Day, DD, LBX)", if possible I would like those removed as well, as it breaks the IMDB lookup routines of some of the STV's.

[xred]

Originally Posted by grooves12

Also, another item is... my provider adds data to some of its PPV listing... some of the titles are preceded by "Family:" or one of the following is added to the end of the title "(All Day)","(All Day, DD)", or "(All Day, DD, LBX)", if possible I would like those removed as well, as it breaks the IMDB lookup routines of some of the STV's.

wow this is pretty specific, you're looking to get specific strings removed from a specific set of channels. Even if a script could be written, I think it would need to be updated every time the channels change.

I saw that you asked for this in the Mytheatre forums as well and I think thats the right place to look as the only way to get what you're after is on the DVB Parser / exporter side.

[StephaneM]

Hello,

Originally Posted by grooves12

Here is my post from another thread I created on the topic, any help would be greatly appreciated....

Can you send me one of your XMLTV file at lmgestion@hotmail.com.

Regards,
Stéphane.

[technazz]

I;ve tried this script and I When I import then go to the edit the channels I get No Channels in xml file. Ive attached the script output.

THanks

Technazz

[StephaneM]

Hi,

Originally Posted by technazz

I;ve tried this script and I When I import then go to the edit the channels I get No Channels in xml file. Ive attached the script output.

Well, first you do not need this script to run the Importer, I made it for xred (avoiding him nigthmares the night before a lineup change ). So if you use it I guess that you know what you are trying do to with it and why.

Especially : the script is not aware of the Importer at all, it's only job is take xmltv files as input and outputs a channels.xml file. You can then use this channels.xml file to replace the one produced by the Importer (should be in the importer installation directory), either by replacing it or giving the right path directly in the script.

But : this will not prevent the Importer to still create / modify the channels.xml. So if you don't see anything in the Importer Channel Editor, this is because the Importer delete the channels it cannot found in your XMLTV files. And that means that the script and the Importer are not seeing the same XMLTV files.

Regards,
Stéphane.

[StephaneM]

Hi,

Originally Posted by grooves12

Here is my post from another thread I created on the topic, any help would be greatly appreciated....

This should do what you want.

Extract the content of the zip file in a folder of your choice, then edit the constantes.inc file in Notepad and change the sEPGFile to the path to your XMLTV files.

Then launch the script FixXMLTV.wsf, it could take some time to execute due to the size of your XMLTV file.

Regards,
Stéphane.

[grooves12]

Originally Posted by StephaneM

Hi,

This should do what you want.

Extract the content of the zip file in a folder of your choice, then edit the constantes.inc file in Notepad and change the sEPGFile to the path to your XMLTV files.

Then launch the script FixXMLTV.wsf, it could take some time to execute due to the size of your XMLTV file.

Regards,
Stéphane.

Thanks for your hard work and responsiveness...

I am running into the problem where I am getting an error because of the xmltv.dtd file not being found, similar to the one seen by xred above. I tried copying one I found on my system from another program... but it gives me another error saying it was expecting different information.

It starts processing the file if I remove that line from the .xml file, but that sort of defeats the purpose and point of scripting, because I would like to add a task that modifies the xml data immediately after downloading it.

What are the ramifications of removing the line that points to the xmltv.dtd file in the .xml file?? Does it hurt the import process at all?? If not, why is the system looking for it and crying??

[StephaneM]

Originally Posted by grooves12

I am running into the problem where I am getting an error because of the xmltv.dtd file not being found, similar to the one seen by xred above. I tried copying one I found on my system from another program... but it gives me another error saying it was expecting different information.

You have done the right thing : the dtd should be in the folder where the XMLTV file is. If the script is giving you another error, this is because the XMLTV file does not follow strictly the DTD

What are the ramifications of removing the line that points to the xmltv.dtd file in the .xml file?? Does it hurt the import process at all?? If not, why is the system looking for it and crying??

There are no problems runing without the DTD. The DTD is describing (for humans and computers) how the XML should be; that is what elements it should contains and how it relates.

When a XML file does have a DTD, the software that load the XML file should validate the XML file against this DTD. If the XML document validate then the software can safely assume that it will find what it expect, and if it fails the software must abort processing the file.

I guess that the XML parser used by the script is validating against the DTD by default whereas in the Importer, I'm not validating the document (though the DTD is needed to load the XML file). If I remember correctly the XML parser in the script will not tolerate elements added into the XMLTV file that are not described in the DTD...

Anyway, here is a version that does not validate the XMLTV file against the DTD. All you have to do is download the updated script or add :

oEPGFile.validateOnParse = False

just after the line

Set oEPGFile = WScript.CreateObject("Msxml2.DOMDocument")


Regards,
Stéphane.

[technazz]

Originally Posted by StephaneM

Hi,

Well, first you do not need this script to run the Importer, I made it for xred (avoiding him nigthmares the night before a lineup change ). So if you use it I guess that you know what you are trying do to with it and why.

Especially : the script is not aware of the Importer at all, it's only job is take xmltv files as input and outputs a channels.xml file. You can then use this channels.xml file to replace the one produced by the Importer (should be in the importer installation directory), either by replacing it or giving the right path directly in the script.

But : this will not prevent the Importer to still create / modify the channels.xml. So if you don't see anything in the Importer Channel Editor, this is because the Importer delete the channels it cannot found in your XMLTV files. And that means that the script and the Importer are not seeing the same XMLTV files.

Regards,
Stéphane.

Thanks its working perfectly now. Good work!

Technazz

[grooves12]

Originally Posted by StephaneM

You have done the right thing : the dtd should be in the folder where the XMLTV file is. If the script is giving you another error, this is because the XMLTV file does not follow strictly the DTD



There are no problems runing without the DTD. The DTD is describing (for humans and computers) how the XML should be; that is what elements it should contains and how it relates.

When a XML file does have a DTD, the software that load the XML file should validate the XML file against this DTD. If the XML document validate then the software can safely assume that it will find what it expect, and if it fails the software must abort processing the file.

I guess that the XML parser used by the script is validating against the DTD by default whereas in the Importer, I'm not validating the document (though the DTD is needed to load the XML file). If I remember correctly the XML parser in the script will not tolerate elements added into the XMLTV file that are not described in the DTD...

Anyway, here is a version that does not validate the XMLTV file against the DTD. All you have to do is download the updated script or add :

oEPGFile.validateOnParse = False

just after the line

Set oEPGFile = WScript.CreateObject("Msxml2.DOMDocument")


Regards,
Stéphane.

Thanks, is it possible to also modify the channels.xml creation script so that is does not validate against the xmltv.dtd?? I am using two different xmltv files netiher of which get a .dtd file created for them, and the one that comes with Sage does not work properly for them.

Also, I am still a little confused about setting up Sage and this importer with this script.
I have one tuner that uses Two different .xml files, and would like to use this script to set the channel numbers up.
Can someone give a quick step by step on getting this started?? I am trying to move from Demedius' importer.

[StephaneM]

Originally Posted by grooves12

Thanks, is it possible to also modify the channels.xml creation script so that is does not validate against the xmltv.dtd??

Yes, I have updated the script to skip the validation against the DTD.

Also, I am still a little confused about setting up Sage and this importer with this script. I have one tuner that uses Two different .xml files, and would like to use this script to set the channel numbers up.

You should first setup the XMLTV Importer and learn the way it works first, you'll find the step by step guide on my site.

The crucial step is the creation of lineup and channels edition. You'll have to create the lineups you need. I always suggest to create a lineup by feed type, for instance : Satellite Lineup / Cable Lineup etc... When you create a lineup you'll notice that they are numbered (you'll need this number to use the script afterward).

Right after lineup creation, go to the "channels" tab, the Importer will list all channels it can find in all your XMLTV files. There you have two choices :

* either edit the channels list : give a name, call-sign, channel number and choose the lineup for each channels. The channel number is what SageTV will use to tune the channels.
* either close the channel editor (by clicking on the "Ok" button) to validate the change even though you do not have changed anything and use the scripts afterward to create the "channels.xml" file for you.

The script is then supposed to be run only when your lineup change (so you use it for the initial setup and each time you have new channels or channels number are changing). Look at the initial post (#16) for the script it explains what you need to change in the constantes.inc file for it to be working. Basically you have to tell the script where to find your XMLTV files (all of them of course), what is the lineup number where to put the channels of each XMLTV files and where to save the channels.xml file.

Also remember, this is NOT a general purpose script. It is only intended to avoid editing your channels manually if you are in the same situations as Xred. But the script will not allows all the things you can do with the XMLTV Importer channel editor.

Note: you'll save you a great deal of time if you can reduce dramatically the size of your XMLTV file. For instance to reduce the overal import time on your first setup it will be smart to use a one day only XMLTV file (the XMLTV file you send me was 59 MB, if it is this large because there is many channels, many days and many programs, the importer will need at least one hour to process all this data, if you can reduce the size for you first setup, it will be easier for you to verify that all is working in a matter of minutes).

Regards,
Stéphane.

[grooves12]

I got this script up and running, and I must say this works GREAT!!! Should be an option in the standard application for autonumbering channels. And for those of us running Mytheatre data as a subset of Zap2it data, this script is AMAZING. Cuts setup time by about 90%

[grooves12]

StephaneM,

It's been a while, but the script you made for modifying my Mytheatre data has improved the functionality of your importer immensely. Previously I was getting multiple of the same episodes recorded because some had title information and some did not. Now it is finding the same episode from the different sources and marking them as the same.

But, there are still some minor problems. Those PPV programs that had (All day) in the title also have extra information, and are the ONE other program type that has a carriage return in the description. So, it is putting this extra info as an episode title, and I think that is somehow differentiating the data and it is thinking mulyiplr airings of the same movie are different movies.

This:

<programme start="20070112130000 +0100" stop="20070112150000 +0100" channel="C11514.www.mtepgdplu.com">
<title lang="en">Family: Cars (All Day)</title>
<length units="minutes">120</length>
<desc lang="en">G *** - All Day (5:00am - 5:00am Eastern)
Movie. Voice of: Owen Wilson, Voice of: Paul Newman, Voice of: Bonnie Hunt. (2006) A rookie race car learns about life from classic autos in a town along Route 66. Animated. (SAP) (58176) (CC)</desc>
<rating system="MPAA">
<value>G</value>
</rating>
<star-rating>
<value>3/5</value>
</star-rating>
<category lang="en">Movie</category>
<date>20060101</date>
</programme>

is being turned into this:

<programme start="20070112150000 +0100" stop="20070112170000 +0100" channel="C11514.www.mtepgdplu.com">
<title lang="en">Cars </title>
<length units="minutes">120</length>
<desc lang="en">Movie. Voice of: Owen Wilson, Voice of: Paul Newman, Voice of: Bonnie Hunt. (2006) A rookie race car learns about life from classic autos in a town along Route 66. Animated. (SAP) (58176) (CC)</desc>
<rating system="MPAA">
<value>G</value>
</rating>
<star-rating>
<value>3/5</value>
</star-rating>
<category lang="en">Movie</category>
<date>20060101</date>
<sub-title lang="en">*** - All Day (5:00am - 5:00am Eastern)</sub-title></programme>

So it is removing the first character, which is usually the rating PG to G, and then putting that whole line as an episode title. It would be better if it just got rid of all of that info completely, as some have different times ie: 6-6, 5:30-5:30, etc that may make a difference in the importer determining if they are the same.

I THINK all of the descriptions with this problem are also ones that have "Family:" preceding the title and/or the "(All Day)","(All Day, DD)", or "(All Day, DD, LBX)" after the title. So, maybe we can just make the script delete the info that is being shifted to the episode title area, if it also has one of those characteristics in the title.

Another MINOR problem is that for episodes that have no description the " -" is still remaining. So, this is that the data looks like after the script is ran:

<programme start="20070119173000 +0100" stop="20070119180000 +0100" channel="C11122.www.mtepgdplu.com">
<title lang="en">Paid Programming</title>
<length units="minutes">30</length>
<desc lang="en"> - </desc>
<rating system="MPAA">
<value>NR</value>
</rating>
<star-rating>
<value>0/5</value>
</star-rating>

I think a better solution in this case would be to delete the entire desc field completely, wouldn't it?? That way, since most episodes with no description info are repeats there could be a way of marking them as such in the importer.

And... similarly, episodes with no episode title, the " -" still remains.

<programme start="20070115010000 +0100" stop="20070115020000 +0100" channel="C10804.www.mtepgdplu.com">
<title lang="en">Book TV</title>
<length units="minutes">60</length>
<desc lang="en"> - Education. Non-fiction books and authors.</desc>
<rating system="MPAA">
<value>NR</value>
</rating>
<star-rating>
<value>0/5</value>
</star-rating>

Once again, thanks for your hard work on the importer, and thanks for helping out with my special requests.

[grooves12]

I just had to do a complete reinstall... and now I can not get this script to work. It is properly creating the channels.xml and it is placed in the importer directory, but the importer is not using it. When I click "edit channels" in the importer the channels are not being read as they are in the channels.xml.

[StephaneM]

Hello,

Originally Posted by grooves12

I just had to do a complete reinstall... and now I can not get this script to work. It is properly creating the channels.xml and it is placed in the importer directory, but the importer is not using it. When I click "edit channels" in the importer the channels are not being read as they are in the channels.xml.

Well, it's obviously a wrong configuration of the constants in the script. You should review them, especially the sLineupNumbers one, you are supposed to put here then lineup numbers created in the Importer.

So you need to run the channel editor of the Importer first, then modify the script constants to match your configuration. If the channels.xml file is getting written with the right lineup numbers then I don't see any reasons why it shouldn't be used by the Importer.

Note : I updated the FixXMLTV.zip archive in this thread for your "movies" problem (hower I will not fix your "remaining dash" problem as it will mostely screw everything else in your XMLTV file).

Regards,
Stéphane.