New XMLTV Import Tool

[doc]

Hi Stephane

We have 3 virtual tuners all set to tune to the same channel, that way we can record every show on a channel with a few minutes padding either side. That way we we never miss a recording.

I currently have the XMLTV Importer set up so that certain programmes always record - using the 'First Run & Rerun Optimization' screen.

I've added some shows and selected the option 'This show is generic'. This works great.

I have a slight problem where they sometimes change the show names and so they don't always get recorded. They are not picked up by the 'This show is generic' rule and are not recorded as the Show ID's are the same as a previous recording.

Is there a way to set something up so that every show on a channel is set to 'generic' ? Either that or have an option for the Show name to start with, or include certain words, so that shows like 'NZ Voice: Show Today' and 'NZ Voice: News Tomorrow' will always get recorded.

I currently edit the EPG file and add a category 'SageRecord' to all shows I want to record, and then have a favourite in Sage set up to record this category.

If I need to pre-process the EPG XML file I am happy to knock up some code to do that as I already pre-process it.

Any help appreciated.

Doc

[doc]

I've just noticed in the Category & Ratings section that there is the option

'shows of this category are generics (neither first run nor rerun)'

Should this do the trick and make Sage record all programs with this Category no matter what?

[StephaneM]

Hi,

Originally Posted by doc

I've just noticed in the Category & Ratings section that there is the option 'shows of this category are generics (neither first run nor rerun)'. Should this do the trick and make Sage record all programs with this Category no matter what?

Yes this is exactly the purpose of this option.

Usually the Importer will make a show a "generic" if it is not a series or a movie (this is deduced by the categories) and if it doesn't show an episode title.

With the category editor you can decide what categories are a series or generic (and the rules editor is here to define exceptions show by show)

Regards,
Stéphane.

[doc]

Originally Posted by StephaneM

Hi,
Yes this is exactly the purpose of this option.

Usually the Importer will make a show a "generic" if it is not a series or a movie (this is deduced by the categories) and if it doesn't show an episode title.

With the category editor you can decide what categories are a series or generic (and the rules editor is here to define exceptions show by show)

Regards,
Stéphane.

I think I was using it the wrong way round

It's working great now, all programs are scheduled to record no matter what

Thanks again.

Doc

[fac13]

Back at post #777 Edgley reported this error after moving to a different computer: -

Error - Unexpected error :Padding is invalid and cannot be removed.

I got the same error when I recently renamed my computer; the same fix worked.

Renaming a machine isn't something that people do often of course, but I thought I'd report the error anyway.

[Black Sky]

Hi,
Quick query on epg updates.
A new (1mnth old) install on Win7 - working ok, but one specific thing on epg updates I have noticed so far....
My epgdata.xml has the correct info for an airing on 14 March on BBC1 - Sunday AM 11:10
Bahrain Grand Prix. Jake Humphrey introduces live coverage of the Bahrain Grand Prix .....(longer blurb cut)
In my Sage Program Guide I have different info for the same airing on 14 March BBC1 - Sunday AM 11:10
Bahrain Grand Prix - Qualifing. Jake Humphrey introduces live coverage of qualifing for the Bahrain Grand Prix .....(longer blurb cut).....
(bolded words are different)
The Sage Program guide also has the following info down the bottom
Motoring - First Run
Original Air date: 13 March 2010
ShowID: EP0031850004

Now this entry is exactly the same as the tv guide data for a "correct" one on the previous day - Saturday 13th March BBC1 - actual Saturday entry for the actual qualifing session), which is now getting repeated for the Sunday airing.

Is there anyway to do a flush/total refresh of the epgdata within SageTV.
Have done the re-imports, remove readd channel, remove and readd source etc, but no joy. Is this correctable, as I want F1 as a favourite, or will I have to do manual recordings or add soem sort of rule to teh importer and or SageTV.

Thx...

[StephaneM]

Originally Posted by Black Sky

Is there anyway to do a flush/total refresh of the epgdata within SageTV.

Yes empty the "GuideData" subfolder of the Importer Config & Logs directory and do the Import again. This will remove previously imported data (cached) that can prevail on new data.

Regards,
Stéphane.

[bigbrotrader]

I have the donators edition, installed on WHS. Scheduled to update every day at 00:00.

Does not update automatically... do the XMLTV importer have to be running H24 for the auto-schedule-update to start?

[StephaneM]

Hi,

Originally Posted by bigbrotrader

I have the donators edition, installed on WHS. Scheduled to update every day at 00:00.Does not update automatically... do the XMLTV importer have to be running H24 for the auto-schedule-update to start?

No the Importer doesn't have to run h24. However, by default, the scheduled task only runs when a user is logged on. As you are using it on WHS, I suppose this is a headless server and a user is rarely logged on.

So, you have to edit the scheduled task created by the Importer to allow it to run when no user is logged and assign it a username (windows) and password.

Regards,
Stéphane.

[bigbrotrader]

Thank you! That worked.

I now also tried to make the webserver (v2.25) auto-force update sagetv epg, that does not work from auto-schedule in xmltv-importer. Maybe in that case a user have to be logged on due to WHS?

[StephaneM]

Originally Posted by bigbrotrader

I now also tried to make the webserver (v2.25) auto-force update sagetv epg, that does not work from auto-schedule in xmltv-importer. Maybe in that case a user have to be logged on due to WHS?

No, I think you must have to downgrade the Web Server plug-in because with some version (I use 2.19) the "Force EPG Update" feature is broken.

To verify this, navigate to the Web Server, on the home page find the [Force EPG update] link, click on it and check what is happening (if you receive an error or not or if the EPG has been reloaded or not)

Regards,
Stéphane.

[ytulpan]

Hi,

I have around 1200 recorded programs. I intend to keep some of them "forever" together with their metadata.

Unfortunately, when re-installing the importer some time ago, I probably chose the wrong option and had to redefine everything. The lookup.xml file was lost and I have not regularly backed it up.

This went unnoticed for some time, until I found out that recorded old movies are linked to the wrong metadata. When one of my kids tried to watch Leni Rifenstahl's "Triumph of the Will" for a school project, Sage showed Chaplin's "Great Dictator". I was speechless and confused while bowing to Sage's greater intelligence.

Unfortunately I found out that tens of other shows have the same personality shift, and pinpointed the problem to the EPGID reuse and the updating of metadata.

Is there a way out ? Can I tell XMLTV importer not to overwrite existing EPGID's ? Maybe I can tell it to use a "starting" number bigger than the previous range, or maybe start the new EPGiDs with a letter, i.e. "SHa000010001" instead "SH0000010001" or anything else to that effect.

I also understand that the same reuse problem might emerge anyway after one year even if I do not start the lookup.xml fresh.

Thanks for any suggestion

[StephaneM]

Hi,

Originally Posted by ytulpan

Is there a way out ? Can I tell XMLTV importer not to overwrite existing EPGID's ? Maybe I can tell it to use a "starting" number bigger than the previous range, or maybe start the new EPGiDs with a letter, i.e. "SHa000010001" instead "SH0000010001" or anything else to that effect.

I also understand that the same reuse problem might emerge anyway after one year even if I do not start the lookup.xml fresh.

There is no real way to recover from this...

You can choose a "starting" number by editing the lookup.xml file, this node is what you want to edit:

<property d2p1:name="showindex" d2p1:value="11541" xmlns:d2p1="http://tempuri.org/Lookup.xsd" />

the "showindex" / "value" is the most greater index used. In the example it means that 11542 is the next free index.

Another way would be to "disconnect" your recording from the SageTV database (but with 1200 recording this will be a pain)

Also there should be no problems in the long term (I'm running the Importer for many years now and I havn't faced any wrong EPGID ever, though I don't archive recordings, but still had some recordings kept more than one year). EPGID get reused when all IDs have been exhausted, though the Importer doesn't reuse active ones...

The only missing part is that the Importer doesn't know what EPGID are still used within SageTV only. I'll think about this (the WebEPG will be needed)

Regards,
Stéphane.

[ytulpan]

Thanks, this is something.

So, my plan is:

1. Load the corrupted (current) wiz.bin and export the recorded shows in XML with Nielm's server.

2. Delete the shows older than the importer reinstall date from the above XML.

3. Translate the EPGIDs to another number by adding (at least) the highest index value from before the install, thus making EPGIDs unique (I'll have to write a small program for that)

4. Revert to a Sage and XLMTV importer snapshot prior to the outage.

5. Update the last index as you described above, to be above the highest value of the translated EPGIDs

6. Restart Sage and import the "translated" show data from 3 above

Does it make sense ?

In another order of ideas, the lookup.xml grows huge since it "remembers" all shows from all channels in the past. Is it possible to delete lines that have no recordings associated with them older than, say, 3 months. This will obviously reduce the file size enormously.

[StephaneM]

Originally Posted by ytulpan

Thanks, this is something.

So, my plan is:

1. Load the corrupted (current) wiz.bin and export the recorded shows in XML with Nielm's server.

2. Delete the shows older than the importer reinstall date from the above XML.

3. Translate the EPGIDs to another number by adding (at least) the highest index value from before the install, thus making EPGIDs unique (I'll have to write a small program for that)

4. Revert to a Sage and XLMTV importer snapshot prior to the outage.

5. Update the last index as you described above, to be above the highest value of the translated EPGIDs

6. Restart Sage and import the "translated" show data from 3 above

Does it make sense ?

I'm not sure... It should be better to only try to recover a lookup.xml with records that does make sense...

If I understand you want to change the EPGID of your old recordings, whereas the ideal would be to change the EPGID of the new ones.. or even better none at all: except for the one that have already been shifted to another description those are the only shows that have received a conflicting EPGID

So I would compare the lookup.xml files to see what is conflicting or not and try to reconcile them (with a quick and dirty VBScript)... There are two kind of conflicts:

* showid and epgid being reused for the wrong show (and as such updating an old recording incorrectly)

* hash being assigned a new showid / epgid : not resulting in a conflict yet, but it means that you have lost the history for the show.

In another order of ideas, the lookup.xml grows huge since it "remembers" all shows from all channels in the past. Is it possible to delete lines that have no recordings associated with them older than, say, 3 months. This will obviously reduce the file size enormously.

3 months is not enough. The lookup file is already cleaned up, the showid / epgid for a show that has not been updated in a year are wiped out (a year because usually shows does have usually offer new episodes after one year)

And the lookup.xml can be big, it only consumes RAM when the Importer makes use of it but it doesn't reduce the speed of the import (except the final stage when it does clean the lookup file)

Regards,
Stéphane.

[ytulpan]

You are right, I meant to change the epgid of the *new* recordings, and only in wiz.bin (actually in the xml derived from it) so as to save the recordings.

I will leave a "hole" in the lookup.xml and lose the hash of the new shows. I don't much care about that.

You see, the big problem is having a consistent wiz.bin and making sure that lookup.xml won't overwrite it later.

[StephaneM]

Originally Posted by ytulpan

You are right, I meant to change the epgid of the *new* recordings, and only in wiz.bin (actually in the xml derived from it) so as to save the recordings.

I will leave a "hole" in the lookup.xml and lose the hash of the new shows. I don't much care about that.

You see, the big problem is having a consistent wiz.bin and making sure that lookup.xml won't overwrite it later.

It will overwrite them later when it will run out of showid (though it could be in ten years...)

The way it works is this: when the import ends, the importer review all showids that were not updated since last year and delete them as well as all related epgid. So the lookup table is filled with holes.

When the importer needs a showid it increase the "index" one by one until it finds a free slot (so the "index" is the last free slot used). So let say your index has reached 999999, the importer will restart at 1, the next free is 10, it will use that.

If you want to be on the safe side, you can add the hash from the new show in the old lookup.xml file with your forged epgid (if the hash doesn't exist yet of course), only copy the "programs" record.

Also remember that EPGID is like this : EP 000006 0136, that is a two letter prefix "MV = movie", "EP = Series episode", "SH = generic show"; "000006" is the showid (it does identify the series), "0136" is the episode id for the show id...

Regards,
Stéphane.

[ehanekla]

Hello,

I have a very weird issue with SageTV:
The EPG is drawn overlapping for a few channels and the list of upcoming movies shows the same movie on the same channel at the same time about 21 times!

Together with SageTV support we could figure out that the affected channels have a negative stationID. This should not happen because these are used for special internal voodoo in the SageTV core.

Is this a bug of the importer? I used the new channel number remapping feature of the XMLVTImporter.

Thanks
Eyk

[StephaneM]

Originally Posted by ehanekla

Together with SageTV support we could figure out that the affected channels have a negative stationID. This should not happen because these are used for special internal voodoo in the SageTV core. Is this a bug of the importer?

Not a bug per say, as nothing in the spec of the SageTV API it says the station ID cannot be negative... So I don't care if the ID I provide is positive or not (but I don't see how it can be negative).

Anyway, you cannot directly change the station ID, you can only change the channel ID in the Importer lineup and this will change the station ID.

Can you please tell me what is the channel ID (the first column in the XMLTV lineup) whose station ID is negative?

Regards,
Stéphane.

[ehanekla]

Thank you for your quick reply and sorry for my late reply.
I created a screenshot of my lineup configuration. The affected channels are "RTL" and "Das Vierte". The channel IDs look strange, but they originate from the channel scan of SageTV and were imported into XmltvImporter.

Thanks
Eyk