Feature Request: use Sage EPG and XMLTV data simultaneously

[xred]

I'm not sure if this has been requested already or not, but would it be possible to allow Sage to use both its internal EPG data and external xmltv plugin data simultaneously?

It should be fairly simple using the zip / postal code to tell sage which one to use. By entering 00000 in to the zip for a tuner, Sage would look at the XMLTV plugin data. By entering ANY other valid ZIP or Postal code, it would use its internal data. So for example I would add

Tuner 1 -> Zip 14214 -> Using channels 100 - 399 & 600 - 900
Tuner 1 -> Zip 00000 -> Using channels 400 - 599

This should work as none of the XMLTV plugins I have used require a valid zip (in fact I think some require the 00000). This would allow users to import data that the internal EPG doesn't support (PPV, NHL, NBA, MLB packages), and still use the internal Sage EPG for other channels which include important data like first run etc.

Make sense to anyone?

TIA

[Motofreak75]

yes this would make life easier for a lot of people.



Or even make selection to use epg or import plugin per tuner card and we can edit out the chanels that we want.

Because I have a DVB-S card using free to air satilite on a motor (over 11 bird are beening picked (about 75 free channel),USB plextor for cable and ATSC HDTV card for local DV via antenna. And its a real pain to make the epg work. Well its only working for the DVB-S. everything else i selected no guide info for now....

thanks

[gabrielcab]

Originally Posted by xred

I'm not sure if this has been requested already or not, but would it be possible to allow Sage to use both its internal EPG data and external xmltv plugin data simultaneously?

It should be fairly simple using the zip / postal code to tell sage which one to use. By entering 00000 in to the zip for a tuner, Sage would look at the XMLTV plugin data. By entering ANY other valid ZIP or Postal code, it would use its internal data. So for example I would add

Tuner 1 -> Zip 14214 -> Using channels 100 - 399 & 600 - 900
Tuner 1 -> Zip 00000 -> Using channels 400 - 599

This should work as none of the XMLTV plugins I have used require a valid zip (in fact I think some require the 00000). This would allow users to import data that the internal EPG doesn't support (PPV, NHL, NBA, MLB packages), and still use the internal Sage EPG for other channels which include important data like first run etc.

Make sense to anyone?

TIA

So, you talking about 400'599 channels... can you extract epg data from them? if so, how? im very desperatly trying to get data from ppv channels on my dish network setup...

[technazz]

Bump this one.

I just found out the hard way that using the XMLTV import removes my internal EPG guide once it refreshed.

The WAF went way down when the tuner #1 (PVR-250 cable set to grab SageTV internal EPG) disppeared because I recently implemented A Multi Linup XMLTV plug in for My DVB-S card... ugg. now i have to make a XML import for my cable tuner if I want to keep my multi line up for DVB-s

Technazz

[xred]

Originally Posted by technazz

Bump this one.

I just found out the hard way that using the XMLTV import removes my internal EPG guide once it refreshed.

The WAF went way down when the tuner #1 (PVR-250 cable set to grab SageTV internal EPG) disppeared because I recently implemented A Multi Linup XMLTV plug in for My DVB-S card... ugg. now i have to make a XML import for my cable tuner if I want to keep my multi line up for DVB-s

Technazz

There's about 6 separate threads going on about this in one form or another.... (PPV plugin, 2 separate feature requests threads yadda yadda yadda) to my knowledge no one from the Sage side has definatively said it is or is not possible to have Sage use its internal EPG AND XMLTV simultaneously... on the face of it, it looks like Sage can use data from 2 sources (Zap2it internal and TVTV) so theres no reason it shouldn't support internal and xmltv... I just wish the Sage guys would definatively come out and say whether we're holding out for something that is or isn't doable within Sage... I'm done venting, back to Iron Chef =)

[technazz]

Originally Posted by xred

There's about 6 separate threads going on about this in one form or another.... (PPV plugin, 2 separate feature requests threads yadda yadda yadda) to my knowledge no one from the Sage side has definatively said it is or is not possible to have Sage use its internal EPG AND XMLTV simultaneously... on the face of it, it looks like Sage can use data from 2 sources (Zap2it internal and TVTV) so theres no reason it shouldn't support internal and xmltv... I just wish the Sage guys would definatively come out and say whether we're holding out for something that is or isn't doable within Sage... I'm done venting, back to Iron Chef =)

Yes.. Ive been through them all.. pulled my hair out setting this thing up to get my PPV data for Dish.(MT Plugin, MT Export XML, XMLTV.exe,Demideus Import XML in).

Finally got it working and then this happens. Although its set up now..my wife thinks im crazy for setting this up (It took me 3 hrs to do it and about 5 hrs of reading)

Technazz

[xred]

Originally Posted by technazz

Yes.. Ive been through them all.. pulled my hair out setting this thing up to get my PPV data for Dish.(MT Plugin, MT Export XML, XMLTV.exe,Demideus Import XML in).

Finally got it working and then this happens. Although its set up now..my wife thinks im crazy for setting this up (It took me 3 hrs to do it and about 5 hrs of reading)

Technazz

I've been following the other threads as well... I'm hesitant to try since _none_ of the XMLTV plugins I've used so far are able to provide first run and other meta as well as the internal Sage grabber does, and that data is critical to getting Sage to record the correct shows, and NOT record lots of wrong stuff.

On the other hand, not having the meta data for PPV data is no biggie.. anything is preferable to the current "no data" situation for PPV channels.... but for regular channels / TV series its a deal breaker.

[rsagetv99]

Originally Posted by xred

I've been following the other threads as well... I'm hesitant to try since _none_ of the XMLTV plugins I've used so far are able to provide first run and other meta as well as the internal Sage grabber does, and that data is critical to getting Sage to record the correct shows, and NOT record lots of wrong stuff.

On the other hand, not having the meta data for PPV data is no biggie.. anything is preferable to the current "no data" situation for PPV channels.... but for regular channels / TV series its a deal breaker.

I thought the same thing when I first started using XMLTV. You should try my rerun-detect script, it will fix the first-run issue you are having with the XMLTV data.

[xred]

Originally Posted by rsagetv99

I thought the same thing when I first started using XMLTV. You should try my rerun-detect script, it will fix the first-run issue you are having with the XMLTV data.

I've been tracking the threads on this and it looks great, but do you have to use the demideus xmltv importer to make it work? I took a quick look at Demideus' setup, but its pretty complex to say the least

If I understand correctly, the process (including your scripts) for me every time I wanted EPG data would go something like this;

Grab xmltv data from zap2it for my cable provider -> run it through your firstrun / rerun scripts -> Import it in to Demideus as the first provider

Then grab xmltv data from XMLTV2MTEPG -> run it through your firstrun / rerun script (and possibly the categorizing scripts as well....) -> Import it in to DemiDeus as a second provider.

Am I understanding the process correctly?

[rsagetv99]

Originally Posted by xred

I've been tracking the threads on this and it looks great, but do you have to use the demideus xmltv importer to make it work? I took a quick look at Demideus' setup, but its pretty complex to say the least

If I understand correctly, the process (including your scripts) for me every time I wanted EPG data would go something like this;

Grab xmltv data from zap2it for my cable provider -> run it through your firstrun / rerun scripts -> Import it in to Demideus as the first provider

Then grab xmltv data from XMLTV2MTEPG -> run it through your firstrun / rerun script (and possibly the categorizing scripts as well....) -> Import it in to DemiDeus as a second provider.

Am I understanding the process correctly?

The process I use nightly is,

1a. Grab XMLTV data from zap2it for both of my providers, (satellite and cable). These are downloaded as 2 different XMLTV files.
1b. Run through the rerundect script, and the channel numbering script.
2a. Grab xmltv data from XMLTV2MTEPG.
2b. Run through channel numbering script. I do not think first-run/original air-date is provided through MT.
3. When the EPG updates nightly it updates the guide through the Demedius importer.

The rerun detect script adds the <previously shown> tag to any show that originally aired more than 8 days ago (you can adjust the time) and does not already have the <previously shown> tag,
I also have it mark shows with no <date> tag as <previously shown> since shows with no original listed airdate are typically reruns, but you can turn this feature off.

Since the rerun detect script is modifying the XMLTV file, you can use it with any importer as long as the importer interprets the <previously shown> tag as a rerun. I know that Demedius' importer does, haven't tried the others.

BTW, you should reconsider Demedius' importer, with it you can make it as simple or complex as you like.

[technazz]

Originally Posted by rsagetv99

The process I use nightly is,

1a. Grab XMLTV data from zap2it for both of my providers, (satellite and cable). These are downloaded as 2 different XMLTV files.
1b. Run through the rerundect script, and the channel numbering script.
2a. Grab xmltv data from XMLTV2MTEPG.
2b. Run through channel numbering script. I do not think first-run/original air-date is provided through MT.
3. When the EPG updates nightly it updates the guide through the Demedius importer.

The rerun detect script adds the <previously shown> tag to any show that originally aired more than 8 days ago (you can adjust the time) and does not already have the <previously shown> tag,
I also have it mark shows with no <date> tag as <previously shown> since shows with no original listed airdate are typically reruns, but you can turn this feature off.

Since the rerun detect script is modifying the XMLTV file, you can use it with any importer as long as the importer interprets the <previously shown> tag as a rerun. I know that Demedius' importer does, haven't tried the others.

BTW, you should reconsider Demedius' importer, with it you can make it as simple or complex as you like.

I have the same setup but for you option 1b(Channel renumbering) and 2b do you have to run this nightly? Or just for the one time setup of the xmltv.properties?

Technazz

[rsagetv99]

Originally Posted by technazz

I have the same setup but for you option 1b(Channel renumbering) and 2b do you have to run this nightly? Or just for the one time setup of the xmltv.properties?

Technazz

I decided to run it nightly, and point my xmltv.properties to include the file that is output by the script. This way, if my lineup ever changes, by me going to Zap2it and adding channels, or by changing what channels are included in the MT export, I don't have to worry about running it again.

The channel script is very fast, at least on my PC, YMMV.

[rsagetv99]

Originally Posted by rsagetv99

The process I use nightly is,

1a. Grab XMLTV data from zap2it for both of my providers, (satellite and cable). These are downloaded as 2 different XMLTV files.
1b. Run through the rerundect script, and the channel numbering script.
2a. Grab xmltv data from XMLTV2MTEPG.
2b. Run through channel numbering script. I do not think first-run/original air-date is provided through MT.
3. When the EPG updates nightly it updates the guide through the Demedius importer.

The rerun detect script adds the <previously shown> tag to any show that originally aired more than 8 days ago (you can adjust the time) and does not already have the <previously shown> tag,
I also have it mark shows with no <date> tag as <previously shown> since shows with no original listed airdate are typically reruns, but you can turn this feature off.

Since the rerun detect script is modifying the XMLTV file, you can use it with any importer as long as the importer interprets the <previously shown> tag as a rerun. I know that Demedius' importer does, haven't tried the others.

BTW, you should reconsider Demedius' importer, with it you can make it as simple or complex as you like.

Also forgot to mention during step 1b I run my HDTV script, which prepends the title of HDTV broadcasts shows based on a list of HDTV channels that I have available.

[xred]

Originally Posted by rsagetv99

The process I use nightly is,

1a. Grab XMLTV data from zap2it for both of my providers, (satellite and cable). These are downloaded as 2 different XMLTV files.
1b. Run through the rerundect script, and the channel numbering script.
2a. Grab xmltv data from XMLTV2MTEPG.
2b. Run through channel numbering script. I do not think first-run/original air-date is provided through MT.
3. When the EPG updates nightly it updates the guide through the Demedius importer.

The rerun detect script adds the <previously shown> tag to any show that originally aired more than 8 days ago (you can adjust the time) and does not already have the <previously shown> tag,
I also have it mark shows with no <date> tag as <previously shown> since shows with no original listed airdate are typically reruns, but you can turn this feature off.

Since the rerun detect script is modifying the XMLTV file, you can use it with any importer as long as the importer interprets the <previously shown> tag as a rerun. I know that Demedius' importer does, haven't tried the others.

BTW, you should reconsider Demedius' importer, with it you can make it as simple or complex as you like.

Well I don't have the issues with channel ordering that some others do, so no need to run that script on my side.

The scripts seem pretty straightforward, but the DemiDeus xmltv importer has a somewhat byzantine set of config options. I had no problem setting up NielM's multi-lineup plugin, but DemiDeus is quite a bit more complicated. I suppose the real issue is that I am dreading the trial / error / re-try cycle its going to take to get it up and running...

[technazz]

Originally Posted by xred

Well I don't have the issues with channel ordering that some others do, so no need to run that script on my side.

The scripts seem pretty straightforward, but the DemiDeus xmltv importer has a somewhat byzantine set of config options. I had no problem setting up NielM's multi-lineup plugin, but DemiDeus is quite a bit more complicated. I suppose the real issue is that I am dreading the trial / error / re-try cycle its going to take to get it up and running...

For me, it seem overwelming to set up the DemiDeus xmltv importer without haveing the xmltv.properties. My properties only has 5 or 6 lines in it very simple. I copied it from MotoFreak75 and juest edited it.

configurations=xmltv.properties
provider.name=DVB-S source
provider.id=1

xmltv.files=cable.xml,bev.xml,ppv.xml

max.stars=4

max.subcategories=1
translate.category.Action=Movie
translate.category.film=Movie
translate.category.Comedy-drama=Movie
translate.category.Sport=Sports event
translate.category.Romance=Movie
translate.category.Suspense=Movie
translate.category.Crime drama=Movie
translate.category.Romance-comedy=Movie
translate.category.Science fiction=Movie

Technazz

[Motofreak75]

here is a upto date one ( I'm not combining any Zapit data) I feel that the ZAPit data is okay, and its it clearly not the same from what we get from Sagetv. The public xml data from Zapit, I find is missing certian shows or any updated or last minute info changes from the providers. (eg. last minute sports events) Sure its missing first run / rerun stuff. But I can live with that, and so I have do a little extra setting up the favorites, no big deal. it works for me when I have 11 birds flying above and I can get all th info I need.


(heres a note to all)
Just be thankful you have options that work. I just hate seeing people complain about something their getting for free. (these feeling were also passed on from the programer Dom (that made this xml exporter) after we both have recieved complants about this free program. And yes this is starting to feel like a certain group when Davetv went down. Dont be that group.

Please.


Here is my Demides files.

the only problem I'm having is just Strong Sexual Content error in the error.log when importing into sage. But I'll figure it out when I have more time,or someone else can.



-----------------------------------------------------
(xmltv.properties)
-----------------------------------------------------
configurations=xmltv.properties,xmltv2.properties,xmltv3.properties
provider.name=DVB-S source
provider.id=1

xmltv.files=epgdata.xml

rating.advisory.Language=PG-13
rating.advisory.Adult Situations=PG-13
rating.advisory.Brief Nudity=PG-13
rating.advisory.Graphic Violence=PG-13
rating.advisory.Mild Violence=PG-13
rating.advisory.Strong Sexual Content=AO
rating.*.NR=NR
rating.*.AO=AO
rating.*.NC-17=NC-17
rating.*.R=R
rating.*.PG-13=PG-13
rating.*.PG=PG
rating.*.G=G
rating.*.AO=AO, Strong Sexual Content
rating.*.Nudity=AO
rating.*.Violence=PG-13

max.stars=5

max.subcategories=1
translate.category.film=Movie

include=xmltv_channel_ids.txt

-------------------------------------------------------
(xmltv2.properties) ZAPIT data
-------------------------------------------------------
configurations=xmltv2.properties
provider.name=CABLE source
provider.id=2

xmltv.files=ROGERS.xml

rating.*.NR=NR
rating.*.AO=AO
rating.*.NC-17=NC-17
rating.*.R=R
rating.*.PG-13=PG-13
rating.*.PG=PG
rating.*.G=G
rating.*.AO=AO, Strong Sexual Content
rating.*.Nudity=AO
rating.*.Violence=PG-13
rating.advisory.Language=PG-13

max.stars=5

max.subcategories=1
translate.category.film=Movie
translate.category.Action=Movie
translate.category.film=Movie
translate.category.Comedy-drama=Movie
translate.category.Sport=Sports event
translate.category.Romance=Movie
translate.category.Suspense=Movie
translate.category.Crime drama=Movie
translate.category.Romance-comedy=Movie
translate.category.Science fiction=Movie

include=ROGERSxmltv_channel_ids.txt
-------------------------------------------------
xmltv3.properties ZAPIT data
-------------------------------------------------
configurations=xmltv3.properties
provider.name=ATSC source
provider.id=3

xmltv.files=UHF-atsc.xml

rating.*.NR=NR
rating.*.AO=AO
rating.*.NC-17=NC-17
rating.*.R=R
rating.*.PG-13=PG-13
rating.*.PG=PG
rating.*.G=G
rating.*.AO=AO, Strong Sexual Content
rating.*.Nudity=AO
rating.*.Violence=PG-13
rating.advisory.Language=PG-13

max.stars=5

max.subcategories=1
translate.category.film=Movie
translate.category.Action=Movie
translate.category.film=Movie
translate.category.Comedy-drama=Movie
translate.category.Sport=Sports event
translate.category.Romance=Movie
translate.category.Suspense=Movie
translate.category.Crime drama=Movie
translate.category.Romance-comedy=Movie
translate.category.Science fiction=Movie

[rsagetv99]

Originally Posted by Motofreak75

here is a up to date one ...the only problem I'm having is just Strong Sexual Content error in the error.log when importing into sage. But I'll figure it out when I have more time,or someone else can.

Looks good to me, thanks for posting your file, and the one before, I started mine with your example. One thing I had to do was remove the category translations because for the Zap2it data, the plugin was marking a lot of things as movies, even though they weren't movies. It seems that Zap2it likes to mark regular series as Comedy, or Crime-Drama, so these shows would show up as movies in the guide (pinkish purple). It still misses marking some movies, but when I get time I may write another script, that looks at the length of a show, and the category to determine if a show is really a movie or just a Crime-Drama series like CSI.

If I had to guess what's causing your error I'd say maybe the line where you used expanded ratings (see below).
Also, I think you can just put include statements in the second and third properties files so you wouldn't have to repeat the ratings and categories over again. Just be sure to specify any parameter that you want different in the second two files and only that parameter will be overriden, everthing else will be included from your main xmltv.properties file. See below.

-----------------------------------------------------
(xmltv.properties)
-----------------------------------------------------
configurations=xmltv.properties,xmltv2.properties,xmltv3.properties
provider.name=DVB-S source
provider.id=1

xmltv.files=epgdata.xml

rating.advisory.Language=PG-13
rating.advisory.Adult Situations=PG-13
rating.advisory.Brief Nudity=PG-13
rating.advisory.Graphic Violence=PG-13
rating.advisory.Mild Violence=PG-13
rating.advisory.Strong Sexual Content=AO
rating.*.NR=NR
rating.*.AO=AO
rating.*.NC-17=NC-17
rating.*.R=R
rating.*.PG-13=PG-13
rating.*.PG=PG
rating.*.G=G
rating.*.AO=AO, Strong Sexual Content <<<-----Just guessing, either remove the expanded rating, or try without the space after the "AO,"
rating.*.Nudity=AO
rating.*.Violence=PG-13

max.stars=5

max.subcategories=1
translate.category.film=Movie

include=xmltv_channel_ids.txt

-------------------------------------------------------
(xmltv2.properties) ZAPIT data
-------------------------------------------------------
configurations=xmltv2.properties
provider.name=CABLE source
provider.id=2

max.subcategories=1
translate.category.film=Movie
translate.category.Action=Movie
translate.category.film=Movie
translate.category.Comedy-drama=Movie
translate.category.Sport=Sports event
translate.category.Romance=Movie
translate.category.Suspense=Movie
translate.category.Crime drama=Movie
translate.category.Romance-comedy=Movie
translate.category.Science fiction=Movie

xmltv.files=ROGERS.xml

include=xmltv.properties
include=ROGERSxmltv_channel_ids.txt

-------------------------------------------------
xmltv3.properties ZAPIT data
-------------------------------------------------
configurations=xmltv3.properties
provider.name=ATSC source
provider.id=3

xmltv.files=UHF-atsc.xml

include=xmltv.properties
include=xmltv2.properties

[technazz]

Since, we are trying fix the small problems we are facing here with MT-Sage-XMLTV I have a couple of questions.

1. How do you guys have multiple configurations for XMLTV.exe. I just have 3 different directories for each XML output (Cable ZAP2IT and DISH/BEV) and Id like to clean it up. Can you post you batch file to incorporate this?

2. How do you handle the channels changing transponders in MT, I rescan in MT and get a copy of a channel and I have to manually place into my SageTV Fav's. This seems to be a pain to get working. And it seems they are moving channels every other month.

3. For the XMLTV2MTEPG script, which I have outputting my PPV data for DISH/BEV how do you get the guide to update when you havent tuned into a transponder to get the EPG. I have my batch file running at 1:am if I dont have lets say a dish transponder turned on for that day I get no data. But If want PPV guide data I have to tune to a channel then run the script. Is this right?

I just need to get the WAF up before I do the full switch. Ive been tweaking a night and then copying over my SageTv Backup folder to go back to the SageTV internal guide for WAF! Uggh..

Technazz

[rsagetv99]

Originally Posted by technazz

Since, we are trying fix the small problems we are facing here with MT-Sage-XMLTV I have a couple of questions.

1. How do you guys have multiple configurations for XMLTV.exe. I just have 3 different directories for each XML output (Cable ZAP2IT and DISH/BEV) and Id like to clean it up. Can you post you batch file to incorporate this?

Not sure what you mean by cleaning it up. I have all of my Zap2it XML files go to the C:\XMLTV directory, copy them over to my SageTV directory, and then run rerundetect, which also calls the channel number and HDTV scripts.

c:
cd \xmltv
xmltv tv_grab_na_dd --config-file c:\xmltv\1.conf --days 13 --output c:\xmltv\1.xml
xmltv tv_grab_na_dd --config-file c:\xmltv\2.conf --days 13 --output c:\xmltv\2.xml
xcopy "C:\xmltv\1.xml" "c:\Program Files\SageTV\SageTV\" /Y
xcopy "c:\xmltv\2.xml" "c:\Program Files\SageTV\SageTV\" /Y
"e:\Kixtart\kix32" "e:\Kixtart\rerundetect.kix"
exit

2. How do you handle the channels changing transponders in MT, I rescan in MT and get a copy of a channel and I have to manually place into my SageTV Fav's. This seems to be a pain to get working. And it seems they are moving channels every other month.

I've been having some problems with MT, (switch related) so I haven't been using it to tune channels lately. I only use it for the guide data right now. In the past I used MS Access and Excel to make changing favorites or renumbering channels a little easier.

3. For the XMLTV2MTEPG script, which I have outputting my PPV data for DISH/BEV how do you get the guide to update when you havent tuned into a transponder to get the EPG. I have my batch file running at 1:am if I dont have lets say a dish transponder turned on for that day I get no data. But If want PPV guide data I have to tune to a channel then run the script. Is this right?

Right now what I do is leave MT tuned to the TVGuide channel which is on the transponder that provides the EPG. When I was tuning with MT I had a manual recording on the EPG channels/transpoders every night to keep the EPG database up to date.
I also put the line "run.before=C:\\xmltv\\UpdatePPVData.bat" in the xmltv.properties file, which ran a batch file that calls the MT-EPG exporter. By adding this line, if at any point I run out of guide data on the MT channels, I could just go to Setup Video Sources, select any source, and enable/disable any channel which would force an EPG update and refresh the data for those channels.

I just need to get the WAF up before I do the full switch. I've been tweaking at night and then copying over my SageTv Backup folder to go back to the SageTV internal guide for WAF! Uggh..

I gave up on WAF a long time ago, they are never satisfied.

[technazz]

Thanks Guys!

Ive got it wokring pretty good.

Ive had some error with XmltV.exe where it says Bad XML Data from DD. Ill wait and see if its ZAP2IT servers are messed up.

Sure is alot of work! But worth it..

technazz