SageTV Community - Updated XMLTV Plugin with compile steps

SageTV Community (http://forums.sagetv.com/forums/index.php)

- SageTV EPG Service (http://forums.sagetv.com/forums/forumdisplay.php?f=7)

- - Updated XMLTV Plugin with compile steps (http://forums.sagetv.com/forums/showthread.php?t=66899)

Updated XMLTV Plugin with compile steps

In an effort for some IPTV channels I am working for importing into SageTV I updated the plugin for more use cases. The changes below should work with most XMLTV grabbers. Also for future development there are easy steps for compiling the plugin on the Unraid Docker image. The Unraid Docker Image already has the required java_jdk installed for compiling. In the future I might look into the the ICON working as the data is also stored inside the XMLTV file.

(Warning me venting)
The original was not touched since the beginning of SageTv and impossible to use without remapping or using scripting for remapping channels. It is almost like the developers of SageTV wanted everyone to use their guide data or SD. As for SD the plus is that it gives 2 weeks of data with more info than XMLTV grabbers thus being a good value. Also to be fair to SageTV they also did give a good run(~15 years) with their guide data. Any case with SageTV on its last leg this needs to change as other DVR development have XMLTV native. Viva La SageTV :)

1. Stop SageTV Server
2. Rename on add/modify/ xmltv_EXAMPLE.properties examples
3. Copy all files and folder contents(not folder) of folder SAGETV_SERVER_ROOT to SageTV folder. Only Jar and *.proerties are required if not compiling required
4. Add the following line in Sage.properties epg/epg_import_plugin=xmltv.XMLTVImportPlugin
5. Start SageTV Server
6. Monitor in server folder xmltv.log, sagetv_0.txt,Sage.properties. If epg/epg_import_plugin=xmltv.XMLTVImportPlugin is removed from Sage.properties something is installed not correctly
7. In SageTV guide setup use XMLTV with zipcode 00000. If ask for license use TRIAL

#######Commnds for UnRaid Docker to Moditfy Sage.properties for Plugin
# Stop Sage Server
sudo -E "PATH=$PATH" -u sagetv /usr/local/bin/stopsage &
# Property Setting for EPG Plugin
sudo sed -i 'epg/epg_import_plugin' /opt/sagetv/server/Sage.properties
sudo echo 'epg/epg_import_plugin=xmltv.XMLTVImportPlugin' >> /opt/sagetv/server/Sage.properties
# Start Sage Server
sudo -E "PATH=$PATH" -u sagetv /usr/local/bin/startsage &

#########Commnds for UnRaid Docker Compile of Plugin
cd /opt/sagetv/server/xmltv
sh ondocker_build.sh

Code:

 

/* Change Log

##V2.0  01/13/2022

1.  Add code if display-name or channel id number(example 5, 5.1 5-1) then add as the channel number for import.  Examples below import as channel 2.1 and 1162 respectively below

##2.02  01/14/2022

1.  Fixed first channel * bug

2.  Fixed double year in title when title already had a date

3.  add tms filed for episode-num instead of creating unique id

4.  Do not generate SxEx if default case is found ("onscreen", "common")

5.  Fix Display for SxEx in descriptions

6.  Fix short name being set as channel number

7.  Start code for reading Logo Icon for channel (WIP) 

##2.03  01/14/2022

1.  Fixed new Integer((int) to remove compile deprecated warning

2.  Switched org.xml.sax.helpers.XMLReaderFactory to javax.xml.parsers.SAXParserFactory; to remove compile deprecated warning

##2.04 01/16/2022

1.  Added .properties entry for xmltv.channel.display-name.ShortNameIndex 

                 default=0 with no entry .properties file

                 init.ChannelShortNameIndex0:=shortest found without spaces

                 init.ChannelShortNameIndex>0: XMLTV file index found in order starting with 1 <display-name>CBS</display-name>

2.  Added .properties entry for "xmltv.channel.xmltv.channel.ShortNameRegEx"                

                default=.* with no entry .properties file will find anything

3.  Added .properties entry for xmltv.channel.display-name.LongNameIndex 

                default=0 with no entry .properties file

                init.ChannelLongNameIndex0:=longest found with spaces

                init.ChannelLongNameIndex>0: XMLTV file index found in order starting with 1 <display-name>CBS</display-name>

4.  Added .properties entry for xmltv.channel.NumberTag 

                 default="" No entry .properties file.  

                 init.ChannelNumberTag="" None will try to detect channel number from <channel id=XXX> or <display-name>XXX<display-name>

                 init.ChannelNumberTag=channel then <channel id=XXX> 

                 init.ChannelNumberTag=TAG_NAME then <TAG_NAME>xxx<TAG_NAME>

5.  Added .properties entry for init.ChannelNumberTagIndex                

                default=0 with no entry .properties file

                init.ChannelNumberTagIndex=0 None will try to detect channel number from <channel id=xxx> or <display-name>

                init.ChannelNumberTagIndex>0 XMLTV file index found in order starting with 1

6.  Added .properties entry for "xmltv.channel.xmltv.channel.NumberTagRegEx"                

                default=.* with no entry .properties file will find anything

7.  Added .properties entry for "xmltv.channel.IconDownload"

                 default=false with no entry .properties

                 false=Do Note download image file

                 true=Download image file from channel <icon src= >

##2.06 01/18/2022

1.  Added the following .properties entry  for enabling or disabling logging parts.  Defualt with no entry is true

                log.show=false

                log.channel=false

                log.configuration=false

                log.defaults=false

2.        Changed the way ChannelId is calulated. The previous way caused collections.  The new way uses the ProviderID as part of the encoding. The new limit limits the Provider ID to 998. 

                 2147483647        Max Number

                 1000000000

                 1030002100

                      99999        SD MAX it Sends

                  998           Upper Part is Listing ProviderID 1-998 999 used if none provided (* 1000000)

                     123456     Max XMLTV Number without Decimal 999999   

                 1   123456     Max XMLTV Number with Decimal 9999.99 When Decimal (* 100 +1000000000)  

3.  Remove non file characters so Channel icons could be downloaded the following examples where failing: "Cats 24/7" or "Dogs 24/7"        

##2.08 01/28/2022

1.  Filter if desc tag is from channel section.  This would crash the plugin if not done.

2.  Add show if it is up to 8 hours previous. Fixes on first import no data problem

3.  Add setting this.show.showId on <episode-num system="pluto"> tag

4.  Fixed freeFormEpisodeNumber on <episode-num system="onscreen"> and <episode-num system="common"> tags

5.  Add new classs to hold all the init. variables from .properties file

6.  Added xmltv.channel.NumberOffset to offset channel numbers on import.

7.  Added if *.xmltv.properties is found in server directory use it as properties without requirement of configurations= in xmltv.properties file

*/

https://forums.sagetv.com/forums/att...1&d=1642548450

The auto mapping works great.

The problem that remains for me is how sagetv reads the epg provided by Channels dvr. As @nyplayer pointed out over in the hardware thread, sagetv does some strange things with the names of movies and the titles of series when the epg comes form channels dvr. See the attached pictures. The same thing happens when I send the epg from channels dvr through xteve.

Quote:

Originally Posted by boukmandutty (Post 632571)

Have you inspect the XMLTV file if is store like that? Also do you sample XMLTV file?

I have attached a sample. I dont know where to access the Schedules Direct version for comparison though. I can confirm from experimentation that the current format from the channels dvr server works flawlessly with nextpvr and even with mythtv.

Quote:

Originally Posted by boukmandutty (Post 632571)

You got it working on linux ?

Quote:

Originally Posted by nyplayer (Post 632575)

You got it working on linux ?

Yeah the auto-mapping works with the files provided in the attachment above. The only problem continues to be the funky thing that happens with the names.

Quote:

Originally Posted by boukmandutty (Post 632576)

Yeah the auto-mapping works with the files provided in the attachment above. The only problem continues to be the funky thing that happens with the names.

I guess I will wait ...
Yeah the names and epgid etc ...also season and episode even though it is there is not recognized by SageTV. So I went with a sched direct west coast and east coast lineup.... and manually mapped channels I wanted.

Quote:

Originally Posted by boukmandutty (Post 632573)

Since I do not know which entries are not correct can you provide and example file and test case is wrong? Also it would help if in the photos you highlight the problem.

Thanks

Quote:

Originally Posted by nyplayer (Post 632575)

You got it working on linux ?

The plug in not Linux dependent it should also work with windows.

Quote:

Originally Posted by hvymetal (Post 632579)

The plug in not Linux dependent it should also work with windows.

Great I will test it on my Windows test machine ...

I know for sure there is no original airdate parsed season episode parsed as misc.... and epgid data is wrong ...

https://forums.sagetv.com/forums/showthread.php?t=66885

The pictures are below.

The xmltv info for The Angry Birds is:

Quote:

<programme start="20220114020000 +0000" stop="20220114040000 +0000" channel="6081">
<title>The Angry Birds Movie 2 (2019)</title>
<desc>The birds unite with the green pigs when fowl from an ice-covered island plan to destroy them both.</desc>
<category>Movie</category>
<category>Feature Film</category>
<category>Animated</category>
<category>Children</category>
<category>Comedy</category>
<category>Adventure</category>
<icon src="https://tmsimg.fancybits.co/assets/p14568986_v_v12_ac.jpg?w=480&h=720"></icon>
<date>20190802</date>
<previously-shown></previously-shown>
<episode-num system="tms">MV010390220000</episode-num>
<credits>
<director>Thurop Van Orman</director>
<actor>Jason Sudeikis</actor>
<actor>Josh Gad</actor>
<actor>Leslie Jones</actor>
</credits>
</programme>

The Family Guy episode:

Quote:

<programme start="20220114023000 +0000" stop="20220114030000 +0000" channel="6082">
<title>Family Guy</title>
<sub-title>Peter's Progress</sub-title>
<desc>A psychic reveals that Peter had a past life in the 17th century as Griffin Peterson.</desc>
<category>Episode</category>
<category>Series</category>
<category>Animated</category>
<category>Sitcom</category>
<icon src="http://tmsimg.fancybits.co/assets/p184483_b_h6_bb.jpg"></icon>
<series-id system="tms">184483</series-id>
<date>20090517</date>
<previously-shown></previously-shown>
<episode-num system="tms">EP002960010132</episode-num>
<episode-num system="onscreen">S7E16</episode-num>
<episode-num system="xmltv_ns">6.15.</episode-num>
<credits>
<director>Brian Iles</director>
<director>Peter Shin</director>
<actor>Seth MacFarlane</actor>
<actor>Alex Borstein</actor>
<actor>Seth Green</actor>
</credits>
</programme>

I have attached for each a marked-up picture of what it looks like with the xmltv plugin. There is also a picture of what it looks like when Schedules Direct is used through the tedious mapping.

@nyplayer I didn't see your recent posts on your attempt to fix this until just now when you linked to it. If you have managed to fix any part of what I have here let me know.

WTF :bang:

Code:

         String title = this.show.title;

                // FIXME handling rerun shows correctly with SageTV 9

                // String rerunTitle = null;

                String rerunTitle = this.show.title;

                String episodeName = this.show.episodeName;

                if (this.initcapAllChannelIds

                        || this.initcapChannelIds

                                .contains(this.channel.xmltvId)) {

                    if (this.initcapTitle) {

                        title = initcap(title);

                    }

                    if (this.initcapEpisodeName) {

                        episodeName = initcap(episodeName);

                    }

                }

                if (title != null) {

                    if (this.show.date != null

                            && isActivated(categories,

                                    this.dateTitleDecorationCategories)) {

                        title = this.dateTitleDecoration.format(

                                new Object[] {title, this.show.date},

                                new StringBuffer(), null).toString();

                    } else if (this.titleAddYear

                            && this.show.year != null

                            && isActivated(categories,

                                    this.titleAddYearCategories)) {

                        title += " (" + this.show.year + ")";

                    }

                    if (this.show.quality != null

                            && this.show.quality.startsWith("HD")

                            && isActivated(this.channel.id,

                                    this.hdTitleDecorationChannels)) {

                        title = this.hdTitleDecoration.format(

                                new Object[] {title}, new StringBuffer(), null)

                                .toString();

                    }

                }

                if (this.show.rerun) {

                    rerunTitle = title;

                    // FIXME handling rerun shows correctly with SageTV 9

                    //title = null;

Quote:

Originally Posted by boukmandutty (Post 632583)

@nyplayer I didn't see your recent posts on your attempt to fix this until just now when you linked to it. If you have managed to fix any part of what I have here let me know.

Update should fix the problems listed. Update is in top post.

Quote:

Originally Posted by nyplayer (Post 632580)

https://forums.sagetv.com/forums/showthread.php?t=66885

Not sure if I fixed the problem because I could not find it but some of the other changes might have solved this one also.

I am getting this error ... on Windows. All I want to test is bringing in epgdata.xml which is in the SageTV root directory ... no XTEVE yet.

14-01-2022 04:47:29,860 java.lang.NoClassDefFoundError: xmltv/Channel
at xmltv.XMLTVImportPlugin.startElement(XMLTVImportPlugin.java:1431)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator.startElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
at xmltv.XMLTVImportPlugin.updateGuide(XMLTVImportPlugin.java:1299)
at xmltv.XMLTVImportPlugin.updateGuide(XMLTVImportPlugin.java:1204)
at sage.EPG.pluginExtractGuide(EPG.java:1625)
at sage.WarlockRipper.extractGuide(WarlockRipper.java:973)
at sage.EPGDataSource.expand(EPGDataSource.java:709)
at sage.EPG.run(EPG.java:773)
at java.lang.Thread.run(Unknown Source)
14-01-2022 04:47:29,860 java.lang.NoClassDefFoundError: xmltv/Channel
at xmltv.XMLTVImportPlugin.updateGuide(XMLTVImportPlugin.java:1311)
at xmltv.XMLTVImportPlugin.updateGuide(XMLTVImportPlugin.java:1204)
at sage.EPG.pluginExtractGuide(EPG.java:1625)
at sage.WarlockRipper.extractGuide(WarlockRipper.java:973)
at sage.EPGDataSource.expand(EPGDataSource.java:709)
at sage.EPG.run(EPG.java:773)
at java.lang.Thread.run(Unknown Source)

Quote:

Originally Posted by nyplayer (Post 632598)

I am getting this error ... on Windows. All I want to test is bringing in epgdata.xml which is in the SageTV root directory ... no XTEVE yet.

I think this was caused because the docker I'm using is Java16 and I did not compile for lower version of java. Can you test the attached version it was compiled for 8 or higher of java.

These are the error logs and my xmltv.properties... I might be doing something wrong in the setup.

What Java are using? Also how did you install? You should only need the .Jar file copied to you servers JARs folder. Make sure you do not have older versions of the plugin in the same folder(.jar or folders). Also any xmltv folder in the root of the Server or in the JARs referencing the older version need to be renamed

Quote:

Originally Posted by hvymetal (Post 632606)

jre1.8.0_191

I am going to try a full fresh install of SageTV there might be some stuff from the old XMLTV around.

Quote:

Originally Posted by nyplayer (Post 632608)

jre1.8.0_191

I am going to try a full fresh install of SageTV there might be some stuff from the old XMLTV around.

It worked with the following setup in a Window 7 64bit VM
1. Installed with latest SageTv 64bit installer
2. Ran SageTV First time
3. Stop SageTV
4. copy the XMLTVImportPlugin.jar in SageTv Jars folder
5. Add the following line in Sage.properties epg/epg_import_plugin=xmltv.XMLTVImportPlugin
6. Copied a XMLTV FILE name epgdata.xml
7. Start SageTV
8. Source Setup->EPG Lineup
9. Zip Code 0000
10. Will list XMLTV Lineup
11. Use extended

The plugin will work with out any .properties file if epgdata.xml in the root of the SageTV server directory. It will give error in xmltv.error.log for not finding the .properties but still work(This is a bug and need correcting). The .properties are used if you want to do customizations and add multiple lineups. The examples I provided 2 examples that give the guide a different name and location of the XMLTV file.