SageTV Community

SageTV Community (http://forums.sagetv.com/forums/index.php)
-   SageTV EPG Service (http://forums.sagetv.com/forums/forumdisplay.php?f=7)
-   -   Updated XMLTV Plugin with compile steps (http://forums.sagetv.com/forums/showthread.php?t=66899)

hvymetal 01-13-2022 09:07 AM

Updated XMLTV Plugin with compile steps
 
2 Attachment(s)
In an effort for some IPTV channels I am working for importing into SageTV I updated the plugin for more use cases. The changes below should work with most XMLTV grabbers. Also for future development there are easy steps for compiling the plugin on the Unraid Docker image. The Unraid Docker Image already has the required java_jdk installed for compiling. In the future I might look into the the ICON working as the data is also stored inside the XMLTV file.

(Warning me venting)
The original was not touched since the beginning of SageTv and impossible to use without remapping or using scripting for remapping channels. It is almost like the developers of SageTV wanted everyone to use their guide data or SD. As for SD the plus is that it gives 2 weeks of data with more info than XMLTV grabbers thus being a good value. Also to be fair to SageTV they also did give a good run(~15 years) with their guide data. Any case with SageTV on its last leg this needs to change as other DVR development have XMLTV native. Viva La SageTV :)

1. Stop SageTV Server
2. Rename on add/modify/ xmltv_EXAMPLE.properties examples
3. Copy all files and folder contents(not folder) of folder SAGETV_SERVER_ROOT to SageTV folder. Only Jar and *.proerties are required if not compiling required
4. Add the following line in Sage.properties epg/epg_import_plugin=xmltv.XMLTVImportPlugin
5. Start SageTV Server
6. Monitor in server folder xmltv.log, sagetv_0.txt,Sage.properties. If epg/epg_import_plugin=xmltv.XMLTVImportPlugin is removed from Sage.properties something is installed not correctly
7. In SageTV guide setup use XMLTV with zipcode 00000. If ask for license use TRIAL


#######Commnds for UnRaid Docker to Moditfy Sage.properties for Plugin
# Stop Sage Server
sudo -E "PATH=$PATH" -u sagetv /usr/local/bin/stopsage &
# Property Setting for EPG Plugin
sudo sed -i 'epg/epg_import_plugin' /opt/sagetv/server/Sage.properties
sudo echo 'epg/epg_import_plugin=xmltv.XMLTVImportPlugin' >> /opt/sagetv/server/Sage.properties
# Start Sage Server
sudo -E "PATH=$PATH" -u sagetv /usr/local/bin/startsage &

#########Commnds for UnRaid Docker Compile of Plugin
cd /opt/sagetv/server/xmltv
sh ondocker_build.sh

Code:


/* Change Log
##V2.0  01/13/2022
1.  Add code if display-name or channel id number(example 5, 5.1 5-1) then add as the channel number for import.  Examples below import as channel 2.1 and 1162 respectively below
##2.02  01/14/2022
1.  Fixed first channel * bug
2.  Fixed double year in title when title already had a date
3.  add tms filed for episode-num instead of creating unique id
4.  Do not generate SxEx if default case is found ("onscreen", "common")
5.  Fix Display for SxEx in descriptions
6.  Fix short name being set as channel number
7.  Start code for reading Logo Icon for channel (WIP)
##2.03  01/14/2022
1.  Fixed new Integer((int) to remove compile deprecated warning
2.  Switched org.xml.sax.helpers.XMLReaderFactory to javax.xml.parsers.SAXParserFactory; to remove compile deprecated warning
##2.04 01/16/2022
1.  Added .properties entry for xmltv.channel.display-name.ShortNameIndex
                default=0 with no entry .properties file
                init.ChannelShortNameIndex0:=shortest found without spaces
                init.ChannelShortNameIndex>0: XMLTV file index found in order starting with 1 <display-name>CBS</display-name>
2.  Added .properties entry for "xmltv.channel.xmltv.channel.ShortNameRegEx"               
                default=.* with no entry .properties file will find anything
3.  Added .properties entry for xmltv.channel.display-name.LongNameIndex
                default=0 with no entry .properties file
                init.ChannelLongNameIndex0:=longest found with spaces
                init.ChannelLongNameIndex>0: XMLTV file index found in order starting with 1 <display-name>CBS</display-name>
4.  Added .properties entry for xmltv.channel.NumberTag
                default="" No entry .properties file. 
                init.ChannelNumberTag="" None will try to detect channel number from <channel id=XXX> or <display-name>XXX<display-name>
                init.ChannelNumberTag=channel then <channel id=XXX>
                init.ChannelNumberTag=TAG_NAME then <TAG_NAME>xxx<TAG_NAME>
5.  Added .properties entry for init.ChannelNumberTagIndex               
                default=0 with no entry .properties file
                init.ChannelNumberTagIndex=0 None will try to detect channel number from <channel id=xxx> or <display-name>
                init.ChannelNumberTagIndex>0 XMLTV file index found in order starting with 1
6.  Added .properties entry for "xmltv.channel.xmltv.channel.NumberTagRegEx"               
                default=.* with no entry .properties file will find anything
7.  Added .properties entry for "xmltv.channel.IconDownload"
                default=false with no entry .properties
                false=Do Note download image file
                true=Download image file from channel <icon src= >
##2.06 01/18/2022
1.  Added the following .properties entry  for enabling or disabling logging parts.  Defualt with no entry is true
                log.show=false
                log.channel=false
                log.configuration=false
                log.defaults=false
2.        Changed the way ChannelId is calulated. The previous way caused collections.  The new way uses the ProviderID as part of the encoding. The new limit limits the Provider ID to 998.
                2147483647        Max Number
                1000000000
                1030002100
                      99999        SD MAX it Sends
                  998          Upper Part is Listing ProviderID 1-998 999 used if none provided (* 1000000)
                    123456    Max XMLTV Number without Decimal 999999 
                1  123456    Max XMLTV Number with Decimal 9999.99 When Decimal (* 100 +1000000000) 
3.  Remove non file characters so Channel icons could be downloaded the following examples where failing: "Cats 24/7" or "Dogs 24/7"       
##2.08 01/28/2022
1.  Filter if desc tag is from channel section.  This would crash the plugin if not done.
2.  Add show if it is up to 8 hours previous. Fixes on first import no data problem
3.  Add setting this.show.showId on <episode-num system="pluto"> tag
4.  Fixed freeFormEpisodeNumber on <episode-num system="onscreen"> and <episode-num system="common"> tags
5.  Add new classs to hold all the init. variables from .properties file
6.  Added xmltv.channel.NumberOffset to offset channel numbers on import.
7.  Added if *.xmltv.properties is found in server directory use it as properties without requirement of configurations= in xmltv.properties file
*/


https://forums.sagetv.com/forums/att...1&d=1642548450

boukmandutty 01-13-2022 06:26 PM

2 Attachment(s)
The auto mapping works great.

The problem that remains for me is how sagetv reads the epg provided by Channels dvr. As @nyplayer pointed out over in the hardware thread, sagetv does some strange things with the names of movies and the titles of series when the epg comes form channels dvr. See the attached pictures. The same thing happens when I send the epg from channels dvr through xteve.

hvymetal 01-13-2022 06:36 PM

Quote:

Originally Posted by boukmandutty (Post 632571)
The auto mapping works great.

The problem that remains for me is how sagetv reads the epg provided by Channels dvr. As @nyplayer pointed out over in the hardware thread, sagetv does some strange things with the names of movies and the titles of series when the epg comes form channels dvr. See the attached pictures. The same thing happens when I send the epg from channels dvr through xteve.

Have you inspect the XMLTV file if is store like that? Also do you sample XMLTV file?

boukmandutty 01-13-2022 06:52 PM

1 Attachment(s)
I have attached a sample. I dont know where to access the Schedules Direct version for comparison though. I can confirm from experimentation that the current format from the channels dvr server works flawlessly with nextpvr and even with mythtv.

nyplayer 01-13-2022 07:08 PM

Quote:

Originally Posted by boukmandutty (Post 632571)
The auto mapping works great.

The problem that remains for me is how sagetv reads the epg provided by Channels dvr. As @nyplayer pointed out over in the hardware thread, sagetv does some strange things with the names of movies and the titles of series when the epg comes form channels dvr. See the attached pictures. The same thing happens when I send the epg from channels dvr through xteve.

You got it working on linux ?

boukmandutty 01-13-2022 07:14 PM

Quote:

Originally Posted by nyplayer (Post 632575)
You got it working on linux ?

Yeah the auto-mapping works with the files provided in the attachment above. The only problem continues to be the funky thing that happens with the names.

nyplayer 01-13-2022 07:18 PM

Quote:

Originally Posted by boukmandutty (Post 632576)
Yeah the auto-mapping works with the files provided in the attachment above. The only problem continues to be the funky thing that happens with the names.

I guess I will wait ...
Yeah the names and epgid etc ...also season and episode even though it is there is not recognized by SageTV. So I went with a sched direct west coast and east coast lineup.... and manually mapped channels I wanted.

hvymetal 01-13-2022 07:34 PM

Quote:

Originally Posted by boukmandutty (Post 632573)
I have attached a sample. I dont know where to access the Schedules Direct version for comparison though. I can confirm from experimentation that the current format from the channels dvr server works flawlessly with nextpvr and even with mythtv.

Since I do not know which entries are not correct can you provide and example file and test case is wrong? Also it would help if in the photos you highlight the problem.

Thanks

hvymetal 01-13-2022 07:39 PM

Quote:

Originally Posted by nyplayer (Post 632575)
You got it working on linux ?

The plug in not Linux dependent it should also work with windows.

nyplayer 01-13-2022 07:46 PM

Quote:

Originally Posted by hvymetal (Post 632579)
The plug in not Linux dependent it should also work with windows.

Great I will test it on my Windows test machine ...


I know for sure there is no original airdate parsed season episode parsed as misc.... and epgid data is wrong ...

https://forums.sagetv.com/forums/showthread.php?t=66885

boukmandutty 01-13-2022 09:21 PM

5 Attachment(s)
The pictures are below.

The xmltv info for The Angry Birds is:

Quote:

<programme start="20220114020000 +0000" stop="20220114040000 +0000" channel="6081">
<title>The Angry Birds Movie 2 (2019)</title>
<desc>The birds unite with the green pigs when fowl from an ice-covered island plan to destroy them both.</desc>
<category>Movie</category>
<category>Feature Film</category>
<category>Animated</category>
<category>Children</category>
<category>Comedy</category>
<category>Adventure</category>
<icon src="https://tmsimg.fancybits.co/assets/p14568986_v_v12_ac.jpg?w=480&amp;h=720"></icon>
<date>20190802</date>
<previously-shown></previously-shown>
<episode-num system="tms">MV010390220000</episode-num>
<credits>
<director>Thurop Van Orman</director>
<actor>Jason Sudeikis</actor>
<actor>Josh Gad</actor>
<actor>Leslie Jones</actor>
</credits>
</programme>
The Family Guy episode:
Quote:

<programme start="20220114023000 +0000" stop="20220114030000 +0000" channel="6082">
<title>Family Guy</title>
<sub-title>Peter's Progress</sub-title>
<desc>A psychic reveals that Peter had a past life in the 17th century as Griffin Peterson.</desc>
<category>Episode</category>
<category>Series</category>
<category>Animated</category>
<category>Sitcom</category>
<icon src="http://tmsimg.fancybits.co/assets/p184483_b_h6_bb.jpg"></icon>
<series-id system="tms">184483</series-id>
<date>20090517</date>
<previously-shown></previously-shown>
<episode-num system="tms">EP002960010132</episode-num>
<episode-num system="onscreen">S7E16</episode-num>
<episode-num system="xmltv_ns">6.15.</episode-num>
<credits>
<director>Brian Iles</director>
<director>Peter Shin</director>
<actor>Seth MacFarlane</actor>
<actor>Alex Borstein</actor>
<actor>Seth Green</actor>
</credits>
</programme>
I have attached for each a marked-up picture of what it looks like with the xmltv plugin. There is also a picture of what it looks like when Schedules Direct is used through the tedious mapping.

@nyplayer I didn't see your recent posts on your attempt to fix this until just now when you linked to it. If you have managed to fix any part of what I have here let me know.

hvymetal 01-13-2022 10:17 PM

WTF :bang:

Code:

        String title = this.show.title;
                // FIXME handling rerun shows correctly with SageTV 9
                // String rerunTitle = null;
                String rerunTitle = this.show.title;
                String episodeName = this.show.episodeName;
                if (this.initcapAllChannelIds
                        || this.initcapChannelIds
                                .contains(this.channel.xmltvId)) {
                    if (this.initcapTitle) {
                        title = initcap(title);
                    }
                    if (this.initcapEpisodeName) {
                        episodeName = initcap(episodeName);
                    }
                }
                if (title != null) {
                    if (this.show.date != null
                            && isActivated(categories,
                                    this.dateTitleDecorationCategories)) {
                        title = this.dateTitleDecoration.format(
                                new Object[] {title, this.show.date},
                                new StringBuffer(), null).toString();
                    } else if (this.titleAddYear
                            && this.show.year != null
                            && isActivated(categories,
                                    this.titleAddYearCategories)) {
                        title += " (" + this.show.year + ")";
                    }
                    if (this.show.quality != null
                            && this.show.quality.startsWith("HD")
                            && isActivated(this.channel.id,
                                    this.hdTitleDecorationChannels)) {
                        title = this.hdTitleDecoration.format(
                                new Object[] {title}, new StringBuffer(), null)
                                .toString();
                    }
                }
                if (this.show.rerun) {
                    rerunTitle = title;
                    // FIXME handling rerun shows correctly with SageTV 9
                    //title = null;


hvymetal 01-14-2022 03:22 AM

Quote:

Originally Posted by boukmandutty (Post 632583)
@nyplayer I didn't see your recent posts on your attempt to fix this until just now when you linked to it. If you have managed to fix any part of what I have here let me know.

Update should fix the problems listed. Update is in top post.

hvymetal 01-14-2022 03:24 AM

Quote:

Originally Posted by nyplayer (Post 632580)

Not sure if I fixed the problem because I could not find it but some of the other changes might have solved this one also.

nyplayer 01-14-2022 06:58 AM

I am getting this error ... on Windows. All I want to test is bringing in epgdata.xml which is in the SageTV root directory ... no XTEVE yet.

14-01-2022 04:47:29,860 java.lang.NoClassDefFoundError: xmltv/Channel
at xmltv.XMLTVImportPlugin.startElement(XMLTVImportPlugin.java:1431)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator.startElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
at xmltv.XMLTVImportPlugin.updateGuide(XMLTVImportPlugin.java:1299)
at xmltv.XMLTVImportPlugin.updateGuide(XMLTVImportPlugin.java:1204)
at sage.EPG.pluginExtractGuide(EPG.java:1625)
at sage.WarlockRipper.extractGuide(WarlockRipper.java:973)
at sage.EPGDataSource.expand(EPGDataSource.java:709)
at sage.EPG.run(EPG.java:773)
at java.lang.Thread.run(Unknown Source)
14-01-2022 04:47:29,860 java.lang.NoClassDefFoundError: xmltv/Channel
at xmltv.XMLTVImportPlugin.updateGuide(XMLTVImportPlugin.java:1311)
at xmltv.XMLTVImportPlugin.updateGuide(XMLTVImportPlugin.java:1204)
at sage.EPG.pluginExtractGuide(EPG.java:1625)
at sage.WarlockRipper.extractGuide(WarlockRipper.java:973)
at sage.EPGDataSource.expand(EPGDataSource.java:709)
at sage.EPG.run(EPG.java:773)
at java.lang.Thread.run(Unknown Source)

hvymetal 01-14-2022 08:08 AM

Quote:

Originally Posted by nyplayer (Post 632598)
I am getting this error ... on Windows. All I want to test is bringing in epgdata.xml which is in the SageTV root directory ... no XTEVE yet.


I think this was caused because the docker I'm using is Java16 and I did not compile for lower version of java. Can you test the attached version it was compiled for 8 or higher of java.

nyplayer 01-14-2022 09:07 AM

1 Attachment(s)
These are the error logs and my xmltv.properties... I might be doing something wrong in the setup.

hvymetal 01-14-2022 09:22 AM

What Java are using? Also how did you install? You should only need the .Jar file copied to you servers JARs folder. Make sure you do not have older versions of the plugin in the same folder(.jar or folders). Also any xmltv folder in the root of the Server or in the JARs referencing the older version need to be renamed

nyplayer 01-14-2022 10:19 AM

Quote:

Originally Posted by hvymetal (Post 632606)
What Java are using? Also how did you install? You should only need the .Jar file copied to you servers JARs folder. Make sure you do not have older versions of the plugin in the same folder(.jar or folders). Also any xmltv folder in the root of the Server or in the JARs referencing the older version need to be renamed

jre1.8.0_191


I am going to try a full fresh install of SageTV there might be some stuff from the old XMLTV around.

hvymetal 01-14-2022 10:43 AM

Quote:

Originally Posted by nyplayer (Post 632608)
jre1.8.0_191


I am going to try a full fresh install of SageTV there might be some stuff from the old XMLTV around.

It worked with the following setup in a Window 7 64bit VM
1. Installed with latest SageTv 64bit installer
2. Ran SageTV First time
3. Stop SageTV
4. copy the XMLTVImportPlugin.jar in SageTv Jars folder
5. Add the following line in Sage.properties epg/epg_import_plugin=xmltv.XMLTVImportPlugin
6. Copied a XMLTV FILE name epgdata.xml
7. Start SageTV
8. Source Setup->EPG Lineup
9. Zip Code 0000
10. Will list XMLTV Lineup
11. Use extended

The plugin will work with out any .properties file if epgdata.xml in the root of the SageTV server directory. It will give error in xmltv.error.log for not finding the .properties but still work(This is a bug and need correcting). The .properties are used if you want to do customizations and add multiple lineups. The examples I provided 2 examples that give the guide a different name and location of the XMLTV file.


All times are GMT -6. The time now is 09:27 PM.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2003-2005 SageTV, LLC. All rights reserved.