PrimeNetEncoder - HDHomeRun Prime Network Encoder
 

Hi,

First I would like to thank all of the forum members that beta tested for me. Their feedback was very helpful in ensuring that my first release was as stable as possible. I need to give special thanks to nyplayer. He went above and beyond in his testing efforts and helped me work through many of the issues.

Overview

PrimeNetEncoder is a network encoder for HDHomeRunPrime. It is designed to run on both windows and linux and to have minimal requirements. The encoder uses the hdhomerun_config application and ffmpeg to capture the stream. Since ffmpeg is only capturing the stream and writing to a file PrimeNetEncoder has very low overhead.

PrimeNetEncoder can be configured to handle any number of HDHomeRunPrime tuners. If you are going to use more than one HDHomeRun Prime you will need to modify the tuners.count property, and create new entries for the tunners in the PrimeNetEncoder.properties file, and create new tuner entries in the sage.properties. Forum members have tested and confirmed that PrimeNetEncoder can also work with other HDHomeRun tuners. Please check the forums for notes on compatibility.

PrimeNetEncoder also offers the option to transcode to h.264 in realtime. This does requires a reasonable amount of CPU resources to accomplish. Transcoding is configurable at the tunner level. There are settings available for quality and performance.

Guides
Note: Installation instructions are included in the download.
There is a very good guide to setting up the PrimeNetEncoder on Linux by SageWizdom http://forums.sagetv.com/forums/showthread.php?t=62737
Below this post is a guide on setting up Non Prime DLNA OTA tuners by nyplayer
Known Issues

1. Please read the install guide for specific issues.

Upgrading

See any specific upgrade notes for the version. To upgrade from a previous version you should replace the PrimeNetEncoder.jar with the new one. You should run the program once so that it can add the new properties to the PrimeNetEncoder.properties file.

Changelog

Version 2.0.0 - Download

• Added quit command. This will attempt to exit gracefully if the tuners are not currently recording.
• Added an local address override to the properties file. This is to workaround some issues with certain Linux distros, and possibly machines with multiple IP addresses
• Change property mediaserver.transfer default value from true to false. Some machines seem to be struggling with transferring to sage using this method.

Version 1.2.5 (RC1) - Download

1. Increased UDP receive buffer size to 262140
2. Added properties for ffmpeg.outputbuffersize, ffmpeg.inputbuffersize and mediaserver.outputbuffersize
3. Set default size for input/output buffers to 32768
4. Added signal quality output to the STATUS command
5. Set a short timeout on UDP read to try and reduce latency
6. Added a PrimeNetEncoder.bat file with recommended settings to pass to java

Version 1.2.4 (Beta) - Download

1. Fixed mediaserver.transfer = false.
2. Added the ability for the OutputWatcher to reset the tuner if data is not produced in a timely fashion
3. Enhanced the exception handling in the TunerOutput thread. This was causing issues with keeping the UDP port open
4. Set thread priority to TunerOutput to MAX. Set thread priority to Watcher to MIN
5. Cleaned up some of the code that was double buffering streams. This might have been causing performance issues
6. Added timeout to the UDP connections

Note: PrimeNetEncoder.properties was removed from the download. Users will need to run the application once to generate a default properties file. Documentation was updated to reflect this.

Version 1.2.2 (Beta) - Download

1. Added media server transfer
2. Using STDIN/STDOUT to transfer data between ffmpeg and PrimeNetEncoder
3. PrimeNetEncoder is handling Stream from HDHomeRun instead of ffmpeg
4. Added ffmpeg setting analyzeduration and probesize
5. Added tuner locking (Note: I do a force unlock when ever I start a new recording)
6. Modified discovery logic to only respond to discovery requests if there is not an active connection with SageTV server

Note: If you are upgrading from a previous version I recommend that you remove the current tuner entries from Sage.properties file and allow SageTV to use network encoder discovery to add the tuners. This ensures that the right properties are added to SageTV. If you are using Windows Firewall please review the install guide for addition ports that need to be opened.

Version 1.0.1 (Beta) - Download

1. Better error handling when binding port is already in use
2. Added setting for delay to wait for ffmpeg to fully start
3. Added scaling property to the transcoding to allow resizing video
4. Adding setting for changing codec used for transcoding

Version 1.0.0 - Download

1. Initial Release

Attached none Prime DLNA Tuners
 

Imortant*************************

The properties Included in the Doc are wrong they will be corrected once the Coder gets a chance. The correct ports are below. I have attatched the corrected properties file PrimeNetEncoder.zip


Attached is the doc on setup of Non Prime DLNA OTA tuners..... HDHR4-2US and HDTC-2US

Looking forward to giving this a try this weekend. I'm hoping this will resolve my random pixelation and sound pops.

Couple of questions. Why would I want to transcode to H264? And why would I want to use this for the non-prime HDHR?

Also, does this replace SageDCT and does it need to be removed from my installation? Which version of Silicondust Firmware does this work with? I've been reluctant to update not knowing if it would break SageDCT.

-uberpixel

The answer to the first answers the second. Space saving, and better playback on devices (many of which no longer support MPEG-2).

I am on the latest firmware as Silicondust has done so many changes that if you are not on the latest beta you should be for EX. Hd Homerun View for Kodi and android.. Fuzzy answered your other questions. I completely removed SAGEDCT and am quite happy I did. As far as none Prime tuners Sage remuxes the mpg and it messes it up that you cannot even extract CC. Also you can esaily update to the latest FFMPG simply by dropping the FFMPEG exe in the folder.

I get no pixelation what so ever since I switched to this encoder.

PS if you are reluctant then stay where you are as some people fear change.

You do not need to remove SageDCT. I would suggest you stop the SageDCT service while you are testing my application though. The machine I developed this on had SageDCT installed on it while I was developing and testing this.

Josh

Well, after trying to set this up I can report that it is not idiot proof...

I'm not an IT guy (Mechanical Engineer) so it's possible that I missed some implied steps in the install process (which I thought I followed correctly) but I'm having problems nonetheless.

The odd behaviour starts after I add the new tuner. At first it shows 1 "Digital Tuner" but when I select it, it pauses and then another "Digital Tuner" option shows up (see attached screenshot). If I select either - the wizard moves on to the next step. Then when I get back to the source summary page, it lists 3 identical versions of the same Prime tuner (see attached screenshot).

If I go back and remove the new source and add it again, it shows 4 "Digital Tuner" options to select...

If I try to tune a channel with the new source it says no signal.

I have not removed the legacy SageDCT tuners from my source list but I have disable the service.

I'm running the latest stable firmware from HDHomerun 20141210. I know you said latest Beta but I was hedging my bets in case this didn't work so I could go back to a previously stable SageDCT solution. Java 8. More system info in my signature.

Any ideas where I may have screwed up? I can post my properties files if that would help.

-uberpixel

By any chance are you using Diamond or Gemstone? I am asking this because I saw this when using Diamond but not the default STV. If you stop and restart the SageTVService you will see that it is a mirage. I had the same problem when using the firewire Network tuner.

Gemstone...

I'll try the install again with the default STV and see what happens.

Not sure why that would cause the "No Signal" result but I'll give it a whirl.

-uberpixel

That would not cause the no signal just the duplicate showings of tuners the no signal is usually a firewall issue. Make sure ffmpeg is defined in the firewall I gave full access to the ffmpeg in the PrimeNetEncoderv1.0.0 folder. As I added tuners I did not want to keep opening ports.

Same result with the default SageTV 7 STV.

Still getting multiple instances of the same tuner on the same port.

Showing 3 all on port 7000 for example. Then no signal when I try to tune.

-uberpixel

You cannot do a preview it will not work. Is there a way for you to test without the firewall? turn off firewall go to the guide and view a channel. For now ignore the duplicate tuners you are seeing just set 1 up. for tuner 0 tuner 1 tuner 2. It will not affect anything as far as recording etc... If you simply disable Gemstone you will still see the duplicate tuners ... I have no answer for this but it is only a visual problem not a working problem.

Ps I am also using HDHomerun Prime + (Comcast)

No joy with the firewall off.

I'm starting to wonder if my issue lies in the channel setup and guide. While trying to preserve my existing installation I may be adding a layer of complexity to this install that is tripping me up. I think I may use a spare server license and setup a clean development install.

I have a feeling I'll be doing plenty of fiddling with sage in the future so having another server to play with may be just the ticket!

Looks like we're neighbors... We could get together over a pint of Hefeweizen and reminisce about classic moments in the SageTV saga. Or not... :goodjob:

-uberpixel

If you have time could you please post PrimeNetEncoder logs and properties, and sagetv properties. That way I can see if there are any potential setup issues.

Yes you are correct that the process to setup is not the easiest. I wish I could get the auto discovery to work properly with Sage. It would make this process a lot easier.

Josh

Logs attached...

Thanks for taking a look!

-uberpixel

Edited... Scratch what I said. Still looking at logs.

It looks like everything is setup correctly. Can you confirm that when you switch the channel that a file is being created in the recording directory. Can you also confirm that the file is growing in size, and is not staying at 0 bytes.

If the file is not generating correctly can you confirm that you have the right version of ffmpeg in the SageTV directory.

The recording shows up in the directory but does not grow beyond 0 bytes. Also a file with this name is created: "HDHomeRunPrimeTuner0on1921681137000DigitalTVTuner-0.mpgbuf"

For ffmpeg I've moved only the ffmpeg.exe into the SageTV directory - I left all of the other files behind since the folder structure didn't seem to jive with your instructions to place ffmpeg.exe in the SageTV directory. I downloaded the latest ffmpeg package from here: http://ffmpeg.zeranoe.com/builds/ and I'm using the static 64-bit version.

I mentioned earlier that I am not using the latest beta from SiliconDust. Could that be an issue?

The .mpgbuf file are beging created when you are trying to preview the channels when adding the tuners to Sage. Previewing or calibrating the tuners from inside of Sage is not supported.

I do not think the firmware should matter. I have two things you can look at.

Can you confirm that hdhomerun_config is located where the settings are pointing to: C\:\\Program Files\\Silicondust\\HDHomeRun\\hdhomerun_config.exe

Why did you download ffmpeg from the location that you did? Why not the current release from ffmpeg.org?

hdhomerun_config is in the correct location.

I followed the link to the Windows static build from the ffmpeg.org site and it sent me to the link previously shown. I just now tried downloading the .tar from ffmpeg.org and there isn't an ffmpeg.exe in that package.

-uberpixel

Think I found your issue.

You did not change tuner ids in the PrimeNetEncoders.properties.

You need to change the below setting for each tuner. Your ID is 1313BED0

That was it.

As it turns out I copied the properties file to the Sage directory and then proceeded to edit the original file in my downloads folder... :blush:

Watching basketball tonight and so far the only glitches I've seen look like they are coming from the satellite truck broadcasting the game. That's a big improvement from what I had been seeing!

I did see some strange behaviour when I added the second and third tuners (channel tuning failure and pixelated video) but a full system restart (server, and Prime) cleared it up.

Thanks guys for all of your help, and thanks to jvl711 for building this encoder! It's nice to know I don't need to live in fear of my Prime no longer working due to a future incompatibility.

-uberpixel

Well, there is something nasty about my system. When I try to test out my setup by doing a 'java -jar PrimeNetEncoder.jar status', I get repeating this:

java.lang.NullPointerException
at jvl.primenetencoder.Tuner.run(Tuner.java:100)

Any clue what is the problem here?

C:\Sagetv\PrimeEncoder>java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) Client VM (build 24.65-b04, mixed mode, sharing)

C:\Sagetv\PrimeEncoder>

I know the tuner works because I was using the HDHomeRun VIEW program to watch a couple of channels.

Are you running a batch file if you are can you post it this is how I am running. I have everything in C:\PrimeNetEncoderv1.0.0

encoder.bat

Nope, no batch file. Just straight Command Line:

C:\Sagetv\PrimeEncoder>java -jar PrimeNetEncoder.jar status
PrimeNetEncoder>java.lang.NullPointerException
at jvl.primenetencoder.Tuner.run(Tuner.java:100)
java.lang.NullPointerException
at jvl.primenetencoder.Tuner.run(Tuner.java:100)
java.lang.NullPointerException
at jvl.primenetencoder.Tuner.run(Tuner.java:100)
java.lang.NullPointerException
at jvl.primenetencoder.Tuner.run(Tuner.java:100)
java.lang.NullPointerException
at jvl.primenetencoder.Tuner.run(Tuner.java:100)
java.lang.NullPointerException
at jvl.primenetencoder.Tuner.run(Tuner.java:100)
java.lang.NullPointerException
at jvl.primenetencoder.Tuner.run(Tuner.java:100)
java.lang.NullPointerException
at jvl.primenetencoder.Tuner.run(Tuner.java:100)
java.lang.NullPointerException
at jvl.primenetencoder.Tuner.run(Tuner.java:100)

Specifying the location of java didn't help either
C:\Sagetv\PrimeEncoder>"c:\Program Files\Java\jre7\bin"\java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) Client VM (build 24.65-b04, mixed mode, sharing)

C:\Sagetv\PrimeEncoder>

Why not just use a batch like the one I posted above I see your "c:\Program Files\Java\jre7\bin"\java -version
java version "1.7.0_67" is wrong quotes are around the directory not the full jave.exe.

you can just create the encoder.bat and drop it in the directory where you have the encoder properties jar and ffmpeg.

I copied your batch file that you posted above, modified the path for java (to remove the (x86) since I'm running 32 bit Win7) and used that. Same result...

You definitely have something wrong maybe you should just zip the whole folder. Could be the properties file.

Dropped you a private message.

Here is the correct batchfile. When you run it you can issue commands such as status etc. at the console window. Everything else is ok. Just replace the one you have.

put your runEncoder.bat file into my PrimeEncoder directory and ran it. Same stinking Exception. I'm thinking we're gonna need to look at that .java file to see where the exception is being thrown and what object is Null.

Thanks for your help so far, but bedtime for bonzo.... Have to pick this up again another evening....

You sure you don't already have 1 running check for a task java.exe
My System ran ok with your stuff must be your Java. Goodnight

04/07/2015 21:32:37 - Setting local IP address to: 192.168.0.104
04/07/2015 21:32:37 - Starting encoder thread: 131B4C09-0

04/07/2015 21:32:37 - Setting local IP address to: 192.168.0.104
04/07/2015 21:32:37 - Starting encoder thread: 131B4C09-1

04/07/2015 21:32:37 - Setting local IP address to: 192.168.0.104
04/07/2015 21:32:37 - Starting encoder thread: 131B4C09-2

Does this eliminate the delay between back to back recordings?

No not network encoder does. The only way to eliminate this is to have SageTV natively support Primes.

Just curious, but you lose the seamless recording transitions with this right?

Yes, there's no mechanism in the current network encoder system to handle directly transitioning a stream to a new file seamlessly. Might be something that could be improved once we're into the code, its just that no real work was done on the network encoder spec for a very long time before seamless recording came to be a thing. It should be noted, that there is NO implementation of the Prime (SageDCT or this) that support the seamless transitions, as both are network encoders.

That said, the HDHR starts and stops streams very quickly, such that it is not a very long gap.

I think that's in the "advanced" network encoder protocol that Sage uses when you setup an Encoding server, but was never documented for the "public" protocol.

Right but I dropped my cablecard and now I just use everything Clear QAM native in Sage, but as noted above, Sage doesn't have the best handling of transport streams, so I'm curious to try this out and see if it solves some of my issues. However the IRPStackSize tweak may have taken care of 90%+ of the errors I was seeing so my motiviation has dropped a bit.

If nothing else, by using this and passing the stream through ffmpeg, it fixes (at at least drops) bad packets in the steram. This can turn an unplayable stream into one that just has a bit of corruption here or there. Being able to record direct to H.264 has other great benefits.

I will say, that the network encoder interface is the first thing I plan on looking at when we get into this thing, hopefully some of the features are there that we need to improve the system and it will just take some documentation to expand those features to these 3rd party encoders. If not, I guess I'll be adding some features as well... I had, early on, talked about wanting to roll this solution (the Prime encoder) into Sage, but I am thinking more and more that a more modular support, where nearly all future encoders are network based, will be more versatile and robust going forward - but the API would have to add in some things like scanning and/or retrieving channels lists and such.

Well... Not sure if it's related to this new encoder but I've started experiencing occasional tuner issues like "Halt Detected" and "Recording Missed Due to Conflict". Worse yet I've seen a few lock-ups and reboots on my server. I'm running a few other things on the server (Blue Iris as well as serving files) so it's possible that the reboots aren't related to Sage.

With the Halt's it typically drops around 5 minutes of the program and then picks it back up again and continues on.

For the tuner conflict, in some cases, it's the only program being recorded so I know I don't have any schedule conflict. I suspect the tuner is dropping out entirely and Sage reports a conflict.

Anyone else seeing this behavior with the new encoder?

-uberpixel

Mine has been rock solid since the beginning no problems at all.

The halt detected I was getting a lot of. I reinstalled sage and that seemed to fix the problem I have only had three halt detected since then. Two of them ere back to back recordings and required a reboot of the prime so that one I don't think was the encoder problem. Maybe try a reinstall of sage not the whole server, if you can find the time to do it.