Native HDHomeRun Prime Support

[jvl711]

I was wondering if anyone is working on this for either Windows or Linux. If not I am willing to try and put some effort into it, but I am not sure where to even start. On windows it appears that the first issue happens very early on in the DirectShow discovery of the device. I am sure even after that is resolved there would still be a ton of additional issues to sory out.

Are there different options to support the HDHomeRun Prime that might be more cross platform than suing DirectShow? Should libhdhomerun be utilized on both Windows and Linux? Would there be some benefit to using it over Direct Show?

[Fuzzy]

I think going forward most of these type of support should be in the form of a network encoder, NOT natively built into the core. This will future proof things far better. Of course, I really different tasks being segregated completely. I think that currently, PrimeNetEncoder is still relying on some windows .exe's, but I think all the features that are required are in the web interface of the HDHomeRun, including streaming the video, which ffmpeg could pull in. this would allow full cross platform capability, and not depend on any drivers or software being installed on a given device.

In all honesty, I am hoping to approach silicon dust about actually building network encoder support directly into the HDHomeRun firmware, which would be quite awesome... This can't be done until there is an extension made to the network encoder profile to enable streaming from an encoder to the server and having the write that stream to disk itself. With the current network encoder profiles, it will require the encoder (the HDHomeRun in this case) to have write access to network share, and I'm thinking that would be far more complication than it'd be worth.

[Narflex]

Originally Posted by Fuzzy

This can't be done until there is an extension made to the network encoder profile to enable streaming from an encoder to the server and having the write that stream to disk itself. With the current network encoder profiles, it will require the encoder (the HDHomeRun in this case) to have write access to network share, and I'm thinking that would be far more complication than it'd be worth.

This is already in the protocol. There's an 'uploadID' (seen in NetworkCaptureDevice.java) which can be passed to network encoders which then can be used when the encoder connects to the SageTV Media Server (MediaServer.java) which then grants it write access over the protocol. This is the same server that streams to SageTVClient...it has various extra commands that you probably never knew about such as directory listings (recursive as well), read/write capabilities and also transcode streaming (if that part even works, I don't recall what it was ever used for).

[Fuzzy]

Ahh, yeah, I had seen the UploadID, but didn't dig too deep into it, as it looked like it was tied deeply into the core network encoder/client connection. I'll have to do some reading.

[KryptoNyte]

When I read "native" support, I was thinking about the ability to play the stream directly in Sage, which is currently hit or miss (the "dump" method as SageDCT calls it, the networkencoder doesn't offer this option). If Sage was capable of rendering the direct stream, wouldn't that be a good precursor to whatever the tuner solution may be?

[mechling-burgh]

Fuzzy if we would do network capture with the built network encoder would we not be just capturing the native stream like SageDCT. If so would we not be also caring over the same problems with comcast where sagetv would not play them sometimes. I know we can update the SageTV Clients with the newest mplayer builds but if I have been reading the forums correctly we won't be able to correct them in the extenders since they can not be upgraded them to the newest mplayer, correct.

[Fuzzy]

Before I knew about the above mentioned uploadID network encoder PUSH method, I had intended the future STREAM network encoder command to leverage ffmpeg as the stream sink on the server, which would result in fixing streams on the way to the HDD. I can't test the 'broken' streams that comcast sends out, as I don't have comcast. I had used dump for a few years with Charter and not experienced much of a problem - I now use ffmpeg, but mostly as a result of switching to PrimeNetEncoder instead of SageDCT.

[Tiki]

One downside to the network encoders is I think they still do not support the "gapless" playback of live tv (you get a pause/glitch when the program you are watching is scheduled to end that lasts for a couple of seconds). Sage v7 added a fix for the pause with native recording sources, but not for network tuners.

[Fuzzy]

Originally Posted by Tiki

One downside to the network encoders is I think they still do not support the "gapless" playback of live tv (you get a pause/glitch when the program you are watching is scheduled to end that lasts for a couple of seconds). Sage v7 added a fix for the pause with native recording sources, but not for network tuners.

I think they might. There is a SWITCH statement, that is supposed to change to a new file. It'd be up to the encoder to break the file at a good point and continue the stream in the next file for it to work.

[mwnswiss]

Originally Posted by jvl711

I was wondering if anyone is working on this for either Windows or Linux. .... Should libhdhomerun be utilized on both Windows and Linux? Would there be some benefit to using it over Direct Show?

I did some work on Linux with the native/so/HDHomeRun2.0 library, made it to compile on OS/X, added with support from Taddeusz the libhdhomerun version 20150615, removed the hardcoded 2 tuner limit so it discovers now 4 tuners on respective models, fixed an issue with the DUAL EU model to properly handle DVB-T versus DVB-C. All should be merged in soon, else you can get it from my fork if you like to further test.

I don't have a PRIME, but if you would test the server on a linux (ubuntu 14.04 would be best) and describe what is not working, I may help. You would need to compile the HDHomeRun2.0 as debug, edit the HDHomeRun2.0 Makefile and add a line "DEBUG=1" on the top and make a clean.sh and buildall.sh afterwards. With debugging enabled on the SageTV server (setting in Sage.properties or via placeshifter) you will get the sagetv_0.txt logfile which will support identifying issues.

Be aware that there is still the blocker of recording video streams on 64bit, all the rest seems now pretty OK.

I don't think the libhdhomerun is used on Windows, but I didn't spend too much time. I recall windows is mainly using the BDA driver installed through the Windows HDHomeRun software.

All my changes here refer to the native plain libhdhomerun usage, NOT to the dvbhdhomerun kernel module usage!

[Narflex]

Originally Posted by Tiki

One downside to the network encoders is I think they still do not support the "gapless" playback of live tv (you get a pause/glitch when the program you are watching is scheduled to end that lasts for a couple of seconds). Sage v7 added a fix for the pause with native recording sources, but not for network tuners.

You can do gapless playback/recording with network encoders...it's done on Google Fiber. As Fuzzy mentioned, the SWITCH command can be used for that. You also need to set the "fast_network_encoder_switch" property for that capture to enable it since most network encoders won't support it by default.

Ideally we should probably extend the protocol to query the network encoder itself for this capability, as that would be easier to setup (of course if its setup using network encoder discovery, then it can control the config that way).

[jvl711]

Originally Posted by Narflex

You can do gapless playback/recording with network encoders...it's done on Google Fiber. As Fuzzy mentioned, the SWITCH command can be used for that. You also need to set the "fast_network_encoder_switch" property for that capture to enable it since most network encoders won't support it by default.

Ideally we should probably extend the protocol to query the network encoder itself for this capability, as that would be easier to setup (of course if its setup using network encoder discovery, then it can control the config that way).

Jeff,

Would you mind explaining a little bit how to use the UploadID with the MediaServer. I assume the NetworkEncoder client gets both the filename and uploadID from the server. It can then either place the file directly to the path itself, or use the MediaServer to upload the file. Besides not needing direct access to the storage path and possibility of gapless playback, is there other benefits to using this methodology. Would this enable quicker start of playback?

As mentioned by Fuzzy I am using FFMPEG to clean up the stream. I am not exactly sure what is wrong with the "pure" stream from comcast, but it can causes pixelation, audio pops, and sometime no video playback. For some reason when the stream is put through FFMPEG with no conversion it is able to fix the stream in such a way that Sage has no problems with it. Should/Could this type of stream fix be moved to SageTV? I think if this was fixed by Sage it might speedup the time it takes for Live TV playback to start.

Thanks,
Josh

[Fuzzy]

The UploadID isn't required for the gapless playback. All that is required is ensuring the streams are uninterrupted between files, meaning the transition happens at the right frame in the stream, and that the encoder is configured in sage as supporting this. What it essentially does, I believe, is since sage 'knows' it is the same stream simply continuing i a new file, it doesn't tear-down and recreate the playback graph.

The UploadID is a different thing, and yes, allows the encoder to push the stream to the server. I've been messing with it off and on the last couple days. Essentially, the encoder receives the uploadID with the START request, and then can EITHER write directly to the file, or open up a channel to the MediaServer port (8171). It requests permission to stream to the file via WRITEOPEN filename UploadID\r\n. If sage recognizes the uploadID, it allows it, and awaits "WRITE offset length\r\nDATA".

It seems to work on the server end for me, I'm having issues on my encoder end getting the data cleanly from ffmpeg. I am using Perl, trying to make my encoder as light as possible, and capturing the piped output from ffmpeg. PERL doesn't seem to have the umpf to do this without dropping data, so the stream gets corrupted. Going to have to move to a compiled c program for the streaming, most likely. In any case, avoiding samba share issues will be worth it if I can get this implementation to work.

[Narflex]

Originally Posted by jvl711

As mentioned by Fuzzy I am using FFMPEG to clean up the stream. I am not exactly sure what is wrong with the "pure" stream from comcast, but it can causes pixelation, audio pops, and sometime no video playback. For some reason when the stream is put through FFMPEG with no conversion it is able to fix the stream in such a way that Sage has no problems with it. Should/Could this type of stream fix be moved to SageTV? I think if this was fixed by Sage it might speedup the time it takes for Live TV playback to start.

Thanks,
Josh

Looks like Fuzzy answered the question about uploadID.

Regarding stream repair...SageTV has extensive code for doing this already. It's what's been always used with any of the Digital TV systems we support. It actually remuxes the transport stream into a program stream and then writes that out. There is a facility in sage.media.format.MPEGParser for doing the remuxing from a Java level...but I don't know if that specific path of code still works because we stopped using that awhile ago. At the native layer though, it definitely works.

Ideally, we would add this to the MediaServer so that you can send in the MPEG2-TS that way (along with some config parameters for setting up certain aspects of the remux) so that it'll do it as it's receiving the data before it writes it to the file. This would then provide this general remuxing service to any network encoder easily.

You may want to give the remuxing code in MPEGParser a shot and see if it works...it's pretty easy to use, and if it works...then that's a huge step forward regarding this.

[jvl711]

Thanks Jeff and Fuzzy. This is great info. I am going to try and implement all of it. I agree that if we could pull all of those things off it would be pretty good implementation. I also would like to add tuner locking support as well. I would like to add all the other features first though.

Josh

[Taddeusz]

One thing that would be nice to support is being able to do is "gang" tuners if what is scheduled to record is in separate programs of the same physical QAM channel. I know Cox cable here in OKC has some of the local networks that way. I know it's also a feature that the DVB crowd has requested in the past.

[Fuzzy]

Originally Posted by Taddeusz

One thing that would be nice to support is being able to do is "gang" tuners if what is scheduled to record is in separate programs of the same physical QAM channel. I know Cox cable here in OKC has some of the local networks that way. I know it's also a feature that the DVB crowd has requested in the past.

That's not something that would be, architecturally, impossible. But it would take changes in both the capture device/lineup systems, AND the scheduling system. There would need to be a way to identify channels that are over the same stream, which is not incredibly easy for sage at this point with the way remapping is done, which would obfuscate simply using tuning information. The scheduler would also need a check early on in it's algorithm to see if a given channel is already being recorded from a previous scheduler pass and use it. That's code changes touching parts of sage that are VERY important, and VERY complicated, so this is a down the road goal, but not something to be rushed into.

[jvl711]

I have made some pretty good progress. Here is how I am currently recording to Sage.

hdhomerun_config.exe (UDP) -> ffmpeg (STDOUT) -> SageTV MediaServer (TCP). This is working pretty well. I also made some tweaks to ffmpeg. I am setting analyzeduration and probesize. Channel changes are now taking ~3-4 seconds instead of ~6 seconds.

I also have discovery working properly. I even have discovery working correctly when PrimeNetEncoder is started as a runnable class from Sage.

Now I am looking to see if I can utilize the MPEGParser class to remux the file instead of ffmpeg. I have a few questions on how to use the class that I am hoping you guys might be able to help me with.
1. Is this library available in Windows environments? What file is this library compiled into.
2. What is the mode variable. I am assuming it is either REMUX_TS or REMUX_PS. Is that the input or output format?
3. What is the channel variable? I see in remuxFile it is set to 0.
4. Here is what I think the usage would look like. openRemux(REMUX_TS, 0, outputStream), pushInitData for first 8192 bytes, muxy.seek(0), pushData starting a position 0 till the end of stream, muxy.close().

[Narflex]

Originally Posted by jvl711

Now I am looking to see if I can utilize the MPEGParser class to remux the file instead of ffmpeg. I have a few questions on how to use the class that I am hoping you guys might be able to help me with.
1. Is this library available in Windows environments? What file is this library compiled into.
2. What is the mode variable. I am assuming it is either REMUX_TS or REMUX_PS. Is that the input or output format?
3. What is the channel variable? I see in remuxFile it is set to 0.
4. Here is what I think the usage would look like. openRemux(REMUX_TS, 0, outputStream), pushInitData for first 8192 bytes, muxy.seek(0), pushData starting a position 0 till the end of stream, muxy.close().

1. Yes, should be....it appears to be in the DShowPlayer DLL file.

2. I'd think it would be output format...it should autodetect input format.

3. It's likely for multiple program transport streams; 0 would mean use the 'primary' one (whatever that would be defined as).

4. Yeah, looks right....the remuxFile method in there is what'd I'd use as my guide to try it out (and looks like you already were thinking that).

[jvl711]

Thanks Jeff,

I will write a test program and load it as a runable in Sage to make sure the library is loading in Windows. I was having a hard time tracing everything that was going on in the C/C++ code, so I was not sure if/when the calls were available.

Do you have any interest or opposition if I were to attempt to add functionallity to sage to allow for tuner locking? I was thinking of adding a get/set method to CaptureDevice that would default to return not locked. I know that the sage core would need to be modified to check to see if the tuner is locked prior to utilizing the tuner for live or recorded tv.

Originally Posted by Narflex

1. Yes, should be....it appears to be in the DShowPlayer DLL file.

2. I'd think it would be output format...it should autodetect input format.

3. It's likely for multiple program transport streams; 0 would mean use the 'primary' one (whatever that would be defined as).

4. Yeah, looks right....the remuxFile method in there is what'd I'd use as my guide to try it out (and looks like you already were thinking that).