OpenDCT Java Network Encoder

[spencerw]

I've been a happy user of SageDCT for years and before that SageMCTuner (thanks Andy!) but at some point I ran into some issues with SageDCT's FFmpeg capture mode on my PC which unfortunately could not be resolved so I decided to write a pure Java network encoder which interfaces with the FFmpeg libraries via JavaCPP. This was before SageTV became open source and thus was not something many people would likely be interested in since SageTV activity had dwindled.

Now that SageTV is open source and the community is coming back I believe OpenDCT could be of use to some members of our community so I will be releasing it as an open source project in the near future. I saw recently that EnterNoEscape has begun work on a Java network encoder (http://forums.sagetv.com/forums/showthread.php?t=62538) and jvl711 created PrimeNetEncoder which is also written in Java. Perhaps a single network encoder with the features of all three encoders makes sense rather than having three different ones :-). I look forward to contributing to such an effort in any way I can.

Here are the features of OpenDCT:

- Works with InfiniTVs and Primes under Windows and Linux.
- Native remuxing via JavaCPP using FFmpeg libraries to produce clean recordings most tools can work with.
- Optimistic FFmpeg probing for faster remux startup. Starts with a small probe size and only increases it if needed (up to the default of 5MB).
- Can be configured to reserve connections to tuners at startup for quicker tuning. (Not recommended if tuners are shared.)
- Suspend/resume support in Windows via JNA.
- Suspend/resume in Linux is not yet supported so a 24x7 system is needed.
- Supports network encoder discovery.
- Runs as a service.
- Low CPU and memory overhead.
- Circular receive buffer used to reduce garbage creation.
- Large RTP receive buffers processed on high priority threads to reduce recording hiccups.
- Can be configured to avoid tuning for a list of channels such as those with DRM.
- Responds to SageTV "START" requests with immediate "OK" responses allowing SageTV to start multiple recordings more quickly.

Requirements:

- Java Service Wrapper Community Edition (32-bit only for Windows, 32-bit or 64-bit for Linux).
- Java Runtime Environment 7 or higher. Must use a 32-bit JRE under Windows due to JSW. Linux users can use 32-bit or 64-bit as selected for JSW above.
- Uses the following open source projects:

Cling 2.0.1
JavaCPP 1.0 (which uses FFmpeg)
JNA (Java Native Access) v4.2.0

If you're interested in beta testing this then keep in mind that there is no installer yet and the installation will require some effort. Roughly you'll have to do the following...

1) The OpenDCT service can work on 32-bit Linux, 64-bit Linux or 32-bit Windows.
64-bit Windows is not supported since Java Service Wrapper Community Edition is not available for 64-bit Windows.
Linux users must decide whether to go 32-bit or 64-bit and download consistently in the steps below.

2) Download and install the Java from http://java.com

3) Download and install Java Service Wrapper Community Edition from the appropriate link below:

a) 32-bit Linux: http://wrapper.tanukisoftware.com/do...-3.5.27.tar.gz
b) 64-bit Linux: http://wrapper.tanukisoftware.com/do...-3.5.27.tar.gz
c) 32-bit Window: http://wrapper.tanukisoftware.com/do...-32-3.5.27.zip

3) Extract opendct.zip (to be supplied) into a directory such as /usr/local/bin (Linux) or c:\bin (Windows).
Avoid C:\Program Files and C:\Program Files (x86). The extracted tree will have an opendct sub-directory with config and lib sub-directories.

4) The opendct/lib directory contains the following jars... Skip to step 5 if you don't need details on the jars.

Four jars below from 4thline.org's Cling v2.0.1. See http://4thline.org/projects/cling/ for details on Cling.

http://4thline.org/m2/org/fourthline...core-2.0.1.jar
http://4thline.org/m2/org/seamless/s...util-1.1.0.jar
http://4thline.org/m2/org/seamless/s...http-1.1.0.jar
http://4thline.org/m2/org/seamless/s...-xml-1.1.0.jar

Licensing details for Cling: See bottom of http://4thline.org/projects/cling/

GNU Lesser General Public License: http://www.gnu.org/licenses/lgpl-2.1.html
Common Development and Distribution License (optional): http://opensource.org/licenses/CDDL-1.0

Three jars for JavaCPP v1.0

http://central.maven.org/maven2/org/...avacpp-1.0.jar
http://central.maven.org/maven2/org/...-2.7.1-1.0.jar

One of the following depending on whether you're using 32-bit or 64-bit Linux or Windows
(only 32-bit Windows is supported by Java Service Wrapper Community Edition):

a) 32-bit Linux: http://central.maven.org/maven2/org/...-linux-x86.jar
b) 64-bit Linux: http://central.maven.org/maven2/org/...nux-x86_64.jar
c) 32-bit Windows: http://central.maven.org/maven2/org/...indows-x86.jar

Licensing details for JavaCPP:

GPLv2 with Classpath exception http://www.gnu.org/software/classpath/license.html

Two jars for Java Native Access v4.2.0

http://repo1.maven.org/maven2/net/ja.../jna-4.2.0.jar
http://repo1.maven.org/maven2/net/ja...form-4.2.0.jar

Licensing details for JNA: https://github.com/java-native-acces...master/LICENSE

LGPL, version 2.1 http://www.gnu.org/licenses/licenses.html
ASL, version 2 http://www.apache.org/licenses/LICENSE-2.0.txt

Your opendct/lib directory should look like this:

11/06/2014 04:38 PM 686,501 cling-core-2.0.1.jar
10/10/2015 11:39 AM 10,093,702 ffmpeg-2.7.1-1.0-linux-x86.jar
10/10/2015 11:40 AM 10,345,541 ffmpeg-2.7.1-1.0-linux-x86_64.jar
10/01/2015 07:15 PM 14,846,566 ffmpeg-2.7.1-1.0-windows-x86.jar
10/01/2015 07:15 PM 229,121 ffmpeg-2.7.1-1.0.jar
10/01/2015 07:15 PM 243,318 javacpp-1.0.jar
09/28/2015 01:58 AM 1,092,828 jna-4.2.0.jar
09/28/2015 01:58 AM 1,844,529 jna-platform-4.2.0.jar
10/10/2015 01:14 PM 82,986 opendct-sage-0.9.1.jar
11/06/2014 04:38 PM 21,646 seamless-http-1.1.0.jar
11/06/2014 04:38 PM 94,456 seamless-util-1.1.0.jar
11/06/2014 04:38 PM 63,142 seamless-xml-1.1.0.jar

5) Customize the opendct/config/opendct.properties file with various information about your
digital cable tuner such as the serial number and the name of the network interface used to
access it. The existing file contains documentation to help with this.

6) Edit the opendct/config/wrapper.conf file.

7) Change the value of the set.JAVA_HOME property to the full path for your Java installation to use for OpenDCT. Use forward slashes
even for Windows. The value should be something like this for Windows: c:/Program Files (x86)/Java/jre1.8.0_60

8) Change the value of the set.OPENDCT_HOME property to the full path for the opendct directory such as /usr/local/bin/opendct or c:\bin\opendct.

9) Change the value of the set.WRAPPER_HOME property to the full path for Java Service Wrapper which you installed in step 3.

10) Find the wrapper.java.classpath.8 property and set its value to the appropriate platform specific ffmpeg jar from the following list:

ffmpeg-2.7.1-1.0-linux-x86.jar
ffmpeg-2.7.1-1.0-linux-x86_64.jar
ffmpeg-2.7.1-1.0-windows-x86.jar

11) Save the wrapper.conf file.

12) Install the OpenDCT service.

For Windows this is easy. Open a command prompt as an administrator and type a command similar to this however update the paths according to your environment:

"C:\Program Files (x86)\JavaServiceWrapper\bin\wrapper" --install C:\bin\opendct\config\wrapper.conf

For Linux it appears to be a bit more difficult. Some instructions are available here:

https://wrapper.tanukisoftware.com/d...aunch-nix.html

Linux users with a network tuner such as a Prime or InfiniTV6 ETH should not set reserve-connections=true for that tuner
in opendct.properties if your system might go to sleep. This is because OpenDCT doesn't know how to detect suspend/resume
in Linux and thus won't release the connections to the tuners at suspend time and won't re-reserve the connections at resume time.
I'll work on improving this in a future Linux release.

Linux users: If you can detect suspend/resume events on your system then you may want to
see the following page for starting/stopping a service via Java Service Wrapper and issue the appropriate command(s) for these events:

https://wrapper.tanukisoftware.com/d...x.html#scripts

13) Stop the SageTV service.

14) Edit Sage.properties and search for the network_encoder_discovery property. If its value is false then set it to true as shown below:

network_encoder_discovery=true

15) Save Sage.properties

16) Start the SageTV service.

17) Start the OpenDCT service. It will appear as SageTV OpenDCT in the Service control panel app. You can also use "net start opendct" from and administrative command prompt if you prefer.

18) Your tuners should appear in the SageTV GUI. Using the GUI configure SageTV to use your new tuners and associate a channel lineup with them.

OpenDCT has been quite stable under Windows so I feel comfortable it is ready for beta testing. If you feel very comfortable with the installation steps above and have a stable system with good signals, etc. and experience with network encoder setup in sage.properties such as with SageDCT then please let me know and we can work out how to get opendct.zip to you. Thanks.

-Bill

[EnterNoEscape]

Wow, I wish I knew about this happening. I have spent an insane amount of time over the past month doing almost exactly what you just did. I even implemented the TS packet processing to locate I frames entirely in Java. I learned a lot more than I ever knew about the MPEG standards. My game plan was to start with support for all of the DCT's via UPnP and then later add other types of capture devices that don't already exist inside of SageTV. I guess I will just stop now since I really don't want to duplicate your efforts. Don't take this the wrong way, I really appreciate that someone actually released this very much needed program. I might have some good contributions once the project source is available.

Since I will likely not be posting my project because I agree that we don't need 3 different versions of roughly the same thing and you beat me to the punch. (I was just thinking yesterday that I need to set milestones for myself or I would never release anything.) I will just explain a little about what my ambitions are and basically ask if we are on the same line of thought.

While I always appreciated that other people would take a lot of their time to create these very good network encoders, they were never feature complete from SageTV's perspective. I know in the past this had to do with not knowing what SageTV was looking for when it sent certain commands. The projects often stopped at "will it record?" and would only fix things that effected that part of encoder. I wanted to create an encoder that could actually work at a fully compliant level. Namely, I wanted to do the channel scanning/preview correctly, etc.

I wanted to add the ability to plug in an analog source when the channel was not copy freely. The analog source would be another source that would be using the same tuner (like an Xbox 360 using WMC via HD-PVR) so you really can only use one or the other and this would allow you to always get the best possible quality.

Joe

[spencerw]

Hi Joe,

I apologize for not sending you a PM earlier but I was wrestling with whether or not to even release OpenDCT due to my current workload at the office for the remainder of the year. It sounds like our plans mesh well to create a more complete offering for the community. Having two people working on a common goal is much better than one. Having three or more would be great too :-).

I had thought of something a little different than your Xbox 360/HD-PVR plan because I don't have an HD-PVR :-(. I thought about creating an external tool which could be invoked on a show from the SageTV guide. It would send a C2R request to WMC to record channels with DRM that way you don't have to switch to WMC to setup recordings. The disadvantage is that it would require WMC to view it and it wouldn't show as a scheduled recording in SageTV. I think your approach is better.

I'm sure your network encoder has features OpenDCT doesn't and vice versa. I'd like to PM you with more details, hopefully later today, so we can exchange code and work on establishing an open source repository for our future work. Thanks for your time and hard work on this so far. Believe me, I know how much work it is. I look forward to working together to build a more complete encoder than either of us would have the time or energy to build alone.

-Bill

[EnterNoEscape]

It's ok; I'm over the shock now and I agree that it is best for all of us to focus on one great solution vs. multiple good solutions. I can also relate to the whole do I have the time to even support and code this situation. My wife has been less than happy about the amount of hours I have spent on this.

As for the recording alternative for DRMed channels, one of the nice things about having multiple people is that they will likely have other hardware. So together we can try to cover a larger user base.

On a related note. A while ago I wrote a rather unpolished C# program for myself that could capture the video playback from WMC. It would resize the WMC window to the desired capture resolution, then record the video and audio. The only drawbacks are the WMC window can't be hidden/covered by another application and you need a decent CPU. I was recently playing with the latest FFmpeg builds and built one myself with the latest Intel QuickSync codecs. I think that could be a way to get around the CPU usage when using high quality for high resolution captures at least for the Intel crowd. On my Ivy bridge processor I could get 4 streams going at the same time and still maintained over 60fps on bluray 1080p content. I have not spent much time on figuring out how to capture hidden windows similar to how Playon does it. I also know you can have multiple WMC windows open and playing at the same time if they are on different profiles, this makes sense since the Xbox is really just a special RDP session and has it's own profile. By having multiple profiles, you can record multiple instances of WMC simultaneously, but that's only if I get around to tackling recording a hidden window. I don't know if there are any legal ramifications to this level of circumvention, but I would assume it falls under analog capture since we are not capturing the original bytes. Also instead of sending a C2R, I was directly manipulating the WMC session. Basically I was telling it to go to live TV and tune a specific channel via Windows Media Center TCPIP Controller. It was very reliable since I could also verify that it tuned into the correct channel, the only thing I could not do was tell it what tuner to use. When you send a C2R to WMC, can you tell it what tuner to use? It doesn't look like that's an option.

I look forward to collaborating with you.

Joe

[spencerw]

Joe and I have agreed to merge our network encoders into the OpenDCT project. The merged implementation will consist primarily of his design and implementation and as such it doesn't make sense to beta test my implementation so please disregard the original post for this thread. I'll work on integrating my FFmpeg remuxing and suspend/resume support. Joe's vision for this project is far grander than mine so it's clear he will be the better person to coordinate this effort going forward. He'll post a message here when the project is uploaded to GitHub hopefully within the next week or two.

-Bill

[EnterNoEscape]

I thought I would provide an update since it's been a few weeks and even though no one other than Bill and I have been posting on this thread, I'm sure other people may have read it. We've been making good progress and a lot of the major functionality is beta ready. Raw streaming is working about as good as it gets and SageTV server communication is working correctly.

The biggest hurdle we are working on right now is FFmpeg integration. This wouldn't take long ordinarily, but we have been focused on doing it via JavaCPP (instead of STDIN, STDOUT) for a much more controlled approach.


Here are some of the designs and goals for this new network encoder:

1) OpenDCT was written from the start to be an open source universal network encoder for anyone who wants to interface a "capture device" with SageTV. The starting focus is to have the DCT functionality working as well as possible, but it is designed to be easy to expand to support many other devices in the future. The desired end result is a network encoder that is easy to extend and offers flexible functionality.

2) In OpenDCT, a capture device can be anything from a live web-stream to screen capturing to a native capture device. A capture device can even be comprised of other capture devices that are selected based on conditions. So despite the name, the project should cover a lot of ground. On the surface, OpenDCT doesn't look like anything new, but the forward thinking code is what I think will enable it to be great.

3) The method of capturing the raw bytes is separated from the method that makes that stream available to SageTV (uploadID/direct file writing). When I built this project, I designed it using a producer/consumer design. All the producer needs to focus on is collecting the data and making the bytes available to the consumer in a format the consumer can understand. All the consumer needs to focus on is processing the data (Ex. remuxing, transcoding, etc.) and sending it off directly to a file or SageTV server via an uploadID. This allows a developer to focus on what they want to accomplish and not need to worry about other parts that are often redundant between these kinds of projects. Also if for example someone writes a new or better consumer, all of the producers can immediately use it. The producers and consumers are selected using a reflection based factory and the selections are saved in a properties file. The factories have a safe default in case someone gets a name wrong in properties and the problem will show up in the logs.

4) The program core is written in Java for ease of cross-platform support and to allow it to one day be a plugin for SageTV if the ability to use a network encoder as a plugin is implemented. We will be supporting Windows and Linux. On the Linux side we will primarily support use on Ubuntu and Fedora. It will also be licensed under Apache License version 2.0 in case any parts would be helpful in the mainline open source SageTV project. I know other licenses would be compatible too, but this just keeps you from thinking about it.


Other notes:

In the future we will be adding a Jetty web-server so you can monitor and configure the capture devices via a web interface. I think this will be the most user friendly way to handle configuration vs. editing the properties file. I was going to do a normal GUI, but that would not work out for headless servers without any desktop installed.

Needless to say, this is a little ambitious. More than 50% of my time has probably been spent commenting and creating/adjusting interfaces and abstract classes so they are flexible and hopefully easy to use. So if this interests you; hang in there.

Joe

[rlvogel322]

Thanks for the update!

[stuckless]

While I don't have a need for OpenDCT, I am interesting in what you are creating. I've done a couple of proof of concept Network Tuners over the years, and I'm always interested when other people build out network tuners.

I'm looking forward to browsing the code, and possibly learning a thing or two about JavaCPP... I've used JNA a few times, so I'm curious about JavaCPP and whether or not it performs better or worse, etc. I know RoboVM team rolled their own Java to Native bridge as well instead of using JNA.

[HelenWeathers]

....and even though no one other than Bill and I have been posting on this thread, I'm sure other people may have read it.

I am very interested in the work you are doing & have been watching this thread. There will be many who can benefit from your work when it's ready.
Keep it up!

[EnterNoEscape]

We are also using JNA, but we are just using that for detecting when Windows is going into and out of standby. I ported Bill's implementation from his project into OpenDCT. When the computer goes into standby it unloads the tuners and when it comes out, it reloads them almost like the program just started for the first time. I felt this was the safest way to handle standby since it's not a requirement to create a capture device with standby in mind, but it is a requirement for them to shutdown properly when requested and this decision makes that a non-issue. We also may have some ground work for Linux standby support too.

I hate to think about wasted time, but the JavaCPP code might not even be needed in most cases if we get the uploadID route set up to be configured to do remuxing on a raw MPEG-TS stream as it comes in. To me, the tuning performance on the raw stream using uploadID is even faster than SageDCT. I think we could even optimize the tuning methods to get just a little more performance out of there (this is part of why standby detection is important). However, I'm focused on correct operation and functionality first.

[KryptoNyte]

I'm also very interested and just watching, following your progress.

[EnterNoEscape]

Thanks for the support guys. Sometimes it hard to tell if you're doing something just for yourself.

[uberpixel]

Originally Posted by KryptoNyte

I'm also very interested and just watching, following your progress.

+1

[jgsouthard]

As a frustrated user of PrimeNetEncoder (and previously SageDCT) with HDHomeRun Prime tuners, I would welcome a more robust solution. I appreciate the efforts folks have made to keep SageTV working since Google killed development, but we really haven't yet had a reliable solution for the Prime. SageDCT worked reasonably well up until it was broken by a SiliconDust firmware update, but I've been fighting "Halt detected in recording" errors and pixelation artifacts with PrimeNetEncoder ever since.

[Naylia]

Don't worry about that...definitely eager to see what you guys create!

[ptoal]

Mark me as another user who is potentially interested. Not just in using, but also contributing.

[PLUCKYHD]

Have you actually been successful in capturing WMC DRM content via an active or inactive window? I saw this discussed at some point was curious if it was still just a theory or actually have had success at doing this?

[EnterNoEscape]

Originally Posted by PLUCKYHD

Have you actually been successful in capturing WMC DRM content via an active or inactive window? I saw this discussed at some point was curious if it was still just a theory or actually have had success at doing this?

I have outside of this project with a visible (not necessarily active) window. I may eventually get to that since it's something that interests me. I was diving back into the world of C++ since it should be the most versatile way to do that without the window being visible and then remembered why I don't use the language for rapid prototyping.

At this exact moment, we are just stabilizing things for beta. I needed to do a last minute paradigm shift on where buffering happens and it looks like it has introduced some unexpected issues that I need to weed out.

We finally got most of the FFmpeg bits working reasonably well. I have a little more time than Bill to play around and a few days ago I just became much more acquainted with how he was doing FFmpeg processing in his own implementation. I definitely don't enjoy anything native from a debugging perspective. Although, using what he already wrote is certainly speeding things up.

[wayner]

Originally Posted by PLUCKYHD

Have you actually been successful in capturing WMC DRM content via an active or inactive window? I saw this discussed at some point was curious if it was still just a theory or actually have had success at doing this?

Isn't this pretty much what Playon does - captures an inactive browser window playing Netflix, or whatever?

[EnterNoEscape]

Originally Posted by wayner

Isn't this pretty much what Playon does - captures an inactive browser window playing Netflix, or whatever?

Yes. I love PlayOn, but it is closed source. If anyone can get it to record a Windows Media Center window, I will help integrate it the into this network encoder. Based on how you create custom content, I suspect it's not possible without the solution being a hack.