Has network encoding changed with V9?

[sgx2]

I've been trying to get my Linux-based (64-bit) V9 instance to be a network encoder to a Linux-based (32-bit) V7 instance. It's been a challenge as I've not found a straight-forward step-by-step.

Unfortunately, I still can't get them to talk -- so, I'm wondering if something has changed in the new code. Here's what I've done so far:

1. I configured the tuning device (an HD PVR) on the V9 encoder host.
2. I shut down both sage instances.
3. I changed the enable_encoding_server= value to true in the Sage.properties file on the V9 encoder host. I noticed that it set encoding_server_port= to 6969 and noted that.
4. I restarted V9 sage on the encoder host.
5. I copied the 76 lines representing the device I was interested in from the V9 encoder Sage.properties file to the Sage.properties file on the V7 host. I had to change the number after mmc/encoder/ to be unique on that host, but that was easy (:1020,1096s/-278648693/123456/)
6. I edited the pasted in mmc/encoders/123456/encoding_host= value to the hostname\:port of the encoder (e.g. encoder\:6969) in the Sage.properties file on the v7 host.
7. I set network_encoder_discovery= to true in the Sage.properties file on the v7 host.
8. I restarted v7 sage on that host.
9. I waited 15 minutes, then tried to add a new video source -- but the network encoder is not there :( ...

Any tips?

[Narflex]

Try it again, but exclude steps 5 & 6. Those should be done automatically for you as part of the network encoder discovery process....and that might be what is tripping things up. If you're still having issues...then check your firewall settings. And if that doesn't help; then post your log files from both SageTV instances here and we should be able to figure it out.

[sgx2]

Originally Posted by Narflex

Try it again, but exclude steps 5 & 6. Those should be done automatically for you as part of the network encoder discovery process....and that might be what is tripping things up. If you're still having issues...then check your firewall settings. And if that doesn't help; then post your log files from both SageTV instances here and we should be able to figure it out.

Thanks!

I tried writing a network encoder, but wasn't sure of all the API details. Modified a perl-based hdhomerun_prime encoder I found, got it to work (kinda, but it's not reliable), so I think I know what *should* happen.

OK, I've removed the lines as recommended, and restarted. There are no firewalls on either machine, and I can "telnet zotac 6969" (zotac being the name of the encoder host) and enter NOOP or similar. It's almost as if the discovery process is not working.

sagetv_0.txt from v9 host (encoder):
Tue 3/1 15:07:25.662 [main@735b5592] V4L: getV4LCardType /dev/null
Tue 3/1 15:07:25.670 [main@735b5592] java.lang.NullPointerException
Tue 3/1 15:07:25.673 [main@735b5592] V4L: Testing type for /dev/video1
Tue 3/1 15:07:25.675 [main@735b5592] V4L: getV4LCardType /dev/video1
Tue 3/1 15:07:25.676 [main@735b5592] V4L: getV4LInputName /dev/video1 0
Tue 3/1 15:07:25.677 [main@735b5592] V4L: getV4LInputName /dev/video1 1
Tue 3/1 15:07:25.681 [main@735b5592] V4L: getV4LInputName /dev/video1 2
Tue 3/1 15:07:25.682 [main@735b5592] V4L: getV4LInputName /dev/video1 3
Tue 3/1 15:07:25.725 [main@735b5592] detect 3
Tue 3/1 15:07:27.162 [Scheduler@75402cce] V4L: createEncoder /dev/video1
Tue 3/1 15:07:27.181 [Scheduler@75402cce] V4L: detected hdpvr
Tue 3/1 15:07:27.614 [Scheduler@75402cce] V4L: getV4LCardType /dev/video1
Tue 3/1 15:07:27.615 [Scheduler@75402cce] V4L: getV4LInputName /dev/video1 0
Tue 3/1 15:07:27.616 [Scheduler@75402cce] V4L: setInput0 0 1 1 1
Tue 3/1 15:07:27.617 [Scheduler@75402cce] V4L: setting standard to 0x3000
Tue 3/1 15:07:27.639 [Scheduler@75402cce] V4L: setting audio to input 2
Tue 3/1 15:07:27.668 [Scheduler@75402cce] V4L: updateColors0 b=128 c=128 h=128 s=128

I just restarted the v7 instance and got these lines (I think this is what you're looking for!):
Tue 3/1 15:09:03.768 [EncodingServerConn@a1260d9] Error with EncodingServer socket:java.io.EOFException
Tue 3/1 15:09:03.771 [EncodingServerConn@a1260d9] java.io.EOFException
Tue 3/1 15:09:03.773 [EncodingServerConn@a1260d9] at java.io.DataInputStream.readByte(DataInputStream.java:267)
Tue 3/1 15:09:03.774 [EncodingServerConn@a1260d9] at sage.Sage.readLineBytes(Sage.java:2105)
Tue 3/1 15:09:03.776 [EncodingServerConn@a1260d9] at sage.EncodingServer$EncodingServerConnection.run(EncodingServer.java:244)
Tue 3/1 15:09:03.777 [EncodingServerConn@a1260d9] at sage.Pooler$PooledThread.run(Pooler.java:252)
Tue 3/1 15:09:03.781 [EncodingServerConn@a1260d9] Error2 closing:java.net.SocketException: Socket closed


sagetv_0.txt from v7 host (main):

Tue 3/1 15:09:03.617 [main@1e7f8de] Error discovering servers:java.net.SocketTim
eoutException: Receive timed out
Tue 3/1 15:09:04.349 [main@1e7f8de] detect 3
Tue 3/1 15:09:08.152 [SageTV@1a515d3] ::INFO: Logging to sagex.jetty.log.JettyS
tarterLogger@c5ce37 via sagex.jetty.log.JettyStarterLogger
Tue 3/1 15:09:08.152 [SageTV@1a515d3] ::INFO: Jetty Starter plugin version 2.3.
0.14
INFO - Configured Root Logger
Tue 3/1 15:09:08.178 [SageTV@1a515d3] LOG4J: Configured Root Logger
INFO - Configured Logging for: sagex-api using file: sagex-api.log4j.properties
Tue 3/1 15:09:08.207 [SageTV@1a515d3] LOG4J: Configured Logging for: sagex-api u
sing file: sagex-api.log4j.properties
Tue 3/1 15:09:08.248 [SageTV@1a515d3] [[SageTCPServer]]: Plugin constructor of v
ersion 2.3.5
Tue 3/1 15:09:08.737 [SageTV@1a515d3] ::INFO: Starting Jetty
Tue 3/1 15:09:08.738 [SageTV@1a515d3] ::INFO: Jetty Plugin log level: INFO
Tue 3/1 15:09:08.873 [SageTV@1a515d3] ::INFO: jetty-6.1.19
Tue 3/1 15:09:08.921 [SageTV@1a515d3] ::INFO: Deploy /opt/SageTV/server/jetty/c
ontexts/apps.xml -> org.mortbay.jetty.webapp.WebAppContext@1c38a88{/apps,/opt/Sa
geTV/server/jetty/webapps/apps.war}
Tue 3/1 15:09:08.928 [SageTV@1a515d3] ::INFO: Deploy /opt/SageTV/server/jetty/c
ontexts/mediastreaming.xml -> org.mortbay.jetty.webapp.WebAppContext@ef8db9{/str
eam,/opt/SageTV/server/jetty/webapps/MediaStreaming.war}

etc...

[Narflex]

Set debug_logging=TRUE in the Sage.properties files on both systems and then generate the logs again. Sorry, I forgot to mention that before. (and make the change while SageTV is not running) And please post the complete files too.

[sgx2]

Originally Posted by Narflex

Set debug_logging=TRUE in the Sage.properties files on both systems and then generate the logs again. Sorry, I forgot to mention that before. (and make the change while SageTV is not running) And please post the complete files too.

Ah, very good. OK, the two log files are attached (zipped due to size restrictions, sorry)

OK, with the debugging on I seem to have been able to see the device on the main host, but it isn't actually working...The first time the HD PVR is accessed it works, but subsequent actions (ie. channel change) do not. No recording passes through.

[Narflex]

OK, now I see the problem. It's trying to use the uploading feature of the network encoder and apparently that was never included in the linux capture code...so when it tries to open the network path for streaming the content to the regular server, it's failing there.

There's 2 solutions to this problem:

1. Update the Linux code to include this feature (which means you need to be a software developer)

2. Set this property in the encoding server: encoding_server_uploading=false and then also ensure that the path that is being used for recording on the regular server is also accessible at the same location on the encoding server. For your case, it looks like you are using /var/media/tv for recording. So if on your regular server, you expose that as an NFS share, and then on your encoding server you mount that at /var/media/tv, and it will solve the problem.

[sgx2]

Originally Posted by Narflex

OK, now I see the problem. It's trying to use the uploading feature of the network encoder and apparently that was never included in the linux capture code...so when it tries to open the network path for streaming the content to the regular server, it's failing there.

There's 2 solutions to this problem:

1. Update the Linux code to include this feature (which means you need to be a software developer)

2. Set this property in the encoding server: encoding_server_uploading=false and then also ensure that the path that is being used for recording on the regular server is also accessible at the same location on the encoding server. For your case, it looks like you are using /var/media/tv for recording. So if on your regular server, you expose that as an NFS share, and then on your encoding server you mount that at /var/media/tv, and it will solve the problem.

Ah! That makes perfect sense. I've already set up an NFS mount for /var/media/tv from my main box to the new encoder, so I'll go ahead and shut down sage on the encoder, make that change, and start it up again. I'll report back on stability in about a day.

Thanks!

sgx2

[sgx2]

Originally Posted by Narflex

OK, now I see the problem. It's trying to use the uploading feature of the network encoder and apparently that was never included in the linux capture code...so when it tries to open the network path for streaming the content to the regular server, it's failing there.

There's 2 solutions to this problem:

1. Update the Linux code to include this feature (which means you need to be a software developer)

2. Set this property in the encoding server: encoding_server_uploading=false and then also ensure that the path that is being used for recording on the regular server is also accessible at the same location on the encoding server. For your case, it looks like you are using /var/media/tv for recording. So if on your regular server, you expose that as an NFS share, and then on your encoding server you mount that at /var/media/tv, and it will solve the problem.

From what I can tell so far, unless I have debug_logging set to true in the encoding server Sage.properties file, I get all manner of errors and the network encoding doesn't work...

I've attached sagetv_1.txt, the log from the encoder when debug_logging is set to false

EDIT ... actually, it's not working at all. File size zero. restarted everything dozens of times. it may have worked once or twice, but it's not stable. cat /dev/video1 > test.ts produces proper files. I've attached the current sagetv_0.txt file for debugging too.
Ah well. I'll have to find some other way to record -- can I import myth recordings into sage?

[Narflex]

I don't see anything wrong at all in that log file from the network encoder (the exceptions in those logs are fine). Can you post both the logs again from the network encoder and the regular server when you reproduce the problem? And there's no problem leaving debug_logging on all the time...I always do on all of my systems.

[sgx2]

Originally Posted by Narflex

I don't see anything wrong at all in that log file from the network encoder (the exceptions in those logs are fine). Can you post both the logs again from the network encoder and the regular server when you reproduce the problem? And there's no problem leaving debug_logging on all the time...I always do on all of my systems.

Great! Let's see what happens when I try again

OK, same issue -- here's the files!

[sgx2]

On a lark I installed the hd pvr on an old winXP laptop, got it working with a sage v9 install. connected it to the linux based v7 host as a network encoder -- and the incoming file sizes are all zero again.

logs attached.

I'm noticing a lot of this kind of exception -- on both encoder platforms -- could this be the culprit?

Sat 3/5 14:28:09.032 [EncodingServerConn@11c59d5] Error with EncodingServer socket:java.net.SocketTimeoutException: Read timed out
Sat 3/5 14:28:09.033 [EncodingServerConn@11c59d5] java.net.SocketTimeoutException: Read timed out
Sat 3/5 14:28:09.034 [EncodingServerConn@11c59d5] at java.net.SocketInputStream.socketRead0(Native Method)
Sat 3/5 14:28:09.034 [EncodingServerConn@11c59d5] at java.net.SocketInputStream.read(Unknown Source)
Sat 3/5 14:28:09.034 [EncodingServerConn@11c59d5] at java.net.SocketInputStream.read(Unknown Source)
Sat 3/5 14:28:09.034 [EncodingServerConn@11c59d5] at java.io.BufferedInputStream.fill(Unknown Source)
Sat 3/5 14:28:09.034 [EncodingServerConn@11c59d5] at java.io.BufferedInputStream.read(Unknown Source)
Sat 3/5 14:28:09.034 [EncodingServerConn@11c59d5] at java.io.DataInputStream.readByte(Unknown Source)
Sat 3/5 14:28:09.034 [EncodingServerConn@11c59d5] at sage.Sage.readLineBytes(Sage.java:2105)
Sat 3/5 14:28:09.034 [EncodingServerConn@11c59d5] at sage.EncodingServer$EncodingServerConnection.run(EncodingServer.java:244)
Sat 3/5 14:28:09.034 [EncodingServerConn@11c59d5] at sage.Pooler$PooledThread.run(Pooler.java:252)

[Narflex]

That exception isn't a problem...it's a normal part of the timing out of the connection (which it always retries). I really can't tell what is wrong at all. Maybe we introduced some network encoder incompatibility between V7 and V9...but I figured that would stand out more if we did that rather than manifest itself as an issue such as this.

[sgx2]

Thanks for looking!

I will try with another windows machine. It also has a firewire card in it that I can try as a capture device, if that works well in v9...

sgx2

[BobPhoenix]

I can't test again until this weekend.

I had the following problem with network encoding in V9

Windows 7 network encoder 9.0.4.232
Linux Mint 17.3 server 9.0.4.232 compiled with update for "smbmount" fix

I could setup the tuners from the Win7 network encoder and scan for available channels - but it wouldn't record successfully just got errors. It did record from HDHRs connected directly to the Linux Mint server just not from the tuners (AVerMedia Duet's) installed in the Win7 network encoder.

I kind of eliminated my need to go to 64 bit linux anyway by deleting my existing Wiz.bin and recreating all of my favorites again. Lost watched data but I went from a boot time of 3 hours to minutes with the new Wiz.bin. In addition to the slow start up my java heap was hovering between 700-1038 used all the time with the old Wiz.bin.

[sgx2]

Tried with solid Win7 box - Sage v9 as network encoder with HDPVR capture device. Again the file is created on the v7 linux system with a file size of zero.

I can confirm that recording directly on the v9 box works okay.

I think an interesting test would be to map an smb share from the linux box on the windows host and have the recorded video stored there. I seem to have forgotten the setting I'll need in the Sage.properties file on my Windows host to do this, however...

[sgx2]

Originally Posted by sgx2

Tried with solid Win7 box - Sage v9 as network encoder with HDPVR capture device. Again the file is created on the v7 linux system with a file size of zero.

I can confirm that recording directly on the v9 box works okay.

I think an interesting test would be to map an smb share from the linux box on the windows host and have the recorded video stored there. I seem to have forgotten the setting I'll need in the Sage.properties file on my Windows host to do this, however...

I just remembered that it is "encoding_server_uploading=false" ... does this go on the encoder or main host? I just put it on the encoder and tested -- the zero-size file still gets created on the main host, nothing gets created on the encoding host...hmmm is this insight into what's happening?

Is there a "known good" procedure for mapping an smb share from a linux box on a windows encoder as the shared directory?

[jjyec]

I've been playing around with network encoders these days and ran into the same issue. I thought it was related to my UNRAID setup but it looks like it isn't.

I got it working between two linux machines, and for a short while I got it working between a Linux server and a windows network encoder but for some reason I haven't been able to recreate that one.

I *think* it's related to SAMBA. From what I could see, when the server starts the recording, it looks like it creates the recoding file in the recording directory, then it signals the network encoder to start recording. I believe the server/samba holds a lock on the file and the network encoder can't write to it. As a test I initiated a recording, noticed the file size stayed at 0. I then killed both sage server and network recorder. For some period of time, I couldn't write anything to this file but suddenly the "lock" was released and I was able to. Permissions looked good on both the server and the network encoder.

With an NFS share, I didn't see this problem.

[jpwegas]

Originally Posted by sgx2

I tried writing a network encoder, but wasn't sure of all the API details. Modified a perl-based hdhomerun_prime encoder I found, got it to work (kinda, but it's not reliable), so I think I know what *should* happen.

sgx2,
Were you trying to use https://github.com/jwittkoski/prime_encoder ?

If so, please let me know (or open a Github issue) where you were experiencing reliability issues. It has been rock solid for me using the HDHR Prime and if there are reliability issues for others I would like to try and improve the code.

Thanks,

--John

[sgx2]

Originally Posted by jpwegas

sgx2,
Were you trying to use https://github.com/jwittkoski/prime_encoder ?

If so, please let me know (or open a Github issue) where you were experiencing reliability issues. It has been rock solid for me using the HDHR Prime and if there are reliability issues for others I would like to try and improve the code.

Thanks,

--John

Hi there -- I think I looked at your prime_encoder as a reference, as the instability was in my implementation. Never fear.

You know, if only my Cable provider allowed us to use cablecard, I'd switch to using an HDHRprime before you could blink.