Network encoder for DBOX 2 DVB set top boxes...

[potpiejimmy]

Hi!

This is a Sage network encoder for the widely-used DBOX 2 DVB-C/DVB-S
set top box.

It uses "ggrab" to digitally grab the video/audio data from the box by
using the streaming capabilities over the ethernet. I called it
"grabrecorder" (maybe it will work with different "grabbing" programs
(and maybe even other STBs) sometime in the future).

It automatically handles the tuning of channels, so you don't need
to configure a separate tuner plugin or anything.

Of course, you need to have a linux image on your box to use the
streaming features (Neutrino, not sure about Enigma).

For now, grabrecorder only records one audio-PID, because Sage doesn't
play mpegs correctly if it contains multiple audio tracks. It could
record all audio PIDs including AC3, of course, but I disabled it
because Sage didn't play it correctly, don't know why... I still need to
figure that out...
edit: as of Sage version 4.1.6, playing files with multiple audio tracks works fine.
Use -a to record all audio pids.

This is a first alpha version without any guarantees :-) But I'd
appreciate your feedback on this...

This is how you set it up:

Download the zip and extract it to a folder. Make sure you have your
system configured to resolve the host name "dbox" to your dbox's IP
address.
edit: Or use the -dh command line option to specify the
IP address.

Start grabrecorder.exe.

It should dump the channel list of your dbox to the console.

Shutdown sage & sage service and open the Sage.properties file. Add the
following lines after the last line that starts with mmc/encoders/...

mmc/encoders/12345/1/0/available_channels=
mmc/encoders/12345/1/0/brightness=130
mmc/encoders/12345/1/0/contrast=135
mmc/encoders/12345/1/0/device_name=
mmc/encoders/12345/1/0/hue=125
mmc/encoders/12345/1/0/last_channel=101
mmc/encoders/12345/1/0/provider_id=601
mmc/encoders/12345/1/0/saturation=105
mmc/encoders/12345/1/0/sharpness=170
mmc/encoders/12345/1/0/tuning_mode=
mmc/encoders/12345/1/0/tuning_plugin=
mmc/encoders/12345/1/0/tuning_plugin_port=0
mmc/encoders/12345/1/0/video_crossbar_index=0
mmc/encoders/12345/1/0/video_crossbar_type=6
mmc/encoders/12345/audio_capture_device_name=
mmc/encoders/12345/audio_capture_device_num=0
mmc/encoders/12345/audio_processor=
mmc/encoders/12345/capture_config=
mmc/encoders/12345/default_device_quality=Great
mmc/encoders/12345/device_class=
mmc/encoders/12345/encoder_merit=0
mmc/encoders/12345/encoding_host=127.0.0.1\:6970
mmc/encoders/12345/last_cross_index=0
mmc/encoders/12345/last_cross_type=6
mmc/encoders/12345/live_audio_input=
mmc/encoders/12345/multicast_host=
mmc/encoders/12345/never_stop_encoding=false
mmc/encoders/12345/video_capture_device_name=DBOX II GrabRecorder
mmc/encoders/12345/video_capture_device_num=0
mmc/encoders/12345/video_encoding_params=Great
mmc/encoders/12345/video_processor=

The number 12345 can be replaced by any number, but it must be unique,
so if it is already used, choose something different.

Start Sage and configure your new video source. Assign the channel
numbers according to your dbox's channel list.

Good luck!

/potpiejimmy

******* edit 2006/01/20 *******

Version 0.2

- use modified ggrab_mod to avoid 99% cpu usage problem (no reading from stdin)
- interrupt ggrab_mod using CTRL+C event instead of piped "q\r\n".

***************************

******* edit 2006/01/24 *******

Version 0.3

- fixed ggrab bug for audio grabbing
- fixed ggrab Pufferfehler bug
ggrab should work for all channels now :-)
- fixed bug grabrecorder terminating itself in some cases

***************************

******* edit 2006/01/25 *******

Version 0.4

- mod ggrab to support audio packet up to 16384 instead of 9000
- changed ggrab back to toggle sectionsd at startup
- removed sectionsd toggle at shutdown because it always produced a core dump
- added command line switches to grabrecorder for dbox host/ip and grabrecorder listening port
- added command line switch to record all available audio pids
- added command line switch to selectively record preferred pids by name

***************************

******* edit 2006/01/26 *******

Version 0.5

- start ggrab_mod with "above normal" priority
- added command line switch to lock dbox remote while recording

***************************

******* edit 2006/01/28 *******

Version 0.6

- do not use UDP streaming for ggrab by default
- optionally enable UDP streaming with command line option -udp

***************************

******* edit 2006/02/23 *******

Version 0.7

- support udrec TS streaming *)

*) Notes:
Specify -udp for udrec TS streaming!
Make sure your dbox is in SPTS mode!
Make sure you have Sage version 4.1.7 or later!
Use -patpmt to record PAT and PMT pids to make the resulting files complete and "VLC-playable" - Sage, however, may have problems with the PAT/PMT of some programs/channels. In this case, drop the -patpmt option. This will result in time-consuming file scans by sage, channel-surfing impossible, but recording should work.

edit 2006/02/24:
Version 0.7.1

- bugfix for spaces in filenames

***************************

******* edit 2006/03/27 *******

Version 0.9

- automatically switch box to SPTS mode while recording using udrec (-udp)
- new command line option -sr switches box to record mode while recording
- new command line option -np to start grabbing process in normal priority class instead of above-normal priority class

Note: It is recommended to have your box in PES mode by default and let grabrecorder switch the AVIA mode to SPTS only during recording. I would also recommend to use the -sr option so that the box is switched to record mode when recording (avoids accidentally switching to another channel on a different bouquet and you can see the dbox is recording on its OSD and LCD...). Hope you like the new version

edit 2007/05/13:
As of Sage 6.1, playing udrec-recorded ts files with pat and pmt works fine, so I would recommend using grabrecorder with the following options:

grabrecorder -udp -a -sr -patpmt

***************************
[Source: http://www.potpiejimmy.de/grabrecorder-0.9-src.zip]

[Alfiegerner]

I'd really like to try this

Make sure you have your
system configured to resolve the host name "dbox" to your dbox's IP
address.

Could you point in the right direction of how do this? I've had a look trhough my router config and can't see a way to make a name resolve to an ip address. Do I need to set up a dns server?

[potpiejimmy]

Originally Posted by Alfiegerner

Could you point in the right direction of how do this? I've had a look trhough my router config and can't see a way to make a name resolve to an ip address. Do I need to set up a dns server?

Simply add a line like this:

192.168.0.202 dbox

to your c:\WINDOWS\system32\drivers\etc\hosts file.

[Alfiegerner]

Okay thanks got it up and running now

It works for me on most channels but ggrab doesn't seem able to stream from the dbox for certain channels. I've had this problem before with ggrab don't know what's wrong with it on these problem channels.

If I manually specify the ggrab options like so:

ggrab -host 192.168.1.110 -s 0 -e "" -p 2074 2073 -o "D:\Tv\TestFile.mpg"

I get the following output:

ptr:19178 lIn:19022 lOut:13672 len:200000
Pufferfehler SearchStreamID
m_fill_video: timeout waiting with all buffers filled

After somewhere between 5 and 10 seconds ggrab goes to 90% plus cpu and hangs at that.

I recorded a 10 meg file off one of these problem channels via a direct file recording on a shared pc folder to see if you can see what the problem is with it: http://www.megaupload.com/?d=5J6D93DE

As I have had problems using ggrab before I have toyed with the idea of using direct file recording as a means of getting the dbox integrated with sage. Do you think this would be a feasible path to follow? I was thinking of changing channels, setting and stopping recordings through the yweb http interface.

[potpiejimmy]

Originally Posted by Alfiegerner

As I have had problems using ggrab before I have toyed with the idea of using direct file recording as a means of getting the dbox integrated with sage. Do you think this would be a feasible path to follow?

I was thinking about that, too... but there are two problems to solve and I just don't know how (maybe you have an idea):

1) With direct file recording, the file is opened by the dbox, but the recorded must be redirected to a file with a specific name. The file names are handled by Sage and are passed on to the network encoder which then must record to the given file name. Gotta find a way to assign a file name to each direct recording command...

2) Sage doesn't play the files correctly if they contain multiple audio tracks. Is there a way to specify a single audio-pid for direct-recording?


As for your ggrab Pufferfehler, I don't know what the problem is, but I will checkout the newest ggrab sources from the cvs and compile it. (Or have you already done this?) The problem is in the cbuffer.cpp and there is a log entry about a larger buffer, don't know how old the ggrab.exe is that I bundled with my zip.file. Maybe I can also increase some buffer sizes... will see...

[Alfiegerner]

Thanks for clarifying that for me. Those are tricky things to get around for direct file recording. When I get a chance I'll see if i can fathom anything from the tuxbox code.

I'm haven't tried compiling the ggrab source myself. I've just downloaded it now and will have a look at it - the latest version I found is from 2003 though from: http://www.menzebach.de/ggrab/

I'll have a tinker with it but I'm a novice at C/C++ so it may take me a while.

Are you familiar with udrec? I know from previous attempts at streaming from my dbox that for some reason udrec works where ggrab fails. Is it feasible to have udrec do the streaming rather than ggrab?

[potpiejimmy]

Originally Posted by Alfiegerner

Are you familiar with udrec? I know from previous attempts at streaming from my dbox that for some reason udrec works where ggrab fails. Is it feasible to have udrec do the streaming rather than ggrab?

Actually udrec was my first choice. It would be great to use for streaming to TS files. I had it running and the result was Sage hanging using 99% cpu time. I took a while for me to find out that it's the missing PMT information in the TS files that Sage (and other media players, VLC for instance) didn't like. I haven't found a solution for it yet, and since udrec also doesn't multiplex mpegs, I had to switch to ggrab...

[Alfiegerner]

Ah, okay - that sounds like a tricky thing to discover. Thanks for letting me know. Back to working out what ggrab doesn't like ...

[potpiejimmy]

Seems 0.22a is the newest version, so recompiling from current CVS sources won't help, I guess.

Also had ggrab hang at 99% cpu usage today... it happened somewhere in the middle of a recording (it seems)... Going to have a closer look at the ggrab source code...

[Alfiegerner]

I've just dug up some investigations from the last time I looked at ggrab. It looks like its related to problems processing the audio stream. If i try and just grab video from one of my problem channels it works fine :

ggrab -host 192.168.1.110 -s 0 -e "" -p 081A -o "D:\Tv Project\temp.mpg"

Doing a scout about on some english forums I found this quote :"Some people have reported similar recording issues with channels that use high or low bitrates for audio."

Will have a look at the code later.

[potpiejimmy]

I haven't solved the Pufferfehler issue yet, but I fixed the 99% cpu usage problem by compiling and using my own ggrab_mod.exe that doesn't create a pthread to read from stdin. So at least ggrab exits when an error occurs.

You can find the new version 0.2 in my original posting.

I've also found a channel that doesn't record at all and always produces the streaming error. It is like you said related to the audio data, recording only the video pid works fine. Workin' on it....

[potpiejimmy]

Found and fixed the ggrab bug. Recording works fine now for ALL channels

[Alfiegerner]

WOW!!!!!

Tested (briefly) on all my problem channels and now recording perfectly.

Well done mate! This issue had been around so long I thought it might be unfixable. Will do some more testing now but in the meantime:

[Alfiegerner]

What can I say? so far - working absolutely flawlessly. i can't tell you how excited I am with this

Ready for the first feature request? I'm thinking about getting another dbox now. Ideally I'm thinking about running two grabrecorder.exe's and specifing which dbox address they stream from and what port each should listen on. What do you think?

Not that I mean to sound ungrateful - just getting large and greedy eyes

edit: Just ran into a couple of problems:

changing channels just now seemed to cause my dbox to crash. Have you ever had this? It might just be flaw in the image I'm using. I've heard that the neutrino nhttpd server can be a little flakey though. I'll see if i can replicate it on this and other images - even better, I'll find an image that I can't replicate it with .

Also, have you ever seen an audio-sych problem using grabrecorder? Now and then I get the audio out of time with the video - tried it through other media players and still seen the audio delay so it looks like a ggrab issue.

I'll try and get some more info for you tomorrow on these. Great work to get us this far!

[potpiejimmy]

Originally Posted by Alfiegerner

Not that I mean to sound ungrateful - just getting large and greedy eyes

No problem, I was going to implement a couple of command line switches including host names and ports anyway. Just wanted to fix that ggrab thing first...

Originally Posted by Alfiegerner

edit: Just ran into a couple of problems:

Aah, it just occured to me I also changed some other lines of codes that I shouldn't have tampered with. I guess this could explain the instability when changing channels. Need to change that back to the original state...

Have you seen the audio-sync problem before version 0.3? Maybe it's also caused by my last changes. I haven't had a sync problem with my dbox (yet)....

[marneb17]

Aloha,
just wanted to let you know, that I am also interested. Right now my dbox2 is connected via svideo to one of my pvr150 and I created a batch file for the exetuner plugin, so that a http request is transmitted to the box for channel change.
The image I use automatically selects an english audio track, if one is transmitted. I would use your plugin, but then I could only record the first audio track, right?

What happens if all the apids are recorded? Maybe it's just something small. Because that, of course would be the perfect solution ;-)

Marneb

[Alfiegerner]

Originally Posted by potpiejimmy

Have you seen the audio-sync problem before version 0.3? Maybe it's also caused by my last changes. I haven't had a sync problem with my dbox (yet)....

That's good to know - I'll have look tonight. I'm also going to to load the most stable image I can find. Any recommendations on an image?

[Alfiegerner]

Suggestion for you: can you set ggrab_mod to run in above normal or maybe even high priority? Occasionally I'm getting pixelated smearing in the image when other processes are running. If I increase the running ggrab_mod priority this fixes it.

[potpiejimmy]

Originally Posted by Alfiegerner

Ready for the first feature request? I'm thinking about getting another dbox now. Ideally I'm thinking about running two grabrecorder.exe's and specifing which dbox address they stream from and what port each should listen on.

You can specify the dbox host name/ip address with the -dh option and the listening port for grabrecorder with the -p option.

Originally Posted by Alfiegerner

changing channels just now seemed to cause my dbox to crash. Have you ever had this? It might just be flaw in the image I'm using. I've heard that the neutrino nhttpd server can be a little flakey though. I'll see if i can replicate it on this and other images - even better, I'll find an image that I can't replicate it with .

Please check if it is fixed now with version 0.4.

[potpiejimmy]

Originally Posted by marneb17

The image I use automatically selects an english audio track, if one is transmitted. I would use your plugin, but then I could only record the first audio track, right?

What happens if all the apids are recorded? Maybe it's just something small. Because that, of course would be the perfect solution ;-)

Hi Marneb17,

I added a command line switch to grabrecorder that allows recording all available audio PIDs. I haven't found an audio-decoder (tried all kinds of decoders in Detailed Setup) that plays the files correctly with multiple audio tracks. Only the VideoLan player plays the files and offers the selection of the different tracks. Other players have problems, too, it seems they think there's only one audio track available and the audio is distorted (turn down the volume before you try to play a file ) The option is -a.

If you don't use -a to record all pids, you may now specify an audio-preference list of "PID"-names by using the -ap command line option. You can say, for example, -ap "englisch,English,dolby" and it will prefer the audio PID that is contained in the list.