OpenDCT: An open source digital cable tuner.

[wit713]

Originally Posted by wit713

Discovery channel recorded successfully but had a halt on another channel.

No, I am changing sagetv.device.<unique_id>.consumer=opendct.consumer.DynamicConsumerImpl to opendct.consumer.FFmpegTransSageTVConsumerImpl for each capture device.

I found the above change on another post. I will keep everyone updated.

The above changed worked for me. I have not had a halted recording since this change.

[MacDaddy]

I'm trying to isolate the source of an image creep in my unRaid docker implementation. I'm using the dockers SageJava8, cAdvisor, LogitechMediaServer and OpenDCT. For the OpenDCT container, I've noticed a number (150+) of directories in the /tmp directory that are of the format "javacppXXXX" where XXXX is some seemingly random number. The contents of the directories appear to be the same and each has the following contents:

/tmp/javacpp929315595318257:
total 26988
drwxr-xr-x 1 root root      462 Jul  1 03:02 .
drwxrwxrwt 1 root root     5442 Aug  5 06:38 ..
-rw-r--r-- 1 root root 18617968 Jul  1 03:02 libavcodec.so.56
-rw-r--r-- 1 root root  1527880 Jul  1 03:02 libavfilter.so.5
-rw-r--r-- 1 root root  4260424 Jul  1 03:02 libavformat.so.56
-rw-r--r-- 1 root root   342048 Jul  1 03:02 libavutil.so.54
-rw-r--r-- 1 root root   584336 Jul  1 03:02 libjniavcodec.so
-rw-r--r-- 1 root root   291072 Jul  1 03:02 libjniavfilter.so
-rw-r--r-- 1 root root   513776 Jul  1 03:02 libjniavformat.so
-rw-r--r-- 1 root root   554664 Jul  1 03:02 libjniavutil.so
-rw-r--r-- 1 root root    44184 Jul  1 03:02 libjnipostproc.so
-rw-r--r-- 1 root root    49272 Jul  1 03:02 libjniswresample.so
-rw-r--r-- 1 root root    77216 Jul  1 03:02 libjniswscale.so
-rw-r--r-- 1 root root   126904 Jul  1 03:02 libpostproc.so.53
-rw-r--r-- 1 root root   103712 Jul  1 03:02 libswresample.so.1
-rw-r--r-- 1 root root   511496 Jul  1 03:02 libswscale.so.3

Is it possible these are artifacts left over when the container is restarted? Is it possible to delete the older ones without hurting anything?

I've got some more research to do on the other docker images, but this was one quick thing that caught my eye and I thought I would look for some better insights. Thanks.

[Fuzzy]

Originally Posted by MacDaddy

I'm trying to isolate the source of an image creep in my unRaid docker implementation. I'm using the dockers SageJava8, cAdvisor, LogitechMediaServer and OpenDCT. For the OpenDCT container, I've noticed a number (150+) of directories in the /tmp directory that are of the format "javacppXXXX" where XXXX is some seemingly random number. The contents of the directories appear to be the same and each has the following contents:

/tmp/javacpp929315595318257:
total 26988
drwxr-xr-x 1 root root      462 Jul  1 03:02 .
drwxrwxrwt 1 root root     5442 Aug  5 06:38 ..
-rw-r--r-- 1 root root 18617968 Jul  1 03:02 libavcodec.so.56
-rw-r--r-- 1 root root  1527880 Jul  1 03:02 libavfilter.so.5
-rw-r--r-- 1 root root  4260424 Jul  1 03:02 libavformat.so.56
-rw-r--r-- 1 root root   342048 Jul  1 03:02 libavutil.so.54
-rw-r--r-- 1 root root   584336 Jul  1 03:02 libjniavcodec.so
-rw-r--r-- 1 root root   291072 Jul  1 03:02 libjniavfilter.so
-rw-r--r-- 1 root root   513776 Jul  1 03:02 libjniavformat.so
-rw-r--r-- 1 root root   554664 Jul  1 03:02 libjniavutil.so
-rw-r--r-- 1 root root    44184 Jul  1 03:02 libjnipostproc.so
-rw-r--r-- 1 root root    49272 Jul  1 03:02 libjniswresample.so
-rw-r--r-- 1 root root    77216 Jul  1 03:02 libjniswscale.so
-rw-r--r-- 1 root root   126904 Jul  1 03:02 libpostproc.so.53
-rw-r--r-- 1 root root   103712 Jul  1 03:02 libswresample.so.1
-rw-r--r-- 1 root root   511496 Jul  1 03:02 libswscale.so.3

Is it possible these are artifacts left over when the container is restarted? Is it possible to delete the older ones without hurting anything?

I've got some more research to do on the other docker images, but this was one quick thing that caught my eye and I thought I would look for some better insights. Thanks.

I think they appear to be left behind when the service crashes out or otherwise is stopped without cleanly shutting down (I don't believe stopping the docker is cleanly shutting down the service before killing it). In my experience, if the service is actually stopped (which will close down the docker once cleanly stopped), it deletes it's last 'in use' tmp/javacpp... folder.

you can test this by running:

docker exec -ti OpenDCT service opendct stop

which will run the service stop command within the container, and cleanly shut down opendct. Because of the way the service is managed, I was unable to consistently get docker to send the SIG_TERM command to opendct to get it to cleanly stop, when the docker itself is told to stop. Once opendct itself was changed to periodicly save the properties file, it became less important, so I never dug in deeper - I hadn't realized the unclean shutdown was also leaving behind artifacts like this.

[MacDaddy]

Thanks for the quick reply. I use the nightly backup plugin so I'm sure that the docker shutdown is not clean. I'm away from my server at the moment but will try what you suggest on my return. IIRC, I think the backup plugin allows for some pre and post scripts to fire. I'm going to dig in that a little more.

[MacDaddy]

Using

docker exec -ti crazifuzzy-opendct service opendct stop

showed a clean shutdown.

I'm not sure if a natural crash or the daily backup produced all those artifacts. With over 150 instances it would roughly correlate to the number of times the daily backup routine has run.

I put this script as a custom stop script in the backup plugin.

#!/bin/bash
docker exec -ti crazifuzzy-opendct service opendct stop > /mnt/cache/appdata/Community_Applications_Scripts/stop/crazifuzzy-opendct-stop.log

Thanks again for your help and I'll let you know if this cleans it up.

[Fuzzy]

Originally Posted by MacDaddy

Using

docker exec -ti crazifuzzy-opendct service opendct stop

showed a clean shutdown.

I'm not sure if a natural crash or the daily backup produced all those artifacts. With over 150 instances it would roughly correlate to the number of times the daily backup routine has run.

I put this script as a custom stop script in the backup plugin.

#!/bin/bash
docker exec -ti crazifuzzy-opendct service opendct stop > /mnt/cache/appdata/Community_Applications_Scripts/stop/crazifuzzy-opendct-stop.log

Thanks again for your help and I'll let you know if this cleans it up.

For comparison, you can run a stop on the docker container itself, and observe that the artifacts remain. Pretty sure if you are stopping the docker every night, that's where the artifacts are coming from. Stopping it as above before the dockers are stopped should definitely reduce the number of artifacts left behind (should only be from crashes or other times the docker is stopped (like upgrades, etc).

[RangerZ]

I have been working on 2 issues, both of which have brought me back to this thread.

1 - Music Choice Channels
2 - Closed Captioning

1 - Music Choice Channels - I am running the .5.28 version. If I tune to a Music Choice Channel (RCN Provider) it takes about 15 seconds to tune, it then display "no signal" for about 8 seconds, and finally plays for about 15 seconds before it stops(?). If I record the show, I can then play it back with no issue. Limited testing and only tried playback after the recording is complete.

Log attached. relevant entries at about 17:05

2 - Closed Caption - I am not clear if I have configured this correctly, but I enabled CCextractor ( consumer.ffmpeg.ccextractor_enabled=true) and installed the LAV filters, which I set for H.264 n Video\audio.

CCExtractor Options:

consumer.ffmpeg.ccextractor_all_streams=true

This enables extracting of both CC1 and CC2. If this is off, it will output only CC1. You will likely want to leave this set to true.

consumer.ffmpeg.ccextractor_custom_options=

This can be any combination of valid CCExtractor options with the exception of -12, -1, -2, -in=, -out= and -o. Some people might want to change how roll-up captions are written for example. These options are sanitized to prevent execution of other programs, so you will not be able to do that from this option.

consumer.ffmpeg.ccextractor_enabled=false

This enables CCExtractor when set to true.

I installed the ATI mpeg decoder (atimpenc.dll) and set this for the MEG decoder, and get captions for CC1. I do notice the reference to ffmpeg above. Do I need to be using ffmpeg for h.264?

Should I be able to see captions for exsisitng recordings or only new recordings?

[KarylFStein]

Originally Posted by RangerZ

1 - Music Choice Channels - I am running the .5.28 version. If I tune to a Music Choice Channel (RCN Provider) it takes about 15 seconds to tune, it then display "no signal" for about 8 seconds, and finally plays for about 15 seconds before it stops(?). If I record the show, I can then play it back with no issue. Limited testing and only tried playback after the recording is complete.

Log attached. relevant entries at about 17:05

This is a known issue with the way the music streams are broadcast. There's some discussion in this thread about it. Some adjustments have been made to make this better.

[EnterNoEscape]

Originally Posted by RangerZ

I have been working on 2 issues, both of which have brought me back to this thread.

1 - Music Choice Channels
2 - Closed Captioning

1 - Music Choice Channels - I am running the .5.28 version. If I tune to a Music Choice Channel (RCN Provider) it takes about 15 seconds to tune, it then display "no signal" for about 8 seconds, and finally plays for about 15 seconds before it stops(?). If I record the show, I can then play it back with no issue. Limited testing and only tried playback after the recording is complete.

Log attached. relevant entries at about 17:05

2 - Closed Caption - I am not clear if I have configured this correctly, but I enabled CCextractor ( consumer.ffmpeg.ccextractor_enabled=true) and installed the LAV filters, which I set for H.264 n Video\audio.



I installed the ATI mpeg decoder (atimpenc.dll) and set this for the MEG decoder, and get captions for CC1. I do notice the reference to ffmpeg above. Do I need to be using ffmpeg for h.264?

Should I be able to see captions for exsisitng recordings or only new recordings?

Music Choice has always been a pain and making those channels come in faster would very negatively affect the consistency of other channels. The SageTV remuxer does ok with these streams, but I think the FFmpeg remuxer build into OpenDCT works a little better. You'll still have a delay, but the stream shouldn't cut off. The only exception is if you're using the Windows full client; it doesn't pace correctly with the live stream and will constantly get ahead of it causing pauses.

I believe you're using the Docker version of OpenDCT. In that case, you'll want to change over to FFmpeg by changing the lines that look like this to this:

sagetv.device.parent.<unique_id>.consumer=opendct.consumer.FFmpegTransSageTVConsumerImpl

After you make that change, .srt files will be created. CCExtractor was added so that you wouldn't lose all of the captions if you were transcoding which is why it's just an FFmpeg feature.

[RangerZ]

Originally Posted by EnterNoEscape

.......I believe you're using the Docker version of OpenDCT. In that case, you'll want to change over to FFmpeg by changing the lines that look like this to this:

sagetv.device.parent.<unique_id>.consumer=opendct.consumer.FFmpegTransSageTVConsumerImpl

I am running a Windows 7 64 bit client\server and OpenDCT .5.28. Does this still apply?

Originally Posted by EnterNoEscape

After you make that change, .srt files will be created. CCExtractor was added so that you wouldn't lose all of the captions if you were transcoding which is why it's just an FFmpeg feature.

So if I understand this, then I do not need to set the

consumer.ffmpeg.ccextractor_enabled=false

to true, and if I do it's only preserving the CC during the transcoding.

[EnterNoEscape]

Originally Posted by RangerZ

I am running a Windows 7 64 bit client\server and OpenDCT .5.28. Does this still apply?

It doesn't because you would already be using FFmpeg by default in that case.

So if I understand this, then I do not need to set the

to true, and if I do it's only preserving the CC during the transcoding.

The issue is that the CCExtractor option isn't compatible with using the SageTV media server because the media server will not allow OpenDCT to upload more than the file it said was ok. You will also need to change

consumer.ffmpeg.upload_id_enabled=true

to false to get the .srt files. Transcoding isn't required for it to work.

[RangerZ]

Making progress. I am generating .srt files.

Here is what I have in summary
LAV filter installed and set for H.264
ATI atimpenc.dll installed and set for MPEG

Stop and disable SageTV Service
Stop Open DCT (from Menu)
update C:\ProgramData\OpenDCT\config\opendct.properties
consumer.ffmpeg.upload_id_enabled=false (tried with and without)
consumer.ffmpeg.ccextractor_enabled=true
Start Open DCT (from Menu)
Enable and Start SageTV Service

I now see 2 new entries in the "Recording Details" window
1 Subtitles check box
2 Subtitle language menu

The Subtitle Language menus contains
1 - No subtitles
2 - #1 CC1 (Default)
3 - #2 CC2

Ticking the subtitles checkbox in the recordings windows is linked to the radio button in the sub-menu.

I also see a different menu on the right side "Caption and Subtitle options" which offers more configuration options than the default menu.

Not clear on what the difference is between Closed Caption and Subtitiles. The subtitle menu refers to CC.

There are some issues:

SRT Files are not deleted for live TV (recordings) when the recording is deleted.

Two files are created, CC1 and CC2. CC1 appears to grow in size, but is still small (however as of this note I do not have a full hour show rerecorded successfully with working subtitles). CC2 appears to remain a 1KB. Both files appear to have a time that is the beginning of the show, not the end.

I began testing during an episode of Dr. Who. I had issues with subtitles coming and going and not returning reliably with the subtitle checkbox toggle. I then recordered a complete episode of StartTrek Next Gen. I left the subtitle checkbox off (default) and made no changes during the recording. I went back after the show was recordered and noticed that the subtitles displayed were from the previous episode of Dr. Who. In addition there were 4 options in the subtitle menu for Star Trek, 2 each for #CC1 and #CC2. None of the other selections displayed subtitles. See attached jpg and files.

Subtitles appear to just stop being displayed. I was watching NCIS, it went to commercial which displayed subtitles, the second commercial did not display subtitles directly after the transition. The subtitles did not return when NCIS did.

Issues persist after stopping and restarting both the sage services and OpenDCT.

Subtitles are not working for MPEG programs, only H.264 files. I was getting captions for MPEG files (but not clear all files have them?).

Channel tuning appears to be slower.

If I understand this, subtitles must be generated during the recording process (not applicable to existing recordings) where as Close Captions appear to work (with no .srt files) based upon the existing files (MPEG only).

More testing required, but any advise on the MPEG config or expectations would be helpful.

[EnterNoEscape]

Originally Posted by RangerZ

Making progress. I am generating .srt files.

Here is what I have in summary
LAV filter installed and set for H.264
ATI atimpenc.dll installed and set for MPEG

Stop and disable SageTV Service
Stop Open DCT (from Menu)
update C:\ProgramData\OpenDCT\config\opendct.properties
consumer.ffmpeg.upload_id_enabled=false (tried with and without)
consumer.ffmpeg.ccextractor_enabled=true
Start Open DCT (from Menu)
Enable and Start SageTV Service

I now see 2 new entries in the "Recording Details" window
1 Subtitles check box
2 Subtitle language menu

The Subtitle Language menus contains
1 - No subtitles
2 - #1 CC1 (Default)
3 - #2 CC2

Ticking the subtitles checkbox in the recordings windows is linked to the radio button in the sub-menu.

I also see a different menu on the right side "Caption and Subtitle options" which offers more configuration options than the default menu.

Not clear on what the difference is between Closed Caption and Subtitiles. The subtitle menu refers to CC.

There are some issues:

SRT Files are not deleted for live TV (recordings) when the recording is deleted.

Two files are created, CC1 and CC2. CC1 appears to grow in size, but is still small (however as of this note I do not have a full hour show rerecorded successfully with working subtitles). CC2 appears to remain a 1KB. Both files appear to have a time that is the beginning of the show, not the end.

I began testing during an episode of Dr. Who. I had issues with subtitles coming and going and not returning reliably with the subtitle checkbox toggle. I then recordered a complete episode of StartTrek Next Gen. I left the subtitle checkbox off (default) and made no changes during the recording. I went back after the show was recordered and noticed that the subtitles displayed were from the previous episode of Dr. Who. In addition there were 4 options in the subtitle menu for Star Trek, 2 each for #CC1 and #CC2. None of the other selections displayed subtitles. See attached jpg and files.

Subtitles appear to just stop being displayed. I was watching NCIS, it went to commercial which displayed subtitles, the second commercial did not display subtitles directly after the transition. The subtitles did not return when NCIS did.

Issues persist after stopping and restarting both the sage services and OpenDCT.

Subtitles are not working for MPEG programs, only H.264 files. I was getting captions for MPEG files (but not clear all files have them?).

Channel tuning appears to be slower.

If I understand this, subtitles must be generated during the recording process (not applicable to existing recordings) where as Close Captions appear to work (with no .srt files) based upon the existing files (MPEG only).

More testing required, but any advise on the MPEG config or expectations would be helpful.

SageTV doesn't know to clean up .srt files, but you can configure the Comskip plugin to take care of that for you.

Often CC2 will remain empty because there just isn't anything there in the video.

The first thing I would always suggest is not being exactly at live or you might never see any subtitles. That's because they have to be written to disk and they are always marginally behind what SageTV considers live.

The second thing is that SageTV doesn't support multi-part files with subtitles. That means if you stop playback on a live feed, then start it again so that there is now a new file, SageTV will only read the first .srt file and not the one for the next segment of the recording. I looked at fixing this, but it started to get daunting enough that I all but gave up.

I don't know what to say about the Dr. Who subtitles showing up for TNG.

[RangerZ]

Thanks for the additional info. all this tracks with my testing.

I'll delete the srt files and reboot and try to get some clean results.

any additional testing scenarios you can suggest would be helpful.

Without setting the parameter

consumer.ffmpeg.upload_id_enabled=

to false, I was getting Close Captions for MPEG files with the ATI decoder. As this displayed CC for old recordings I assume that they CC is embedded in the file.

Do you know if this is similarly true for H.264 and it's a matter of finding the right decoder, or am I making an invalid inference?

[Tiki]

Originally Posted by RangerZ

Thanks for the additional info. all this tracks with my testing.

I'll delete the srt files and reboot and try to get some clean results.

any additional testing scenarios you can suggest would be helpful.

Without setting the parameter

consumer.ffmpeg.upload_id_enabled=

to false, I was getting Close Captions for MPEG files with the ATI decoder. As this displayed CC for old recordings I assume that they CC is embedded in the file.

Do you know if this is similarly true for H.264 and it's a matter of finding the right decoder, or am I making an invalid inference?

The difference is not the file type (mpeg vs h.264). The difference is the source and how it was captured.

If you recorded from antenna or cable, CC (closed captions) were probably embedded in the broadcast. If you used a digital tuner (HDHR, Prime, etc.) that grabs the broadcast stream in its original format WITHOUT transcoding, the cc should still be there. However, if you use a capture device that re-encodes the show in a new format (HDPVR, Colossus, etc.), the captions usually get lost in the conversion.

[RangerZ]

I have been using a HDHR-CC on RCN cable for about 3 years. I have gone from SageDCT to Primenet encoder on v7 to a clean install of OpenDCt on V9. I did have an HDHR-US capturing OTA, but am not reinstalling this due to poor signal quality on most recordings.

I reset the MPEG encoder to the ATI one and now get CC with no issues on MPEG recordings.

Thinking more since my last post, I expect that CCextractor (full version) can be used to create SRT files for existing H.264 files that I want to see, so yah the info is there. It's just not as transparent on the H.264 files. I think if I record H.264 with out interrupting it, that this config should work.

As I seem to get both, what determines if I get an H.264 or MPEG file? I have never seen a VC-1 file (that I know of).

I akso note that the recordings menu is different for H.264 and MPEG. MPEG is missing the left side subtitle entries and the enhanced right side caption options.

[Tiki]

My understanding of digital video recording is this:
Audio, video, and text (cc) are all encoded separately using different codecs.
-Audio uses codecs such as MP3, aac, AC-3, etc.
-Video uses codecs such as mpeg-1, mpeg-2, h.264, h.265, etc.
- not sure about text
These separate items then get multiplexed (muxed) together and stored in a container file. The container file acts sort of like a zip file, keeping everything together.
Common container file extensions are:
.mpg, .mkv, .ts, .mp4

Different container types support different features (for example, some support menus, playlists, multiple audio tracks, etc.).

This gets more confusing because "mpeg" can mean a video codec (mpeg1 and mpeg2) or an audio codec (MP3), or a container type. Also, there are at least two types of mpeg containers: .mpg (mpeg program stream) and .ts (mpeg transport stream).

Because the container and the codec are separate things, you can't always tell what video codec was used by looking at the file name extension. For example, a .mkv file could contain video encoder as MPEG2 or h.264. You need to use a program like media info to peer inside the container and see what's really in there.

When the US switched from analog to digital for broadcast tv, they adopted a standard called ATSC. The original version of this standard used MPEG2 for video encoding. So all broadcast tv uses this format now. There are newer revisions to the ATSC standard now that support more advanced codecs, but they have not been adopted by TV broadcasters yet (they are slow to change, since TVs need to be updated to work with any new standards).

Most cable and satellite in the US used the same standards initially. But many of them have started transitioning some or all channels to h.264. Some may even be using h.265. These newer codecs are more efficient and require less bandwidth to transmit. Because they have a closed system where they control the equipment, they can do this.

When you record with a device like the HDHR, you get the raw data in its original format. So if it was broadcast as h.264, that's what you get. Sage remuxes it and stores it in a file on your computer. By default it uses an mpeg transport stream container (.ts) for the HDHR, but you can change it to use an mpeg program stream container (.mpg) if you want by editing a property.

If you use OpenDCT to control your HDHR or Prime, it can be configured to use ffmpeg to remux or even transcode with a different codec, but if you do this, you may lose some information (such as the closed captions).

If you use a set top box (STB) with a device like the HD-PVR, the STB decodes the video and puts out an analog stream. The HDPVR grabs the analog stream and converts it to digital format. The analog stream coming out of the STB can't handle multiple separate streams, so there is no closed captioning information. If you want cc, you must use your STB remote to turn it on. Then the STB will mix them into the video and they will be on all the time (you won't be able to turn them on and off when you play back the recording). So in this case you are just capturing exactly what is coming out of your cable box: one audio stream and one video stream.

[RangerZ]

@Tiki, this explanation is GOLD, thank you.

The recording info gives me all the info, which is an MPEG2-TS container with either the MPEG2 or H.264 video, and 2 Dolby Digital audio tracks (5.1 and stereo).

When you refer to broadcaster above, not sure if you mean the station (CBS, USA) or in my case RCN. I think I am noticing virtually all USA shows are H.264, and most CBS shows are MPEG2. Current have a 2000 vintage SVU coming thru in h.264 on USA and last night a 2016 NCIS in MPEG2.

Do I care about the container (TS vs MPEG2)?

Back to the original issue, I am noticing that some time after deleting the .srt files I lost the menu options for subtitles on H.264. They did not return after a reboot. I added comskip, but I believe after this issue started.

[MacDaddy]

Originally Posted by MacDaddy

I'll let you know if this cleans it up.

Just a quick follow up. Closing OpenDCT cleanly before the backup plugin closes the docker appears to have eliminated the artifacts. Thanks again.

[gordyp]

Originally Posted by RangerZ

Back to the original issue, I am noticing that some time after deleting the .srt files I lost the menu options for subtitles on H.264. They did not return after a reboot. I added comskip, but I believe after this issue started.

When there is no .srt file, Sage does not show the Subtitle check box or the Subtitle menu options, since as far as it is concerned, there are no subtitles to display...