Android MiniClient - Subtitle + CC support

[HelenWeathers]

Originally Posted by stuckless

.....Out of curiousity, how does one go about enabling CC?

While watching a recording, bring up the on screen menu (you can use the Microphone button on the Shield) and one of the choices is for CC.

[TechBill]

Originally Posted by stuckless

Thanks Bill, I haven't done any work, yet, on enabling CC in the MiniCient. Unfortunately it's not a trivial task Out of curiousity, how does one go about enabling CC?

Playback a recorded show and bring up the menu, you will see two caption options on it. One for Captions On / Off and second one for Caption font size.

To enable Closed Caption - change Captions from Off to On.

Bill

[TechBill]

Any progress on this?

I am hoping to be able to play my HDHomeRun recorded shows with CC enabled on mini-client some day

Bill

[stuckless]

Originally Posted by TechBill

Any progress on this?

I am hoping to be able to play my HDHomeRun recorded shows with CC enabled on mini-client some day

Bill

Sorry Bill, no new progress.

[TechBill]

Just popping in to see if any new progress on this yet?

Right now we can only playback DirectTV recording since the boxes does the decoding and burn the CC into the video itself during recording.

The HDHomerun does not decode and depends on the SageTV to do the decoding when play back so all of our OTA shows have to be watched on HD300 if we want to watch it with Closed Captioning.

Bill

[Fuzzy]

Originally Posted by TechBill

Just popping in to see if any new progress on this yet?

Right now we can only playback DirectTV recording since the boxes does the decoding and burn the CC into the video itself during recording.

The HDHomerun does not decode and depends on the SageTV to do the decoding when play back so all of our OTA shows have to be watched on HD300 if we want to watch it with Closed Captioning.

Bill

What capture device are you using for your OTA tuner? I'm wondering if it would be possible to extract the CC out to .srt files as it is captured. OpenDCT has the option of running the stream through ffmpeg, so parameters might be added to do the extraction there.

[EnterNoEscape]

Originally Posted by Fuzzy

What capture device are you using for your OTA tuner? I'm wondering if it would be possible to extract the CC out to .srt files as it is captured. OpenDCT has the option of running the stream through ffmpeg, so parameters might be added to do the extraction there.

The binaries that I compiled of CCExtractor for OpenDCT could easily be ported to a plugin like the Comskip plugin (and I really think they should, I'm a little lacking in experience here). OpenDCT uses CCExtractor with FFmpeg because if you do live transcoding, you'll lose the captions when the video is converted. OpenDCT actually sends a remuxed copy of the original video directly to CCExtractor while it's writing the transcoded video to storage. Both copies are fully synced up, so the result is that the captions are preserved as subtitles (.srt).

[TechBill]

Originally Posted by Fuzzy

What capture device are you using for your OTA tuner? I'm wondering if it would be possible to extract the CC out to .srt files as it is captured. OpenDCT has the option of running the stream through ffmpeg, so parameters might be added to do the extraction there.

Originally Posted by EnterNoEscape

The binaries that I compiled of CCExtractor for OpenDCT could easily be ported to a plugin like the Comskip plugin (and I really think they should, I'm a little lacking in experience here). OpenDCT uses CCExtractor with FFmpeg because if you do live transcoding, you'll lose the captions when the video is converted. OpenDCT actually sends a remuxed copy of the original video directly to CCExtractor while it's writing the transcoded video to storage. Both copies are fully synced up, so the result is that the captions are preserved as subtitles (.srt).

This is the OTA tuner I am using

https://www.amazon.com/SiliconDust-H...ords=hdhomerun


CCextractor probably will work and I have use it before on other OTA tuners but there one thing I don't like about CCextractor.

CC extractor convert CC to subtitle mean it become a subtitle which is always display at the bottom of the screen with either white or yellow fonts with no border backgrounds.

If a snowy scene come up on video then the white subtitle become nearly invisible and impossible to read same with yellow if there something yellowish in the video which also mask it out.

Another reason I don't like subtitle is the placement that it always at the bottom of the screen. CC is displayed at the bottom of the screen most of the time but if there something that is displayed on the bottom of the screen by the TV station (person's name, scorecards, weather alert, school closing etc) then CC jump up to the top of the screen temporary until the announcement at the bottom of the screen disappears then the CC will resume back to the bottom of the screen. With subtitle, I have to pause, rewind, disable subtitle, play, pause, enable subtitle, play just to be able to read the announcement at the bottom of the screen.

My family always prefer the tuner or device doing the decoding before it recorded to a file, reason for this is that the original CC is preserved and it become "Open Captioned" meaning you cannot turn it off/on since it burn and become part of the video overlay and we can stream it over the internet without worrying about which video player have a built in CC decoder to watch it. This is what we do with DirectTV boxes, enable the CC decoder on it then recorded both video and CC on it to a file allowing us to play it back on any video player even if it doesn't have CC or subtitle support built in.

We did contact HDHomeRun asking them if they would every consider adding a decoder in their devices but they said TV the one that does the decoding which is true but it would been cool if OTA tuner manufacture would add decoder on their hardwares.

Same thing with DVD players, when the progressive DVD players start coming out and CC will not pass thur a HDMI. It was a big loss for the Deaf culture because we had to stay using a composite cables to watch our movies in lesser quality until some of the DVD manufacturer like Sony start adding CC decoder in their hardware which decode the CC and overlay it on the movie before it sent over the HDMI cable preserving both CC and HD video.

Mplayer can decode CC without using CCextractor but not sure about FFmpeg. I have used Mplayer to watch my recording that had CC embedded into it back in the days since there was no other video player which could playback CC then later VLC and other player started addeding CC support to their players but it not always reliable and sometime distort the CC.

Bill

[TechBill]

I would like to add one more thing,

If using CCextractor is only option to get CC on our recording, we would be okay using CCextractor on our recording if we could somehow overlay it on the video before it recorded to a file which I been trying to do for years. And also added a black border background behind the texts.

Anything that will allow us to enjoy our recordings, we will accept it no matter what as long we can read what being said in the recordings.

Thank you for all your effort into this. I really hope we find a way to get CC or subtitle to work on android miniclient from the OTA tuners. I use the miniclient with my DirectTV boxes recordings but I use the HD300 with my HDHomeRun OTA recordings.

Bill

[EnterNoEscape]

I presume that the lack of the black background could be fixed via the STV. Considering that the subtitles worked without stuckless doing anything special, I have to assume the server is telling the miniclient to just overlay them, just like when you see the timeline. Does anyone know much about this?

[stuckless]

Originally Posted by EnterNoEscape

I presume that the lack of the black background could be fixed via the STV. Considering that the subtitles worked without stuckless doing anything special, I have to assume the server is telling the miniclient to just overlay them, just like when you see the timeline. Does anyone know much about this?

I do know for SRT files, there is nothing I need to do... but for embedded titles, I'd actually have to parse them from the stream, and then use an API to tell sagetv what the captions are. I haven't had time to actually look into this, and when I do, I'll have to do in the native player implementation. I know that ExoPlayer supports parsing the subtitles, but, I think they are only supporting Webttv (sor something like that).

It would be cool if the server could optionally parse the subtitles on the fly as a client is playing the file. Right now, sagetv it just reading the files and sending it to the client, but if could spawn a separate thread and "decode" the subtitles as well, and then feed it to the subtitle handler, then the miniclient(s) would not need to know how to handle it. (Maybe the thread would simply decode and create the .srt files, on the fly, and then that part of the code handle the viewing, since it does it already)

[EnterNoEscape]

Originally Posted by stuckless

I do know for SRT files, there is nothing I need to do... but for embedded titles, I'd actually have to parse them from the stream, and then use an API to tell sagetv what the captions are. I haven't had time to actually look into this, and when I do, I'll have to do in the native player implementation. I know that ExoPlayer supports parsing the subtitles, but, I think they are only supporting Webttv (sor something like that).

It would be cool if the server could optionally parse the subtitles on the fly as a client is playing the file. Right now, sagetv it just reading the files and sending it to the client, but if could spawn a separate thread and "decode" the subtitles as well, and then feed it to the subtitle handler, then the miniclient(s) would not need to know how to handle it. (Maybe the thread would simply decode and create the .srt files, on the fly, and then that part of the code handle the viewing, since it does it already)

As was indirectly pointed out, there's more detail in close captions than the .srt format captures. Such as the actual positions of the captions. CCExtractor has better formats that it can export. It even has it's own binary format that basically has 100% of the original detail. It can also actually generate .xml + .png files for display with the correct offsets that should be used (which was one of Bill's complaints). .srt just happens to be something SageTV already supports.

[Narflex]

Originally Posted by stuckless

It would be cool if the server could optionally parse the subtitles on the fly as a client is playing the file. Right now, sagetv it just reading the files and sending it to the client, but if could spawn a separate thread and "decode" the subtitles as well, and then feed it to the subtitle handler, then the miniclient(s) would not need to know how to handle it. (Maybe the thread would simply decode and create the .srt files, on the fly, and then that part of the code handle the viewing, since it does it already)

I'm pretty sure this would work if somebody tried to do it. The main thing that's tricky is to keep them in sync..when the server parses the CC data may be quite a few seconds off from when the corresponding video frame is actually displayed. However, I wrote some CC injection stuff at one point which would take all of the CC byte data from the recording (after it was already extracted by another tool), load it into memory, and then would inject it into the system with proper timing (it was embedded inside MiniPlayer as another thread and used the timing information from MiniPlayer to determine what CC byte pair was up for decoding right then and it would do this continually during playback). That same method could be adapted to take input from the FastMpeg2Reader instead (which has an MPEG parser in it and handles all the video data coming from disk that's sent to the miniclient; but it doesn't go down to the level of reading the user_data header where the CC info is stored...but it would not be all that difficult to extend it to doing that although it would require disabling many of the optimizations in it).

So this could all be done at the Java layer inside of SageTV and it would probably work fine. If anybody is interested in working on it as a dev, feel free to ping me and I can give you guidance on implementing it. Although I have this funny feeling there's a reason I didn't do this before...maybe because it would only work for the extenders/placeshifters and not for other types of playback...or maybe there's other issues I'm forgetting about...but it should be possible to deal with anything that comes up.

[MattHelm]

Maybe the .srt could be converted to Advanced SubStation Alpha and background added automatically? I do not know how to do this, but am aware of this format from trying to clean up some subtitles in an anime. There is a free open source program called Aegisub that supports editing, and maybe it supports some automation??? Wish I knew more to help with this.

If no one else has time to look into it, I'll try and look this weekend. I know this will add more complexity, but if the computer does it all in the background, maybe it'll help you.

[Fuzzy]

Originally Posted by Narflex

I'm pretty sure this would work if somebody tried to do it. The main thing that's tricky is to keep them in sync..when the server parses the CC data may be quite a few seconds off from when the corresponding video frame is actually displayed. However, I wrote some CC injection stuff at one point which would take all of the CC byte data from the recording (after it was already extracted by another tool), load it into memory, and then would inject it into the system with proper timing (it was embedded inside MiniPlayer as another thread and used the timing information from MiniPlayer to determine what CC byte pair was up for decoding right then and it would do this continually during playback). That same method could be adapted to take input from the FastMpeg2Reader instead (which has an MPEG parser in it and handles all the video data coming from disk that's sent to the miniclient; but it doesn't go down to the level of reading the user_data header where the CC info is stored...but it would not be all that difficult to extend it to doing that although it would require disabling many of the optimizations in it).

So this could all be done at the Java layer inside of SageTV and it would probably work fine. If anybody is interested in working on it as a dev, feel free to ping me and I can give you guidance on implementing it. Although I have this funny feeling there's a reason I didn't do this before...maybe because it would only work for the extenders/placeshifters and not for other types of playback...or maybe there's other issues I'm forgetting about...but it should be possible to deal with anything that comes up.

Seems to me the BEST final solution would be to choose the most robust caption/subtitle format out there (one that can handle text and image overlays, plus include placement data - I think ASS does this, but I am not certain, as I don't have a lot of experience in subtitling), and store the data within the sage.jar process. Then have it rendered via STV widgets, which would allow it to function on any sagetv display. It's not like the conversion of subtitle formats is in any way a complex task for the server to do, even if it has to extract it from the file at playback time. This end-goal would then support all potential subtitle formats going INTO the sage server, including direct digital CC streams, convert it internally to the master format for synchronized display with the playback.

(I'll be honest, I'm a big fan of standardizing everything on the server end, so the clients don't have to even think. There was a time after SageTV LLC's disappearance that I started drawing up a (much simpler) replacement, and it was very much focused on this, such that on import or recording of a file, it was going to be immediately converted to MKV with a very narrow set of possible combinations).

[valnar]

Is there a way to script a process that extracts the CC's into a subtitle file similar to the way we post process commercial detection?

[stuckless]

Originally Posted by valnar

Is there a way to script a process that extracts the CC's into a subtitle file similar to the way we post process commercial detection?

https://github.com/CCExtractor/ccextractor

Never used it, but it looks like it does what you want. Maybe you can try it out, and report back your findings.

[valnar]

I was hoping the Sage Gods would bless us mortals with automatic code for that.

[EnterNoEscape]

Originally Posted by stuckless

https://github.com/CCExtractor/ccextractor

Never used it, but it looks like it does what you want. Maybe you can try it out, and report back your findings.

OpenDCT does SRT in realtime via CCExtractor if you're using the FFmpeg consumer. There is one downside to this. The caption positioning data is lost. The subtitles always show up at the bottom of the screen potentially getting in the way.

[valnar]

Can it be configured to do JUST that?