Resolution off on android miniclient with samsung plasma

[jeffoffline]

Okay first off just wanted to thank stuckless for his work on the miniclient, I can't begin to explain how stoked I am to be able to use it to play back recorded tv and have comskip working

Now on to my issue, if there is a better place to post this my apologies. I'm using a Nexus Player and Samsung Plasma TV for playback, the TV is native 1024x768 and when playing back dvd's ripped as mp4's it displays them as stretched vertically. All my other android apps do not do this and when using SageTV through my htpc which is set to 1366x768 (connected to the same plasma) everything plays properly. Recorded/Live TV (ATSC) also displays properly.

I realize the TV is at the point outdated and the resolution is a bit of an oddball but wanted to mention it incase it was an easy fix.

I've tried both the default and exoplayer and both result in the same stretching during playback.

[stuckless]

Originally Posted by jeffoffline

Okay first off just wanted to thank stuckless for his work on the miniclient, I can't begin to explain how stoked I am to be able to use it to play back recorded tv and have comskip working

Now on to my issue, if there is a better place to post this my apologies. I'm using a Nexus Player and Samsung Plasma TV for playback, the TV is native 1024x768 and when playing back dvd's ripped as mp4's it displays them as stretched vertically. All my other android apps do not do this and when using SageTV through my htpc which is set to 1366x768 (connected to the same plasma) everything plays properly. Recorded/Live TV (ATSC) also displays properly.

I realize the TV is at the point outdated and the resolution is a bit of an oddball but wanted to mention it incase it was an easy fix.

I've tried both the default and exoplayer and both result in the same stretching during playback.

Do things get better/worse when you bring up the on screen nav (long press select), and then select the aspect icon?

[jeffoffline]

I tried cycling through the different aspect ratios, 3 looked the same and the 4th one just seemed to zoom it in more so there were no black bars on the screen but everything was still stretched a bit vertically.

[jeffoffline]

ahh, it appears to be an issue with the way the mini client handles the video resolution of the file and it's not something to do with the tv resolution. I could replicate the issue on my phone (LG G4) The mp4 files are 720x360.

Here's a screenshot I took on my laptop playing the video with SageTV (this is how the video should look)


and here's one i took on my phone playing via the miniclient where it's stretched vertically.

[Fuzzy]

SageTV has settings for display aspect ratio that need to be set properly for it to properly scale the video. That said, I want to say that aspect ratio correction in the miniclient is not fully implemented, and that Stuckless is sort of just handing the file off to the player and letting it deal with it. I'm sure android has some sort of display aspect ration settings as well, but it is likely hidden somewhere (as android devices are typically tied directly to a given screen).

[stuckless]

Originally Posted by Fuzzy

SageTV has settings for display aspect ratio that need to be set properly for it to properly scale the video. That said, I want to say that aspect ratio correction in the miniclient is not fully implemented, and that Stuckless is sort of just handing the file off to the player and letting it deal with it. I'm sure android has some sort of display aspect ration settings as well, but it is likely hidden somewhere (as android devices are typically tied directly to a given screen).

Well it is fully implemented... in that unless there are bugs, there isn't much more that I'm doing (And this seems like a bug).

The code for doing the AR stuff is here...
https://github.com/OpenSageTV/sagetv...ectHelper.java

If someone wants to review it, ask questions, suggest edits, etc... I'm game. I didn't write this code, I lifted it from the IJKPlayer sample player.

I could be that all my videos are stretched and I didn't notice I looked at the 2 videos posted and I thought the top one was the problem... which is likely a sign that I'm watching too many stretched videos

[Taddeusz]

It appears to be an issue with anamorphic video. The "stretched" display of that 720x360 video is correct if you display it at a 1:1 pixel ratio. In order to display correctly the DAR of the video needs to be respected. Not sure how Android deals with anamorphic video?

[stuckless]

Originally Posted by Taddeusz

It appears to be an issue with anamorphic video. The "stretched" display of that 720x360 video is correct if you display it at a 1:1 pixel ratio. In order to display correctly the DAR of the video needs to be respected. Not sure how Android deals with anamorphic video?

Thanks... I can see that in the MiniClient... I'm not calling setVideoSampleAspectRatio() which includes sar/dar information, so that is likely what is causing the stretching for TVs with non-square pixels. I'll open a defect and hopefully get a fix out on the weekend.

[Fuzzy]

Originally Posted by stuckless

Thanks... I can see that in the MiniClient... I'm not calling setVideoSampleAspectRatio() which includes sar/dar information, so that is likely what is causing the stretching for TVs with non-square pixels. I'll open a defect and hopefully get a fix out on the weekend.

Unfortunately, that call is a bit inconvenient. It'd be nicer if IJKPlayer had an overloaded call to pass the actual float value in, which is what Sage stores after setup in videoframe/display_aspect_ratio, but instead, it only seems to have a call that takes Numerator and Denominator. You'll have to reverse engineer a fraction of integers from the float to make it work. Luckily, it looks like IJK never cares if it is 16/9, or 16000/9000, or 17777778/10000000, so the conversion may not be that difficult.

[stuckless]

Originally Posted by Fuzzy

Unfortunately, that call is a bit inconvenient. It'd be nicer if IJKPlayer had an overloaded call to pass the actual float value in, which is what Sage stores after setup in videoframe/display_aspect_ratio, but instead, it only seems to have a call that takes Numerator and Denominator. You'll have to reverse engineer a fraction of integers from the float to make it work. Luckily, it looks like IJK never cares if it is 16/9, or 16000/9000, or 17777778/10000000, so the conversion may not be that difficult.

IJKplayer gets those 2 values from it's player instance (after video starts)... So, getting the 2 values is pretty straight forward from an IJKPlayer point of view. I looked at the ijk sample player and it passed the video size and those 2 values (all from the player instance)... I'm only passing the video size.

I'm sure ExoPlayer is doing something completely different.

At this point the MiniClient hasn't hooked into allowing sagetv to toggle AR... there's another open defect around that as well, and I think in that case, sagetv would be sending me the "float value" and I'll have to figure out how to do something with it.

[EnterNoEscape]

Originally Posted by Fuzzy

Unfortunately, that call is a bit inconvenient. It'd be nicer if IJKPlayer had an overloaded call to pass the actual float value in, which is what Sage stores after setup in videoframe/display_aspect_ratio, but instead, it only seems to have a call that takes Numerator and Denominator. You'll have to reverse engineer a fraction of integers from the float to make it work. Luckily, it looks like IJK never cares if it is 16/9, or 16000/9000, or 17777778/10000000, so the conversion may not be that difficult.

Using fractions is actually the most correct way to work with AV. FFmpeg uses fractions for almost everything that could be a decimal. While a fraction can be represented as a decimal, often in AV, that's not accurate enough (especially when it comes to framerate). It actually uses the fraction to make calculations, so any valid fraction should work. FFmpeg also has functions to reduce fractions to their least common denominator, so providing something that's not optimal will not cause any real issues. However, I would probably try to snap the decimals to their most likely fraction when possible (ie. 1.778 should be 16/9 and not 889/500) so you don't end up with some really strange, but technically correct fractions causing subtle distortion.

[Fuzzy]

Originally Posted by stuckless

IJKplayer gets those 2 values from it's player instance (after video starts)... So, getting the 2 values is pretty straight forward from an IJKPlayer point of view. I looked at the ijk sample player and it passed the video size and those 2 values (all from the player instance)... I'm only passing the video size.

I'm sure ExoPlayer is doing something completely different.

At this point the MiniClient hasn't hooked into allowing sagetv to toggle AR... there's another open defect around that as well, and I think in that case, sagetv would be sending me the "float value" and I'll have to figure out how to do something with it.

The issue is that going off of the metrics provided by the android OS is not necessarily the best option, otherwise his setup would be working already. The varying video pipelines and displays out there are WHY sagetv has it's own setting of display aspect ratio. Sage's setting also allows fine-tuning the ratio very easily from within the UI.

Originally Posted by EnterNoEscape

Using fractions is actually the most correct way to work with AV. FFmpeg uses fractions for almost everything that could be a decimal. While a fraction can be represented as a decimal, often in AV, that's not accurate enough (especially when it comes to framerate). It actually uses the fraction to make calculations, so any valid fraction should work. FFmpeg also has functions to reduce fractions to their least common denominator, so providing something that's not optimal will not cause any real issues. However, I would probably try to snap the decimals to their most likely fraction when possible (ie. 1.778 should be 16/9 and not 889/500) so you don't end up with some really strange, but technically correct fractions causing subtle distortion.

Oh, I fully understand that fractions are the 'right' way to do it, but sage already has a method for setting the display aspect ratio, and, because it's able to be tweaked very finely (0.005 steps), and not just 16/9, 4/3, etc., it is stored as decimal in the sage.properties file.

[jeffoffline]

Originally Posted by stuckless

I could be that all my videos are stretched and I didn't notice I looked at the 2 videos posted and I thought the top one was the problem... which is likely a sign that I'm watching too many stretched videos

ha that probably wasn't the best spot for me to screenshot, it's certainly more noticeable in some scenes vs others but it sounds like you guys know what's going on now...

[stuckless]

Originally Posted by Fuzzy

The issue is that going off of the metrics provided by the android OS is not necessarily the best option, otherwise his setup would be working already. The varying video pipelines and displays out there are WHY sagetv has it's own setting of display aspect ratio. Sage's setting also allows fine-tuning the ratio very easily from within the UI.

I get that... but, I think that this particular issue, can be solved by setting the the AR values from the video stream (as the sample does). IJK player has been forked 3000 times and it's used in a lot of android apps... if it had a fundamental AR issue my guess would be that it's solved, and I think that I need to update the player with the AR values.

I'm not 100% sure how I'd use the sagetv AR information. I'm not seeing where sagetv is passing that to me. ie, I request to play a video, and it sends me a video stream, I don't have any other information about the video, except what the player parses from the video stream.

That being said, SageTV does have an AR toggle, and there is an open defect to support using the AR toggle. I'm still investigating exactly what I'm supposed to so there... since it seems like I tell sagetv what AR modes I support and the AR toggle just sends me a command with the AR mode that the user selected, and then I still need to apply it.

[Fuzzy]

I just think a SageTV solution should honor the SageTV DAR setting, and ignore the device's settings. Things like 2.35:1 anamorphic projector lenses and such need to be accountable for, and that is the proper way it is handled in a sagetv installation.

That said, I believe this is all handled by BasicVideoFrame.java. It does all the math that sage desires, knowing the AR of the stream, the DAR of the display, and the chosen AR mode. With that information, BasicVideoFrame.refreshVideoSizing() does the math for where and what size to make the video rectangle. I think this is the information that is sent to you (via setVideoRectangles, and the GFXCMD_SETVIDEOPROP), so for it to be properly used, you'd need to tell the actual renderer to stretch the video to that rectangle, and NOT use its own AR math.

[stuckless]

Originally Posted by Fuzzy

I just think a SageTV solution should honor the SageTV DAR setting, and ignore the device's settings. Things like 2.35:1 anamorphic projector lenses and such need to be accountable for, and that is the proper way it is handled in a sagetv installation.

That said, I believe this is all handled by BasicVideoFrame.java. It does all the math that sage desires, knowing the AR of the stream, the DAR of the display, and the chosen AR mode. With that information, BasicVideoFrame.refreshVideoSizing() does the math for where and what size to make the video rectangle. I think this is the information that is sent to you (via setVideoRectangles, and the GFXCMD_SETVIDEOPROP), so for it to be properly used, you'd need to tell the actual renderer to stretch the video to that rectangle, and NOT use its own AR math.

I spent all of yesterday (and some of today) understanding how sagetv supports aspects... and I wish I could say that there was nice solution... but I think I missing something...

1. setVideoRectangles does not directly send an aspect ratio... SageTV will calculate the src/dest rectangles and a client can use that to position/scale the video. SageTV manages a list of apsects, and it will adjust the rectangles accordingly. This sort of works from what I can tell, but, it's not supported on MiniClients. ie, when you issue AR_TOGGLE it checks if you are a miniclient and if you are, then it expects that you've implemented the "VIDEO_ADVANCED_APSPECT_LIST". So, it appears that having sagetv "figure out" the aspect ratio is not something that is supported on MiniClients.

2. MiniClient can use VIDEO_ADVANCED_ASPECT and VIDEO_ADVANCED_ASPECT_LIST. The client can send down a bunch of arbitrary strings, like, "Source;Fill;Zoom;Stretch", and when AR_TOGGLE is sent, sagetv will just toggle between the list, sending the MiniClient a value, such as "Fill", in the VIDEO_ADVANCED_ASPECT property... the client is expected to "handle" that in some way. It will also send setVideoRectangles with a source rectangle of 0,0,0,0.

So, in neither case is SageTV sending the video aspect ratio... in the first, it just figures out the size of the video rectangles, and in the second, it just sends me a "name" of an aspect mode that I then need to apply to the video, by figuring the source video, video bounds, etc (which is what the current MiniClient aspect switcher does today)... This just allows the sagetv AR_TOGGLE to toggle it, instead of the on screen nav.

I many ways it would be nice if I could use #1... but it appears that if you are MiniClient you need to figure it out on your own.

I've tried to advertise the MiniClient as a Desktop UI and the automatic aspect video switching works... but, sadly, lots of other things no longer work, since, sagetv thinks I'm a desktop client.

So, I'm likely not going to have a solution for this, this weekend. I'll continue to look at things, but I think I'll need Jeff's help to fully understand what I need to do.

[Fuzzy]

Yeah, sorry, I have not analyzed step by step how lt is done, and honestly don't think it would handle it differently for miniclients. The videobounds seemed like it would be enough, and the client/rednerer shouldn't even need to know the aspect ratio of the video or the display, just where to draw it and how big.

As far as I can guess, the lack of handling the bounds is also why preview doesn't work in a window in the Android miniclient.

[stuckless]

Originally Posted by Fuzzy

Yeah, sorry, I have not analyzed step by step how lt is done, and honestly don't think it would handle it differently for miniclients. The videobounds seemed like it would be enough, and the client/rednerer shouldn't even need to know the aspect ratio of the video or the display, just where to draw it and how big.

As far as I can guess, the lack of handling the bounds is also why preview doesn't work in a window in the Android miniclient.

Yean, the Android MiniClient has never honoured the setVideoBounds... but I have it implemented now... There is still an issue with the preview in the that still doesn't allow the video to show through the UI... I'm looking at that as well... (since there is bug for that as well )

I may make a change to the Sage core to allow miniclient to use the "standard" AR toggle UNLESS the client actually passes the advanced AR modes, and chooses to do it own its own. For some reason, there is code in there to specifically NOT allow the miniclient to use the standard AR toggling, so maybe there's a reason for that.

[stuckless]

Just an update... It's been a productive 2 days... but unfortunately I won't be releasing anything this weekend. But, I do have the SageTV AR toggle working (so the AR modes in the MiniClient will now just delegate to the SageTV system). And I have the video preview windows working. I have one small issue that I'm currently working though, but I've run out of time for this weekend, so likely I'll solve it later in the week and maybe release something next weekend.

[Fuzzy]

Excellent news. Did it take any core changes to get it to use the server side aspect controls, or was it still passing them them to the client that way as well as the client-side options?