Touch Interface

[jusjoken]

I wanted to thank those who have contributed to my Indiegogo Campaign with 16 backers so far plus 3 prior to the campaign starting

So here is a sample of what I have been busy with on Gemstone 2. This new touch interface is not complete but it is working in most parts of Gemstone. A new toggle touch button is located in the bottom right next to the SageTV logo that will turn on the enhanced footer and also expand the header to make touching the buttons easier.

I have also been working to ensure the menus and Flows are Touch/Mouse enabled so you can also simply touch or click a poster or item to select it as well.

Note: some may recognize the interface as I borrowed the concept from MeInMaui from the SageMC TouchScreen addon from 2008 and tweaked it a lot by incorporating it into the footer and changing the layout, adding buttons etc.

Samples....

New touch toggle button in bottom right...


Touch enabled on the Main Menu

Other examples in Flows...




Using the Gemstone Options with the touch interface....


More to come....

k

[stuckless]

Looks great!

[KryptoNyte]

Beautiful. I see 5 days left on that campaign, we have just a little ways to go folks!

[reggie14]

I hate to offer any sort of criticism... it's amazing to see what jusjoken and stuckless have accomplished. But, I have to say that trying to use the miniclient on a tablet is an incredibly frustrating experience right now, and I'm concerned this isn't going to be that much better...

I realize that the miniclient architecture makes UI design a very challenging task, as the miniclient doesn't necessarily know what kind of screen you're on to change the behavior of the controls. But, to the the absolute extent possible, it seems like it would be best if a touch-friendly interface didn't rely on on-screen remotes to function well. I'd like to think that's possible even given the current constraints, although it would probably break a bunch of stuff in current interfaces...

So here are some ideas, which I realize are just as much comments on the miniclient as the STV. But I think a touch-friendly interface would need changes to both.

First, I really dislike the current model where you have to double tap to select things. The Gemstone interface gets around this in the home menu simply by having mouse-over events, but this isn't going to work for just about everything else. I sure hope I'm not double tapping everywhere to select items. So, to me it seems like it would be preferable to have single tap focus and select an item.

Similarly, every screen that has a long list is a pain to use. While I'd love to be able to touch-and-drag to scroll, I'm guessing that's not going to work well with the miniclient architecture. If that's true, then I'd be inclined to say scrolling should always function as a page-up or page-down. If we knew every button on the screen was going to be mouse/touch friendly, then a single-finger swipe up/down could be a page-up and page-down. Any scrollable list should probably also have page-up/down buttons on-screen, too.

I'm not sure how you deal with horizontal scrolling. So, I'm inclined to say (almost) everything should scroll vertically. The main place that wouldn't work would be the Program Guide. Could the miniclient interpret swipe-left/right as left/right-arrow, and then simply have the STV code respond to those commands by doing the equivalent to a page-left/right?

I'm not sure how to handle "special" things, like the options menu. Could this be implemented as a long press? That is, a long press would focus and hit the options button.

Behavior would need to change for the video screen, which the current miniclient already does. This is a case where you'd need on-screen buttons, but only for skipping, RW/FW, and play/pause.

Are these crazy suggestions, either because they wouldn't work, they'd be too difficult to build, or generally too disruptive?

[Wirenut]

I have to agree with reggie14.

I totally appreciate all the fine work applied to this project. I use my Nexus 7 tablet with this app almost every day on my eliptical workout.

Thanks,

Wirenut

[stuckless]

@reggie14 - I can appreciate your frustration in using the MiniClient on a tablet/phone. It's not ideal. This is a system designer for up/down/left/right/select on a remote and translating that into touch, as is, it not ever going to be a pleasant experience.

I think that while the MiniClient can work on the Phone/Tablet, it is not the primary objective for the Android project. It is more of a last resort... ie, You can use it a placeshifter, but, it's never going to feel like a native app on a phone, but it'll get the job done.

While changes to the STV could make this better, it will never be great, but it will usable to some extent.

At first glance, it seems that "tap" should equal "select". I started out that way. The problem is much of sagetv UI is not designed for that kind of navigation. Out of the box, if "tap" is "select" then the main menu becomes a static list of 6-8 items, and you can not navigate to the sub menus (or the sub-sub-menus). So, tap becomes focus, and double-tap becomes select, and it's something to get used to if you intend to spend time on a tablet.

If there is a ever a "mobile" themed STV created, I'll consider adding an option in the configuration to allow tap equals select, but until then, the UI, as is, is not usable in that way.

[jusjoken]

Originally Posted by reggie14

I hate to offer any sort of criticism... it's amazing to see what jusjoken and stuckless have accomplished. But, I have to say that trying to use the miniclient on a tablet is an incredibly frustrating experience right now, and I'm concerned this isn't going to be that much better...

First, let me say that comments, criticisms and suggestions are always welcome so keep them coming

Originally Posted by reggie14

I realize that the miniclient architecture makes UI design a very challenging task, as the miniclient doesn't necessarily know what kind of screen you're on to change the behavior of the controls. But, to the the absolute extent possible, it seems like it would be best if a touch-friendly interface didn't rely on on-screen remotes to function well. I'd like to think that's possible even given the current constraints, although it would probably break a bunch of stuff in current interfaces...

The Touch controls shown here is step 1 in likely a long road of UI changes to get Mouse/Touch working well in Gemstone. And as with everything I try to do in Gemstone, it will be an option, and perhaps an option that will allow an earlier release to deal with at least some level of usability on a touch device while I find time to do everything on my todo list.

Originally Posted by reggie14

So here are some ideas, which I realize are just as much comments on the miniclient as the STV. But I think a touch-friendly interface would need changes to both.

First, I really dislike the current model where you have to double tap to select things. The Gemstone interface gets around this in the home menu simply by having mouse-over events, but this isn't going to work for just about everything else. I sure hope I'm not double tapping everywhere to select items. So, to me it seems like it would be preferable to have single tap focus and select an item.

I still need to dig into this model where a touch/click gives focus vs performs a select. First, this is already an option in SageTV/Gemstone. See "Require focus for mouse selection" in Detailed Setup/Customize. This setting toggles this behavior but keep in mind Gemstone is not yet even mouse friendly so this is not likely consistently implemented on all Gemstone screens. As well, this approach as stuckless suggests, may not be valid where horizontal or vertical lists need scrolling before selecting.

So I still need to experiment with what can be done in the UI and what works best of those options.

Originally Posted by reggie14

Similarly, every screen that has a long list is a pain to use. While I'd love to be able to touch-and-drag to scroll, I'm guessing that's not going to work well with the miniclient architecture. If that's true, then I'd be inclined to say scrolling should always function as a page-up or page-down. If we knew every button on the screen was going to be mouse/touch friendly, then a single-finger swipe up/down could be a page-up and page-down. Any scrollable list should probably also have page-up/down buttons on-screen, too.

Again on this one I need to experiment. My initial plan is to make the FF/RW touch buttons perform page scroll right/left/down/up dependent on horizontal vs vertical and skipFW/skipRW to go to the last/first item in a list. That will make it workable for a first release and then we can go from there.

Originally Posted by reggie14

I'm not sure how you deal with horizontal scrolling. So, I'm inclined to say (almost) everything should scroll vertically. The main place that wouldn't work would be the Program Guide. Could the miniclient interpret swipe-left/right as left/right-arrow, and then simply have the STV code respond to those commands by doing the equivalent to a page-left/right?

As above, left right will be handled with the FF/RW etc buttons to and they will know to do left vs up based on the screen/list they are on in Gemstone...at least in a first release.

You have mentioned Swipes a few times. Swipes are not currently captured by listeners available in the STV code so until someone (perhaps me in the future) writes a new set of listeners for touch, the STV will not handle swipes. There is however a MouseDrag that I will experiment with but so far I have not figured a good way to use it. It is used in the STV in scroll bars where you drag the graphic that is the scrollbar which then move the list. There may be a way for me to exploit this but I have not spent enough time on that one....yet!

Originally Posted by reggie14

I'm not sure how to handle "special" things, like the options menu. Could this be implemented as a long press? That is, a long press would focus and hit the options button.

Long press in the STV is interpreted as a middle mouse button click (MouseClick = 2 in the code). A right click with a mouse is MouseClick = 3 and by default the STV will execute the Options function with any RightClick...however, I have not figured out yet on a Touch device how to cause the RightClick (MouseClick = 3). So I have a few options here and still experimenting.

Originally Posted by reggie14

Behavior would need to change for the video screen, which the current miniclient already does. This is a case where you'd need on-screen buttons, but only for skipping, RW/FW, and play/pause.

I have not tackled the OSD (for the video screen) yet but there are challenges there. My preference is to have nothing on the screen until the screen is tapped/clicked and then the OSD is displayed and touch/click enabled. A small OSD on screen all the time could be done as an option but nothing decided here yet....open to suggestions.

As for the comment re "which the current miniclient already does". Stuckless and I will need to work together when a Gemstone release is ready to determine if any of his workarounds need to be undone or toggled off when the STV better supports touch as once the STV handles touch better then these workarounds could negate the new functions (or be confusing). Time will tell as I have not tackled the OSD yet.

Originally Posted by reggie14

Are these crazy suggestions, either because they wouldn't work, they'd be too difficult to build, or generally too disruptive?

Keep the suggestions coming. I am in early stages and the only one working on the UI in Gemstone so it all will take time so I can adjust as I go and get other ideas.

And keep in mind, this is only a first crack at a way to make touch workable in Gemstone. I am well into a new horizontal/vertical combo menu option (similar in some ways to the 360 Flow style) that will expand the menus to optionally include Flow/View content so right from the menu you can scroll your movies or tv shows or recently recorded etc views. I plan to incorporate a hotkey function like the flows have so you can jump to a letter in those media lists quickly and likely an on screen keyboard/alpha list to jump using a mouse/touch interface.... all in the works along with numerous fixes and requests.

Now...to find more time for this hobby

k

[reggie14]

Originally Posted by stuckless

I think that while the MiniClient can work on the Phone/Tablet, it is not the primary objective for the Android project. It is more of a last resort... ie, You can use it a placeshifter, but, it's never going to feel like a native app on a phone, but it'll get the job done.

Understood, and the miniclient works great on an AndroidTV with a remote- particularly when you use a remote with a full set of media buttons. I figured tablets/phones were mostly just along for the ride.

Out of curiosity, do you think a webGUI would ultimately be a better solution for tablets/phones?

Originally Posted by stuckless

If there is a ever a "mobile" themed STV created, I'll consider adding an option in the configuration to allow tap equals select, but until then, the UI, as is, is not usable in that way.

Sure, that makes perfect sense. I think I understand why navigation in the miniclient works as it does now. As you said, it basically needs to given the way the current STVs work.

I hope one of the main takeaways from my post is that STV designers shouldn't need to assume that the current user-input methods would map to the same commands that they do now. If mobile STVs were explicitly designed for touch, and not by selecting items with directional buttons on remotes, then everything could work a little differently. I think that could provide big usability improvements overall.

[stuckless]

Originally Posted by reggie14

Understood, and the miniclient works great on an AndroidTV with a remote- particularly when you use a remote with a full set of media buttons. I figured tablets/phones were mostly just along for the ride.

Out of curiosity, do you think a webGUI would ultimately be a better solution for tablets/phones?

I do think that a Web/Native UI would offer a better experience on a mobile device with limited features.

I've given this some thought, and started some preliminary work on it. My feeling here is that what we need for a phone/tablet is more of a "player". ie, Browse/Search shows/movies, select a show/movie and play it. The actual player will likely have to delegate to the full miniclient due to the protocols, etc, so this initial web/app would really be a better front end to browsing and selecting a media item to play.

I'm not sure I see the value in replicating EVERYTHING in the STV in a Web UI... and I think that if you want to schedule recordings or resolve conflicts, etc, you can use miniclient directly for that.

That's sort of my thoughts around providing a "player" for SageTV.

Sure, that makes perfect sense. I think I understand why navigation in the miniclient works as it does now. As you said, it basically needs to given the way the current STVs work.

I hope one of the main takeaways from my post is that STV designers shouldn't need to assume that the current user-input methods would map to the same commands that they do now. If mobile STVs were explicitly designed for touch, and not by selecting items with directional buttons on remotes, then everything could work a little differently. I think that could provide big usability improvements overall.

I think that Jusjoken has a pretty good grip on this. I think to get a much better experience it will take some collaboration between myself and him (which has been happening for quite some time now ).

I'm curious to see how the next version Gemstone works in this regard. I've used Gemstone on a tablet a couple times during a couple of treadmill sessions... I feel your pain

[reggie14]

Originally Posted by jusjoken

The Touch controls shown here is step 1 in likely a long road of UI changes to get Mouse/Touch working well in Gemstone. And as with everything I try to do in Gemstone, it will be an option, and perhaps an option that will allow an earlier release to deal with at least some level of usability on a touch device while I find time to do everything on my todo list.

That makes sense. It looks like the path you're going down initially will certainly be easier to use than what's currently available.

The one thing I worry about is that a good mouse/touch-friendly UI might be fundamentally incompatible with a good remote-friendly UI. I wonder about the negative consequences of trying to do a single STV that handles both.

I'm not sure how much of a problem that could be. One broad implication of what I'm proposing is that interfaces would not be able to rely on focus for important navigation decisions. As stuckless mentioned, the home screen is a good example of this. Focus opens submenus, while selection opens new screens. This distinction wouldn't work if a touch-friendly miniclient and STV made a single tap focus-and-select.

If there's only a small number of things like that, then maybe it's not a big deal- you could configuration options/workarounds for those individual screens. But I'm pretty sure that's not the only example.

Originally Posted by jusjoken

As well, this approach as stuckless suggests, may not be valid where horizontal or vertical lists need scrolling before selecting.

I think it could work with lists, but there would need to be other changes. First, I think scrolling through lists line-by-line would be really painful, and should be avoided. And I think it can be avoided, without too many big changes.

If everything in a list is clickable, then you only need a way to scroll through all the options in the list. I think that would be better accomplished through paging through lists, rather than scrolling through lists line-by-line. This is already supported in almost every screen, so all you need to do is give users a way to easily send a page-up/down command (or skip, for horizontal scrolling).

Again, if everything in the STV is clickable, then you don't need to support directional buttons. That frees up the swipe up/down/left/right to gestures to be mapped to page-up/down and skip back/forward commands, respectively.

Originally Posted by jusjoken

You have mentioned Swipes a few times. Swipes are not currently captured by listeners available in the STV code so until someone (perhaps me in the future) writes a new set of listeners for touch, the STV will not handle swipes. There is however a MouseDrag that I will experiment with but so far I have not figured a good way to use it. It is used in the STV in scroll bars where you drag the graphic that is the scrollbar which then move the list. There may be a way for me to exploit this but I have not spent enough time on that one....yet!

Just to be clear, I'm not suggesting that the STV be made to handle swipes. To the STV these would just look like page-up/down, skip-back/forward commands.

Originally Posted by jusjoken

Long press in the STV is interpreted as a middle mouse button click (MouseClick = 2 in the code). A right click with a mouse is MouseClick = 3 and by default the STV will execute the Options function with any RightClick...however, I have not figured out yet on a Touch device how to cause the RightClick (MouseClick = 3). So I have a few options here and still experimenting.

OK. I think Android supports various gestures, so there ought to be something that you could map to middle and right click...

That being said, I think it would be preferable to not overload that too much. When it comes to selecting/tapping things, I think you probably only want to support single-tap and long-press.

As a side note, I had no idea the STV interpreted long-presses!

Originally Posted by jusjoken

I have not tackled the OSD (for the video screen) yet but there are challenges there. My preference is to have nothing on the screen until the screen is tapped/clicked and then the OSD is displayed and touch/click enabled. A small OSD on screen all the time could be done as an option but nothing decided here yet....open to suggestions.

Sorry if my previous post was misleading on this. I absolutely agree with you- a single tap should bring up the OSD and media control buttons. This would be consistent with just about every media app out there.