webserver + vlc + android + streaming = error

[jrjbertram]

I have Nielm's Webserver V2.25 installed on top of SageTV 6.6 (pretty sure) and have VLC 1.05 installed in order to get streaming media.

Works great from my PC (localhost and laptop on my LAN). The webserver works great from my android phone. Streaming works great from my LAN. Streaming even works great from a Windows PC outside my LAN (ie, from work).

Streaming from an android phone (a Droid) does not work (from LAN or from 3G connection).

It seems to be a problem with the download of the playlist, though, not from video encoding settings.

From the phone, I can navigate to one of my recordings, I can select streaming, and I can select one of the encodings (doesn't matter which). When I click on one of the 'Play in External Player' links, the phone's browser wants to save the playlist file as a downloaded file ("PlaylistGenerator.m3u" for example) but when the phone tries to download, the browser gets an error:

"Cannot download. The size of the item cannot be determined."

I can't find any useful information googling for the error... am not sure if it's an issue with the phone's browser or something with the web browser. (The browser can stream video from YouTube, and I've been able to set up Darwin streaming media server and successfully stream videos to my phone over RTSP if I get the video encoded correctly with HandBrake).

I think that maybe the webserver on the download of the playlist is perhaps not including the size of the playlist when it is downloaded. Does that sound reasonable? Maybe a missing/unsupported HTTP header or something?

I've been poking through the webserver's config files and don't see anything that looks like I can configure this option. Wondering if anyone's been down this road and figured it out already..?

Thanks much,
- Josh.

[jrjbertram]

Here's what I see on wireshark for one of the files that are downloaded normally as part of the page reload. Note the "Content-Length" HTTP header in the response.

  
  GET /sage/menu_items.js HTTP/1.1
  Host: 192.168.1.198:8080
  User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.8) Gecko/20100202 Firefox/3.5.8 (.NET CLR 3.5.30729)
  Accept: */*
  Accept-Language: en-us,en;q=0.5
  Accept-Encoding: gzip,deflate
  Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
  Keep-Alive: 300
  Connection: keep-alive
  Referer: http://192.168.1.198:8080/sage/Detai...aFileId=300316
  If-Modified-Since: Wed, 15 Jul 2009 04:29:52 GMT
  Authorization: Basic anJqYmVydHJhbTpncmV0Y2h5
  Cache-Control: max-age=0
  
  HTTP/1.1 304 Not modified
  Date: Sun, 14 Mar 2010 04:16:14 GMT
  MIME-Version: 1.0
  Server: Rogatkin's JWS based on Acme.Serve/$Revision: 1.26 $
  Last-modified: Wed, 15 Jul 2009 04:29:52 GMT
  Connection: close
  Content-Type: application/x-javascript
  Content-Length: 6381

Here's what I see when I click on the playlist generator links. Note no Content-Length header. Can that header be added to however PlaylistGenerator works? I suspect this would fix my android browser problem.

GET /sagepublic/PlaylistGenerator?Command=generate&pltype=wvx&fntype=url&MediaFileId=300316&TranscodeOpts=mode%3Dvlc%26vc%3Dh264%26vb%3D192%26ac%3Dmp3%26ab%3D64%26scale%3D0.3%26mux%3Dts%26deint%3D1 HTTP/1.1
Host: 192.168.1.198:8080
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.8) Gecko/20100202 Firefox/3.5.8 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://192.168.1.198:8080/sage/Detai...aFileId=300316

HTTP/1.1 200 Ok
Date: Sun, 14 Mar 2010 04:12:20 GMT
Content-Disposition: attachment; filename=playlist.wvx
Content-Encoding: gzip
MIME-Version: 1.0
Server: Rogatkin's JWS based on Acme.Serve/$Revision: 1.26 $
Connection: close
Content-Type: video/x-ms-wvx

<binary data>

[jrjbertram]

I don't see anything in 4.3 or 4.4 of:

http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html

...that makes me think that Content-Length is required. Guessing that android is a little too strict?

- J.

[jrjbertram]

Ended up finding:
http://forums.sagetv.com/forums/showthread.php?t=44680

Converting to Jetty / new webserver app fixed the Content-Length issue. New issues, but I'll start another thread for that.

[jrjbertram]

I have the Jetty webserver installed on top of SageTV 6.6 (pretty sure) and have VLC 1.05 installed in order to get streaming media.

(Installed Jetty per http://forums.sagetv.com/forums/showthread.php?t=44680. Used nielm_websrv_jetty_2_x.zip.)

Works great from my PC (localhost and laptop on my LAN). The webserver works great from my android phone. Streaming works great from my LAN. Streaming even works great from a Windows PC outside my LAN (ie, from work).

Streaming from an android phone (a Droid) does not work (from LAN or from 3G connection).

It seems to be a problem with the download of the playlist, though, not from video encoding settings.

From the phone, I can navigate to one of my recordings, I can select streaming, and I can select one of the encodings (doesn't matter which). When I click on one of the 'Play in External Player' links, the phone's browser wants to save the playlist file as a downloaded file ("PlaylistGenerator.html").

The phone downloads the file as an HTML file to its temporary downloads, but it seems to have lost track that it's actually a playlist file extension -- I can't get it to save as anything other than HTML, so I don't think there's a way for another program / browser plugin to play it.


Note: This thread was spawned from... once I switched to Jetty, I got farther.
http://forums.sagetv.com/forums/showthread.php?t=47384

[jreichen]

Originally Posted by jrjbertram

Ended up finding:
http://forums.sagetv.com/forums/showthread.php?t=44680

Converting to Jetty / new webserver app fixed the Content-Length issue. New issues, but I'll start another thread for that.

I see in your wireshark log that the response is being gzipped. Jetty's gzip filter is apparently handling the Content-Length header better than TJWS does.

[jreichen]

Originally Posted by jrjbertram

It seems to be a problem with the download of the playlist, though, not from video encoding settings.

From the phone, I can navigate to one of my recordings, I can select streaming, and I can select one of the encodings (doesn't matter which). When I click on one of the 'Play in External Player' links, the phone's browser wants to save the playlist file as a downloaded file ("PlaylistGenerator.html").

The phone downloads the file as an HTML file to its temporary downloads, but it seems to have lost track that it's actually a playlist file extension -- I can't get it to save as anything other than HTML, so I don't think there's a way for another program / browser plugin to play it.

Take a look at the response headers and see if they contain a Content-Type. When I click on an m3u playlist, the content type is audio/x-mpegurl. The iPhone just returns back to the page after clicking on the link but I haven't looked up its officially supported playlist types.

If your content type is wrong, check out this wiki page and see if you can get the results you want: http://tools.assembla.com/sageplugin...ustomMimeTypes.

[jrjbertram]

I've been looking into the mime.types suggestion you'd made, but can't seem to get it to work. I'm currently getting back the following:

GET /sage/public/MediaFile?MediaFileId=315951&Segment=0 HTTP/1.1
Host: --edit--:--edit--
Accept-Encoding: gzip
Referer: http://--edit--/sage/DetailedInfo?MediaFileId=315951
Accept-Language: en-US
User-Agent: Mozilla/5.0 (Linux; U; Android 2.0.1; en-us; Droid Build/ESD56) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17
Cookie: grid_num_hours=6; grid_num_chans=999
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Charset: utf-8, iso-8859-1, utf-16, *;q=0.7

HTTP/1.1 200 OK
Date: Sat, 20 Mar 2010 02:35:25 GMT
Content-Length: 1903091
Last-Modified: Mon, 15 Mar 2010 15:00:25 GMT
Accept-Ranges: bytes
Content-Range: bytes 0-1903090/1903091
Content-Disposition: attachment; filename=Make-MusicMachines-42109-0.mp4
Server: Jetty(6.1.19)

Content-Type is missing as you suggested.

When I look at a random sample mpeg I found on the internet, I see:

GET /downloads/tomb.raider.sample.mpg HTTP/1.1
Host: www.kvcd.net
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.1.249.1036 Safari/532.5
Referer: http://kvcd.net/portal/articles.php?lng=en&pg=60
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

HTTP/1.1 200 OK
Date: Sat, 20 Mar 2010 01:43:34 GMT
Server: Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8l DAV/2 mod_auth_passthrough/2.1 FrontPage/5.0.2.2635
Last-Modified: Tue, 23 Nov 2004 08:10:04 GMT
ETag: "ae504da-999048-3e9888017cf00"
Accept-Ranges: bytes
Content-Length: 10063944
Keep-Alive: timeout=10, max=30
Connection: Keep-Alive
Content-Type: video/mpeg

The random internet download downloads as an mpeg file, and is recognized by the media player on the phone (but won't play due to wrong encoding, probably).

Followed directions at:
http://tools.assembla.com/sageplugin...ustomMimeTypes

1) Created SageTV/SageTV/jetty/user/mime.types
2) Created the Set directive you describe (copied & pasted) in SageTV/SageTV/jetty/contexts/nielm_websrv.context.xml
2) Put random garbage into the mime.types file (wasn't sure what the file format should be) to verify I had the right path -- was assuming I'd get a failure or jetty page load failure, but got no change. Random garbage currently is:

sadfasdfjsd


asdlkfjasdf

Googling, I came found Jetty documentation about setting mime types directly in the webdefault.xml file. For ex:
http://docs.codehaus.org/display/JETTY/webdefault.xml
http://account.pacip.com/jetty/doc/MimeTypes.html
http://forums.atlassian.com/thread.j...503&#257257503

Tried adding in the following to SageTV/SageTV/jetty/etc/webdefault.xml:

  <!-- ==================================================================== -->
  <!-- Default MIME mappings                                                -->
  <!-- The default MIME mappings are provided by the mime.properties        -->
  <!-- resource in the org.mortbay.jetty.jar file.  Additional or modified  -->
  <!-- mappings may be specified here                                       -->
  <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  -->
  <!-- UNCOMMENT TO ACTIVATE
  <mime-mapping>
    <extension>mysuffix</extension>
    <mime-type>mymime/type</mime-type>
  </mime-mapping>
  -->
  <mime-mapping>
    <extension>mp4</extension>
    <mime-type>video/mpeg</mime-type>
  </mime-mapping>

Replaced my garbage text in mime.types with (for lack of a better idea):

  <mime-mapping>
    <extension>mp4</extension>
    <mime-type>video/mpeg</mime-type>
  </mime-mapping>

The file I'm downloading is named "F:\sage recordings\Make-blahblah.mp4" as displayed in the sage web interface.

(Note, I don't think it matters, but this is a converted file that's 3 seconds in length which was encoded with 'iPod-Fair Quality'. I don't think this encoding will work on my Droid, but was going to figure that out after I got the file to download correctly.)

What am I missing?

Thanks
- J.

[jrjbertram]

Hmm.. of course as soon as I hit send, I get some success..

If I go back to the original file (an *.mpg) I get the right content type:

GET /sage/public/MediaFile?MediaFileId=310403&Segment=0 HTTP/1.1
Host: --edit--:--edit--
Accept-Encoding: gzip
Referer: http://--edit--/sage/DetailedInfo?MediaFileId=310403
Accept-Language: en-US
User-Agent: Mozilla/5.0 (Linux; U; Android 2.0.1; en-us; Droid Build/ESD56) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17
Cookie: grid_num_hours=6; grid_num_chans=999
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Charset: utf-8, iso-8859-1, utf-16, *;q=0.7

HTTP/1.1 200 OK
Date: Sat, 20 Mar 2010 03:29:23 GMT
Content-Type: video/mpeg
Content-Length: 807550976
Last-Modified: Mon, 15 Mar 2010 15:30:00 GMT
Accept-Ranges: bytes
Content-Range: bytes 0-807550975/807550976
Content-Disposition: attachment; filename=Make-MusicMachines-42109-0.mpg
Server: Jetty(6.1.19)

So I guess by default somewhere Jetty knows how to handle mpg files but not mp4, and whatever I was doing didn't work for the new file type.

I'll go down the route of trying to convert to a format that'll work on the droid and see how far that gets me.

- J.

[jrjbertram]

Sooo.. finally got this working with brute force, but would like to figure out the "right" way.

Details included for anyone interested.

So I dug around in the jetty jar file and found mime.properties. It has a format like:

mp2	= audio/mpeg
mp3	= audio/mpeg
mpeg	= video/mpeg
mpe	= video/mpeg
mpga	= audio/mpeg
mpg	= video/mpeg

I tried inserting something in this format for mp4 in my mime.types file, but no dice.

Soooo... I used the Java 'jar' tool to extract jetty's jar file into a temp directory (ie, 'jar -xf jetty[vernum].jar', then modified mime.properties directly inserting:

mp4	= video/mpeg

Then used jar tool to re-create the jar file ('jar -cf jetty_test.jar org META-INF') with my modified mime.properties.

Clicking on the mp4 files now more or less works. The mime type is correctly set to video/mpeg, and the phone will now download the files as *.mpeg.

(Android phones seem to be particular on their file extensions -- I tried overriding the file on the save as <file>.mp4, but it saved it as <file>.mpeg anyway... whatever..)

In any case, the file extension didn't matter.. the built in android media player was able to play the file with an mpeg file extension. For the Droid users out there, this sort of surprised me, but the iPod-Fair Quality convert setting worked on the Droid. I haven't looked at the encoder settings, but this apparently must be one of the supported codecs on Droid.

So this works pretty well if you're willing to convert files for testing, but requires that you completely download the file before playing it. I was using converted files that were 3 seconds in length, and that was maybe 500KB download depending on the file.

I think the next step is determining why the playlists are also being saved as HTML files, rather than as playlist files. Probably a mime type issue as well???

If I get the playlist issue figured out, then I believe that streaming to Droids (and possibly all Androids) should work?

- J.

[jreichen]

I didn't intend to leave you here answering all your own questions over the weekend, I was on vacation

I gave you the wiki page that app developers use to make sure their app picks up the mime types correctly. I should have given you this one that's intended for users.

Details
Jetty delivers a default set of mime types and that is what you found in the jar file. However, I found it to be somewhat limited so those are used as defaults and can be overridden in the following ways (without "surgery" ) :

1. User overrides are stored in jetty/user/mime.types. This gives the user ultimate control and is not overwritten when the plugin is upgraded.
2. The context xml for a web app can specify a mime.types file. This is mainly for compatibility for nielm's web interface to keep reading the webserver/mime.types file after a user has switched from the TJWS server to the Jetty server.
3. The plugin will at some point be delivered with some defaults. mp4 is probably a good place to start. These will be in jetty/etc/mime.types.

There is some information written to SageTV's log to help you follow which files are being loaded.

The format of the files is that used by nielm's web server (mime types first) and is opposite of Jetty's default format (file extensions first) that you found on the Internet. The plugin overrides Jetty's default and instead recognizes nielm's format for compatibility.

Try creating a mime.types file in the jetty/user folder and putting this in the file:

video/mp4  mp4

It's possible to use webdefault.xml but it will be easier for users to modify a regular text file than an xml file. Plus webdefault.xml gets overwritten every time Jetty is upgraded. The jetty/user folder does not.

To Summarize
If you find you need a new mime type, add it to jetty/user/mime.types. If it's a pretty common mapping, then report it over on the Jetty support thread and I'll make a default in the next release of the Jetty plugin.

[jrjbertram]

No problem, using this as more of a note taking of what works / doesn't.. so your replies are just bonus.

I will move my mime types to the files you recommended. I'm hitting a dead end on this route because the media player on the droid for some reason can't seem to parse or doesn't like any of the playlist files. Googled and experimented for quite a while before giving that up for now.

In other "progress" I've cobbled together something that works.. sort of the brute force approach. Would like to refine, as it doesn't transcode on the fly..

I installed sage job queue and set up the client to run some basic bat files to use handbrake to convert the files into a format suitable for my droid, and to then post process them with a tool called mp4box that adds "hints" for streaming. I then have a perl script that builds up a static page that I have jetty serving which contains rtsp:// links for each file that I've transcoded. And _then_ I have darwin streaming media server serving the videos to my phone over rtsp. (Takes breath)

So it works... downside is I have to process each video fully before it can be viewed, and then save off the file. Plus I need to build something that will prune out files using some sort of algorithm (age, whether orig sage file deleted, etc). But... its kind of hacked together.

Have also been looking into vlc, which it appears should be able to transcode on the fly similar to how its done in nielm's original stuff.. but I'm thinking ill probably need to set up vlc to stream over rtsp. Vlc's command line is complicated, and the forums are full of mixed results..... so that'll probably take a while.

How hard would it be to modify the webserver to insert special/custom rtsp links along with the file link for a given properties view of a sage recording? That'd be a convenient way to build my recording list if I can't get vlc working.

Is there an easy way to dig out the vlc command lines the webserver is currently using?

Got any other ideas?

Thanks,
-J

[jrjbertram]

Have a similar thread going on over in the VLC forum.. talks about my (lack of) success with getting VLC's "video on demand" functionality working.

http://forum.videolan.org/viewtopic....243835#p243835

Conclusion.. maybe the darwin solution I've got working is about all I'm going to get for now. But.. surely it works, because youtube can do it.

[jrjbertram]

Once I figured out how to start process explorer as administrator, I was able to fish this VLC command line out after selecting one of the transcoded streams from the web interface.

"C:\Program Files (x86)\VideoLAN\VLC\vlc.exe" -I dummy --dummy-quiet --quiet --no-sub-autodetect-file "F:\sage_recordings\Make-FireSculpture-35212-0.mpg" :sout=#transcode{vcodec=h264,vb=96,scale=0.25,acodec=mp3,ab=32,channels=2,threads=2,deinterlace}:duplicate{dst=std{access=http,mux=ts,dst=127.0.0.1:8000}}

That gives me some hints for testing. One of the things I'm not doing is opening ports 8000 on my server (or router firewall). I bet that might be my problem.

[jrjbertram]

Sooo... definitely had problem with my ports. The darwin server was taking TCP 8000 and 8001 for IceCast (I think). So updated that and verified 8000-8100 were unused. Verified that when I click the transcode link that VLC is launched, verfied with 'netstat -ao' that the PID listed matched VLC. Verified that the links work with Windows media player on the local host.

Still no dice. Noticed that when I launch from windows (tried with both chrome and IE) that Sage kicks off the VLC process. Noticed that when I click the link from the phone that the VLC process is not getting kicked off.

Started looking at the differences in the playlist. (Both wvx seemed to be recognized by both phone and windows, so started comparing those.) On windows, playlist looks like:

<asx Version = "3.0">
<title>Downloaded Playlist</title>
<entry>
<title>"Make" at Mon, Mar 8 10:00 AM on 32-2 KRINDT2</Title>
<ref href="http://localhost:8888/sage/public/MediaFileTranscode?MediaFileId=300316&Segment=0&mode=vlc&vc=mp4v&vb=256&ac=mp3&ab=96&scale=0.3&mux=ts&deint=1"/>
</entry>
</Asx>

On the android, the XML is the same, but the link is truncated to:

http://<ipaddr>:8888/sage/public/MediaFileTranscode?MediaFileId=300316&Segment=0&mode=vlc

(This was from clicking the same link [encoder, bit rate, playlist] on the web page.)

I tried plugging that link directly into the phone browser, and I get a jetty error as the result. Interesting part pasted in:

Exception while processing MediaFileTra java.lang.NumberFormatException: null java.lang.NumberFormatException: null at java.lang.Integer.parseInt(Unknown Source) at java.lang.Integer.parseInt(Unknown Source) at net.sf.sageplugins.webserver.MediaFileTranscodeServlet.doServletGet(MediaFileTranscodeServlet.java:110) at net.sf.sageplugins.webserver.SageServlet.doGet(SageServlet.java:86) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389) at

Makes sense I guess -- it looks like an invalid URL.

That begs the question though.. why is the server giving the android phone a bad URL in the first place?

Does the server build the URL within the playlist based on the browser's user agent?

When I make the full URL available to the phone, it begins downloading an mpeg file. The way the phone works, I'm unable to test that file until it's done downloading (as far as I can tell).

I feel like I'm close! In theory, figure this out, choose the right encoder and settings, and maybe this will actually work.

- Josh.

[Fonceur]

Interesting findings. I was afraid I might have to add some transcoding engine directly to SageTCPServer or such... Using VLC for the heavy lifting sounds like a much better idea.

[jrjbertram]

Its already built in to the sage webserver.. thank nielm, I guess.

[jreichen]

Thanks for the info. You might look at the request logs in the jetty/logs folder and see if there are any differences in the requests coming from the desktop and Android.

I'll see if I can reproduce it this weekend and also try to comment on some of your recent posts.

IIRC Emok did a fair share of the VLC work.

[jrjbertram]

Ill try to get some wireshark traces, too, in case there's something useful.

-J

[Fonceur]

I was able to fish this VLC command line out after selecting one of the transcoded streams from the web interface.

Regarding that VLC string, from testing it seems that 127.0.0.1 makes it available only locally, so you need to use the actual IP if you want to be able to access it from another machine, like say an Android phone... Not sure if it really makes sense, but I couldn't do vlc <--> vlc from two different pc's otherwise... Of course that doesn't help you, but if others want to test on their own...

EDIT: Which on second thought might just be related to the components I removed from XP or services I have disabled...