Cayars divx encoding, cleanup, creation of embedded webpage, more!

[seamonkey420]

Well... i had always wanted to be able to encode my mpg shows i've watched to divx without opening up virtualdub or anything. In addition to encoding the files to a divx format, i also wanted to make those newly encoded/archived avi files available on my network and on my webserver.

Thus seamonkey420's divx encoding/web publishing script
i thought i'd share my results/batches with everyone else cuz i know that i've been trying to do this since i started using sagetv...
_______________

EDIT:
updated!!! now offer both virtualdub (divx avi) or windows media encoder (wmv) batches...

__________________

MPG to DIVX/WMV encoding using VirtualDub/WMencoder, integrating into SageTV / Cayars Ultimate STV!

Installation:

required programs:
-Divx codec installed (not included)
-VirtualDub installed (mpeg version, included in zip )
-SageTV
-Cayars Ultimate STV installed

where to put files
-convert.bat (put in virtual dub root folder)
-tvshows.vcf (put in virtual dub root folder also)
-all the .txt files (ie embedS.txt, embedE.txt, etc..)
-put the Encode.cmd into your cayars\external folder (ie C:\Program Files\Frey Technologies\SageTV\STVs\OriginalV2\cayars\External )

files that need to modified!
-convert.bat
you will need to change the idir and the odir parameters.
the idir is the location sagetv saves your tv recordings
the odir is the locations you want to save your encoded files to (note, this batch will create a new folder in this directory with the date stamp (mmddyyyy format))

also, if you want to have the batch search for a different file type to trigger the encoding, you can do so here too, by default i have it set to look for a .VPrj file.

-tvshows.vcf
open up virtual dub and go to the File > Load Proccessing Settings and open the tvshows.vcf file
modify your video and audio processing settings to your liking, then go to File > Save Processing Settings and save/overwrite it as tvshows.vcf

-.txt files
modify these files to your liking, i would suggest to run the script/batch first and see how things look. these files are basically templates i have my batch use to create the webapges and embed the movie into a webpage.

-Encode.cmd (put and located in C:\Program Files\Frey Technologies\SageTV\STVs\OriginalV2\cayars\External)
edit this batch, so that it points to where you have virtualdub installed.
be sure to first do the drive:
then chdir to the exact folder path of virtual dub

-then once you have all of this done, in Cayars Ultimate STV. you can add a menu item to point to this file.

Right Click in the menu/submenu you want to add the encoding option.

Choose Add New Menu Item > External Program

Then, for external program, use: cmd.exe

Then for arguments/parameters use: /C "C:\Program Files\Frey Technologies\SageTV\STVs\OriginalV2\cayars\External\Encode.cmd"
(you can copy this line, then use the Paste From Clipboard option)

Then enter the menu title/text to show: ie, Encode and Publish

Then once your done, right click again and now Save Menu Definition File.

Hopefully you should now be able to just click the option and the encoding will start automatically in teh background.!!

here's exactly what the batch does:
1. in sagetv, click the new menu item we added that points to the batch/encode script
2. this calls our convert.bat script. this script does it all!
3. it first will check for a .VPrj file in the idir
4. if it finds one, a new folder is created with the current date mmddyyyy (ie 03192005)
4. it will then go and encode all the mpg movies that have a .VPrj file using virtual dub and the processing settings we saved in the tvshows.vcf
5. once it finishes encoding the .VPrj it will go back and then create a webpage file, .html with the show name/episode (ie FamilyGuyFunnyDay-3.html) in the folder that was created (ie 03192005/FamilyGuyFunnyDay-3.html)
6. After the embedded webpage is created, it will go back and then create a main index.html file for that folder (ie 03192005\index.html)
7. After the date index is created, another index is then created, with links to all of the index.html files that might exist.
8. and finally once this is done, the batch will go back and clean up all of the files assocatied with the newly encoded divx avi (ie mpg, ccno, txt, log, etc)
9. and lastly, it will do some quick orphaned file cleanup...

heheh... :-)

the zip files contains the .txt, convert.bat, and even a copy of virtualdub (mpeg build version) .

____________________________________________________

MPG to WMV encoding using Windows Media Encoder, integrating into SageTV / Cayars Ultimate STV!
(updated 4/22/05)

Overview: Well.. its amazing what a little time, thinking, discussing, and testing can accomplish. I know that currently ajuhawk has created an application to do most of this and more, but i'm a fan of the batch file still.. so here's the latest of my batch files..

Notes: This DOES NOT handle compression in SageTV. Movies/shows may be encoded more than once if compression is used.

Installation:

required programs:
-SageTV
-Cayars Ultimate STV installed
-Windows Media Encoder 9 ( click to go to download page)
-PSP Video 9 ( http://pspvideo9.com )
-Videora ( http://videora.com )

_________

where to put files
-wme.bat (put in windows media encoder folder, ie C:\Program Files\Windows Media Components\Encoder)
-psp.prx or whatever you call your windows media encoder profile (put in windows media encoder folder, ie C:\Program Files\Windows Media Components\Encoder)
-all the .txt files (ie embedS.txt, embedE.txt, indexS.txt, etc..) (put in windows media encoder folder, ie C:\Program Files\Windows Media Components\Encoder)
-put the Encode.cmd into your cayars\external folder (ie C:\Program Files\Frey Technologies\SageTV\STVs\OriginalV2\cayars\External )

_________

files that need to modified!
-wme.bat
you will need to change the idir and the odir parameters.
the idir is the location sagetv saves your tv recordings
the odir is the locations you want to save your encoded files to (note, this batch will create a new folder in this directory with the date stamp (mmddyyyy format))

here are the parameters that can be changed/need (with examples) and what they refer to:

set idir=T:\Recorded Shows (where sagetv saves the mpgs of tv shows)
set odir=T:\Encoded Shows (where you want to create your encoded tv show webportal)
set oFile=wmv (file type you want to encode to)
set profile=psp.prx (windows media player profile to use)
set ldir=T:\Recorded Shows\Log (location of log folder, be sure to create it before running!!)
set trigger=VPrj (main trigger file )
set pdir=T:\Recorded Shows\PSP (location that PSP Video 9 looks for movies to auto-convert)

also, if you want to have the batch search for a different file type to trigger the encoding, you can do so here too, by default i have it set to look for a .VPrj file.

-.txt files
modify these files to your liking, i would suggest to run the script/batch first and see how things look. these files are basically templates i have my batch use to create the webapges and embed the movie into a webpage.

-Encode.cmd (put and located in C:\Program Files\Frey Technologies\SageTV\STVs\OriginalV2\cayars\External)
edit this batch, so that it points to where you to where the wme.bat is saved (C:\Program Files\Windows Media Components\Encoder\wme.bat)

-then once you have all of this done, in Cayars Ultimate STV.
you can add a menu item to point to this file.
-Right Click in the menu/submenu you want to add the encoding option.
-Choose Add New Menu Item > External Program
-Then, for external program, use: cmd.exe
-Then for arguments/parameters use: /C "C:\Program Files\Frey Technologies\SageTV\STVs\OriginalV2\cayars\External\Encode.cmd"
(you can copy this line, then use the Paste From Clipboard option)
-Then enter the menu title/text to show: ie, Encode and Publish
-Then once your done, right click again and now Save Menu Definition File.
Hopefully you should now be able to just click the option and the encoding will start automatically in teh background.!!

________

here's exactly what the batch does:
1. in sagetv, click the new menu item we added that points to the batch/encode script
2. this calls our wme.bat script. this script does it all!
3. first it creates a new folder is created with the current date mmddyyyy (ie 03192005)
4. it then will check for a .VPrj file in the idir
5. if it finds one, it will then check the log file folder for a log, if not
6. it will then go and encode all the mpg movies that have a .VPrj file using windows media encoder and the profile we saved in the psp.prx
7. once it finishes encoding the .VPrj it will go back and then create a webpage file, .html with the show name/episode (ie FamilyGuyFunnyDay-3.html) in the folder that was created (ie 03192005/FamilyGuyFunnyDay-3.html)
8. After the embedded webpage is created, it will go back and then create a main index.html file for that folder (ie 03192005\index.html)
9. After the date index is created, another index is then created, with links to all of the index.html files that might exist.
10. and finally once this is done, the batch will go back and clean up all of the files assocatied with the newly encoded wmv (ie , ccno, txt, log, etc) if it successfully encoded. And also move the mpg to the psp folder (for psp video to auto-convert later).
However, if encoding of the mpg failed, the original mpg is left alone.
11. then it will do some quick orphaned file cleanup...


let me know what everyone thinks... hehe... :-)
peace
seamonkey420

[Enormous]

Brilliant! I can't wait to get this installed!

[seamonkey420]

oh yea eric...

i'd recommend to encode the files smaller, mine were like 200mbs and took about 4 minutes to load over our 802.11g network.. so realworld i'd say you would want to get down to about 50-100mbs (dialupers, oh well..)..

what bitrate do you rip your mpg to divx at?
i think i'm using like 500kbs or so...

hope you find it useful!
peace
seamonkey420

[seamonkey420]

Well.. after doing some more work and searching, i've decided to now convert this batch file to use windows media encoder vs virtual dub.. since .wmv files stream WAY better than divx avi's..

the requirements:
-Windows Media Encoder 9 installed
-Cayars17 STV (if you want to be able to call from sagetv) or dynamic menus

the instructions:

-put alll of the files (wme.bat, embedE.txt, etc) except the encode.cmd in your windows media encoder directory
usually, C:\Program Files\Windows Media Compopents\Encoder

-put the encode.cmd into your Cayars external folder:
( C:\Program Files\Frey Technologies\SageTV\STVs\OriginalV2\cayars\External )

-edit the wme.bat in the C:\Program Files\Windows Media Compopents\Encoder folder.

set idir=T:\Recorded Shows
set odir=T:\Encoded Shows
set oFile=wmv

again, i'd recommend testing non-live folders to prevent bad things...
also i'd suggest to call the bat file from a cmd/command prompt window to see whats happening if things don't work correctly..

be sure to modify the idir and odir parameters (idir = where mpgs are, odir = where you want wmvs saved to; creates date folder, then puts files in that folder, oFile = filetype to encode (makes flexible so you can use with virtual dub or anyther encoder that takes commandline parameters)

you can edit how your web pages look by modifyiing the text files, the batch file just uses the 'type' command to echo the text in their to a webpage with the same name as the file...

also, the included tvshows.prx is my profile i use for ripping.... my original mpg files are about 1gb in size, it compresses them down to a 40mb wmv file (sweet eh!) and allows for me to embed the video into a webpage and have it viewable on the internet through a highspeed connection.. encodes to 480x360 size, encoding takes a bit of time though since i go for max compression (about an hour for a 30min video..) but does not cripple my system; i'm still able to do whatever i want and have sagetv record in teh back ground.. avg bitrate of 131 K bits/second..

see my above note about incorporating it into cayars17 stv...

also i've tried to add as many comments in the batch file to explain the sections... and am in the process of making more parameters available to make modifying it easier!..


peace
seamonkey420

[nielm]

I would just like to not for anyone that has not found this out yet that Sage cannot play back WMV's itself...

[ajuhawk]

great work!! now to get it integrated into nielm's webserver...

[Dorien]

I must be a complete noob. Or at least feel like one. From the direction I am unclear where I actually am suppose to place all the *.txt files.

Also, when I run the command with in SAGE (per the directions) the cmd promt flashes but I don't notice it actually doing anything. I think this my be related to where I have those txt files at or I just fat fingured something.

Thanks for any help you guys can provide.

[abasu2003]

maybe also not so clear is that you have to install Windows Media Encoder (from teh windows site) which is different from Windows Media Player. Then you will have a C:\Program Files\Windows Media Encoder directory

[seamonkey420]

Originally Posted by Dorien

I must be a complete noob. Or at least feel like one. From the direction I am unclear where I actually am suppose to place all the *.txt files.

Also, when I run the command with in SAGE (per the directions) the cmd promt flashes but I don't notice it actually doing anything. I think this my be related to where I have those txt files at or I just fat fingured something.

Thanks for any help you guys can provide.

yea, sorry about the vagueness at times of directions..

anywho, i'm heading out of town tonight, but i'll post some more info sun nite.. until then.. can you post what your folder locations are for where you save your movies to (.mpg files, where sagetv saves movie files), and where you want to save the encoded files..
i can modify my bat to work for ya...

however, on a side note.. my latest .prx profile i've been using does crazy awesome compression (990 mb to 40mb) but takes like 1 1/2 hours for each 990mb movie file.. however i get 480 x 360, 195kbit/s (variable bit rate, peak at 300 kbit/s) and it streams well from any high speed location... but i still need to work on getting the encoding time down.. anyone care to post their prx profiles?

peace
seamonkey420

[seamonkey420]

Originally Posted by ajuhawk

now to get it integrated into nielm's webserver...

yea!!! i know, that would be so sweeet!!! i've been thinking of a way but the only thing i can think of trying is to edit the nielm config file to create a menu that would link to the main tvshows page i have my batch create.. but it sure would be nice to be able to view them right from the recordings tab.. hmm... i am also planning on incorporating the date/time stamp thingy i saw in another post incase wmv support is figured out.. just in case..
but if i ever get my lazy butt moving and learn to code, i'll try to create an app to do all of this.. been playing with the media encoder sdk so we'll see.. hehe..

peace
seamonkey420

[ptaylor]

I'm using the original version you posted to create DIVX files and it seems to work well... But, it doesn't seem to integrate as well as I'd like with Sage...

I was hoping that the info about the individual video would be preserved, so it still has the episode description and the "pretty" title that Sage shows in the Video Library...

I guess what I would really rather have is the same functionality that Sage gives me today (with Cayars, of course), except automatically converting MPEG files that are left on the machine for more than a few days to DIVX videos instead...

Paul

[Dorien]

Well I got it working after playing around and looking at all the info, both in this thread and in the bat files. Didnt get a clean video from divx but worked fine with the wmv. Which is fine because that is what I would perfer for streaming as well. I just had a few questions for anyone that might be able to help.

1) like the other poster I would like for it to pull the more user friendly titles. like maybe eps #. Don't know how possible that is but it would be nice.
2)what are the possiblities of having the bat file run in the active record folder, so I don't have to manually move them, and have it only process the files that are older than a preset time? Or ones that have a watched flagged on them. Or have the option to have it go only after files that are not actively being used (created/watched).
3)I would also like the option for it not to do file clean-up afterwards. I may still want to watch the show on the TV and manual cleanup myself.
4) is there a way to have the folder name that it creates something other than the date? can it make folder names of the show instead? I would like to group and sort by show title than by date that it was created within the html page.

What my goal is, is to have task schuduler run in the middle of the night and process that days shows. Best case senerio would be to have it only process shows that I havent already watched. So the next day they will be available for viewing. Of course I might also want to still process shows that I have already watch just to archive them. I figure this could be done with a unmodified bat file.

Anyway, want to say thanks and this is awesome work Two thumbs up!

[ajuhawk]

I've almost got a modified batch file that'll do some (not all) of what you're looking for...I'm in my final testing run now and will post once I know it's working...

Originally Posted by Dorien

Well I got it working after playing around and looking at all the info, both in this thread and in the bat files. Didnt get a clean video from divx but worked fine with the wmv. Which is fine because that is what I would perfer for streaming as well. I just had a few questions for anyone that might be able to help.

1) like the other poster I would like for it to pull the more user friendly titles. like maybe eps #. Don't know how possible that is but it would be nice.
2)what are the possiblities of having the bat file run in the active record folder, so I don't have to manually move them, and have it only process the files that are older than a preset time? Or ones that have a watched flagged on them. Or have the option to have it go only after files that are not actively being used (created/watched).
3)I would also like the option for it not to do file clean-up afterwards. I may still want to watch the show on the TV and manual cleanup myself.
4) is there a way to have the folder name that it creates something other than the date? can it make folder names of the show instead? I would like to group and sort by show title than by date that it was created within the html page.

What my goal is, is to have task schuduler run in the middle of the night and process that days shows. Best case senerio would be to have it only process shows that I havent already watched. So the next day they will be available for viewing. Of course I might also want to still process shows that I have already watch just to archive them. I figure this could be done with a unmodified bat file.

Anyway, want to say thanks and this is awesome work Two thumbs up!

[ajuhawk]

Ok, here's a modified bat file that does more what I invision. I still have some work that I'm going to do to make it do more of what I want, but more on that in a bit.

Mods:
1) The batch file does not delete your original mpg file so that you can still watch and delete this through sagetv (simply a @REM addition).
2) Made it so that the batch file uses the files (.mpg.xml) created when you say that you want a file to be compressed using sagetv (this can be found on the detailed info for a particular show - cayars STV). As such you need to make sure that you set up the jobs folder prior to trying this. Also you need to edit this batch file (as before) with an additonal folder location of your jobs folder. Once the batch file has processed all of the .mpg.xml files in the jobs folder it will cleanup those files and post the web links as before.

Future work:
1) the next item I plan to add is a command to run videoRedo prior to encoding so that all the commercials are removed in the encoded version. I may do that this weekend...
2) as previously mentioned I'd like to see this incorporated into Nielm's webserver...I'm not sure how much help I can be here but I'm going to keep throwing this idea out. This would also fix the previous comment about wanting the output files sorted by show instead of date encoded.
3) we'll see...

Hope this is at least useful to someone else - let me know if you find bugs in what I did.

-Adam

Edit: found a bug - file reuploaded with fix.

[seamonkey420]

Originally Posted by ajuhawk

Ok, here's a modified bat file that does more what I invision. I still have some work that I'm going to do to make it do more of what I want, but more on that in a bit.

Mods:
1) The batch file does not delete your original mpg file so that you can still watch and delete this through sagetv (simply a @REM addition).
2) Made it so that the batch file uses the files (.mpg.xml) created when you say that you want a file to be compressed using sagetv (this can be found on the detailed info for a particular show - cayars STV). As such you need to make sure that you set up the jobs folder prior to trying this. Also you need to edit this batch file (as before) with an additonal folder location of your jobs folder. Once the batch file has processed all of the .mpg.xml files in the jobs folder it will cleanup those files and post the web links as before.

Future work:
1) the next item I plan to add is a command to run videoRedo prior to encoding so that all the commercials are removed in the encoded version. I may do that this weekend...
2) as previously mentioned I'd like to see this incorporated into Nielm's webserver...I'm not sure how much help I can be here but I'm going to keep throwing this idea out. This would also fix the previous comment about wanting the output files sorted by show instead of date encoded.
3) we'll see...

Hope this is at least useful to someone else - let me know if you find bugs in what I did.

-Adam

Edit: found a bug - file reuploaded with fix.

cool..
i'm going to check this out on my laptop.. i like where all of this is going!..
looks like, in advertantly, i'm getting myself ready for a sony psp too.. hehe.. i'm think that wmv files would be perfect for it and i could fit so many ones on a 512mb memory duo pro .. hehe..

peace
seamonkey420

[ajuhawk]

Just to let ya'll know, I've talked myself out of adding videoRedo automation to the batch file. So if you were eagerly anticipating that, let me or seamonkey know and I'm sure one of us can get it put in there fairly easily.

Reasoning - after looking at some of the files I want to keep, I decided that I could probably take the extra time to assure that the commercial cuts were spot on and not risk losing show.

-Adam

[Dorien]

Well I got the app working perfectly with just the right bitrate that I wanted. I also finnally figure out how to server it useing webserve from Nielm. Took me a while since I'm no expert at either java or html. But I now have a stream link in the Neilm webserve page that links to the shows_index.html. Works perfectly.

Only one thing. My girlfriend would like to know if I could make the background of the the html files it creates black and the text white or yellow. Of course I can't so I asking help from the experts.

The rest of the stuff Im going to have to make batch files that are executed threw task sch. to make the whole process more automated. If I come up with anything that I think is not too crappy I will share with you guys.

ajuhawk,
havnent had a chance to try out that batch yet. Though I am looking forward to trying it tomorrow. I wanted to get everything working first and made sure it was working before I started tweaking.

Thanks all for the effort in this thread. Its just such a kewl add-on! Night all!

[seamonkey420]

nice! i was going to look into doing that later on here tonight.. hehe..

but to get the backgrounds of the pages and so on changed.

just modify the
emdedS.txt
and the
indexS.txt
files with html code for bgcolor and text, and so, like so

replace the <body>
with
<body bgcolor='blue' text='white' vlink='red' alink='yellow'>

this would make the background color blue, the main text color white, visited links red, and active links yellow.

its pretty flexible and convenient in that the changes you make apply to all the pages..



another note, got another updated batch file for wme, use the same directions... used ajuhawk's kind of thinking and keeping of original mpg..
-this version now creates a log file with the name of the movie.log (ie TheSimpsonsHomerRules-9329-0.log) if it sees any .wmv or the filetype you use for oFile in the encoded directory.
-if there is is not log file and there is a .trigger file in the recorded shows folder, it will then encode the mpg movie and do teh usual moving, creating and so on..
-also, i removed the delete of the mpg since it is nice to be able to just do that in sagetv

install notes:
-to use Windows Media Encoder, be sure to install it first! (download here!
-create a Windows Media Profile (.prx)
-put the embedE.txt, embedS.txt, indexE.txt, indexS.txt, and wme.bat into the windows media encoder folder (C:\Program Files\Windows Media Components\Encoder)
-also put the .prx profile in the windows media encoder folder (C:\Program Files\Windows Media Components\Encoder)
-Be sure to set Parameters in the wme.bat to your system/config!
idir = the directory that sageTV saves mpg recordings to
odir = folder where wmv files will be saved and webpages
oFile = File extension to be outputted, .wmv by default
ldir = Log file directory
trigger = file to check for to see if programs are done recording/viewed


i've started to use parameters as much as possible anymore to allow for more flexibility.

peace
seamonkey420

[jwaters]

OK, I've added a few lines to automate the removal of commercials prior to processing the WMV... I downloaded the trial of VideoReDo, and based on my results, I'll probably purchase...

I'm running this at night, and putting the WMV's on a 1GB SD Card which I can pop into my PocketPC. I'm not too worried about 100% accuracy in for commercial removal, the automatic detection by comskip works good enough for my purposes.

Anyway, here's the line that removes commercials:

for /f "tokens=1 delims=." %%i in ('dir /s/b "%idir%\*-?.VPrj"') do if not EXIST "%%i-Trimmed.mpg" (cscript //nologo "C:\Program Files\VideoReDo\vp.vbs" "%%i.VPrj" "%%i-Trimmed.mpg")

And my batch file -- this has been modified with my directories, and to delete the VPRJ & trimmed files, but not the original MPG's. This way, it can be automated to run each night without duplicating anything.

[Dorien]

seamonkey420,
Does this new logging prevent it from copying movie files that have already been compressed? Was a little unclear about this part.

Originally Posted by seamonkey420

nice! i was going to look into doing that later on here tonight.. hehe..

but to get the backgrounds of the pages and so on changed.

just modify the
emdedS.txt
and the
indexS.txt
files with html code for bgcolor and text, and so, like so

replace the <body>
with
<body bgcolor='blue' text='white' vlink='red' alink='yellow'>

this would make the background color blue, the main text color white, visited links red, and active links yellow.

its pretty flexible and convenient in that the changes you make apply to all the pages..



another note, got another updated batch file for wme, use the same directions... used ajuhawk's kind of thinking and keeping of original mpg..
-this version now creates a log file with the name of the movie.log (ie TheSimpsonsHomerRules-9329-0.log) if it sees any .wmv or the filetype you use for oFile in the encoded directory.
-if there is is not log file and there is a .trigger file in the recorded shows folder, it will then encode the mpg movie and do teh usual moving, creating and so on..
-also, i removed the delete of the mpg since it is nice to be able to just do that in sagetv

install notes:
-to use Windows Media Encoder, be sure to install it first! (download here!
-create a Windows Media Profile (.prx)
-put the embedE.txt, embedS.txt, indexE.txt, indexS.txt, and wme.bat into the windows media encoder folder (C:\Program Files\Windows Media Components\Encoder)
-also put the .prx profile in the windows media encoder folder (C:\Program Files\Windows Media Components\Encoder)
-Be sure to set Parameters in the wme.bat to your system/config!
idir = the directory that sageTV saves mpg recordings to
odir = folder where wmv files will be saved and webpages
oFile = File extension to be outputted, .wmv by default
ldir = Log file directory
trigger = file to check for to see if programs are done recording/viewed


i've started to use parameters as much as possible anymore to allow for more flexibility.

peace
seamonkey420