Notices

 SageTV Customizations This forums is for discussing and sharing user-created modifications for the SageTV application created by using the SageTV Studio or through the use of external plugins. Use this forum to discuss customizations for SageTV version 6 and earlier, or for the SageTV3 UI.

#61
01-12-2009, 11:37 PM
 megamojo Sage Advanced User Join Date: Apr 2006 Posts: 179
I found a very minor but very annoying problem tonight. It seems that leading zeros in filenames aren't preserved in the title field of the .properties file. Instead of trying to explain it perfectly, I'll just show the example of what's happening to me. I name all my TV show files as like "Heroes 1x04 Collision.avi" and after running this tool the title in the .properties file would be "Heroes 1x4 Collision." This means that in the regular Sage view episode 4 would come after episode 10, not cool. Also, is there a way that we can control what files are output? I don't really need all the different metadata filetypes and they make for lots of clutter.
#62
01-12-2009, 11:58 PM
 Halstead Sage Advanced User Join Date: Jul 2005 Location: Portland, Or Posts: 183
Is /TV/ going to stay hard coded? I currently have all of my shows in a /TV Shows/ top-level directory, and I'd hate to loose all of the 'watched' information.

Needless to say, I understand this probably isn't your top priority right now.
__________________

HTPC/Server: A8 3850; Win 7 Home x86 + Java 1.6.0_26; Sage 7.1.9; Driving Epson 8500 pj @ 1080p | Office: Core2Duo E8500 w/ EFI-X running OS X 10.5 & Win 7 Pro x64 (dual boot) + Java 1.6.0_20; Sage 7.0.23; Driving 30" Samsung 1900x1200 monitor
#63
01-13-2009, 12:51 AM
 evilpenguin SageTVaholic Join Date: Aug 2003 Location: Seattle, WA Posts: 3,696
Quote:
 Originally Posted by megamojo I found a very minor but very annoying problem tonight. It seems that leading zeros in filenames aren't preserved in the title field of the .properties file.
Did you download beta4 that I just put out tonight. It should fix this problem.
Code:
Title=Scrubs 4x03 - My New Game
Quote:
 Originally Posted by megamojo I don't really need all the different metadata filetypes and they make for lots of clutter.
I'll make it easer once i've sorted out all the technical stuff, but until they you can just make a batch file like this...

Code:
@ECHO OFF
set DIRECTORY=G:\mediaScraper

cd /D "%DIRECTORY%"
mediaScraper.exe /profile inputFile /genPropertyFile /outputFile "%~dp1%~n1" /inputFile "%~f1"
pause
Where you change G:\mediaScraper to the path to your folder. Then you can just drag and drop your files onto the batch file.

Quote:
 Originally Posted by Halstead Is /TV/ going to stay hard coded? I currently have all of my shows in a /TV Shows/ top-level directory, and I'd hate to loose all of the 'watched' information.
Bah, my bad, I meant to do this this time, I'll get it in the next beta, but in the meantime just open up
Code:
...\scrapingProfiles\inputFile.scrape
and change where it says TV[\\\/] to TV Shows[\\\/]
Code:
# Profile: Series\Season\Episode format
#
#   INPUT : %%inputFile%%    - Input file with unknown format
#

Profile           =TV_Series_Season_Episode
Encode CLI #1     =?>!showTitle||!seasonNum||!episodeNum<:>TV Shows[\\\/]+($$showTitle$$[^\\\/]+)[\\\/]+[^0-9\\\/]+($$seasonNum$$[0-9]+)[\\\/]+\s*($$episodeNum$$[0-9]{2})<?
Encoder #1        =/use "%%inputFile%%" /noOverWrite
Encode CLI #1     =?>!showTitle||!seasonNum||!episodeNum<:>TV Shows[\\\/]+($$showTitle$$[^\\\/]+)[\\\/]+[^0-9\\\/]+($$seasonNum$$[0-9]+)[\\\/]+\s*S?[0-9][xE]($$episodeNum$$[0-9]{2})<?
Encoder #1        =/use "%%inputFile%%" /noOverWrite
Encode CLI #1     =?>!showTitle||!seasonNum||!episodeNum<:>TV Shows[\\\/]+($$showTitle$$[^\\\/]+)[\\\/]+[^0-9\\\/]+($$seasonNum$$[0-9]+)[\\\/]+\s*[0-9]{1,2}($$episodeNum$$[0-9]{2})<?
Encoder #1        =/use "%%inputFile%%" /noOverWrite

# Profile: Series\SeasonXEpisode format
#
#   INPUT : %%inputFile%%    - Input file with unknown format
#

Profile           =TV_Series_SeasonXEpisode
Encode CLI #1     =?>!showTitle||!seasonNum||!episodeNum<:>TV Shows[\\\/]+($$showTitle$$[^\\\/]+)[\\\/]+.*S?($$seasonNum$$[0-9]+)[xE]($$episodeNum$$[0-9]{2})<?
Encoder #1        =/use "%%inputFile%%" /noOverWrite
Encode CLI #1     =?>!showTitle||!seasonNum||!episodeNum<:>TV Shows[\\\/]+($$showTitle$$[^\\\/]+)[\\\/]+.*($$seasonNum$$[0-9]{1,2})($$episodeNum$$[0-9]{2})<?
Encoder #1        =/use "%%inputFile%%" /noOverWrite
And that should get it working.

Last edited by evilpenguin; 01-13-2009 at 12:54 AM.
#64
01-13-2009, 12:07 PM
 joey3002 Sage Advanced User Join Date: Dec 2008 Posts: 79
Have a problem.
When I select multiple files, it only does 1 of the multiples.

I did update input.properties with correct path (TV Archive)

Attaching the log file.. hmm just read log.. its only doing 1 file of the 2 i selected... so not attaching log because nothing in it.. how can i get it to do multiple files or directories (recursive)?

Thanks and nice app..
#65
01-13-2009, 12:38 PM
 joey3002 Sage Advanced User Join Date: Dec 2008 Posts: 79
Quote:
 Originally Posted by mickp Awesome! I've adapted my old comskip batch file to process all files in a directory. The log files from this should be quite interesting. I'll give the new version a run over one of my download directories and pm the logs. In the meantime i've attached the two batch files required for bulk scraping. Hopefully someone will find them useful. Usage is scrape [unc path] Mick. Edit: Fixed a bug which would cause all shows to be processed every time. Now only files without a .properties file are scraped on subsequent runs.
Just a quick FYI on this.. if you are using long filenames, you need to do the following:

option1:
scrape.cmd "G:\my tvshows\seinfeld\season 1\"
option2:
use short filenames..
scrape.cmd g:\mytvsh~1\seinfeld\season~1\

hope this helps anyone..
#66
01-13-2009, 01:37 PM
 joey3002 Sage Advanced User Join Date: Dec 2008 Posts: 79
wow.. sorry for the many many posts in this forum today..
#67
01-15-2009, 01:25 PM
 evilpenguin SageTVaholic Join Date: Aug 2003 Location: Seattle, WA Posts: 3,696
Now that I'm mostly happy with the scrapping, I'm going to start adding new features and usability improvements. JPG downloading and dragging and dropping multiple files are at the top of my list.
#68
01-15-2009, 01:51 PM
 jaminben Sage Icon Join Date: Sep 2007 Location: Norwich, UK Posts: 1,754
Quote:
 Originally Posted by evilpenguin Now that I'm mostly happy with the scrapping, I'm going to start adding new features and usability improvements. JPG downloading and dragging and dropping multiple files are at the top of my list.
Cool, looking forward to it
__________________
Server - Win7 64bit, 2.4Ghz Intel Core 2 Duo, TBS 6284 PCI-E Quad DVB-T2 Tuner, 3 x HD200 & 1 x HD300 extenders
#69
01-15-2009, 02:00 PM
 bluenote Sage Aficionado Join Date: Nov 2004 Location: Vancouver, canada Posts: 336
Is there a "main" config file? I want to turn off the generation of the file formats that I don't need. I'm browsing the included files but so far I don't see a way to do that.

thanks for this tool,

Cory
#70
01-15-2009, 02:14 PM
 evilpenguin SageTVaholic Join Date: Aug 2003 Location: Seattle, WA Posts: 3,696
Quote:
 Originally Posted by cnovak Is there a "main" config file? I want to turn off the generation of the file formats that I don't need. I'm browsing the included files but so far I don't see a way to do that. thanks for this tool, Cory
That's also high on my todo list
#71
01-15-2009, 03:57 PM
 noobgoldberg New Member Join Date: Dec 2008 Posts: 1
Quote:
I'm not having luck with the batch processing. Individually dragging and dropping seems to work fine, but as soon as I type:

Quote:
into the command line I get:

Quote:
The file it should be looking for in the "Season 3" directory is "S03E01 - Occupation.avi" but it doesn't seem to find it. I've used an administrator command line as well to no avail. Any ideas? I'm hoping I'm just having a stupid moment and it's an easy fix. I haven't updated sageMC to the latest "b" version (still "a") but I can't imagine that would influence this in the least...other than that just an x86 Vista computer here...
#72
01-15-2009, 04:03 PM
 evilpenguin SageTVaholic Join Date: Aug 2003 Location: Seattle, WA Posts: 3,696
Try not including that trailing "\".
#73
01-16-2009, 03:38 AM
 evilpenguin SageTVaholic Join Date: Aug 2003 Location: Seattle, WA Posts: 3,696
Beta5: Now with features!

Just updated to Beta5 and, since it includes actual features, I have release notes. The big ones are in bold.

mediaScraper 1.0 beta5
• Added Drag and Drop support for multiple files and folders
• Added support for VIDEO_TS folders
• Caching webpages so when doing multiple episodes of the same show we don't hammer theTVDB.
• Skip files that already have metadata. Can be overrided with /updateInfo
• Added customizable TV basefolders. (ex: /basefolder "TV Shows")
• Added defaults.txt which you can use to configure your default options.
• Added year to movie titles (ex: 300 (2006))
• Lots of minor RegEx tweaks.

Last edited by evilpenguin; 01-25-2009 at 10:21 PM.
#74
01-16-2009, 04:44 AM
 jaminben Sage Icon Join Date: Sep 2007 Location: Norwich, UK Posts: 1,754
Quote:
You soon banged that one out

One small request that I think would benifit other people and not just me is an option to select where to place fanart images rather than placing an image in the same folder as the TV show. Or rather than naming the fanart background the same as the TV show name it "background" as mine doesnt use the fanart unless I move it or rename it to "background". Other than that it worked first time, only tried Bones so far though, will try more later.

Very Cool Tool

Thankyou

edit:

I've been playing with the tool for awhile now and have noticed some issues with certain files not working and was hoping you maybe able to get these to work as well.

Example files that do work:

burn.notice.s01e06.720p.hdtv.x264.bia
Bones.s03e11.Xvid-Orpheus
stargate.atlantis.s05e20.hdtv.xvid-preair
smallville.s08e01.720p.hdtv.x264-ctu
lost.s04e13-e14.720p.hdtv.x264-ctu
sanctuary.us.s01e13.720p.hdtv.x264-ctu

Example files that dont work:

stargate.atlantis.s05e15.720p.hdtv
Heroes.S02E11.720p.HDTV.X264-DIMENSION
reaper.s01e18.720p.hdtv.x264-ctu
Star Wars The Clone Wars S1E05 HDTV XviD-SC-SDH

As you can see there isnt that much difference between the ones that do work and others that dont. It does appear to timeout on the files that dont work giving a text line of attempt#1, attempt#2 etc. But I could be reading it wrong and I dont really know what I'm looking at anyway. I've attached my scraper log just incase you need it.

Another request I have would be to remove unwanted text from the file name (see attched screenshot). So the original title of a file is "Bones.s03e11.Xvid-Orpheus" and a cleaned up file name would look like "Bones Season 3 Episode 11 The Pain in the Heart".

I also hope that one day the extra data avaiable via a .info file could be imported into Sage as it does seem a shame to waste all that extra information seeing as its available.

Dragging of folders worked superbly for Bones and Smallville I havent tried movies yet but I'll get around to that later.

Other than that, great job

Regards

Ben
Attached Images
 Bones Example.jpg (347.3 KB, 323 views)
Attached Files
 scraper.txt (60.7 KB, 324 views)
__________________
Server - Win7 64bit, 2.4Ghz Intel Core 2 Duo, TBS 6284 PCI-E Quad DVB-T2 Tuner, 3 x HD200 & 1 x HD300 extenders

Last edited by jaminben; 01-16-2009 at 08:18 AM.
#75
01-16-2009, 12:39 PM
 Halstead Sage Advanced User Join Date: Jul 2005 Location: Portland, Or Posts: 183
EP- thought you's like to know that MediaScraper seems to be playing along just fine with DirMon.

Any ETA on a fix/workaround for shows like Termiantor: The Sarah Connor Chronicles?

Oh, and where do i put the /basefolders string? in the defaults file?

Thanks again for all the hard work!
__________________

HTPC/Server: A8 3850; Win 7 Home x86 + Java 1.6.0_26; Sage 7.1.9; Driving Epson 8500 pj @ 1080p | Office: Core2Duo E8500 w/ EFI-X running OS X 10.5 & Win 7 Pro x64 (dual boot) + Java 1.6.0_20; Sage 7.0.23; Driving 30" Samsung 1900x1200 monitor

Last edited by Halstead; 01-16-2009 at 12:47 PM.
#76
01-16-2009, 01:21 PM
 evilpenguin SageTVaholic Join Date: Aug 2003 Location: Seattle, WA Posts: 3,696
Quote:
 Originally Posted by jaminben One small request that I think would benifit other people and not just me is an option to select where to place fanart images rather than placing an image in the same folder as the TV show. Or rather than naming the fanart background the same as the TV show name it "background" as mine doesn't use the fanart unless I move it or rename it to "background". Other than that it worked first time, only tried Bones so far though, will try more later.
I wasn't really sure how to name the jpg I downloaded or where to put it so I just guessed. If you can give me some examples of what you're talking about I can get it in there pretty quickly.

Quote:
 Originally Posted by jaminben Example files that dont work: stargate.atlantis.s05e15.720p.hdtv Heroes.S02E11.720p.HDTV.X264-DIMENSION reaper.s01e18.720p.hdtv.x264-ctu Star Wars The Clone Wars S1E05 HDTV XviD-SC-SDH
All of those, minus "The Clone Wars", is a bug where I thought that 720p was the season/episode. It was an easy fix and you can download just the updated profile and put it in your scrapingProfiles folder. This is just a quick and dirty patch, I'll add something a little more robust this weekend where it tries to strip out a lot of that crap that the encoders groups put in the file name.

The Clone Wars issue is kinda a tough one. theTVDB.com won't recognize it unless its has the ":" (Star Wars: The Clone Wars). Normally I'd fix this by running it through imdb, but imdb is kicking back "Star Wars: Episode II - Attack of the Clones" instead of the TV show. I've got the same issue with "X-Files" and I'm brainstorming for a good solution.
#77
01-16-2009, 01:27 PM
 evilpenguin SageTVaholic Join Date: Aug 2003 Location: Seattle, WA Posts: 3,696
Quote:
 Originally Posted by Halstead Any ETA on a fix/workaround for shows like Termiantor: The Sarah Connor Chronicles?
Getting the metadata should be working now, but I ran into a new issue where it can't save the fanart because the name contains an illegal windows file name character (. I'll get that sorted out this weekend.

Quote:
 Originally Posted by Halstead Oh, and where do i put the /basefolders string? in the defaults file?
Unfortunately, that's not working like I want it to yet. I'll put out a fix tonight and let you know how to add it then.
#78
01-16-2009, 02:54 PM
 jaminben Sage Icon Join Date: Sep 2007 Location: Norwich, UK Posts: 1,754
Quote:
 Originally Posted by evilpenguin I wasn't really sure how to name the jpg I downloaded or where to put it so I just guessed. If you can give me some examples of what you're talking about I can get it in there pretty quickly.
I use a central folder for films and tv shows I record via sage and these are located at \\SAGEWHS\Fanart\Movies & \\SAGEWHS\Fanart\TV. I place all my tv shows in a separate share all placed in their individual folders. All I then have to do is place a background.jpg in each series folder for it to work and I don't then need to rename each file everytime I get a new one. This is a bit long winded but it works.

What would be cool is if your scraper could rename a file to make it more friendly to fanart, something like my example above. I see you've also mentioned this in your first post.

Quote:
This is the way I currently have my fanart setup and is proberly not the best way or what other people would use\like. Another thought would be to allow the user to specify where they place fanart using your default file. I don't know how that would work or even if it is workable.

Quote:
 Originally Posted by evilpenguin I'll add something a little more robust this weekend where it tries to strip out a lot of that crap that the encoders groups put in the file name.
Using your temp file I got all to work except:

stargate.atlantis.s05e15.720p.hdtv

I didn't try starwars.

Regards

Ben
__________________
Server - Win7 64bit, 2.4Ghz Intel Core 2 Duo, TBS 6284 PCI-E Quad DVB-T2 Tuner, 3 x HD200 & 1 x HD300 extenders
#79
01-16-2009, 03:07 PM
 justme Moderator Join Date: Mar 2003 Location: Greater Baltimore/Washington Metropolitan area Posts: 1,481
Very cool

Much Thanx. I'm glad to have a CLI batch scraper again.

2 requests ,please, while you're still fairly early in development...

1) Can we have the metadata files created in a user specified location(either via a settings file, but preferably a command line switch)? This is needed for files that are already archived to DVDs and network shares that lack write acccess permission/ability. I know that 2 of the .my metadata import options for SageTV allow metadata to be put in a subdir, or a common user specified dir. This allows .my files to function alot like VSfilter looks for subtitles. ie
.\metadata would put the files into a subdir of the show folder called "metadata". This is usefull for keeping clutter down in the dir your media files are stored in. With the small downside of adding a metadata subdir to each of your show directories that contain media files. Still, as an example, one subdir. added is "cleaner" than 25 .my files added to a show folder for a series that has 25 episodes.
c:\metadata is obvious where they'd go. This is usefull for the write access issues I mentioned with DVDs and certain network shares. This is a big one for me as I have many shows already archived/burnt to DVD before we started using .my files. I obviously don't wish to reburn those disc, but would love to have your tool be able to make metadata for them.

2) Can you add the ability to handle files that don't use season numbers? Instead they are numbered by absolute Episode count. An Example of something like this would be the american version of "Dragonball Z". It has 276 reqular episodes, not counting movies. So a file like "Dragonball Z-EP201.avi" would be proccessed as the 201st episode(aka "A Dark and Secret Power") instead as episode 1 of season 2(aka "A New Goal... Namek!").
I realize that this would need a switch/setting(so the app knew which way to process the file), as other use may want that same files proccessed as season2 episode 1. Still this was a nice feature in the older TV.com scraper before it was broken by site changes. I would really appreciate if you could add this functionality to you scraper, as I rarely ever use season numbers in either the directory tree structure or filename. IMHO season info is one of the many pieces of info that metadata files are for.

Apologies if your app is already capable of any of these actions. I just found this thread today and my file server is torn down for a rebuild(upgrading CPU and harddrives) this weekend. So I haven't yet gotten a chance to truely test your app of my media collection. I just did some simple tests using fake(empty text files renamed to avi) files.

Thank you for all your time and effort,
/justme

Last edited by justme; 01-16-2009 at 03:14 PM.
#80
01-16-2009, 04:22 PM
 evilpenguin SageTVaholic Join Date: Aug 2003 Location: Seattle, WA Posts: 3,696
Quote:
 Originally Posted by jaminben This is the way I currently have my fanart setup and is proberly not the best way or what other people would use\like. Another thought would be to allow the user to specify where they place fanart using your default file. I don't know how that would work or even if it is workable.
I'm all about supporting what the SageMC dev's and plug-in dev's such as yourself decide is the preferred method. So if that's what you like I'll make sure it gets in there.

Quote:
 Originally Posted by jaminben What would be cool is if your scraper could rename a file to make it more friendly to fanart, something like my example above. I see you've also mentioned this in your first post.
My current TV folder is incredibly ugly, which actually makes it good for my testing . I've already laid all the ground work for auto-organizing files, now I juts have to double and triple check my work to make sure I don't blow out peoples files.

Quote:
 Originally Posted by jaminben stargate.atlantis.s05e15.720p.hdtv
Hmm, thought I tested that one, oh well, I'll get that sorted out over the weekend.

 Tags fanart, imdb, metadata, themoviedb, thetvdb

 Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Forum Rules
 Forum Jump User Control Panel Private Messages Subscriptions Who's Online Search Forums Forums Home Information & Announcements     View the Forum Terms of Service     Announcements     SageTV Downloads & Instructions SageTV Products     SageTV Software     SageTV EPG Service     SageTV Placeshifter     SageTV Media Extender     SageTV HD Theater - Media Player     SageTV Linux         SageTV for unRAID/Docker     SageTV Mac Edition     SageTV Recorder Software SageTV Development and Customizations     SageTV Github Development     SageTV Clients Development     SageTV Studio     SageTV Customizations         SageMC Custom Interface     SageTV v7 Customizations         Batch Metadata Tools         Diamond         Sage My Movies         Phoenix         Gemstone for v7     SageTV v9 Customizations         Gemstone for v9         Phoenix 3.0     Customization Announcements     SageTV Downloads SageTV BETA Release Products     SageTV Beta Test Software Hardware Support     Hardware Support SageTV International User Forums     SageTV Australia/New Zealand     SageTV Canada     SageTV Germany     SageTV France     SageTV Netherlands     SageTV South America     SageTV United Kingdom     SageTV Europe (Miscellaneous) General Discussion     General Discussion     The SageTV Community     SageTV Marketplace

 Similar Threads Thread Thread Starter Forum Replies Last Post JREkiwi SageTV Customizations 251 10-09-2019 11:11 AM stuckless Batch Metadata Tools 2416 09-20-2011 08:03 PM loomdog32 SageMC Custom Interface 8 01-04-2009 12:34 AM mycorona SageMC Custom Interface 2 11-30-2008 01:08 PM Opus4 SageTV Studio 9 02-20-2008 06:35 PM

All times are GMT -6. The time now is 12:13 AM.

 Contact Us - SageTV, LLC - Archive - Privacy Statement - Top