TV.com automatic scraper

[roxy99]

As it turns out I was busy doing some php projects. I miss programming in Python and I miss my hobby.

So next things to look at:
1) TV com new format and broken screen scraper
2) Use sageTv show ID as a means to determine the season/episode correctly.

Bear with me I am little rusty. You guys now know the program better than me and I wrote the dang thing. I'll keep you posted.

[roxy99]

Yup upgrades on TV.com have rendered this version of the scraper obsolete. I am going through each section one at a time...

[megamojo]

Originally Posted by roxy99

2) Use sageTv show ID as a means to determine the season/episode correctly.

If this is just an additional means of detecting the season/episode that is great, but if that were to replace the existing amazing logic it would break it for me, since my files don't have the sage filenames and are just in their own import folder with names like Showname 1x03 Epname.avi for season 1, ep 3 for example. I'm eagerly awaiting your update since my viewing experience is just not the same with out it. Thanks for getting back to us!

[roxy99]

Originally Posted by megamojo

If this is just an additional means of detecting the season/episode that is great, but if that were to replace the existing amazing logic it would break it for me, since my files don't have the sage filenames and are just in their own import folder with names like Showname 1x03 Epname.avi for season 1, ep 3 for example. I'm eagerly awaiting your update since my viewing experience is just not the same with out it. Thanks for getting back to us!

And thank you for your support. I promise I won't break anything - famous last words.

[roxy99]

The changes are extensive. Every html tag is changed in the TV.com website. Also, they are now making heavy use of CSS classes. So an update may be a couple of weeks away.

I want to start thinking about version 2 of this scraper- what is your wish list for this scraper? XBMC is using a similar scraper but I don't know what features it sports. As a command line utility, what core features are missing in your opinion.

If I migrate this to gui, what should the gui be like?

These are the questions I'm asking myself.

[megamojo]

The only thing that's really bugged me so far is the inability to use it with some shows that have odd character's in their titles. Terminator: The Sarah Connor Chronicles is what I'm talking about. I tend to exchange : for - in that situation, other programs use a period instead of a colon. I'm not saying either of those is the solution, just that the issue remains that no matter what I name the folder, I've never been able to use the scraper for that show.

I'm personally a little leery of going to a gui, or if you do I would request that you still make it scriptable since that's how I use it. It drives me crazy when handy utilities like this can only be used through a gui.

[dvd_maniac]

How about returning a list of possible matches to make sure the right show is used without the need to create a showid.txt file? Kinda like IMDB search works in Sage...

[roxy99]

Originally Posted by dvd_maniac

How about returning a list of possible matches to make sure the right show is used without the need to create a showid.txt file? Kinda like IMDB search works in Sage...

The problem with that approach is then the user needs to intervene whenever the utility finds ambiguous matches. For eg: Knight Rider 1986 vs Knight Rider 2008. If the utility finds a video file named justs Knight_Rider_EpName.mpg, it has to stop in the middle and ask the user - which one you mean.

I could make it so that it does all the non-ambigious matches first and then for the ambiguous ones prompt the user, more than one shows match videname.mpg:

Press (A) Knight Rider 1986
(B) Knight Rider 2008

I guess a gui is low on the prioirity scale.

Megamo,
Even when you rename the tilte without the : it doesn't work?
Thanks

[Brent]

I thought I remembered talk about a coming TV.com API. Anyone else no anything about that? It could really help out in the development of this plugin.

[roxy99]

All the necessary plugins are there already. The my plugin in Sage is all that's needed. I only need to modify this utility to run in the background as a monitoring app from any new recordings, execute the tv.com scraper (like comskip monitor by Steep).
Unless you mean Sage is developing their own TV.com scraper to be incorporated into the core product. That would be cool then my utility would not be needed.

What I really need to add is the ability to determine the proper productionid from the Sage mpg recording and then get the exact url from TV.com to scrape the my data.

I found out why Terminator: The Sarah Connor chronicles does not parse. TV.com has the series filed under 'S' for Sarah. The utlity is expecting to find the Terminator episode in the T's.

[megamojo]

Originally Posted by roxy99

The problem with that approach is then the user needs to intervene whenever the utility finds ambiguous matches. For eg: Knight Rider 1986 vs Knight Rider 2008. If the utility finds a video file named justs Knight_Rider_EpName.mpg, it has to stop in the middle and ask the user - which one you mean.

I could make it so that it does all the non-ambigious matches first and then for the ambiguous ones prompt the user, more than one shows match videname.mpg:

Press (A) Knight Rider 1986
(B) Knight Rider 2008

I guess a gui is low on the prioirity scale.

Megamo,
Even when you rename the tilte without the : it doesn't work?
Thanks

I think asking about ambiguous matches is exactly what I would want it to do. That is much better behavior than either ignoring the file or going with the wrong information. Doing all the non-ambiguous files first and then asking would be absolutely wonderful. Another option is to make it automatic most of the time, and then if you pass it an optional parameter, it asks about every show just to be sure. This is how the automatic scraper for movies with imdb works. This makes it automatic most of the time, and selectable when needed.

[justme]

The changes are extensive... So an update may be a couple of weeks away....

Just glad to hear you are willing to keep this great tool alive. Thanks for all the effort and time you've put in to this.

I want to start thinking about version 2 of this scraper- what is your wish list for this scraper?

I'm almost certain I mentioned this request before, so please forgive me if I did. It's been a long time since my last post, though I do watch the forums almost daily. Anyway...

The thing that springs to my mind, other than TV.com to have not broken your scraper , is the ability to redirect output to a different location. Something like a simple command line parameter(-o seems logical and available) that when invoked caused the resulting ".my" files and "videolist.txt" to be placed in the specified location. Of course, if not invoked the app would act just like before.
IE. videometadata.exe -p T:\foo -o C:\bar\metadata
Would run on the files in T:\foo and instead of putting the results there, put the resulting .my files and videolist in to C:\bar\metadata. I don't need you to be able to preserve(recreate in the output location) the subdir structure, if any exists, of the scanned dir although that might be nice. I'm fine with a flat dump of all .my files and the videolist.txt going to one flat dir.

The reasons for this request is for 2 different situations I've run into. Both are really the result of the same basic issue, as you'll quickly see. 1) Running it on files I've archived/burnt off to DVD. 2) Running it on shares that lack write privileges for that account/PC, on my network.

I realize I can work around the network write privilege, but the DVD one is a Huge bummer. I've get many DVDs I burnt before I started to include .my files. I don't want to have to re-burn these DVDs just to add .my file and use discs and time, since I'm lazy. Since there are tools/plug-ins that allow metadata in subdirs and user specified dirs to be used by Sage, I just need your tool to write the metadata.

Thanx for your time,
/justme

[Halstead]

Any word on an update? I really miss this excellent utility!

[alijiwani]

bump! any word on this? I loved this scraper when it worked!

[MasonMan]

This is the item that i'm looking for. let us know if we can help somehow to get this app back working. Thanks for all the work!

MasonMan

[bialio]

Hey roxy99 - it'd be nice once this import is working again for it to look for .ts files also. Just another somewhat common video format now that the HD-PVR is out there recording shows like that!

btl.

[roxy99]

Hi guys,

Been out the scene for a long time and I don't even use SageTV anymore but I'm trying to see if I can revitalize the utility etc. I think I'll start from scratch.

I need some guidance from the users though.

What should the goal be?

The start is take a filename-> ascertain its series-> ascertain episode-> scrape from TV.com-> output to some file format.

The user interface needs a gui providing more customization instead of command line parameters.

I'll try not to drop the ball on the project again this time.
Thanks

[stuckless]

Originally Posted by roxy99

Hi guys,

Been out the scene for a long time and I don't even use SageTV anymore but I'm trying to see if I can revitalize the utility etc. I think I'll start from scratch.

I need some guidance from the users though.

What should the goal be?

The start is take a filename-> ascertain its series-> ascertain episode-> scrape from TV.com-> output to some file format.

The user interface needs a gui providing more customization instead of command line parameters.

I'll try not to drop the ball on the project again this time.
Thanks

Roxy - I'll just throw this out there.... I work on BMT (batch metadata tools) which is metadata lookup facility that currently supports imdb, themoviedb, dvdprofiler, and tvdb. You might consider writing your tv.com provider as a bmt provider and take advantage of some other things that bmt has to offer...

* Automatic metadata fetching when movies/tv is added to the collection
* Web UI for searching/browsing/editing
* You just need to provide the parser implementations, and bmt will do the searching/matching filename scraping, etc.
* Direct updating of the Sage database

Anyways, I'm just throwing it out there, since I'll never create a tv.com parser myself... but I'd happily accept some else's as an add-on

[Fuzzy]

With thetvdb's tv.com scraping, I'm not sure this is really even needed. thetvdb.com will already grab data from tv.com automatically, and then with BMT's thetvdb provider, it's pretty much taken care of.

[roxy99]

Cool. been out the loop for a while. I'll need to look at BMT.

Edit: BMT - wow! Great job. If only I knew Java. If there is any way I can assist, let me know. Just let me know what format my output needs to be in or how these tools can mesh.