STV Import: Imdb Search Interface (7 Jul 2008 v6.10)

[mike1961]

Originally Posted by cpainter

I too would love to have this feature. Not only this but theater show times as well. Just curious if other users of SageTV would be willing to pay a little extra to have all the features of IMDB, including streaming trailers, properly integrated and part of the experience.

Like Nielm mentioned. Whether its IMDB or some other service, to properly (legally) incorporate it, there are significant fees associated with doing so. Yet, it is a great feature.

CPainter - Showtimes are already available with the MovieTimes add on. It gets them from Yahoo and you can select any zip code or zip codes near you.

Mike

[bcjenkins]

Originally Posted by nielm

Ok... ill get onto it. They have taken down the old site layout that I was using to parse...

It may take a while to fix as the HTML parser may have to be re-worked, and depending on how much the layhout has changed this could be a lot of work...

And I truly hate screen scraping HTML with a passion especially on sites like IMDB that are machine generated but poorly written and do not even validate in an [X]HTML parser (882 errors, and just look at line 434!)... (I also maintain the belgian XMLTV grabber which uses screen scraping.)

Note, as mentioned previously: as this plugin is legally dubious, Sage is unlikely to make this a core feature unless they set up a licencing deal with the IMDB ($15,000/yr minimum), which is also unlikely...

Is there a separate method which could be used? http://imdbpy.sourceforge.net/

[jbuszkie]

Originally Posted by bcjenkins

Is there a separate method which could be used? http://imdbpy.sourceforge.net/

Interesting.. IMDB provides a text version of the database.. for personal non-comercial use.

http://www.imdb.com/interfaces.html#plain

I don't know how often the database is updated... but it might be worth looking into.. How would people feel about keeping a 1GB (I'm guessing uncompressed) on their HTPC machine? I know that's less than .5% for me!

[joe123]

Hi.

For about a week now, all (most) of my IMDB information fields are blank.

Has the IMDB format changed and this import no longer picks up the new format?

Anyone else having this problem?

[ke6guj]

did you read the last weeks postings in this thread? IMDB changed the layout of the webpages and nielm's webscraping of the IMDB no longer works. nielm will need to recode the IMDB plugin to work iwth the new pages.

[Zippster]

Originally Posted by nielm

Ok... ill get onto it. They have taken down the old site layout that I was using to parse...

It may take a while to fix as the HTML parser may have to be re-worked, and depending on how much the layhout has changed this could be a lot of work...

And I truly hate screen scraping HTML with a passion especially on sites like IMDB that are machine generated but poorly written and do not even validate in an [X]HTML parser (882 errors, and just look at line 434!)... (I also maintain the belgian XMLTV grabber which uses screen scraping.)

Note, as mentioned previously: as this plugin is legally dubious, Sage is unlikely to make this a core feature unless they set up a licencing deal with the IMDB ($15,000/yr minimum), which is also unlikely...

[nielm]

Originally Posted by jbuszkie

Interesting.. IMDB provides a text version of the database.. for personal non-comercial use.

http://www.imdb.com/interfaces.html#plain

I don't know how often the database is updated... but it might be worth looking into.. How would people feel about keeping a 1GB (I'm guessing uncompressed) on their HTPC machine? I know that's less than .5% for me!

It's not the space, its accessing it...
The text files have to be put into some form of database to make them actually useful.
However then there are the facts that they do not contain all the info that is in the IMDB, there are no sensible 'keys' in the text files for cross-linking, and sometimes the data in the text files is corrupt. Not to mention that the supplied programs either do not compile, do not run, or do not cope with the size that the DB has grown to since they were originally written....

(I wasted a lot of time looking at using the text files when I wanted to be 'honest' about the IMDB plugin)

For a program that does use the text files, see jmdb -- note that this requires an installation of mysql or postgresql...


Anyway... I am half way through... Searching and getting info for people now works, info for titles is in progress..

[joe123]

Originally Posted by nielm

Anyway... I am half way through... Searching and getting info for people now works, info for titles is in progress..

I did not know the details.

Thanks Nielm - your module makes SageTV for us worth while.

[mkanet]

Thank you so much Neil! As you can see, there are a lot of people who love this plugin.

I didnt realize what a pain it is to maintain this. No wonder, you didn't seem too thrilled about my idea to support imdb movie trailers (integrated into SageTV detailed info.

I can't speak for everyone else, but I would gladly donate in a heartbeat for this plugin.. I didn't realize how much I use this plugin until after it stopped working.

Originally Posted by nielm

Ok... ill get onto it. They have taken down the old site layout that I was using to parse...

It may take a while to fix as the HTML parser may have to be re-worked, and depending on how much the layhout has changed this could be a lot of work...

And I truly hate screen scraping HTML with a passion especially on sites like IMDB that are machine generated but poorly written and do not even validate in an [X]HTML parser (882 errors, and just look at line 434!)... (I also maintain the belgian XMLTV grabber which uses screen scraping.)

Note, as mentioned previously: as this plugin is legally dubious, Sage is unlikely to make this a core feature unless they set up a licencing deal with the IMDB ($15,000/yr minimum), which is also unlikely...

[mkanet]

I dont think this supports actor picture, movie cover, or the potential to support flash movie trailers.

Originally Posted by jbuszkie

Interesting.. IMDB provides a text version of the database.. for personal non-comercial use.

http://www.imdb.com/interfaces.html#plain

I don't know how often the database is updated... but it might be worth looking into.. How would people feel about keeping a 1GB (I'm guessing uncompressed) on their HTPC machine? I know that's less than .5% for me!

[jusjoken]

I am not sure if this helps at all, but my understanding (from reading this link... "http://www.avidardik.com/2007/02/27/imdb-interface-changes/") is that the old interface is still available by replacing the "www" with "former" when accessing the site.

Ken

[joe123]

Originally Posted by jusjoken

I am not sure if this helps at all, but my understanding (from reading this link... "http://www.avidardik.com/2007/02/27/imdb-interface-changes/") is that the old interface is still available by replacing the "www" with "former" when accessing the site.

Ken

So what needs to be done to use the "former" version? Or is this a temp fix in that it will eventually be dropped as well?

[JREkiwi]

Originally Posted by joe123

So what needs to be done to use the "former" version? Or is this a temp fix in that it will eventually be dropped as well?

The "former" version was the one being used by the import, and it has been dropped.

John

[joe123]

Originally Posted by JREkiwi

The "former" version was the one being used by the import, and it has been dropped.

John

Ah ok! Now it makes sense. This module is one of the best modules available for SageTV.

I hope SageTV is $giving$ Neil something for his efforts. I am willing to contribute as well.

The wify is using SageTV less now She loved being able to get film info on the spot.

[ldavis]

Is anyone else having an issue with the IMDB search. I have tried with both SageMC and the stock STV. It finds the title and cover art along with some bits of information (quotes) but most of the information is missing. I have tried this on a couple of machines with different installations and get the same result. I wonder if the format on the IMDB website has changed.

* merged *

[Addict]

Yep, I've had the same issue for a while. I figured it had something to do with the new beta code I'm using and that SageMC hasn't been updated to work with the new releases yet...and probably won't be until 1) on official Sage release comes out and 2) the guys get around to updating SageMC.

Of course, I could be wrong on my reasoning as to why it's failing...?

[othy]

Originally Posted by Addict

Yep, I've had the same issue for a while. I figured it had something to do with the new beta code I'm using and that SageMC hasn't been updated to work with the new releases yet...and probably won't be until 1) on official Sage release comes out and 2) the guys get around to updating SageMC.

Of course, I could be wrong on my reasoning as to why it's failing...?

The plugin (and the implementation of it in SageMC) used a legacy version of IMDB that the site has now discontinued. In other words, the plugin has to be corrected so that it scrapes data from the newer version of IMDB.

It's just coincidence that many people noticed this right after they updated their SageTV to the newest beta.

Tim

[nielm]

Released 29 Jul 2008 v6.8 for SageTV v6.4 (Download)

• JAR: Fixed website parsing
• STVI: for Sage 6.4 only
• STVI: Add Google search for movie trailers
• STVI: Add IMDB search to Online Services menu.

NOTE
Users of older versions of Sage, and SageMC users only need to replace the JAR file in their installations, and restart Sage.
There is no need to re-import, unless you are running 6.4 beta, and want to search for trailers


I have completely re-written the website parsing code, and hopefully made it a bit more robust.. This may result in a few problems -- if you find some, please let me know which movie causes the problem...

This re-write resulted in some horrible regular expressions... for instance, the following Java RE for matching and extracting the information for a filmography item for a person:

"^([^<]*)</a>(\\s*\\([0-9?]{4}[^)]*\\))?\\s*(\\(\\p{javaUpperCase}+\\))?(\\s*<small>.*?</small>)?(\\s*<i>\\(<a href=\"http://pro.imdb.com/[^\"]*\">.*?</a>\\)</i>)?(\\s*\\([^)]*\\))?(\\s*\\(\\d+ episodes?, [^)]*\\))?(\\s*<br/>.*)?"

It is better commented in the code, but it still gives me a headache!

[appelm]

Thanks nielm! Works great again! wow did I miss this this.


I updated data for several dvds and all worked except one. I had the following error "IMDB read failed: net.sf.sageplugins.sageimdb.DbFailureException: java.lang.StringIndexOutOfBoundsException: String index out of range: -1"

I think it may have something to do with the movie title length. The movie is "Magnificent Desolation: Walking on the Moon 3D" http://www.imdb.com/title/tt0401623/ .

Thanks again,
Marc

[cryoforever]

Thank You very much for updating this for the rest of us.. I appreciate your time and hard work.