Original Air Date scraping

[evilpenguin]

Since it's prolly related, its worth noting i'm still having my issue where BMT fails to scrape Sage Recordings with no episode title but the proper original airdate.

[stuckless]

Originally Posted by evilpenguin

Since it's prolly related, its worth noting i'm still having my issue where BMT fails to scrape Sage Recordings with no episode title but the proper original airdate.

Have you tried this since I updated the phoenix API last night?

If so then could you post the filename and I'll give it a shot.

[evilpenguin]

Ahh, I didn't see that tucked away in the Plugin->Library. I'll check it tonight.

[ohpleaseno]

The automatic plugin seems to be scraping one day higher than the OAD. For instance, a show that the file name is 2010-11-30 gets metadata from 2010-12-01

2010-12-01 03:46:34,531 - AUTO; MEDIA; \\WHs\dumpster\done\Talk Shows\Chelsea Lately\Chelsea Lately 2010-11-30.avi; Chelsea Lately 2010-11-30; 3550756
2010-12-01 03:46:38,363 - UPDATE; \\WHs\dumpster\done\Talk Shows\Chelsea Lately\Chelsea Lately 2010-11-30.avi; Chelsea Lately; TV; S4; E183; MF3550756; 3550756

Episode 183 is the 12/1/2010 episode.

[evilpenguin]

Ok, at first it looks like that succeeded, but checking the logs it looks like BMT is using the date the shows were recorded rather than the original airing date.

Here's the log for a show and the actual original airdate is 6/16/10...

2010-12-01 10:40:00,279 [Thread-194] INFO  sagex.phoenix.metadata.search.SearchQueryFactory - Applying MediaTitle information: FileMatcher [fanart=null, file=null, fileRegex=[\\/]Tosh0-, mediaType=TV, metadata=sagex.phoenix.metadata.search.ID@1a07336, title=Tosh.0, year=null]
2010-12-01 10:40:00,280 [Thread-194] INFO  sagex.phoenix.metadata.MetadataManager - search(): tvdb; SearchQuery; Type: TV; YEAR:0;ID:97731;EPISODE_TITLE:;CLEAN_TITLE:Tosh.0;RAW_TITLE:Tosh.0;EPISODE_DATE:2010-11-23;FILE:\\Secretivo\Data2\SageRecordings\Tosh0-8094769-0.mpg;PROVIDER:tvdb;
2010-12-01 10:40:00,280 [Thread-194] INFO  sagex.phoenix.metadata.MetadataManager - Searching: Tosh.0 using TVDBMetadataProvider[MetadataProviderInfo [id=tvdb, fanartId=null, mediaTypes=[TV], name=thetvdb.com]]
2010-12-01 10:40:00,282 [Thread-194] INFO  sagex.phoenix.metadata.provider.tvdb.TVDBItemParser - TVDB Series: http://www.thetvdb.com/api/5645B594A3F32D27/series/97731/en.xml
2010-12-01 10:40:00,349 [Thread-194] INFO  sagex.phoenix.metadata.provider.tvdb.TVDBItemParser - TVDB date: http://thetvdb.com/api/GetEpisodeByAirDate.php?apikey=5645B594A3F32D27&seriesid=97731&airdate=2010-11-23
2010-12-01 10:40:00,829 [Thread-194] WARN  sagex.phoenix.metadata.provider.tvdb.TVDBItemParser - Can't do lookup by season/epsidoe for season: null; episode: null
2010-12-01 10:40:00,830 [Thread-194] WARN  sagex.phoenix.metadata.provider.tvdb.TVDBItemParser - Failed while parsing series: MediaSearchResult [extraArgs=CLEAN_TITLE: Tosh.0,ID: 97731,YEAR: 0,RAW_TITLE: Tosh.0,FILE: \\Secretivo\Data2\SageRecordings\Tosh0-8094769-0.mpg,PROVIDER: tvdb,EPISODE_DATE: 2010-11-23,, id=97731, imdbId=null, metadata=null, providerId=tvdb, score=1.0, title=Tosh.0, type=TV, url=null, year=0]

Edit: I'm guessing this explains ohpleaseno's thing too since he probably downloaded/imported the show the next day.

[stuckless]

Originally Posted by ohpleaseno

The automatic plugin seems to be scraping one day higher than the OAD. For instance, a show that the file name is 2010-11-30 gets metadata from 2010-12-01

2010-12-01 03:46:34,531 - AUTO; MEDIA; \\WHs\dumpster\done\Talk Shows\Chelsea Lately\Chelsea Lately 2010-11-30.avi; Chelsea Lately 2010-11-30; 3550756
2010-12-01 03:46:38,363 - UPDATE; \\WHs\dumpster\done\Talk Shows\Chelsea Lately\Chelsea Lately 2010-11-30.avi; Chelsea Lately; TV; S4; E183; MF3550756; 3550756

Episode 183 is the 12/1/2010 episode.

I'm not quite sure about this. I simply pass a date into tvdb and let it resolve the show info. I'm guessing the show much also be incorrect on tvdb??

Originally Posted by evilpenguin

Ok, at first it looks like that succeeded, but checking the logs it looks like BMT is using the date the shows were recorded rather than the original airing date.

Here's the log for a show and the actual original airdate is 6/16/10...

2010-12-01 10:40:00,279 [Thread-194] INFO  sagex.phoenix.metadata.search.SearchQueryFactory - Applying MediaTitle information: FileMatcher [fanart=null, file=null, fileRegex=[\\/]Tosh0-, mediaType=TV, metadata=sagex.phoenix.metadata.search.ID@1a07336, title=Tosh.0, year=null]
2010-12-01 10:40:00,280 [Thread-194] INFO  sagex.phoenix.metadata.MetadataManager - search(): tvdb; SearchQuery; Type: TV; YEAR:0;ID:97731;EPISODE_TITLE:;CLEAN_TITLE:Tosh.0;RAW_TITLE:Tosh.0;EPISODE_DATE:2010-11-23;FILE:\\Secretivo\Data2\SageRecordings\Tosh0-8094769-0.mpg;PROVIDER:tvdb;
2010-12-01 10:40:00,280 [Thread-194] INFO  sagex.phoenix.metadata.MetadataManager - Searching: Tosh.0 using TVDBMetadataProvider[MetadataProviderInfo [id=tvdb, fanartId=null, mediaTypes=[TV], name=thetvdb.com]]
2010-12-01 10:40:00,282 [Thread-194] INFO  sagex.phoenix.metadata.provider.tvdb.TVDBItemParser - TVDB Series: http://www.thetvdb.com/api/5645B594A3F32D27/series/97731/en.xml
2010-12-01 10:40:00,349 [Thread-194] INFO  sagex.phoenix.metadata.provider.tvdb.TVDBItemParser - TVDB date: http://thetvdb.com/api/GetEpisodeByAirDate.php?apikey=5645B594A3F32D27&seriesid=97731&airdate=2010-11-23
2010-12-01 10:40:00,829 [Thread-194] WARN  sagex.phoenix.metadata.provider.tvdb.TVDBItemParser - Can't do lookup by season/epsidoe for season: null; episode: null
2010-12-01 10:40:00,830 [Thread-194] WARN  sagex.phoenix.metadata.provider.tvdb.TVDBItemParser - Failed while parsing series: MediaSearchResult [extraArgs=CLEAN_TITLE: Tosh.0,ID: 97731,YEAR: 0,RAW_TITLE: Tosh.0,FILE: \\Secretivo\Data2\SageRecordings\Tosh0-8094769-0.mpg,PROVIDER: tvdb,EPISODE_DATE: 2010-11-23,, id=97731, imdbId=null, metadata=null, providerId=tvdb, score=1.0, title=Tosh.0, type=TV, url=null, year=0]

That is correct, in that I set the OAD to be the date/time on the recording file. I don't use the AOD that in the metadata, since it might wrong. ie, if recording was previously updated with the wrong data, then by using what is stored with the item, then i'd continue to fetch the wrong data based on what was previously stored.... Maybe I need to update this logic and for new recordings, simply using the OAD in the recording.

[evilpenguin]

Originally Posted by stuckless

That is correct, in that I set the OAD to be the date/time on the recording file. I don't use the AOD that in the metadata, since it might wrong. ie, if recording was previously updated with the wrong data, then by using what is stored with the item, then i'd continue to fetch the wrong data based on what was previously stored.... Maybe I need to update this logic and for new recordings, simply using the OAD in the recording.

For TV you should be using the OAD 100% of the time. The way you're describing will only work if the show was recorded on the same day it originally aired and there are many common reasons why that won't be the case.

This also explains ohpleaseno's thing above. He downloaded/imported the show the day after it aired so i'm guessing if he checks the log BMT won't be using the date from the filenname (11-30-2010) but the date it was imported into Sage (12-01-10).

[stuckless]

Originally Posted by evilpenguin

For TV you should be using the OAD 100% of the time. The way you're describing will only work if the show was recorded on the same day it originally aired and there are many common reasons why that won't be the case.

This also explains ohpleaseno's thing above. He downloaded/imported the show the day after it aired so i'm guessing if he checks the log BMT won't be using the date from the filenname (11-30-2010) but the date it was imported into Sage (12-01-10).

That might be case for ohpleaseno... but I think the scraper overrides the file's date/time.

I'll look at updating the processing of recordings to use the OAD in the recording and not the file date/time. This will open another can of worms... but I'll deal with those later.

[adone36]

I mentioned this previously. If you d/l a tv file, BMT uses the file creation time as the record date. The work around is to sort your recordings by OAD to keep them in proper order. I can't think of a reason to use anything but the OAD. If I d/l a show, it is because I either missed the recording or the show is not available on my provider. If I import into Sage recordings, I would want them in broadcast order.

[evilpenguin]

Hey, FYI, found a new bug that's prolly gonna start popping up soon. If you record an older episode of a TV show the same day as a first run of the same show they all get scrapped as the first run.

Happened to me today where two reruns got scrapped with the same info as the first run that aired tonight.

[stuckless]

Originally Posted by evilpenguin

Hey, FYI, found a new bug that's prolly gonna start popping up soon. If you record an older episode of a TV show the same day as a first run of the same show they all get scrapped as the first run.

Happened to me today where two reruns got scrapped with the same info as the first run that aired tonight.

I forget to fix this in the last build to use the aod of the recording

[phelme]

Might this explain why two recent recording of HBO's "In Treatment" are being renamed to the immediately previous airing? In the example below, "Adele: Week Seven" is being renamed to "Jesse: Week Seven":

2010-12-07 19:01:03,468 [Timer-12] INFO  sagex.phoenix.metadata.MetadataManager - search(): tvdb; SearchQuery; Type: TV; FILE:\\sagetv-server\BigMedia2-Recordings\InTreatment-AdeleWeekSeven-5590329-0.ts;YEAR:0;EPISODE_TITLE:Adele: Week Seven;RAW_TITLE:In Treatment;EPISODE_DATE:2010-12-07;CLEAN_TITLE:In Treatment;
2010-12-07 19:01:03,468 [Timer-12] INFO  sagex.phoenix.metadata.MetadataManager - Searching: In Treatment using TVDBMetadataProvider[MetadataProviderInfo [id=tvdb, fanartId=null, mediaTypes=[TV], name=thetvdb.com]]
2010-12-07 19:01:03,468 [Timer-12] INFO  sagex.phoenix.metadata.provider.tvdb.TVDBSearchParser - TVDB Search: http://www.thetvdb.com/api/GetSeries.php?seriesname=In+Treatment
2010-12-07 19:01:03,483 [Timer-12] INFO  sagex.phoenix.util.url.CachedUrl - Removing Cached Url File: cache\url\8863fbea2b3133b28fff92be44d58fe8.cache
2010-12-07 19:01:06,608 [Timer-12] INFO  sagex.phoenix.metadata.MetadataManager - Fetching Metadata for MediaSearchResult [extraArgs=CLEAN_TITLE: In Treatment,YEAR: 0,EPISODE_TITLE: Adele: Week Seven,RAW_TITLE: In Treatment,FILE: \\sagetv-server\BigMedia2-Recordings\InTreatment-AdeleWeekSeven-5590329-0.ts,EPISODE_DATE: 2010-12-07,, id=81248, imdbId=null, metadata=null, providerId=tvdb, score=1.0, title=In Treatment, type=TV, url=81248, year=2008]
2010-12-07 19:01:06,608 [Timer-12] INFO  sagex.phoenix.metadata.provider.tvdb.TVDBSeriesParser - TVDB Series: http://www.thetvdb.com/api/5645B594A3F32D27/series/81248/en.xml
2010-12-07 19:01:06,624 [Timer-12] INFO  sagex.phoenix.util.url.CachedUrl - Removing Cached Url File: cache\url\3cd2e4f3df32f77ca858fb4e7cb45882.cache
2010-12-07 19:01:07,187 [Timer-12] INFO  sagex.phoenix.metadata.provider.tvdb.TVDBSeriesParser - TVDB Actors: http://www.thetvdb.com/api/5645B594A3F32D27/series/81248/actors.xml
2010-12-07 19:01:07,202 [Timer-12] INFO  sagex.phoenix.util.url.CachedUrl - Removing Cached Url File: cache\url\33334834d3370e0e127d12961041ee3d.cache
2010-12-07 19:01:07,265 [Timer-12] INFO  sagex.phoenix.metadata.provider.tvdb.TVDBItemParser - TVDB date: http://thetvdb.com/api/GetEpisodeByAirDate.php?apikey=5645B594A3F32D27&seriesid=81248&airdate=2010-12-07
2010-12-07 19:01:07,296 [Timer-12] INFO  sagex.phoenix.metadata.provider.tvdb.TVDBItemParser - TVDB Episode: http://www.thetvdb.com/api/5645B594A3F32D27/series/81248/default/3/27/en.xml
2010-12-07 19:01:07,296 [Timer-12] INFO  sagex.phoenix.util.url.CachedUrl - Removing Cached Url File: cache\url\4e7cd65acdbe737107f5da9f636b8719.cache
2010-12-07 19:01:07,937 [Timer-12] INFO  sagex.phoenix.metadata.MetadataManager - Saving Metadata for SageMediaFile [sageId=5774524, sageObject=MediaFile[id=5774524 A[5590329,5585234,"In Treatment",19548@1207.18:30,30] mask=TV host=sagetv-server encodedBy=Hauppauge HD PVR Capture Device on SAGETV-1:6969 Great-H.264 format=MPEG2-TS 0:00:30 2948 kbps [#0 Video[H.264 59.94006 fps 1280x720 16:9 progressive]#1 Audio[AC3 48000 Hz 6 channels 448 kbps MAIN idx=1 id=1100]] \\sagetv-server\BigMedia2-Recordings\InTreatment-AdeleWeekSeven-5590329-0.ts, Seg0[Tue 12/7 18:30:00.021-Tue 12/7 19:01:00.004]]]
2010-12-07 19:01:08,468 [Timer-12] WARN  sagex.phoenix.metadata.persistence.Sage7Persistence - Failed to update tv series info for: In Treatment
2010-12-07 19:01:08,468 [Timer-12] INFO  sagex.phoenix.metadata.MetadataManager - Saving Phoenix Fanart for SageMediaFile [sageId=5774524, sageObject=MediaFile[id=5774524 A[5590329,5585234,"In Treatment",19548@1207.18:30,30] mask=TV host=sagetv-server encodedBy=Hauppauge HD PVR Capture Device on SAGETV-1:6969 Great-H.264 format=MPEG2-TS 0:00:30 2948 kbps [#0 Video[H.264 59.94006 fps 1280x720 16:9 progressive]#1 Audio[AC3 48000 Hz 6 channels 448 kbps MAIN idx=1 id=1100]{EpisodeNumber=27, SeriesInfoID=1009634, SeasonNumber=3, MediaType=TV, Director=Jim McKay, IMDBID=tt1638572, Genre=Drama, Actor=Gabriel Byrne -- Dr. Paul Weston;Dianne West -- Dr. Gina Toll;Alison Pill -- April;Blair Underwood -- Alex Prince;Embeth Davidtz -- Amy;John Mahoney -- Walter;Josh Charles -- Jake;Melissa George -- Laura Hill;Mia Wasikowska -- Sophie, OriginalAirDate=1291708800000, MediaProviderID=tvdb, Writer=Sarah Treem, MediaProviderDataID=81248, Year=2010, Guest=Dane DeHaan, MediaTitle=In Treatment, Description=Paul tries to get Jesse to open up about a recent transgression and his relationship with his father Roberto., EpisodeName=Jesse: Week Seven, RunningTime=1800000}] \\sagetv-server\BigMedia2-Recordings\InTreatment-AdeleWeekSeven-5590329-0.ts, Seg0[Tue 12/7 18:30:00.021-Tue 12/7 19:01:00.004]]]

I also had this happen to a couple of Bob Saget's "Strange Days" episodes as well.

[stuckless]

Originally Posted by phelme

Might this explain why two recent recording of HBO's "In Treatment" are being renamed to the immediately previous airing? In the example below, "Adele: Week Seven" is being renamed to "Jesse: Week Seven":

2010-12-07 19:01:03,468 [Timer-12] INFO  sagex.phoenix.metadata.MetadataManager - search(): tvdb; SearchQuery; Type: TV; FILE:\\sagetv-server\BigMedia2-Recordings\InTreatment-AdeleWeekSeven-5590329-0.ts;YEAR:0;EPISODE_TITLE:Adele: Week Seven;RAW_TITLE:In Treatment;EPISODE_DATE:2010-12-07;CLEAN_TITLE:In Treatment;
2010-12-07 19:01:03,468 [Timer-12] INFO  sagex.phoenix.metadata.MetadataManager - Searching: In Treatment using TVDBMetadataProvider[MetadataProviderInfo [id=tvdb, fanartId=null, mediaTypes=[TV], name=thetvdb.com]]
2010-12-07 19:01:03,468 [Timer-12] INFO  sagex.phoenix.metadata.provider.tvdb.TVDBSearchParser - TVDB Search: http://www.thetvdb.com/api/GetSeries.php?seriesname=In+Treatment
2010-12-07 19:01:03,483 [Timer-12] INFO  sagex.phoenix.util.url.CachedUrl - Removing Cached Url File: cache\url\8863fbea2b3133b28fff92be44d58fe8.cache
2010-12-07 19:01:06,608 [Timer-12] INFO  sagex.phoenix.metadata.MetadataManager - Fetching Metadata for MediaSearchResult [extraArgs=CLEAN_TITLE: In Treatment,YEAR: 0,EPISODE_TITLE: Adele: Week Seven,RAW_TITLE: In Treatment,FILE: \\sagetv-server\BigMedia2-Recordings\InTreatment-AdeleWeekSeven-5590329-0.ts,EPISODE_DATE: 2010-12-07,, id=81248, imdbId=null, metadata=null, providerId=tvdb, score=1.0, title=In Treatment, type=TV, url=81248, year=2008]
2010-12-07 19:01:06,608 [Timer-12] INFO  sagex.phoenix.metadata.provider.tvdb.TVDBSeriesParser - TVDB Series: http://www.thetvdb.com/api/5645B594A3F32D27/series/81248/en.xml
2010-12-07 19:01:06,624 [Timer-12] INFO  sagex.phoenix.util.url.CachedUrl - Removing Cached Url File: cache\url\3cd2e4f3df32f77ca858fb4e7cb45882.cache
2010-12-07 19:01:07,187 [Timer-12] INFO  sagex.phoenix.metadata.provider.tvdb.TVDBSeriesParser - TVDB Actors: http://www.thetvdb.com/api/5645B594A3F32D27/series/81248/actors.xml
2010-12-07 19:01:07,202 [Timer-12] INFO  sagex.phoenix.util.url.CachedUrl - Removing Cached Url File: cache\url\33334834d3370e0e127d12961041ee3d.cache
2010-12-07 19:01:07,265 [Timer-12] INFO  sagex.phoenix.metadata.provider.tvdb.TVDBItemParser - TVDB date: http://thetvdb.com/api/GetEpisodeByAirDate.php?apikey=5645B594A3F32D27&seriesid=81248&airdate=2010-12-07
2010-12-07 19:01:07,296 [Timer-12] INFO  sagex.phoenix.metadata.provider.tvdb.TVDBItemParser - TVDB Episode: http://www.thetvdb.com/api/5645B594A3F32D27/series/81248/default/3/27/en.xml
2010-12-07 19:01:07,296 [Timer-12] INFO  sagex.phoenix.util.url.CachedUrl - Removing Cached Url File: cache\url\4e7cd65acdbe737107f5da9f636b8719.cache
2010-12-07 19:01:07,937 [Timer-12] INFO  sagex.phoenix.metadata.MetadataManager - Saving Metadata for SageMediaFile [sageId=5774524, sageObject=MediaFile[id=5774524 A[5590329,5585234,"In Treatment",19548@1207.18:30,30] mask=TV host=sagetv-server encodedBy=Hauppauge HD PVR Capture Device on SAGETV-1:6969 Great-H.264 format=MPEG2-TS 0:00:30 2948 kbps [#0 Video[H.264 59.94006 fps 1280x720 16:9 progressive]#1 Audio[AC3 48000 Hz 6 channels 448 kbps MAIN idx=1 id=1100]] \\sagetv-server\BigMedia2-Recordings\InTreatment-AdeleWeekSeven-5590329-0.ts, Seg0[Tue 12/7 18:30:00.021-Tue 12/7 19:01:00.004]]]
2010-12-07 19:01:08,468 [Timer-12] WARN  sagex.phoenix.metadata.persistence.Sage7Persistence - Failed to update tv series info for: In Treatment
2010-12-07 19:01:08,468 [Timer-12] INFO  sagex.phoenix.metadata.MetadataManager - Saving Phoenix Fanart for SageMediaFile [sageId=5774524, sageObject=MediaFile[id=5774524 A[5590329,5585234,"In Treatment",19548@1207.18:30,30] mask=TV host=sagetv-server encodedBy=Hauppauge HD PVR Capture Device on SAGETV-1:6969 Great-H.264 format=MPEG2-TS 0:00:30 2948 kbps [#0 Video[H.264 59.94006 fps 1280x720 16:9 progressive]#1 Audio[AC3 48000 Hz 6 channels 448 kbps MAIN idx=1 id=1100]{EpisodeNumber=27, SeriesInfoID=1009634, SeasonNumber=3, MediaType=TV, Director=Jim McKay, IMDBID=tt1638572, Genre=Drama, Actor=Gabriel Byrne -- Dr. Paul Weston;Dianne West -- Dr. Gina Toll;Alison Pill -- April;Blair Underwood -- Alex Prince;Embeth Davidtz -- Amy;John Mahoney -- Walter;Josh Charles -- Jake;Melissa George -- Laura Hill;Mia Wasikowska -- Sophie, OriginalAirDate=1291708800000, MediaProviderID=tvdb, Writer=Sarah Treem, MediaProviderDataID=81248, Year=2010, Guest=Dane DeHaan, MediaTitle=In Treatment, Description=Paul tries to get Jesse to open up about a recent transgression and his relationship with his father Roberto., EpisodeName=Jesse: Week Seven, RunningTime=1800000}] \\sagetv-server\BigMedia2-Recordings\InTreatment-AdeleWeekSeven-5590329-0.ts, Seg0[Tue 12/7 18:30:00.021-Tue 12/7 19:01:00.004]]]

I also had this happen to a couple of Bob Saget's "Strange Days" episodes as well.

I'm going to try and get a fix out this weekend for this... I had it happen to me as well