SageTV Community  

Go Back   SageTV Community > SageTV Development and Customizations > SageTV v7 Customizations > Batch Metadata Tools

Notices

Batch Metadata Tools This forums is for discussing the user-created Batch Metadata Tools for SageTV.

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old 12-24-2011, 02:30 PM
heffneil heffneil is offline
Sage Expert
 
Join Date: Feb 2008
Posts: 747
Comparing to Plex

I have become a big fan of Plex recently but obviously plex doesn't do everything sage does. The question I have is that plex just found all the metadata without me doing much if anything. Also I don't get any of the foreign covers or anything? Is there some repository they are using that we could "tap" for BMT?

Thanks,

Neil
Reply With Quote
  #2  
Old 12-24-2011, 06:33 PM
Fuzzy's Avatar
Fuzzy Fuzzy is offline
SageTVaholic
 
Join Date: Sep 2005
Location: Jurupa Valley, CA
Posts: 9,957
No, they are pulling from theTVdb as well. I think if anything, they have a little more 'liberal' search mechanisms... so if an exact match doens't work, they do something like a google search. BMT is too exact in it's mathcing, so if you have Guide data for "American Dad!" and thetvdb has "American Dad" it may not match. But if you do a google search of thetvdb for "American Dad!", you'll find the right show.

I really like the theme songs that are in Plex, but am wondering where they are pulling them from, and if that source can be contributed to (there aren't many of the kids programs on there, except maybe sponge-bob).
__________________
Buy Fuzzy a beer! (Fuzzy likes beer)

unRAID Server: i7-6700, 32GB RAM, Dual 128GB SSD cache and 13TB pool, with SageTVv9, openDCT, Logitech Media Server and Plex Media Server each in Dockers.
Sources: HRHR Prime with Charter CableCard. HDHR-US for OTA.
Primary Client: HD-300 through XBoxOne in Living Room, Samsung HLT-6189S
Other Clients: Mi Box in Master Bedroom, HD-200 in kids room
Reply With Quote
  #3  
Old 12-24-2011, 08:14 PM
heffneil heffneil is offline
Sage Expert
 
Join Date: Feb 2008
Posts: 747
Well the details are fantastic. I don't know if you tried it but ir requires nothing from me and shows all American box covers! That is a real nuisance for bmt. Too bad bmt can't be as simple and accurate!
Reply With Quote
  #4  
Old 12-29-2011, 06:48 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,554
The title throws me a litttle, since Plex and BMT are completely different things... but oh well...

BMT sends the guide title to tvdb and if tvdb returns no results then there is nothing more that bmt does. But if tvdb does return results then bmt will use a matching algorithm to determine the best match. This is not a simple string compare, but rather, it is an algorithm that includes multiple steps. Typically a "no match" is a result of tvdb sending back no results... or we get a timeout from the server (which happens quite often as well) which gives no results... but bmt will retry a limited number of times in those cases.

The biggest reason why Plex is better at this... they have a community of people working on it. BMT has 1 person and to be honest, that 1 person is not very motivated to be continue working on it. But, like plex, the source code is there, so make it better... that's why I opened the source from the start... but people would rather complain why isn't BMT as good as "INSERT PRODUCT HERE"... rather than rolling up there sleeves and fixing it I suspect that the Plex community is better in that regard, and as a result, the product is better.

You don't have to be a "coder" to help. Suggesting that I use google is a great idea (lots of people suggest this)... but it doesn't really help me. Researching whether or not there is a free and open google api that can be be used would be a better start, but I'm not going to take an "idea" and simply spend hours researching whether or not it's viable... I don't have the time. Perhaps there are other libraries (java preferably) out there that do this better... When I started bmt, there wasn't, but things change, and maybe there is today... but again, I don't have the time to do 100% of the development and 100% of the research

To be fair, over the years, there have been a handful of people that have contributed code, research, or documentation to bmt, and I've been extremely thankful.
Reply With Quote
  #5  
Old 12-29-2011, 10:52 AM
heffneil heffneil is offline
Sage Expert
 
Join Date: Feb 2008
Posts: 747
Yeah the title sucked but given it was in this subsection in the forums I thought the rest could be figured out.

Stuckless, thank you for your contribution of BMT. I am by no means crapping on it. I appreciate it, use it and need it to make sage "good". I am frustrated with it but like you said you do this out of a labor of love and it isn't a business. I didn't expect such a strong response but I guess I was suggesting that whatever plex is doing we (and i use we figuratively because I am no programmer) consider and take a look at. From there I don't know. I am not going to take apart Plex and figure it out or really know what is different or even how BMT works. My comments were only to open up the idea that someone consider looking at Plex's algorithm / solution as a viable aid to BMT. I don't know if you are familiar with plex. I don't know if anyone here is or if even anyone else suggested it. I merely installed it and recognized that "hey all the fan art is there and it looks pretty darn good. That is cool". As for the rest you can ignore me or this thread I was just throwing the idea out there.

Happy New Year,

Neil
Reply With Quote
  #6  
Old 12-29-2011, 01:00 PM
Fuzzy's Avatar
Fuzzy Fuzzy is offline
SageTVaholic
 
Join Date: Sep 2005
Location: Jurupa Valley, CA
Posts: 9,957
Quote:
Originally Posted by stuckless View Post
You don't have to be a "coder" to help. Suggesting that I use google is a great idea (lots of people suggest this)... but it doesn't really help me. Researching whether or not there is a free and open google api that can be be used would be a better start, but I'm not going to take an "idea" and simply spend hours researching whether or not it's viable... I don't have the time. Perhaps there are other libraries (java preferably) out there that do this better... When I started bmt, there wasn't, but things change, and maybe there is today... but again, I don't have the time to do 100% of the development and 100% of the research

To be fair, over the years, there have been a handful of people that have contributed code, research, or documentation to bmt, and I've been extremely thankful.
Shouldn't really need an API to scrape the top result from the google search. I've tried it many times, and it works well (I'm just not much of a scraper/coder type of fella). I have found that using scroogle (a proxy type search designed to avoid the google cookie-monster) is much easier, and supports queries in the http request.

Code:
https://ssl.scroogle.org/cgi-bin/nbbwssl.cgi?Gw=Toot%20and%20puddle%20site:thetvdb.com
https://ssl.scroogle.org/cgi-bin/nbb...te:thetvdb.com
provides a relatively clean page from which to scrap the seriesID. Just pick up what comes after the first "thetvdb.com/?tab=series&id=" on the page.

obviously, only do this when tvdb doesn't return something from it's own api's. but every item that has ever failed in BMT has returned properly for me in this method (unless it genuinly doens't exist on thetvdb.com).
__________________
Buy Fuzzy a beer! (Fuzzy likes beer)

unRAID Server: i7-6700, 32GB RAM, Dual 128GB SSD cache and 13TB pool, with SageTVv9, openDCT, Logitech Media Server and Plex Media Server each in Dockers.
Sources: HRHR Prime with Charter CableCard. HDHR-US for OTA.
Primary Client: HD-300 through XBoxOne in Living Room, Samsung HLT-6189S
Other Clients: Mi Box in Master Bedroom, HD-200 in kids room

Last edited by Fuzzy; 12-29-2011 at 01:15 PM.
Reply With Quote
  #7  
Old 12-30-2011, 06:33 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,554
Quote:
Originally Posted by Fuzzy View Post
Shouldn't really need an API to scrape the top result from the google search. I've tried it many times, and it works well (I'm just not much of a scraper/coder type of fella). I have found that using scroogle (a proxy type search designed to avoid the google cookie-monster) is much easier, and supports queries in the http request.

Code:
https://ssl.scroogle.org/cgi-bin/nbbwssl.cgi?Gw=Toot%20and%20puddle%20site:thetvdb.com
https://ssl.scroogle.org/cgi-bin/nbb...te:thetvdb.com
provides a relatively clean page from which to scrap the seriesID. Just pick up what comes after the first "thetvdb.com/?tab=series&id=" on the page.

obviously, only do this when tvdb doesn't return something from it's own api's. but every item that has ever failed in BMT has returned properly for me in this method (unless it genuinly doens't exist on thetvdb.com).
Again... thanks for the idea. I wasn't aware of scroogle, which would make it much easier to parse, since parsing google's site would be extremely difficult. I really hate writing scapers because they are so fickle (which is why bmt only has one -- imdb). I'll do some research and testing, and if I can do this quickly (ie, hours) then I'll add it in.
Reply With Quote
  #8  
Old 12-30-2011, 07:05 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,554
And this is where I quickly lose interest "researching" this. Before I get bogged down in writing a solution, I first want to understand why it's "not working". So I write a quick junit test for "American Dad!" and "American Dad", they both work (ie resolve to the same show). I then try "Toot and Poodle" and sure enough it doesn't work because the name is "Puddle" and not "Poodle". The guide data shows the correct name, so and when the correct name is used, it works, so i don't see why someone will have "Toot and Poodle" instead of "Toot and Puddle". So, if there is a problem with either of these 2 names, then using the above idea of googling it, will not fix it, since bmt finds it anyways. I need some real TV shows that are in the EPG that bmt doesn't find, so that I write a test case for it, and then prove that it works once a solution is in place.
Reply With Quote
  #9  
Old 12-30-2011, 08:32 AM
panteragstk's Avatar
panteragstk panteragstk is offline
SageTVaholic
 
Join Date: Oct 2008
Location: New Braunfels, TX
Posts: 3,312
I've used both and I will always prefer the way bmt does things. The main reason for that is with the tools that have been created to modify metadata within sage that makes bmt so good. It has always been accurate for me, and plex is nice, but it isn't and will never be sage, but is a decent front end that does most things other than live tv and recording.

I haven't had enough time to play around with plex, but it isn't as easy to correct incorrect metadata as it is with the bmt web interface.

This all really boils down to personal preference. Both are great.
__________________
SageTV Server: unRAID Docker v9, S2600CPJ, Norco 24 hot swap bay case, 2x Xeon 2670, 64 GB DDR3, 3x Colossus for DirecTV, HDHR for OTA
Living room: nVidia Shield TV, Sage Mini Client, 65" Panasonic VT60
Bedroom: Xiomi Mi Box, Sage Mini Client, 42" Panasonic PZ800u
Theater: nVidia Shield TV, mini client, Plex for movies, 120" screen. Mitsubishi HC4000. Denon X4300H. 7.4.4 speaker setup.
Reply With Quote
  #10  
Old 12-30-2011, 12:44 PM
wayner wayner is offline
SageTVaholic
 
Join Date: Jan 2008
Location: Toronto, ON
Posts: 6,613
PiX64 is working on writing some code that will export metadata from SageTV to Plex. This will help for people (like me) that have done a lot of work customizing their metadata. It will also help to get shows into Plex that will never have metadata - like sports.

One thing that I am looking to do is to write some code in SJQ that will rename all of my files in a Plex compatible format - this essentially means adding SXXEXX to filenames of older Recorded TV Files (I now have Sage set to include this in the filename) and Imported TV files.

@stuckless - If you are still are updating BMT it might be a nice future option in BMT to have the ability to add the Season and Episode to the filename if it isn't already there.
__________________
New Server - Sage9 on unRAID 2xHD-PVR, HDHR for OTA
Old Server - Sage7 on Win7Pro-i660CPU with 4.6TB, HD-PVR, HDHR OTA, HVR-1850 OTA
Clients - 2xHD-300, 8xHD-200 Extenders, Client+2xPlaceshifter and a WHS which acts as a backup Sage server
Reply With Quote
  #11  
Old 12-30-2011, 01:10 PM
Fuzzy's Avatar
Fuzzy Fuzzy is offline
SageTVaholic
 
Join Date: Sep 2005
Location: Jurupa Valley, CA
Posts: 9,957
Quote:
Originally Posted by stuckless View Post
And this is where I quickly lose interest "researching" this. Before I get bogged down in writing a solution, I first want to understand why it's "not working". So I write a quick junit test for "American Dad!" and "American Dad", they both work (ie resolve to the same show). I then try "Toot and Poodle" and sure enough it doesn't work because the name is "Puddle" and not "Poodle". The guide data shows the correct name, so and when the correct name is used, it works, so i don't see why someone will have "Toot and Poodle" instead of "Toot and Puddle". So, if there is a problem with either of these 2 names, then using the above idea of googling it, will not fix it, since bmt finds it anyways. I need some real TV shows that are in the EPG that bmt doesn't find, so that I write a test case for it, and then prove that it works once a solution is in place.
If I mentioned "Toot and Poodle" anywhere, that was a typo on my part. thetvdb has improved THEIR search in the last year, which is why those shows don't fail like they used to. It used to not handle ampersands properly, so "Toot & Puddle" would not work, because the show on thetvdb is called "Toot and Puddle". This is just one example, but there are many shows that used to really throw thetvdb for a loop, mostly due to punctuation differences between guide data ('CSI: Crime Scene Investigation' vs. 'CSI - Crime Scene Investigation' vs. 'C.S.I. Crime Scene Investigation'). I have actually seen all three of those over the years of using sagetv, even though technically, the first one is correct. That's why I had suggested the google thing a year or so ago. It's not as important now. And really, it's even better now if you use the new guide data plugin, as it allows those quirks to be fixed in the guide before it reaches sagetv.
__________________
Buy Fuzzy a beer! (Fuzzy likes beer)

unRAID Server: i7-6700, 32GB RAM, Dual 128GB SSD cache and 13TB pool, with SageTVv9, openDCT, Logitech Media Server and Plex Media Server each in Dockers.
Sources: HRHR Prime with Charter CableCard. HDHR-US for OTA.
Primary Client: HD-300 through XBoxOne in Living Room, Samsung HLT-6189S
Other Clients: Mi Box in Master Bedroom, HD-200 in kids room
Reply With Quote
  #12  
Old 12-30-2011, 01:57 PM
wayner wayner is offline
SageTVaholic
 
Join Date: Jan 2008
Location: Toronto, ON
Posts: 6,613
The problem with colons is that they are illegal in filenames so this can cause an issue when filenames are used as the identifying source of the file
__________________
New Server - Sage9 on unRAID 2xHD-PVR, HDHR for OTA
Old Server - Sage7 on Win7Pro-i660CPU with 4.6TB, HD-PVR, HDHR OTA, HVR-1850 OTA
Clients - 2xHD-300, 8xHD-200 Extenders, Client+2xPlaceshifter and a WHS which acts as a backup Sage server
Reply With Quote
  #13  
Old 12-30-2011, 03:51 PM
emveepee emveepee is offline
Sage Aficionado
 
Join Date: Nov 2006
Posts: 319
I've written a similar renaming utility for NextPVR, and I am also a few fan of Plex for devices. There are a few gotcha's with the lookups that I've found,

The Tribune guide data here in North America and TheTVDB data for episodes don't always match so the originalAirDate is important. Grey's Anatomy S08E05 is a good example.

Also some shows won't match because a two hour episode might be originally recorded with one name but it will shown as two episodes on TheTVDB.

The there are normal issues like multiple years for "The Twilight Zone" and multiple countries for "The Office".

Hawaii Five-O is a problem because Tribune data thinks it is Hawaii Five-0. TheTVDB has the original as Hawaii Five-0 and the new one as Hawaii Five-O Your SageTV value is probably okay, because it is new enough.

Martin
Reply With Quote
  #14  
Old 12-30-2011, 03:57 PM
wayner wayner is offline
SageTVaholic
 
Join Date: Jan 2008
Location: Toronto, ON
Posts: 6,613
Quote:
Originally Posted by emveepee View Post
Also some shows won't match because a two hour episode might be originally recorded with one name but it will shown as two episodes on TheTVDB.
There is a similar issue for kids' cartoons where you may have 2-3 titles in a 30 minute time slot. The TVDB preference is to separate into individual titles - this is preferable for DVD rips but it leads to incorrect matching for TV recordings.
__________________
New Server - Sage9 on unRAID 2xHD-PVR, HDHR for OTA
Old Server - Sage7 on Win7Pro-i660CPU with 4.6TB, HD-PVR, HDHR OTA, HVR-1850 OTA
Clients - 2xHD-300, 8xHD-200 Extenders, Client+2xPlaceshifter and a WHS which acts as a backup Sage server
Reply With Quote
  #15  
Old 12-30-2011, 07:07 PM
Fuzzy's Avatar
Fuzzy Fuzzy is offline
SageTVaholic
 
Join Date: Sep 2005
Location: Jurupa Valley, CA
Posts: 9,957
Quote:
Originally Posted by emveepee View Post
Also some shows won't match because a two hour episode might be originally recorded with one name but it will shown as two episodes on TheTVDB.
Plex does actually support handling two-episode recordings. I believe it is named as s01e01-e02 or something like that. It is discussed in the plex naming guide (here)

It also seems to have an answer for just about every other type of problem item, like Daily shows (news, etc) (having a date in the filename instead of a season and episode). About the only type of show it might have a problem with out of the box is sports.
__________________
Buy Fuzzy a beer! (Fuzzy likes beer)

unRAID Server: i7-6700, 32GB RAM, Dual 128GB SSD cache and 13TB pool, with SageTVv9, openDCT, Logitech Media Server and Plex Media Server each in Dockers.
Sources: HRHR Prime with Charter CableCard. HDHR-US for OTA.
Primary Client: HD-300 through XBoxOne in Living Room, Samsung HLT-6189S
Other Clients: Mi Box in Master Bedroom, HD-200 in kids room
Reply With Quote
  #16  
Old 12-30-2011, 07:11 PM
Fuzzy's Avatar
Fuzzy Fuzzy is offline
SageTVaholic
 
Join Date: Sep 2005
Location: Jurupa Valley, CA
Posts: 9,957
Quote:
Originally Posted by wayner View Post
There is a similar issue for kids' cartoons where you may have 2-3 titles in a 30 minute time slot. The TVDB preference is to separate into individual titles - this is preferable for DVD rips but it leads to incorrect matching for TV recordings.
This varries from show to show on thetvdb. It depends on who created the information. Some are listed together (like 'My Friends, Tigger and Pooh'), some are separated (like 'Penguins of Madagascar')

Ultimately, it comes down to the lack of standards being the key problem, which, really, is what makes this stuff so difficult.

And, at least plex DOES support this type of situation, with the above mentioned multi-episode recording filename scheme (The Penguins of Madagascar - s01e01-e02 - Gone in a Flash; Launchtime.mpg)
__________________
Buy Fuzzy a beer! (Fuzzy likes beer)

unRAID Server: i7-6700, 32GB RAM, Dual 128GB SSD cache and 13TB pool, with SageTVv9, openDCT, Logitech Media Server and Plex Media Server each in Dockers.
Sources: HRHR Prime with Charter CableCard. HDHR-US for OTA.
Primary Client: HD-300 through XBoxOne in Living Room, Samsung HLT-6189S
Other Clients: Mi Box in Master Bedroom, HD-200 in kids room
Reply With Quote
  #17  
Old 12-30-2011, 07:43 PM
wayner wayner is offline
SageTVaholic
 
Join Date: Jan 2008
Location: Toronto, ON
Posts: 6,613
I just had this exact issue with Curious George - see this thread on the tvdb forums.
__________________
New Server - Sage9 on unRAID 2xHD-PVR, HDHR for OTA
Old Server - Sage7 on Win7Pro-i660CPU with 4.6TB, HD-PVR, HDHR OTA, HVR-1850 OTA
Clients - 2xHD-300, 8xHD-200 Extenders, Client+2xPlaceshifter and a WHS which acts as a backup Sage server
Reply With Quote
  #18  
Old 12-30-2011, 09:20 PM
emveepee emveepee is offline
Sage Aficionado
 
Join Date: Nov 2006
Posts: 319
Quote:
Originally Posted by Fuzzy View Post
Plex does actually support handling two-episode recordings. I believe it is named as s01e01-e02 or something like that. It is discussed in the plex naming guide (here)
Yes they have done a great job in defining the naming conventions. From that link, they used the split "Heroes" and it shows the problem in the Tribune data it would have been originally recorded as episode "Orientation; Jump, Push, Fall" with id EP008483610067 (http://tvlistings.zap2it.com/tv/heroes/EP008483610067) I think I will see what they do with this. I'll have to watch to see if the semi colon can be used to disambiguate the episodes.

Note for zap2it, that url with the page for the series title less special characters and spaces as dashes followed by epgId is the direct link.

Martin
Reply With Quote
  #19  
Old 12-31-2011, 10:36 AM
wayner wayner is offline
SageTVaholic
 
Join Date: Jan 2008
Location: Toronto, ON
Posts: 6,613
Quote:
Originally Posted by Fuzzy View Post
Plex does actually support handling two-episode recordings. I believe it is named as s01e01-e02 or something like that. It is discussed in the plex naming guide
Is there any way to handle this in Sage? I am guessing not since I believe that episode number is an integer. You would have to stuff this into some other field.
__________________
New Server - Sage9 on unRAID 2xHD-PVR, HDHR for OTA
Old Server - Sage7 on Win7Pro-i660CPU with 4.6TB, HD-PVR, HDHR OTA, HVR-1850 OTA
Clients - 2xHD-300, 8xHD-200 Extenders, Client+2xPlaceshifter and a WHS which acts as a backup Sage server
Reply With Quote
  #20  
Old 12-31-2011, 10:44 AM
Fuzzy's Avatar
Fuzzy Fuzzy is offline
SageTVaholic
 
Join Date: Sep 2005
Location: Jurupa Valley, CA
Posts: 9,957
Correct, sage cannot handle this properly, as it is based on each recording is a single episode. It would be a lot of work to implement this in the UI.
__________________
Buy Fuzzy a beer! (Fuzzy likes beer)

unRAID Server: i7-6700, 32GB RAM, Dual 128GB SSD cache and 13TB pool, with SageTVv9, openDCT, Logitech Media Server and Plex Media Server each in Dockers.
Sources: HRHR Prime with Charter CableCard. HDHR-US for OTA.
Primary Client: HD-300 through XBoxOne in Living Room, Samsung HLT-6189S
Other Clients: Mi Box in Master Bedroom, HD-200 in kids room
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Plex as a front end for Sage wayner General Discussion 228 07-24-2012 01:40 PM
Plex in a TV? What about sage? rwc General Discussion 9 09-05-2010 04:38 PM
Comparing the features of SageTV 7 to SageMC? davidjade SageMC Custom Interface 0 05-22-2010 12:37 AM
Possible to get SageTV to handle my library more like Plex? DAMAC SageTV Mac Edition 0 06-10-2009 10:33 AM
Comparing Sage to Snapstream/Beyond TV? pchallin General Discussion 28 02-12-2004 09:00 AM


All times are GMT -6. The time now is 06:36 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.
Copyright 2003-2005 SageTV, LLC. All rights reserved.