Future of MyMovies and alternatives in SMM

[Ericft]

I agree...i'm already confused with the options of my movies, BMT and so forth..starting to see some issues with my movies database and maybe it's a config problem on my part...but unification in the metadata dept is not a bad idea...maybe collaborating the best of all options into one sagetv plugin and then the phoenix/ortus/native UI will be the icing on the cake...

[jaminben]

The only problem with this is that everyone has their own ideas about how it should work and until a stable standard is formed I don't see these different scrapers becoming unified if ever. There's obviously reasons why we didn't use BMT as our main scraper and vice versa..... in the long run I think one will become the more popular (betamax/vhs) and will ultimately be the platform that future plugins will be based off.

Cheers

Ben

[PLUCKYHD]

Okay first let me explain

1. I am not inventing/writing my own scraper I am going to be using another very well known existing scraper.

2. BMT and ortus don't suit my wants for movies. Everyone has different wants and needs and I get the privilege of coding for mine . I prefer multi disc support/offline support and boxset support. Those are all things I needed and wanted.

3. SMM will work in all STV's phoenix,ortus or the core. It is not tied to the core except to play movies and main menu. Those are easily modified for any new stv. So don't think SMM is fragmented in that what because it isn't.

Now for you icing on the cake Stranger

SMM was originally intended to be a unified SageMediaManager. MyMovies is just the first phase I did. I didn't plan on having to find another avenue to fit my needs so that has put a hitch in the process none the less.

What does the future of SMM hold? (no deadlines but this is the roadmap)

1. Implement new scraper that has been discussed (already in process).

2. Implement core support including default db,bmt,ortus,SMM or the new scraper in one unified UI.

3. Share the main api meaning any other 3rd part developer can also develop for a UI that is metadata independent as well.

So what does this mean

SMM in the future you will have the option between any of the existing scrapers you want to use to get your data into SMM. Our just use the core db with folder jpegs if that is your cup of tea. Of course there will be limits on some features from one scraper to the next.

As well as future developers that chose to use my shared api can build UI's that are also completely open to whatever provider someone wants to use.

In my mind this gives a great open experience and doesn't tie anyone to one metadata scraper. As well as it hopefully lets Stuckless, MyMovies,Ortus and all those great developers stay focused on their great scrapers they make.

Hopefully that explains everything and puts some people at ease or at least they can see my bigger vision.

cheers

pluckyhd

[Fuzzy]

Don't worry stanger, those have been my concerns as well. The main thing I never liked about MyMovies (and SMM in particular) was that it DIDN'T use the built-in sageTV database. There should only ever be one location for that data, and in my opinion, it should be Sage. Sage supports all the custom metadata you could want to throw at it, so there is really no reason NOT to use it. i don't care how many different scrapers/metadata tools are out there, as long as they all store the info in the same place. make any sense? that way whatever scraper you end up using, can store their data in the same place. Whatever UI mod's you choose to use, will pull the data from teh same place. Plucky is VERY good at making the UI views, and I feel he should just focus on that, and let others who have done more work on the metadata scraping/populating side deal with that. That would all ow the community to be segmented, but not fragmented.. if that makes any sense...

Okay.. i'm a few beers in and rambling.. but I think I've made some sense...

[PLUCKYHD]

Originally Posted by Fuzzy

Don't worry stanger, those have been my concerns as well. The main thing I never liked about MyMovies (and SMM in particular) was that it DIDN'T use the built-in sageTV database. There should only ever be one location for that data, and in my opinion, it should be Sage. Sage supports all the custom metadata you could want to throw at it, so there is really no reason NOT to use it. i don't care how many different scrapers/metadata tools are out there, as long as they all store the info in the same place. make any sense? that way whatever scraper you end up using, can store their data in the same place. Whatever UI mod's you choose to use, will pull the data from teh same place. Plucky is VERY good at making the UI views, and I feel he should just focus on that, and let others who have done more work on the metadata scraping/populating side deal with that. That would all ow the community to be segmented, but not fragmented.. if that makes any sense...

Okay.. i'm a few beers in and rambling.. but I think I've made some sense...

Ouch.. I don't handle metadata or scraping or populating to be clear and there was no way for me to get all the mymovies data into the default db and make it usable to me like I wanted. Also if you have all this different plugins populating the default db I am sorry but Stuckless and all of us would spend way too much time helping users when our plugins where conflicting. As it stands now and will stand my work does not interfere with Stuckless or anyone else. I GUARANTEE if I wrote to the db there would be conflicts. You can say again and again don't install this and this but users will do it and expect you to tell them how to fix it.

You say there is no reason not to use it I differ 100% sql is more dynamic and I am more capable of filters/query in sql then I will ever be in the default database. To me it takes to many calls to get what I can get out of one line of query in sql.

So I think people would rather me code in what I am more capable in.

So just because you think there is NO reason doesn't mean others developers thinks that way.

And as always if you don't like it you don't have to use it!

[Fuzzy]

Originally Posted by PLUCKYHD

Ouch.. I don't handle metadata or scraping or populating to be clear and there was no way for me to get all the mymovies data into the default db and make it usable to me like I wanted. Also if you have all this different plugins populating the default db I am sorry but Stuckless and all of us would spend way too much time helping users when our plugins where conflicting. As it stands now and will stand my work does not interfere with Stuckless or anyone else. I GUARANTEE if I wrote to the db there would be conflicts. You can say again and again don't install this and this but users will do it and expect you to tell them how to fix it.

You say there is no reason not to use it I differ 100% sql is more dynamic and I am more capable of filters/query in sql then I will ever be in the default database. To me it takes to many calls to get what I can get out of one line of query in sql.

So I think people would rather me code in what I am more capable in.

So just because you think there is NO reason doesn't mean others developers thinks that way.

And as always if you don't like it you don't have to use it!

My point was that anyone should be able to install any metadata source they choose. (that actually what Sage implemented a metadata plugin infrastructure back in v2 or something). There shouldn't be conflicts, as people should only ever need a single scraper (well, perhaps one for movies, and one for tv.. whatever.. it doesn't matter). What I want to see is a complete decoupling of the UI from the data sources.

And honestly, what sort of complicated queries are you doing that couldn't be done with a combination of FilterByMethod and DataUnion/DataIntersection API calls? I haven't seen anything in (my admittedly limited time messing with) SMM that really has anything more complicated than that.

[PLUCKYHD]

Originally Posted by Fuzzy

My point was that anyone should be able to install any metadata source they choose. (that actually what Sage implemented a metadata plugin infrastructure back in v2 or something). There shouldn't be conflicts, as people should only ever need a single scraper (well, perhaps one for movies, and one for tv.. whatever.. it doesn't matter). What I want to see is a complete decoupling of the UI from the data sources.

And honestly, what sort of complicated queries are you doing that couldn't be done with a combination of FilterByMethod and DataUnion/DataIntersection API calls? I haven't seen anything in (my admittedly limited time messing with) SMM that really has anything more complicated than that.

You answered your own question by saying maybe they would for tv and movies.

Limited time and use is not enough to say you see something that couldn't be done. Yes it probably could with multiple filter after filter to some extent, but as I have said before mymovies has a lot of built in functions that are not possible with sagedb.

For instance on a simple load

I get the movies for the current video type

I apply multiple filters (online,offline,watched etc etc)

I check user restrictions

I sort and sub sort

There is more to it but I am seeing multiple calls in the sage db as to one dynamic line in SQL.

I can't stand passing judgement when you yourself said you haven't used it much. Don't assume.

And again if you don't like or agree stop using it no one is making you.

[dbone1026]

Originally Posted by Fuzzy

My point was that anyone should be able to install any metadata source they choose. (that actually what Sage implemented a metadata plugin infrastructure back in v2 or something). There shouldn't be conflicts, as people should only ever need a single scraper (well, perhaps one for movies, and one for tv.. whatever.. it doesn't matter). What I want to see is a complete decoupling of the UI from the data sources.

And honestly, what sort of complicated queries are you doing that couldn't be done with a combination of FilterByMethod and DataUnion/DataIntersection API calls? I haven't seen anything in (my admittedly limited time messing with) SMM that really has anything more complicated than that.

I have to agree with Plucky on this one. The bigger picture as well is SMM may very well bring new users to SageTV from WMC. One of the biggest roadblocks that keep people from switching or trying a different Front End is essentially having to re-catalog their video collection when they spent a lot of time getting it to where it needs to be with the current front end they are using. I know quite a few people who wouldn't have even considered SageTV if it wasn't for SMM since they could just carry over their WMC catalog. As for myself, I was not in agreement when it was decided to change the way metadata was handled with SageTV with the whole centralized folder, and found BMT to be a mess. That alone discouraged me from SageTV, but SMM has renewed my interest (as a MediaBrowser/MyMovies user my media is set to go).

Ultimately, it is a plugin which is completely "user optional", so not sure what the griping is about

[Fuzzy]

I'm not saying MyMovies isn't a good data source at all. And of course, using that data source is what has the potential to bring in users from elsewhere. My point was that it would be best to decouple the UI from the data provider (which is what this thread seems to be discussing). And don't take it as griping, I've discussed it with plucky in the past.. and I don't 'use' mymovies because I honestly don't have a need for it (though it is installed to attempt to see what the draw is). My concerns are more about the userbase. It was just a suggestion, and a confirmation of Stanger's concerns about fragmentation.

[Fuzzy]

Originally Posted by jaminben

The only problem with this is that everyone has their own ideas about how it should work and until a stable standard is formed I don't see these different scrapers becoming unified if ever. There's obviously reasons why we didn't use BMT as our main scraper and vice versa.

I understand not necessarily not wanting to use BMT as your scraper, there are things I don't like about it as well (the dependance on a WebUI being a big one). But what are the reasons for not storing your metadata in the SageDB? These are the things I'm not understanding about the Ortus and SMM methods.

[Skirge01]

As someone who's still using BMT and SageTV v6, some things immediately come to mind for me. For one, if I decide to use SMM (which I never have yet) or Ortus or Phoenix and utilize their own DBs, each one is independent of each other. This means that I could theoretically use each one's DB with the others, assuming they all allowed such a thing. Secondly, due to this very thing, I can run them ALL concurrently without breaking anything in any of the separate DBs. Third, this would allow for each of the devs to go their own route in the format and/or content of the DB and simply give me more options. For all I know, one might work better under my own set of circumstances (hardware setup, available horsepower, library contents, etc).

Look, I'm all for uniformity and the KISS methodology, but options are good, too. Google is great and I do stick to it almost exclusively, but why shouldn't we have other options for searching? Forget the highly improbable nature of this notion, but what happens if Google decides they no longer want to be in the search business? Oh, there's Bing. The loss of Google is no big deal then. Let's bring it closer to home. What would've happened if MeInMaui gave up on SageMC a couple of years ago? There were no alternatives. We'd pray or beg that someone like PluckyHD or Stuckless or jaminben would pick up the torch, but there's no guarantee that would happen. Again, options are good.

[Skirge01]

Originally Posted by Fuzzy

I understand not necessarily not wanting to use BMT as your scraper, there are things I don't like about it as well (the dependance on a WebUI being a big one). But what are the reasons for not storing your metadata in the SageDB? These are the things I'm not understanding about the Ortus and SMM methods.

One other possible issue that comes to mind is what happens if SageTV changes their DB, thereby breaking the scraper? Each of these devs would be forced to go back and patch (or possibly even entirely rewrite) their scrapers to work with the new format and/or options... well, either that or abandon the project. With their own DB, this is a non-issue. Instead of stopping to fix their scraper to make it work again, they just keep on going and improving it.

[Fuzzy]

Originally Posted by Skirge01

One other possible issue that comes to mind is what happens if SageTV changes their DB, thereby breaking the scraper? Each of these devs would be forced to go back and patch (or possibly even entirely rewrite) their scrapers to work with the new format and/or options... well, either that or abandon the project. With their own DB, this is a non-issue. Instead of stopping to fix their scraper to make it work again, they just keep on going and improving it.

Sage could do whatever they want with the DB format itself. It's accessed only through exposed API calls, so it wouldn't break anything. All I can see changing in the future would be them adding more 'official' tags - not anything that would break current implementations.

[stanger89]

Fuzzy already covered most of this but I'm just going to answer a few of these from my POV....

Originally Posted by jaminben

The only problem with this is that everyone has their own ideas about how it should work and until a stable standard is formed I don't see these different scrapers becoming unified if ever. There's obviously reasons why we didn't use BMT as our main scraper and vice versa..... in the long run I think one will become the more popular (betamax/vhs) and will ultimately be the platform that future plugins will be based off.

Just to be clear, I'm not talking about SMM, SMM v2 or Ortus using BMT as a scraper. I'm talking about decoupling scrapers, UI and the actual storage of the metadata.

I guess the underlying question is why not store data in Sage the way Phoenix does.

Originally Posted by PLUCKYHD

Okay first let me explain

1. I am not inventing/writing my own scraper I am going to be using another very well known existing scraper.

2. BMT and ortus don't suit my wants for movies. Everyone has different wants and needs and I get the privilege of coding for mine . I prefer multi disc support/offline support and boxset support. Those are all things I needed and wanted.

3. SMM will work in all STV's phoenix,ortus or the core. It is not tied to the core except to play movies and main menu. Those are easily modified for any new stv. So don't think SMM is fragmented in that what because it isn't.

Now for you icing on the cake Stranger

SMM was originally intended to be a unified SageMediaManager. MyMovies is just the first phase I did. I didn't plan on having to find another avenue to fit my needs so that has put a hitch in the process none the less.

What does the future of SMM hold? (no deadlines but this is the roadmap)

1. Implement new scraper that has been discussed (already in process).

2. Implement core support including default db,bmt,ortus,SMM or the new scraper in one unified UI.

3. Share the main api meaning any other 3rd part developer can also develop for a UI that is metadata independent as well.

So what does this mean

SMM in the future you will have the option between any of the existing scrapers you want to use to get your data into SMM. Our just use the core db with folder jpegs if that is your cup of tea. Of course there will be limits on some features from one scraper to the next.

As well as future developers that chose to use my shared api can build UI's that are also completely open to whatever provider someone wants to use.

FWIW, this is sort of what I'm getting at, it's the 4th API we've got going now, we've got Sage's API, Phoenix, Ortus, and now SMM.

Originally Posted by PLUCKYHD

So I think people would rather me code in what I am more capable in.

So just because you think there is NO reason doesn't mean others developers thinks that way.

And as always if you don't like it you don't have to use it!

FWIW, I definitely defer to the software developers on the best way to implement things

Originally Posted by Skirge01

As someone who's still using BMT and SageTV v6, some things immediately come to mind for me. For one, if I decide to use SMM (which I never have yet) or Ortus or Phoenix and utilize their own DBs, each one is independent of each other. This means that I could theoretically use each one's DB with the others, assuming they all allowed such a thing. Secondly, due to this very thing, I can run them ALL concurrently without breaking anything in any of the separate DBs. Third, this would allow for each of the devs to go their own route in the format and/or content of the DB and simply give me more options. For all I know, one might work better under my own set of circumstances (hardware setup, available horsepower, library contents, etc).

Look, I'm all for uniformity and the KISS methodology, but options are good, too. Google is great and I do stick to it almost exclusively, but why shouldn't we have other options for searching? Forget the highly improbable nature of this notion, but what happens if Google decides they no longer want to be in the search business? Oh, there's Bing. The loss of Google is no big deal then. Let's bring it closer to home. What would've happened if MeInMaui gave up on SageMC a couple of years ago? There were no alternatives. We'd pray or beg that someone like PluckyHD or Stuckless or jaminben would pick up the torch, but there's no guarantee that would happen. Again, options are good.

Oh I completely agree, options are good, but let me throw this back at you. I haven't really had a chance to really put SMM to good use, why? Because my metadata isn't in a format that works well with it. My metadata is all in DVD Profiler, which BMT/Phoenix support. I've tried getting all my stuff into MM Collection Manager but it's been an excercise in frustration. Sadly the "simplest" option is to just start from scratch but that doesn't work that well. Likewise with Ortus, I'd basically have to start from scratch metadata wise.

So from my perspective, the fact that SMM and Ortus use their own, incompatible databases it's a positive "I can run them all at the same time" it's a negative "I have to redo my metadata from scratch for each one".

This is what I'm getting at, it SMM utilized the Sage database for metadata storage, I'd likely be using it because I could just install it and all the metadata would already be there.

Or say you used SMM and had all your metadata defined there, but wanted to use the Phoenix or Ortus STV. No problem, all your data is there.

Originally Posted by Skirge01

One other possible issue that comes to mind is what happens if SageTV changes their DB, thereby breaking the scraper?

That's the beauty of Sage, they don't provide access to the database, they give you APIs to access the data, so it doesn't matter how Sage structures the database, you just call the APIs. And FWIW, in the history from SageTV 3, they've broken/obsoleted very little, if anything API wise.

Just to let me blue-sky a bit on what I'm thinking of when I think of an "ideal" setup. It would have basically three pieces, the SageTV core, which provides everything we all know about including a metadata database; the "scrapers" or data aggregators, such as BMT or Ortus's scraper, or MMCM, which would collect data and populate the SageTV core database; and finally UI plugins or replacements, like SMM or Phoenix, or Ortus which use the metadata and present it to the user with options for filtering or what not.

Such a system would allow you to use a Phoenix STV with say MMCM movie data and Ortus TV show data. Or if you didn't like the Phoenix STV you could use the Stock STV, or Ortus, or SMM independent of where the data comes from. You could mix-and-match, picking the scraper that best works for your needs and the UI you most like. No having to transfer metadata between database or regrab it if you change UIs.

Plucky, I'd don't know if you've asked at all, but I'm relatively certain Jeff has been quite helpful in updating the Sage APIs for requests from Phoenix (maybe Ortus too) devs to make the SageTV database more usable, so if you haven't you might want to run some of your thoughts/concerns about Sage's sorting/filtering abilities by him/SageTV.

[Skirge01]

Originally Posted by Fuzzy

Sage could do whatever they want with the DB format itself. It's accessed only through exposed API calls, so it wouldn't break anything. All I can see changing in the future would be them adding more 'official' tags - not anything that would break current implementations.

Having done some programming myself, I can tell you that you can never see what the future needs or changes will be. When there's just one person with a goal, things work great. When you invite others to join in your goal, you almost never wind up in the same place. One person's foresight can never account for the masses and I'm sure that neither you or I can come up with every possible future usage scenario. We'd both wind up with the same dilemma: not enough forward thinking. I think that's why these guys are designing/using their own DBs.

[Fuzzy]

Originally Posted by Skirge01

Having done some programming myself, I can tell you that you can never see what the future needs or changes will be. When there's just one person with a goal, things work great. When you invite others to join in your goal, you almost never wind up in the same place. One person's foresight can never account for the masses and I'm sure that neither you or I can come up with every possible future usage scenario. We'd both wind up with the same dilemma: not enough forward thinking. I think that's why these guys are designing/using their own DBs.

Well, as far as SMM, the only reason he's given for using his own UI, was the query mechanics. There's no actual data he's using that can't be stored in sage's DB. I'm not sure on the Ortus side what the limitation they are seeing is. As for future changes, You are right, no one can really see the future, but I don't see the standard tags going anywhere (Title, Original Air Date, Cast/Crew, etc).

FWIW, it seems it would be a lot simpler to spend development time in creating a SQL-Query style wrapper for the sageAPIs, than it would be to create a whole new database system to run in parallel, and contain pretty much the same data.

[Skirge01]

Originally Posted by stanger89

Just to let me blue-sky a bit on what I'm thinking of when I think of an "ideal" setup. It would have basically three pieces, the SageTV core, which provides everything we all know about including a metadata database; the "scrapers" or data aggregators, such as BMT or Ortus's scraper, or MMCM, which would collect data and populate the SageTV core database; and finally UI plugins or replacements, like SMM or Phoenix, or Ortus which use the metadata and present it to the user with options for filtering or what not.

I wrote a bunch of stuff and then realized you nearly made my point for me right there. Everything you described sounds like what I think is being done here by Phoenix, Ortus, and SMM, except for one, tiny thing. Your "SageTV core" doesn't store the metadata. The scrapers store it someplace else. This would allow other devs to utilize any/all of the DBs they choose to.

FWIW, I do understand what you're saying and I see your reasoning. But, again, having done some programming, I can definitely understand why these guys would prefer to design and query their own DB. It's simply more efficient and allows them greater flexibility. They can design it to behave exactly how they want. I have no idea how responsive SageTV is when these devs make a request, but no matter how good/quick/accommodating SageTV's employees are (and comments on the forums make me believe they really are), it would still be quicker to not have to make that request at all.

[PLUCKYHD]

Originally Posted by Fuzzy

I understand not necessarily not wanting to use BMT as your scraper, there are things I don't like about it as well (the dependance on a WebUI being a big one). But what are the reasons for not storing your metadata in the SageDB? These are the things I'm not understanding about the Ortus and SMM methods.

We have been over this again and again MyMovies already had the sql database why should I recreate redundant databases makes no since. If people want mymovies in the default db they simple check the option to have mymovies write properties file and there. You argument has no merit to me.

Ortus has valid reasons as well. If you don't think SQL's are easier and more dynamic well you need to use one compared to the sage db api. I am not knocking their api just saying SQL's to me we the developer have full control over and for the record Ortus does write to the default db as well if that option is on.

So there you have the option to write to the default db for mymovies or ortus for other sources already.

Originally Posted by stanger89

Just to be clear, I'm not talking about SMM, SMM v2 or Ortus using BMT as a scraper. I'm talking about decoupling scrapers, UI and the actual storage of the metadata.


FWIW, this is sort of what I'm getting at, it's the 4th API we've got going now, we've got Sage's API, Phoenix, Ortus, and now SMM.

Or say you used SMM and had all your metadata defined there, but wanted to use the Phoenix or Ortus STV. No problem, all your data is there.

Stranger did you read my goals. My goal is to get rid of some of the fragmentation and allow people to use whatever scraper they want. A combination API if you will. I know fuzzy would say that wouldn't be necessary if we wrote to the db, we don't and we are not going to so this helps combine all the efforts regardless of the UI.

Originally Posted by Fuzzy

Well, as far as SMM, the only reason he's given for using his own UI, was the query mechanics. There's no actual data he's using that can't be stored in sage's DB. .

Again I have told you it just doesn't seem to sink in.

MyMovies already had a database in SQL. They have built in functions there is no way for me to get those functions necessary to bring mymovies data into sage into the db. I would have to query mymovies sql get results then query the default db to get matching results. If I am going to query the sql to perform the functions (which I have to) no reason not to use it. I guarantee if I wrote all the mymovies information to the default db someone would bitch I am writing to much to the default db.

If someone was using ortus or bmt as well guess what with the plugin system listeners it would come down to who sage's api fired last that would get into the database. How in the heck is that ideal? That is not sage's fault but you seem to pass that off as a non issue when I guarantee it would be a huge issue.

Originally Posted by Skirge01

But, again, having done some programming, I can definitely understand why these guys would prefer to design and query their own DB. It's simply more efficient and allows them greater flexibility. They can design it to behave exactly how they want.

Thank you Skirge for understanding. I think all this conversation is doing is confusing the basic users when there is no need for more of that.

I feel like Fuzzy has some strong hate against what we are doing or trying to do and if that is the case that is the case. If you think this is making sagetv fragmented hell I can quit at anytime and spend my free time doing other things. I think what I and ortus and phoenix have done add to the sagetv experience not take away.

[JetreL]

Originally Posted by PLUCKYHD

We have been over this again and again
..snip..
Stranger did you read my goals.
..snip..
Again I have told you it just doesn't seem to sink in. ..snip..
I guarantee if I wrote all the mymovies information to the default db someone would bitch I am writing to much to the default db.
..snip..
I feel like Fuzzy has some strong hate against what we are doing or trying to do and if that is the case that is the case.

I think you are letting this get under your skin. It'll be "Ok!"

[PLUCKYHD]

Originally Posted by JetreL

I think you are letting this get under your skin. It'll be "Ok!"

You would to promise if someone just kept attacking the way you were doing something insisting it was the "wrong" way.