Web GUI extremely slow

[adone36]

I have a Win 7 Sage 7 server which houses TV recordings and another Win 7 computer which contains music and video imports. I installed BMT and it got all the metadata but I have a few problems.

1) If I need to change a show, I go in find the right data and hit save. It can take 5 or more min to save, but usually has the right info on next scan.

2) If I do a batch scan, (say 25 imports formatted "Torchwood S01E01.mkv" it may take hours and come up with 19 good 6 failed. They all have identical format and I can go back and fix the others one at a time and it populates them fine.

I also have the same problems as others with Sage not showing the ep titles etc, just 1-26 "Torchwood" though BMT retrieves these.

[stuckless]

Originally Posted by adone36

I have a Win 7 Sage 7 server which houses TV recordings and another Win 7 computer which contains music and video imports. I installed BMT and it got all the metadata but I have a few problems.

1) If I need to change a show, I go in find the right data and hit save. It can take 5 or more min to save, but usually has the right info on next scan.

When you save a show, it may take some time to grab all the fanart (like about 30 seconds), but I've never seen it take much longer than that. Perhaps you could forward me the logs, i suspect that there are some network timeout issues there, if it's taking that long.

2) If I do a batch scan, (say 25 imports formatted "Torchwood S01E01.mkv" it may take hours and come up with 19 good 6 failed. They all have identical format and I can go back and fix the others one at a time and it populates them fine.

I'll need the logs, when the this happens. It could be that it gets a network error on those 6. BMT does not retry when it gets a network error.

I also have the same problems as others with Sage not showing the ep titles etc, just 1-26 "Torchwood" though BMT retrieves these.

I assume you mean in folder view and not in the recordings view? This is not really a bmt issue, at least I'm not willing to fix it. I recommend installing a plugin, like custom folders, that may show the correct information from the correct fields.

[adone36]

Thanks for your help.

I noticed that tv shows on the server itself update immediately and batches take seconds to run. The ones which crawl are on the remote machine. All paths are unc. I originally could not access the files on the remote because I was running sage as a service on the local machine and had to change to a common admin account. Since then, i can access these remote files in Sage perfectly.

How do I turn on logging to check what's going on??

[stuckless]

Originally Posted by adone36

Thanks for your help.

I noticed that tv shows on the server itself update immediately and batches take seconds to run. The ones which crawl are on the remote machine. All paths are unc. I originally could not access the files on the remote because I was running sage as a service on the local machine and had to change to a common admin account. Since then, i can access these remote files in Sage perfectly.

How do I turn on logging to check what's going on??

It appears that I'm eating some exception like the timeout exception, so I need to go into the code and make sure that I'm losing them totally. This was pointed out by another person, and I may look at implementing a re-try que for automaticly scanned items, if a timeout exception happens.

In terms of just normal logging, you can set the log level to debug for phoenix-api in the Web UI under Configuration -> Log4j. This should write more information to the phoenix.log (in the logs directory). But, exceptions are always logged, so changing the log level from info to debug will mean more noice int he logs, not more exceptions.

[adone36]

A couple questions,

Files are being updated with metadata. Why is it I have a few .properties files in the root directory housing DVDs, and .properties files with the AVI, mkv, etc files, but the overwhelming majority are in the cache directory under the SageTV directory?

I still cannot get the web GUI to work on files on the remote machine. Local files update fine. The remote ones are retrieved instantly, all the options (look-ups, etc, etc) work, but when I hit save I just get the spinning saved logo. The phoenix.log just shows a last entry of: Saving- with all the metadata. After the next scan, the dvd shows with metadata.

phoenix metadata log just shows auto followed by skip for hundreds of dvds. I guess this is just verifying that previous lookups are still there?

[stuckless]

Originally Posted by adone36

A couple questions,

Files are being updated with metadata. Why is it I have a few .properties files in the root directory housing DVDs, and .properties files with the AVI, mkv, etc files, but the overwhelming majority are in the cache directory under the SageTV directory?

I still cannot get the web GUI to work on files on the remote machine. Local files update fine. The remote ones are retrieved instantly, all the options (look-ups, etc, etc) work, but when I hit save I just get the spinning saved logo. The phoenix.log just shows a last entry of: Saving- with all the metadata. After the next scan, the dvd shows with metadata.

phoenix metadata log just shows auto followed by skip for hundreds of dvds. I guess this is just verifying that previous lookups are still there?

The cache area is just that, cached data. Those .properties are used by the caching engine, but are not actually metadata properties, they are cache properties. You can ignore them, delete them, etc.

Unfortunately, i can't test the remote machine scenario. All my files are on my main sagetv server. It seem that it is hitting some sort of issue, and it's add the the phoenix.log doesn't show anything. Do you see any errors in the jetty log or the sagetv_0.txt file?

[adone36]

I called the gui "slow" when I think the real problem is that the gui makes the change and then locks up. Clicking on the screen make the spinning save logo go away and I thought it was done.

Possible bug: I had a tv show in the gui I wanted to transfer to Sage recordings. I hit transfer under tools and it started processing thousands of video files. I though it would just do the one (I guess just checking "sage recording" and saving would be the way to go), but anyhow the files included all of the files in excluded directories.

Where is the sagetv_0.txt file created? I couldn't find it.
How do I turn on jetty logging???

[adone36]

OK serious bug:

This morning I moved tv eps to the server and using the gui I went to each ep and fixed the date, etc and then selected "sage recording" and hit save. On rescan Sage is now adding thousands of video files, including excludes to its recordings. Import videos as recordings is not checked. So I guess it set "import into Wiz.bin" to true to add the show and didn't set it back?

Now all video files are marked as sage recordings. If I restore Wiz.bin and set to restore the recording directories, when I reset the restore switch to false, it will add all the videos back on the next scan. How can I prevent this?

[stuckless]

Originally Posted by adone36

I called the gui "slow" when I think the real problem is that the gui makes the change and then locks up. Clicking on the screen make the spinning save logo go away and I thought it was done.

When you save an item, it does have to download fanart, etc. That process can take anywhere from just a couple of seconds to a couple of minutes depending on how much fanart, connection timeouts, etc.

Possible bug: I had a tv show in the gui I wanted to transfer to Sage recordings. I hit transfer under tools and it started processing thousands of video files. I though it would just do the one (I guess just checking "sage recording" and saving would be the way to go), but anyhow the files included all of the files in excluded directories.

It's not a bug... basically the tools menu acts on your entire collection. I think it does give a warning when you use an item from the tools menu. Perhaps I could provide a better warning.

Where is the sagetv_0.txt file created? I couldn't find it.
How do I turn on jetty logging???

sagetv_0.txt is the core sagetv log and it's in the sagetv home directory. It may only get created if debug logging is enabled (you'd have to go looking for that in the sagetv ui settings)

Originally Posted by adone36

This morning I moved tv eps to the server and using the gui I went to each ep and fixed the date, etc and then selected "sage recording" and hit save. On rescan Sage is now adding thousands of video files, including excludes to its recordings. Import videos as recordings is not checked. So I guess it set "import into Wiz.bin" to true to add the show and didn't set it back?

The setting on the single metadata page is NOT tied and any way to the imprt tv files are recordings. The global import tv as recordings is only used when doing automatic scans. From the webui, you have to explicity tell it to happen when you do the scan. THe other thing, if bmt is scanning 1000s of files, then you did it to scan 1000s of files? I'm assuming somewhere duing this process, you hit "Find Metadata", responded to the options an clicked scan? If not, then are you sure that BMT is the plugin that is adding your files as recordings? The import tv as recordings flag will only import files that are TV files as recordings. It will not import Movies as recordings (at least I don't think it will)

Now all video files are marked as sage recordings. If I restore Wiz.bin and set to restore the recording directories, when I reset the restore switch to false, it will add all the videos back on the next scan. How can I prevent this?

Do you have .properties for each of these videos? If so, Sage will process those automatically. As for BMT, you can disable the automatic update or remove the plugin.

[adone36]

Now that I think about it, when I hit "import videos as recordings" under tools (it does not warn in any way, btw) last night it marked all the videos as recordings THEN. The subsequent scan just put them in recordings. It was not caused by changing the data on a few shows individually the next morning.

I do not have fanart enabled. Almost all DVD folders have a folder.jpg file.
I only use BMT and Commercial skip.

The real bug would seem to be that these imports include ALL of the EXCLUDES. If excluded, BMT should completely ignore the files.

So assuming I restore wiz.bin, how can I remove the current BMT data and have it rescan?

[stuckless]

Originally Posted by adone36

Now that I think about it, when I hit "import videos as recordings" under tools (it does not warn in any way, btw) last night it marked all the videos as recordings THEN. The subsequent scan just put them in recordings. It was not caused by changing the data on a few shows individually the next morning.

I thought there was a warning on that option... but I'll check that tonight. But, if you did use that option, then it will import all videos as recordings.

The real bug would seem to be that these imports include ALL of the EXCLUDES. If excluded, BMT should completely ignore the files.

I'm not sure... the exclude is currently only used during the automatic update. When you do actions from the web ui, like browse folders, etc, the exclude pattern is not used.

So assuming I restore wiz.bin, how can I remove the current BMT data and have it rescan?

I think there is an option in the Tools to clear metadata... you can use that... or if you browse a folder, then you can use the option on the folder view, instead of the option from the tools menu.

[adone36]

I got everything straightened out now.

I still think BMT should completely ignore excluded directories. It does not make much difference if a guy has 50 shows, 30 dvds, and 100 avi, but I have 550 dvds and 4300 media files I exclude. Even with very fast processors this can take time.

I notice that for imported tv files, the "record date" is set as the file creation date. I think the original air date should be used. Most of the time you d/l an episode if you missed recording it or you deleted by accident, etc.. When added into recordings you must use a sort by orig air date to show then in sequence when listed with others. If someone can think of a reason to use the creation date, make it an option as to what date is used for the recording date.

How can I force BMT to update a video without using the GUI one at a time???

[stuckless]

Originally Posted by adone36

I still think BMT should completely ignore excluded directories. It does not make much difference if a guy has 50 shows, 30 dvds, and 100 avi, but I have 550 dvds and 4300 media files I exclude. Even with very fast processors this can take time.

I assume you mean the exclude regex that bmt uses for it's automatic scanner? If so, then then while bmt doesn't currently exclude those from it's view, you will, in future be able to exclude those. BMT piggy backs on Phoenix... actually most of bmt is Phoenix, and as such, when Phoenix is released, there will be more information on customizing views, etc, which will allow you to do what you need to to.

I notice that for imported tv files, the "record date" is set as the file creation date. I think the original air date should be used. Most of the time you d/l an episode if you missed recording it or you deleted by accident, etc.. When added into recordings you must use a sort by orig air date to show then in sequence when listed with others. If someone can think of a reason to use the creation date, make it an option as to what date is used for the recording date.

I'm sure that code can be tweaked, at some time, to use the original air date. It's not making a very high spot on my priority list, though I tend to sort all tv by season/episode, so when it was actually recorded is less important to me. But, I've added an issue, so this doesn't get forgotten.

How can I force BMT to update a video without using the GUI one at a time???

Are you asking how do you update a file one at a time, without using the gui? If so, then the automatic plugin does that.

Or, Are you asking how do you update more than one file, from the gui? If so, then just navigate to the view/folder and then hit the "Update Metadata" button which still in the folder view, and it will find metadata for everything in that folder, and it's children.

[adone36]

Classic example of why I think "exclude" should be more than just a filter:

I had hundreds of DVDs that did not resolve correctly (abbreviations, multiple movies with same name, etc). i fixed them all to BMT naming conventions. After a scan the gui missing metafile tab is useless because it brings up thousands of titles in excluded directories.

I have my Tivo set to download certain shows. A program d/ls the show, removes encryption and commercials, and then saves as an mpg in a sage import directory.

They are in format "TVShowName - EpisodeName (11-14-2010)"

I haven't tested how BMT will parse these, but if you do make changes to "record date", etc, you might want to include parsing for TivoHD.

By forcing an update I meant in terms of the automatic parsing (I can't use the gui on dvds until I figure why it doesn't work properly on remote directories as all my dvds are on a file server). So basically I meant renaming the directory "Gladiator" to "Gladiator (2000)" to get the correct data. This forces sage to use the new folder.jpg if changed and I assume it makes BMT add the "New" DVD.

I just noticed the strangest thing. Yesterday I changed dozens of movies. I added (year) to get them to resolve properly. I had had them originally without the parentheses. I noticed the scan symbol was on for hours. I looked at the phoenix-metadata.log and it showed it was updating 1 movie every 20 minutes give or take. Nearly the same amount of (long) time for each. Attached are logs.

How does BMT add new metadata?? Is it done when Sage scans and then queued??

[stuckless]

The bmt automatic plugin will not block. It basically adds all new files to a queue and then processes them, so that it does not block the sage event queue. So, I'm not sure why sage would have the scanning icon lit up for hours.

I can see in the logs that it did take that long, and I'm not sure why. All files are processed by a single thread, so if it takes a long time to download fanart, etc, then it will back up the bmt queue. There's not indication in the logs as to why it took so long.

[adone36]

Where is the queue file located and can it be edited or examined?

In the logs there are many, many entries: "no file segment for sage file". They all reference show names and tuners. These shows have never been recorded or scheduled. What are these?

[stuckless]

Originally Posted by adone36

Where is the queue file located and can it be edited or examined?

In the logs there are many, many entries: "no file segment for sage file". They all reference show names and tuners. These shows have never been recorded or scheduled. What are these?

You can't actually see the queue... and I'm not sure what those files are... they may actually be airings (from the epg).