Plugin: Sage Artifact Delete (SAD)

[KJake]

Originally Posted by Slugger

It is preconfigured. By default it'll delete every extension (blank means delete them all).

Perfect then!

[ranger]

Not sure what the root cause of this was, but I will tell you what I did and what I found afterwards. I am running SageTV 7.0.11 with multiple plugins installed, among them Jetty, BMT, and SAD. From the BMT web interface this morning I removed a few folders from the video sources in the BMT web interface. When I came home tonight, all the video files in the folders and the artifact files had been deleted. The folders are still intact, but something triggered the deletion of the video files themselves. The sad.log file has a record of all the file deletions along with the artifact file deletions.
I don't know if there is an explanation for the behavior or if it is a bug in one of the 2 plugins. If no one here has any ideas, I'll inquire in the BMT area.
This is an excerpt from sad.log

2010-06-17 07:21:36,541 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Serenity (2005)\Serenity (2005).avi
2010-06-17 07:21:36,555 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Serenity (2005)\Serenity (2005).avi.properties
2010-06-17 07:21:36,570 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Serenity (2005)\Serenity (2005).dvdid.xml
2010-06-17 07:21:36,586 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Serenity (2005)\Serenity (2005).jpg
2010-06-17 07:21:36,603 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Serenity (2005)\Serenity (2005).nfo
2010-06-17 07:21:36,618 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Serenity (2005)\Serenity (2005).tbn
2010-06-17 07:21:36,640 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\The Day the Earth Stood Still (1951)\The Day the Earth Stood Still (1951).avi
2010-06-17 07:21:36,654 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\The Day the Earth Stood Still (1951)\The Day the Earth Stood Still (1951).avi.properties
2010-06-17 07:21:36,670 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\The Day the Earth Stood Still (1951)\The Day the Earth Stood Still (1951).dvdid.xml
2010-06-17 07:21:36,694 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\The Day the Earth Stood Still (1951)\The Day the Earth Stood Still (1951).jpg
2010-06-17 07:21:36,717 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\The Day the Earth Stood Still (1951)\The Day the Earth Stood Still (1951).nfo
2010-06-17 07:21:36,731 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\The Day the Earth Stood Still (1951)\The Day the Earth Stood Still (1951).tbn
2010-06-17 07:21:36,751 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\TVShows\Doctor Who\Doctor Who - 22x2 - Attack of the Cybermen (2).avi
2010-06-17 07:21:36,769 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\TVShows\Doctor Who\Doctor Who - 22x2 - Attack of the Cybermen (2).avi.properties
2010-06-17 07:21:36,787 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Sky Captain and the World of Tomorrow (2004)\Sky Captain and the World of Tomorrow (2004).avi
2010-06-17 07:21:36,806 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Sky Captain and the World of Tomorrow (2004)\Sky Captain and the World of Tomorrow (2004).avi.properties
2010-06-17 07:21:36,820 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Sky Captain and the World of Tomorrow (2004)\Sky Captain and the World of Tomorrow (2004).dvdid.xml
2010-06-17 07:21:36,835 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Sky Captain and the World of Tomorrow (2004)\Sky Captain and the World of Tomorrow (2004).jpg
2010-06-17 07:21:36,848 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Sky Captain and the World of Tomorrow (2004)\Sky Captain and the World of Tomorrow (2004).nfo
2010-06-17 07:21:36,861 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Sky Captain and the World of Tomorrow (2004)\Sky Captain and the World of Tomorrow (2004).tbn
2010-06-17 07:21:36,895 WARN  [FileDeleteTask]: File deleted: \\rangerone\VideoI\TVSeries\Monty Python's Flying Circus\Monty Python's Flying Circus - 3x6 - The War Against Pornography.jpg
2010-06-17 07:21:36,911 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Another Thin Man (1939)\Another Thin Man (1939).avi
2010-06-17 07:21:36,925 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Another Thin Man (1939)\Another Thin Man (1939).avi.properties
2010-06-17 07:21:36,938 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Another Thin Man (1939)\Another Thin Man (1939).dvdid.xml
2010-06-17 07:21:36,955 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Another Thin Man (1939)\Another Thin Man (1939).jpg
2010-06-17 07:21:36,971 WARN  [FileDeleteTask]: File deleted: \\rangerone\Media\Movies\Another Thin Man (1939)\Another Thin Man (1939).nfo

[Slugger]

It would have been better had you had debug level on for the SAD logging, but based on that log output, something was deleted and SAD found all of those artifacts for whatever was deleted. They all look like valid artifacts except for the avi files, I would assume? However, for some reason Sage told SAD that the avi file was NOT a registered SageTV object. SAD only deletes files if they aren't registered as a SageTV object (and to be extra safe, I do that check twice before deleting the file).

	public void run() {
		for(File f : files) { // files is a collection of discovered artifacts
			if(fake)
				LOG.warn("File skipped because SAD is in pretend mode: " + f);
                        // This is the second check with the API before deleting
			else if(API.apiLocalUI.mediaFileAPI.GetMediaFileForFilePath(f) == null) {
				boolean result;
				if(rmNonEmptyDirs)
					result = FileUtils.deleteQuietly(f);
				else
					result = f.delete();
				if(result)
					LOG.warn("File deleted: " + f);
				else
					LOG.error("Delete FAILED: " + f);
			} else
				LOG.warn("File skipped because it is a registered SageTV media object: " + f);
		}
	}

Based on the limited log output, I'd say SAD did what it was suppose to do. Not sure why the avi files were not registered with Sage, but for whatever reason, they weren't and so that's why SAD deleted them. As a precaution, you could add file extensions that SAD should ignore. However, I'd probably try to figure out why the avi files were no longer registered with Sage. I won't say it's a BMT or a Sage problem. Hell, I won't even rule out a SAD bug, but based on the current evidence, I don't see any problems with SAD here (though I'm more than willing to be proven wrong).

[ranger]

Originally Posted by Slugger

I won't say it's a BMT or a Sage problem. Hell, I won't even rule out a SAD bug, but based on the current evidence, I don't see any problems with SAD here (though I'm more than willing to be proven wrong).

Thanks for taking a look. I am not sure what triggered this off either. If I find something that makes sense, I'll post back, but as you say, the log doesn't have enough detail to tell us why.

[nyplayer]

Originally Posted by ranger

Not sure what the root cause of this was, but I will tell you what I did and what I found afterwards. I am running SageTV 7.0.11 with multiple plugins installed, among them Jetty, BMT, and SAD. From the BMT web interface this morning I removed a few folders from the video sources in the BMT web interface. When I came home tonight, all the video files in the folders and the artifact files had been deleted. The folders are still intact, but something triggered the deletion of the video files themselves. The sad.log file has a record of all the file deletions along with the artifact file deletions.
I don't know if there is an explanation for the behavior or if it is a bug in one of the 2 plugins. If no one here has any ideas, I'll inquire in the BMT area.
This is an excerpt from sad.log[/CODE]

The same thing happened to me when I had tested SAD and Sagetv 7 on my test machine and I couldn't figure out why my test import AVI files disappeared. I have to retest it and open up all the logging.

[Slugger]

Originally Posted by nyplayer

The same thing happened to me when I had tested SAD and Sagetv 7 on my test machine and I couldn't figure out why my test import AVI files disappeared. I have to retest it and open up all the logging.

Ok, before you do any more testing, let me add some more logging output to SAD and put a new build in the repository. Basically, I want to log more details of the event that's triggering the delete. Will try to do it today, but may not have time. You may have to nudge me on the weekend if I forget.

[voorhees]

Slugger, I just installed SAD based on your mention in the SJQ thread that the SJQ File Cleaner will eventually be replaced with SAD. After installing and configuring, I picked a few items to test with the artifact finder to see how it works. I picked 5 mpg files where 4 were recordings and 1 was an imported video. The SAD log (below) gives me an error for all items telling me they are not registered SageTV media objects. All five items show up in SageTV, with the 4 recordings showing up in the Recordings area and the 1 imported video showing up in the Videos area. I am assuming the mpg files are what you should be picking as a media object, is that correct? I am also assuming that if SAD says all of my items are not registered SageTV media objects, that it will not delete the artifacts, is that correct? Any thoughts as to why my Recorded TV items are not registered SageTV media objects?

2010-06-20 12:34:48,453 ERROR [Plugin]: Selected file is not a registered SageTV media object! [D:\shares\SageTV\SageTV Recordings\AngelinaBallerina-863707-0.mpg]
2010-06-20 12:37:20,421 ERROR [Plugin]: Selected file is not a registered SageTV media object! [D:\shares\SageTV\SageTV Recordings\TopGear-1258684-0.mpg]
2010-06-20 12:47:01,906 ERROR [Plugin]: Selected file is not a registered SageTV media object! [D:\shares\SageTV\SageTV Recordings\AngelinaBallerina-830251-0.mpg]
2010-06-20 12:50:11,359 ERROR [Plugin]: Selected file is not a registered SageTV media object! [D:\shares\SageTV\Movies and Videos\Movies Recorded On TV\Armageddon.mpg]
2010-06-20 12:51:09,562 ERROR [Plugin]: Selected file is not a registered SageTV media object! [D:\shares\SageTV\SageTV Recordings\Up-2547602-0.mpg]

Thanks.

[Slugger]

This appears to be a bug with the Sage API calls. You can test this by launching studio and going to the expression evaluator and running the following:

GetMediaFileForFilePath("D:\\shares\\SageTV\\SageTV Recordings\\AngelinaBallerina-863707-0.mpg")

Simply use each file path printed in your sad log output. If studio returns null then Sage believes the files aren't media objects (at least via that API call). And if that's the case then that's a bug with the core API. If studio doesn't return null then Greg's wrappers are inadvertently returning null and if that's the case then I'll need to talk with him.

The problem in SAD is that the API call is returning null so SAD then believes the file isn't a media object. I'm not able to reproduce that here, but that's basically what's going on. If studio does not return null with the above expression then we need to know which version of SAD and all its listed dependencies you're running in order to be able and track this down.

[razrsharpe]

Originally Posted by Slugger

Based on the limited log output, I'd say SAD did what it was suppose to do. Not sure why the avi files were not registered with Sage, but for whatever reason, they weren't and so that's why SAD deleted them.

Any chance this could have something to do with it:

Originally Posted by ranger

I removed a few folders from the video sources in the BMT web interface.

I can definitely see the act of removing video source folders unregistering those video objects from the db... Could SAD be getting confusing and thinking that those artifacts should be deleted? Did your testing include removing an import directory and see what SAD did?

[Slugger]

Originally Posted by razrsharpe

I can definitely see the act of removing video source folders unregistering those video objects from the db... Could SAD be getting confusing and thinking that those artifacts should be deleted? Did your testing include removing an import directory and see what SAD did?

SAD simply reacts to MediaFileRemoved events fired from the core. If removing an import directory causes MediaFileRemoved events to be fired then that's the reason. Based on all the info we now have, I'd say that's the case. SAD only ever does something when a MediaFileRemoved event is fired. If no event fires then SAD does nothing. I didn't test removing an import directory because that action in and of itself is irrelevant. SAD listens for and acts upon fired MediaFileRemoved events only.

[voorhees]

Originally Posted by Slugger

This appears to be a bug with the Sage API calls. You can test this by launching studio and going to the expression evaluator and running the following:

GetMediaFileForFilePath("D:\\shares\\SageTV\\SageTV Recordings\\AngelinaBallerina-863707-0.mpg")

Simply use each file path printed in your sad log output. If studio returns null then Sage believes the files aren't media objects (at least via that API call). And if that's the case then that's a bug with the core API. If studio doesn't return null then Greg's wrappers are inadvertently returning null and if that's the case then I'll need to talk with him.

The problem in SAD is that the API call is returning null so SAD then believes the file isn't a media object. I'm not able to reproduce that here, but that's basically what's going on. If studio does not return null with the above expression then we need to know which version of SAD and all its listed dependencies you're running in order to be able and track this down.

Slugger, thanks for the help.

I tried all five of them in the expression evaluator in studio and they all returned null. So, from your statement above, it looks like SageTV believes all of my recorded TV and imported videos are not media objects. Is that correct? It handles everything OK, so how does it believe they are not media objects? That can not be good. So, I am a little unclear, does that mean it is a bug in SageTV and I need to submit a bug report?

Thanks.

[Slugger]

Originally Posted by voorhees

Slugger, thanks for the help.

I tried all five of them in the expression evaluator in studio and they all returned null. So, from your statement above, it looks like SageTV believes all of my recorded TV and imported videos are not media objects. Is that correct? It handles everything OK, so how does it believe they are not media objects? That can not be good. So, I am a little unclear, does that mean it is a bug in SageTV and I need to submit a bug report?

Thanks.

Actually, didn't SAD already delete those files listed in the log? If so, then the calls will return null because the files don't exist anymore. Try the same thing, but with files that are still on the drive. This is one of those weird issues where you don't really know there's a problem until it's too late. Really, you need to test all your current recordings and make sure they don't return null. But then even if they don't, perhaps it's something else that's making the core unregister them as media objects just before the event is fired, which is why SAD ends up deleting them. Bottom line, SAD asks the API (twice) before deleting a file if it's a registered media object. Only if that API call returns null twice does SAD actually delete the file.

This is one of those cases where I really wish it was a SAD bug that was causing this, but I'm just not seeing it. SAD listens for MediaFileRemoved events, searches for artifacts of the media file it's told was deleted, filters out any artifacts that are registered media objects, schedules the remaining files to be deleted, then the files are deleted only after asking the API one more time if it's not a registered media object. If someone sees a bug in my code, I'd love to hear about it (and fix it).

EDIT: Right, you were just testing with those mpgs and SAD told you they weren't registered. You went to studio and ran the expression evaluator and it returned null for each of those API calls (you remembered to double up on the backslashes?). Assuming that's all true, then, yes, Sage believes those files aren't registered media objects and an issue should be opened with Sage.

[Slugger]

Originally Posted by razrsharpe

Any chance this could have something to do with it:

I can definitely see the act of removing video source folders unregistering those video objects from the db... Could SAD be getting confusing and thinking that those artifacts should be deleted? Did your testing include removing an import directory and see what SAD did?

Thinking about this again... I've assumed that a MediaFileRemoved event means the media file was actually physically deleted. I never considered the case where an object is unregistered by the core, but the file is not actually deleted (is this actually possible?). If it is, then this case should be handled (i.e. never delete the file actually referenced in the event). I think it should go one step further... if the media file isn't actually deleted then should SAD even be deleting the artifacts? In other words, if it's just the case where the object is unregistered, but not deleted then shouldn't SAD just ignore the event and not delete any artifacts? Thoughts?

[sic0048]

Originally Posted by Slugger

... if the media file isn't actually deleted then should SAD even be deleting the artifacts? In other words, if it's just the case where the object is unregistered, but not deleted then shouldn't SAD just ignore the event and not delete any artifacts? Thoughts?

I would agree with that statement. If the file exists, then the artifacts probably shouldn't be deleted either. At least I cannot think of a situation where you would want to delete those files.

[Slugger]

The weirdness has been solved, I believe. Turns out core 7.0.11 added some new, undocumented features, which would explain all the behaviour described here.

The fix will be to only delete artifacts if the media file object SAD is being told about was actually deleted (and not just unregistered from wiz.bin), otherwise the event will be ignored and nothing will be deleted.

Fix might be a day or two if I don't have time before I play baseball tonight. I'm on the road to Toronto for work tomorrow, Thursday is busy, and I'm in Ottawa on Friday so if I can't find the time before 6pm tonight then it might be the weekend before it's fixed, but at least we now know the issue and the fix.

Issue 134 has been opened for this. If you want to be notified as soon as the fix is checked in then you'll want to star this issue at the project site.

[Slugger]

I love it when fixes are easier than you expect. Just a few lines (5 to be exact) needed to clean up this issue. A new version of SAD is now in the repository with this fix.

[vividweb]

I haven't tried installing this but am following along as I think it is a great plugin (just like all of Sluggers work).

Would it be possible to move the files that are being triggered for delete to a common folder where you can confirm deletion? Kind of a "Recycle Bin", it would add an extra step, but I know I would feel a lot better about using the plugin if I new I could recover from an accidental delete.

[electronics4lif]

Originally Posted by Slugger

This appears to be a bug with the Sage API calls. You can test this by launching studio and going to the expression evaluator and running the following:

GetMediaFileForFilePath("D:\\shares\\SageTV\\SageTV Recordings\\AngelinaBallerina-863707-0.mpg")

Simply use each file path printed in your sad log output. If studio returns null then Sage believes the files aren't media objects (at least via that API call). And if that's the case then that's a bug with the core API. If studio doesn't return null then Greg's wrappers are inadvertently returning null and if that's the case then I'll need to talk with him.

The problem in SAD is that the API call is returning null so SAD then believes the file isn't a media object. I'm not able to reproduce that here, but that's basically what's going on. If studio does not return null with the above expression then we need to know which version of SAD and all its listed dependencies you're running in order to be able and track this down.

I looks like I am also experiencing this bug. SAD.log says that all of my 'test' files are not media objects. I am subscribed to see if something gets resolved with this bug.

[Slugger]

Originally Posted by electronics4lif

I looks like I am also experiencing this bug. SAD.log says that all of my 'test' files are not media objects. I am subscribed to see if something gets resolved with this bug.

Someone will have to report it to Sage because I am not able to reproduce this bug.

[Slugger]

Originally Posted by vividweb

I haven't tried installing this but am following along as I think it is a great plugin (just like all of Sluggers work).

Would it be possible to move the files that are being triggered for delete to a common folder where you can confirm deletion? Kind of a "Recycle Bin", it would add an extra step, but I know I would feel a lot better about using the plugin if I new I could recover from an accidental delete.

I might be convinced to (optionally) move artifacts to a "recycle bin" (maybe even the actual recycle bin on Windows systems), but cleanup of the recycle bin would have to be manual (i.e. I won't be adding any features to SAD to review this recycle bin directory and delete files from it - too much extra work for something that I'll never use). If that's something that you want then open an issue ticket on the project site. This definitely wouldn't be high priority and wouldn't get tackled until a "rainy day" arrives (i.e. autumn most likely).