Plugin: Sage Job Queue (SJQ) - Page 22

Slugger · #**421** 02-13-2009, 06:39 PM

Quote:

Originally Posted by perfessor101

Hello, Slugger
you were right ... my logs are getting deleted as expected.

I have a lot of recordings from previous SageTV and previous SJQ (or Dirmon) installs. (1,265 recordings)

I got used to using marker files to tag the levels of processing that way when I have to start over with a new install nothing needs reprocessing.

However, I'm guessing all the media files that have never been processed by SJQ 2.1 are getting run through every rule check each time the Queue Loader runs ... and with the number of media files I currently have it seems to be confusing something between SageTV, Jetty and SJQ.

What exactly is the issue? Regardless of whether SJQ has or hasn't processed a media object, it's going to run the tests for every object on every run of the QueueLoader. Media objects processed by older versions of SJQ are remembered as you upgrade (unless you delete your sjq.sqlite db file when upgrading, which you shouldn't do unless otherwise told). Please explain what you mean by, "... it seems to be confusing something between SageTV, Jetty and SJQ."

On a side note, this paragraph of yours got me to open up the source code and have a quick peek. Turns out the code currently loops through the media objects, performs your tests on each object and if it passes all tests attempts to add the task to the queue. If the task has already been completed on the media object then the addition to the queue is dropped. A significant performance gain can be made to the QueueLoader by first checking to see if the task has already been completed before even processing the tests for the object. Issue 51 was created to address the issue. You may want to star the issue to get a notification when a build with the fix is ready. However, as far as I can tell, this bug is just a performance issue bug, as it works now, it should not be getting "confused." Please do elaborate on how you're seeing things getting "confused."

Quote:

I was curious is there a way of marking all tasks for all media files done?
(if there was a command I could type ...)

Another possibility I have been thinking of is changing the rules and the tasks to a set of "blank" rules and "blank" batch files so that SJQ could "process all files"

The way I've always done this when I needed to (during early development and testing of SJQ) is the latter. I'd change the command my task ran to a simple batch file that simply returned success. There is no way to do this automatically in SJQ.

Quote:

I've up, down and re-graded Jetty (1.4, 1.5) and tried sjq-2.1.1.186 and sjq-2.1.1.236 and I'm thinking maybe it's time to switch to linux

Thanks for your time,
Bobby

I think I'm missing something here. Why Linux? Nothing wrong with Linux, but I doubt a switch in OS is going to solve any issues you might be seeing since the SJQ code is common for all platforms unless you think there may be a problem with Jetty on Windows? Again, your "things get confused" statement really needs some details to try and figure things out.

perfessor101 · #**422** 02-14-2009, 01:12 AM

Hello,
The thing that is happening is that when I start SageTV in program mode with Jetty and SJQ running it can peg the cpu utilization on one core (11%-25%, mostly 25%).
If I run SageTV without SJQ\jetty it hovers in the 0%-1% range even while recording. (not that recording would really matter with hardware encoders)

Currently when I log into the SJQ page I'll get 27 pages of completed tasks.
Then after a few clicks, or a few minutes my next click will give me an SJQ page with all fields listed blank. (reponses like from a fresh/clean SJQ install)
Any links I click on SJQ respond with "blank pages" (The SJQ web page "headers" are there ... just no data in the fields)
It's like the connection to the database is dropped.
When this happens I try to "touch" the context file but the only way I seem to get the SJQ data back is by stopping and restarting SageTV.

* also the database has blown up to 405 MB in about the last week
* I'll wait for your response ... but I think I will extract my rules and tasks from the database ... manually process my latest recordings ... and then run media files through all tasks with batch files that return success

When I woke on Wednesday night the SageTV server computer locked up and the Leds on the memory (ballistix tracer ddr2 ram) were moving about once a second. I rebooted the computer and it seemed okay. (SageTV was recording from all tuners and the system seemed to be working)

So I started of thinking of ways to reduce the load on SJQ / Jetty by possibly marking tasks done to speed up the Queue loader process and maybe reduce the Queue's depth.

The SageTV server locked up again on Friday night around 7pm (woke up at 9:30pm) so I restarted the computer and removed Jetty from the SageTV runnable list.
At midnight it finishes the last recording ... at that point I'm going to run memtest and maybe Prime95 for a bit to check the hardware hasn't started to die.

* ran through three passes of memtest 2.01 with no problems

After the hardware testing at midnight I'll put Jetty / SJQ back in the runnable classes line and grab some screen captures of the SJQ web pages to send.

Why linux?
Windows can frustrate me ... Don't get me wrong, Windows XP professional runs pretty smoothly for me ... but as I only have one license for XP Pro and one for MCE 2005.
(and one for Windows Me ... and one for Windows 98 and one for Windows 3.1 and Dos 6.22 and ...)

I get a week off at work during Inventory the last week of February so I'll try reinstalling Windows XP MCE 2005 to see if they have improved compatibility with Service Pack 3.

Thanks for your time and help
Bobby

ps - Also with unRaid I currently have large numbers of files in single UNC shares I'm not sure if the number of media files in a directory would matter?
\\storage\recordings - 3TB

wayner · #**423** 02-14-2009, 08:53 AM

I don't know that I have posted about this but I am having a very similar issue to the post above - when I run Sage with SJQ/jetty it causes SageTVservice to use 50% of my CPU and the SJQ web client freezes after a few minutes in the way that Bobby describes. The only way to fix it is to restart Sage but then this issue comes back after a fw minutes. I have temporarily taken Jetty out of load_at_startup in the sage.properties.

Slugger · #**424** 02-14-2009, 10:18 AM

Quote:

Originally Posted by perfessor101

Hello,
The thing that is happening is that when I start SageTV in program mode with Jetty and SJQ running it can peg the cpu utilization on one core (11%-25%, mostly 25%).
If I run SageTV without SJQ\jetty it hovers in the 0%-1% range even while recording. (not that recording would really matter with hardware encoders)

Currently when I log into the SJQ page I'll get 27 pages of completed tasks.
Then after a few clicks, or a few minutes my next click will give me an SJQ page with all fields listed blank. (reponses like from a fresh/clean SJQ install)
Any links I click on SJQ respond with "blank pages" (The SJQ web page "headers" are there ... just no data in the fields)
It's like the connection to the database is dropped.
When this happens I try to "touch" the context file but the only way I seem to get the SJQ data back is by stopping and restarting SageTV.

Does a core always max out when you run SJQ? I've seen that problem, but it was related to an infinite loop bug with the client parser. SJQ shouldn't be maxing a core - my Q6600 runs at 0-1% CPU while SJQ is active - unless you've somehow got the QueueLoader running on a constant interval of less than two minutes or so (MaxSleep < 2).

I've got 97 pages of completed tasks and have not had any problems. When the pages start to go blank I need to know if the requests to the server are failing or something else. Do you run Firefox? If so, can you install Firebug and view the AJAX calls when this starts happening? I need to see the response from the calls. Panels going blank in SJQ suggests a 500 error being returned from SJQ and if that's the case the response should contain a Java stack trace, which is what I really need to see to even figure out what part of the code is causing you problems. Without some kind of data point the best I can do is guess as to where the problem(s) lie.

Quote:

* also the database has blown up to 405 MB in about the last week
* I'll wait for your response ... but I think I will extract my rules and tasks from the database ... manually process my latest recordings ... and then run media files through all tasks with batch files that return success

This suggests an infinite loop somewhere that is constantly logging an error message to the database.

Download sqlite3 command line interface and open the sjq.sqlite db with it and run this query:

Code:

SELECT * FROM log WHERE taskid = 0 AND mediaid = 0 ORDER BY id DESC;

This will print the server log messages in reverse chronological order. What kind of messages are you seeing here? Is there one that is contantly being repeated? Replace DESC with ASC to print them in chronological order.

Quote:

When I woke on Wednesday night the SageTV server computer locked up and the Leds on the memory (ballistix tracer ddr2 ram) were moving about once a second. I rebooted the computer and it seemed okay. (SageTV was recording from all tuners and the system seemed to be working)

So I started of thinking of ways to reduce the load on SJQ / Jetty by possibly marking tasks done to speed up the Queue loader process and maybe reduce the Queue's depth.

The SageTV server locked up again on Friday night around 7pm (woke up at 9:30pm) so I restarted the computer and removed Jetty from the SageTV runnable list.
At midnight it finishes the last recording ... at that point I'm going to run memtest and maybe Prime95 for a bit to check the hardware hasn't started to die.

* ran through three passes of memtest 2.01 with no problems

After the hardware testing at midnight I'll put Jetty / SJQ back in the runnable classes line and grab some screen captures of the SJQ web pages to send.

Post the screen shots, but as I said above, installing firebug and getting me the AJAX responses when things start to fail is really going to get me pointed in the right direction.

If your basic h/w tests are passing then I'd really refrain from toasting your system installation and starting over. If you can remove SJQ for a few days/a week and everything runs stable then we can say with some confidence that it's SJQ causing all the problems. But without some kind of stack trace, log output, etc. pointing me to the problem area all I can do is guess unless I can reproduce the issue(s) myself, which I just haven't been able to at this point.

Quote:

Why linux?
Windows can frustrate me ... Don't get me wrong, Windows XP professional runs pretty smoothly for me ... but as I only have one license for XP Pro and one for MCE 2005.
(and one for Windows Me ... and one for Windows 98 and one for Windows 3.1 and Dos 6.22 and ...)

I get a week off at work during Inventory the last week of February so I'll try reinstalling Windows XP MCE 2005 to see if they have improved compatibility with Service Pack 3.

I'm a Linux guy myself, if all my HTPC hardware were fully supported in Linux I'd be running my server on Linux, but that's not the case so my server runs on XP Home SP3. As I said above, if you remove SJQ for a few days or a week or whatever and your system becomes stable again then I'd be confident is saying that SJQ is causing your problems and if it is SJQ then I doubt an OS reinstall is necessary. It's just a matter of getting me pointed to the problem area of the code. Once we can do that then I should be able to track down any bugs that may exist.

Quote:

ps - Also with unRaid I currently have large numbers of files in single UNC shares I'm not sure if the number of media files in a directory would matter?
\\storage\recordings - 3TB

This shouldn't matter unless there are limits with the java.io.File package in the JVM, which I doubt. Everything you explain tells me SJQ gets itself into an infinite loop somehow, I just need to know which area of the code is looping - without any nudges in the right direction I can't even venture a decent guess since I've had no such behaviour with SJQ at all.

Slugger · #**425** 02-14-2009, 10:33 AM

Quote:

Originally Posted by wayner

I don't know that I have posted about this but I am having a very similar issue to the post above - when I run Sage with SJQ/jetty it causes SageTVservice to use 50% of my CPU and the SJQ web client freezes after a few minutes in the way that Bobby describes. The only way to fix it is to restart Sage but then this issue comes back after a fw minutes. I have temporarily taken Jetty out of load_at_startup in the sage.properties.

And removing Jetty/SJQ has restored stability in your system? Can you reenable Jetty, but remove SJQ? Does the stability remain? If so, then we can probably say with certainty that SJQ is causing your problems.

What I need from both of you is your sjq.sqlite db files as they stand today, less the log output. Download the sqlite3 command line processor, create a copy of your sjq.sqlite db file and open the copy with sqlite3. Run the following queries:

Code:

sqlite> DELETE FROM log WHERE mediaid <> 0;
sqlite> vacuum;

Those two queries should purge most of the log data and hopefully reduces the size of your db files to something than can be shipped to me. I'm heading out for the day, but if the above reduces the sizes of your dbs to something that Google Code will accept then please attach your files to issue 51 in the project.

EDIT: Please change your SJQ password before copying the db file since the pwd is encrypted in the db so I have no way of accessing the db in SJQ GUI without knowing the password. Reset it to 'sjqadmin' or something, but be sure to post the password if you're able to attach your db file to Google Code.

perfessor101 · #**426** 02-14-2009, 12:04 PM

Hello Slugger,

With SJQ running on my system with my media files and network shares, the CPU usage is always ridding on the high side and I have MaxSleep set to 20 minutes with AfterRec by 5 minutes.

I have found firefox to be much more friendly with SJQ 2 so I'll try installing firebug now ... and I'll try capturing the logs for you (well, us too).

99.999% of the log messages from the command are as below with very few errors at the end. the captured text output from the command is 333,556KB
I can rar and send? (17MB)

Code:

SELECT * FROM log WHERE taskid = 0 AND mediaid = 0 ORDER BY id DESC; 

5381593|0|0|	Line 18: Evaluating 'FileExists != "M:\rebooting.marker"' == true
5381592|0|0|	MediaVar '%t%' == 'NCIS'
5381591|0|0|	MediaVar '%p%' == 'NCIS-SeaDog-6135763-0'
5381590|0|0|	MediaVar '%e%' == 'mpg'
5381589|0|0|	MediaVar '%f%' == 'NCIS-SeaDog-6135763-0.mpg'
5381588|0|0|	MediaVar '%d%' == '\\sagetvserver\tv5'
5381587|0|0|	MediaVar '%c%' == '\\sagetvserver\tv5\NCIS-SeaDog-6135763-0.mpg'
5381586|0|0|	MediaVar '%y%' == ''
5381585|0|0|	MediaVar '%s%' == 'Sea Dog'

with a couple of 
5631|0|0|MediaVariables.getVars(File f): Cannot get MediaFile object from database!
at the end

SageTV is recording for a bit ... so I'll try throwing Jetty/SJQ back in after it finishes.

* tried connecting to SJQ/Jetty and the server won't respond now the CPU usage is running from 25% to 35%
* I keep on getting the timeout error currently
* I'll double check the rules I have backed up against the ones I pulled from the database
* when I wake up tonight I'll drop in the cleaned up sjq.sqlite database to see if that lets it start ...
* if not I'll try deleting the database (I have two backed up copies of the current database)

After deleting the logs and vacuum-ing the sjq.sqlite database it's 359,078KB and I get a 27 MB file after rar'ing it. (and googling 'Google code maximum file size' says 10MB)

If I hadn't bought so much windows software (OS/programs) and knew that my hardware was supported I'd be running on linux now.

I used to reinstall windows on my computers every six months just to keep everything stable and snappy. As I get older I enjoy that less and less.

Hey Wayner do you have any similar hardware or software installed like in my signature?

Bobby

ps -- in firebug what tabs / sections / information would you like me to capture?

Slugger · #**427** 02-14-2009, 12:31 PM

Quote:

Originally Posted by perfessor101

Hello Slugger,

With SJQ running on my system with my media files and network shares, the CPU usage is always ridding on the high side and I have MaxSleep set to 20 minutes with AfterRec by 5 minutes.

I have found firefox to be much more friendly with SJQ 2 so I'll try installing firebug now ... and I'll try capturing the logs for you (well, us too).

99.999% of the log messages from the command are as below with very few errors at the end. the captured text output from the command is 333,556KB
I can rar and send? (17MB)

Code:

SELECT * FROM log WHERE taskid = 0 AND mediaid = 0 ORDER BY id DESC; 

5381593|0|0|	Line 18: Evaluating 'FileExists != "M:\rebooting.marker"' == true
5381592|0|0|	MediaVar '%t%' == 'NCIS'
5381591|0|0|	MediaVar '%p%' == 'NCIS-SeaDog-6135763-0'
5381590|0|0|	MediaVar '%e%' == 'mpg'
5381589|0|0|	MediaVar '%f%' == 'NCIS-SeaDog-6135763-0.mpg'
5381588|0|0|	MediaVar '%d%' == '\\sagetvserver\tv5'
5381587|0|0|	MediaVar '%c%' == '\\sagetvserver\tv5\NCIS-SeaDog-6135763-0.mpg'
5381586|0|0|	MediaVar '%y%' == ''
5381585|0|0|	MediaVar '%s%' == 'Sea Dog'

with a couple of 
5631|0|0|MediaVariables.getVars(File f): Cannot get MediaFile object from database!
at the end

SageTV is recording for a bit ... so I'll try throwing Jetty/SJQ back in after it finishes.

* tried connecting to SJQ/Jetty and the server won't respond now the CPU usage is running from 25% to 35%
* I'll double check the rules I have backed up against the ones I pulled from the database
* when I wake up tonight I'll drop in the cleaned up sjq.sqlite database to see if that lets it start ...
* if not I'll try deleting the database (I have two backed up copies of the current database)

After deleting the logs and vacuum-ing the sjq.sqlite database it's 359,078KB and I get a 27 MB file after rar'ing it. (and googling 'Google code maximum file size' says 10MB)

If I hadn't bought so much windows software (OS/programs) and knew that my hardware was supported I'd be running on linux now.

I used to reinstall windows on my computers every six months just to keep everything stable and snappy. As I get older I enjoy that less and less.

Hey Wayner do you have any similar hardware or software installed like in my signature?

Bobby

Ok, that output tells me you have debug logging enabled - disable that. If you have 1200 recordings each run of the QueueLoader is probably producing dozens of megs of output and hundreds of megs per week.

DELETE FROM log;
vacuum;

Disable debug logging in SJQ then clean up the db file then restart SJQ (with debug logging disabled). Reproduce problems and if they reoccur then attach vacuum'd db file (which should be fine size-wize after this latest vacuum) and any firebug output you can get me.

Slugger · #**428** 02-14-2009, 12:41 PM

Quote:

Originally Posted by perfessor101

ps -- in firebug what tabs / sections / information would you like me to capture?

Watch the console tab, as you click on things you'll see AJAX requests being made to Jetty. When things go blank what's happening? I expect some of those calls are returning an error (code 500). Those are the requests that I'm interested in. From those requests, click on a failed one and I need the output from all of the tabs (params, headers, post, response - they may not all be there depending on which action you're performing).

If the server code is caught in a hard loop you may notice an AJAX call never returns (you'll see the spinner icon beside it in firebug and it never finishes). That's not what I want to see (a real PITA to deal with), but on the other hand, it tells me that my code is looping somewhere for sure and we can proceed from there. If the AJAX calls are stuck and not returning then I really need your vacuum'd db file (with all the logs removed and no debug output).

Slugger · #**429** 02-14-2009, 12:48 PM

That screen shot tells me SJQ is stuck in a loop - the fact that you can't even login leads me to believe the ruleset or client parser is to blame.

I really need your db file to load onto my system and play with it to see if I can reproduce it.

And I am really late leaving the house now.

Try to vacuum the db file size down so you can attach it to GC or if not then can you post it somewhere so I can download it when I get home?

Thanks.

Slugger · #**430** 02-14-2009, 12:53 PM

Quote:

Originally Posted by Slugger

That screen shot tells me SJQ is stuck in a loop - the fact that you can't even login leads me to believe the ruleset or client parser is to blame.

I really need your db file to load onto my system and play with it to see if I can reproduce it.

And I am really late leaving the house now.

Try to vacuum the db file size down so you can attach it to GC or if not then can you post it somewhere so I can download it when I get home?

Thanks.

One last thing, then I really have to leave. If I'm right and one of the config parsers is causing the hard loop then start with a fresh db file (backup your current one first) and create a simple ruleset and simple config file:

Code:

if [Title == "Foo" ] {
   FAKETASK
}

And in the client:

Code:

:MAXPROCS 1
FAKETASK {
   "C:\fake\fake.exe \"%c%\""
}

With that setup pound on the GUI, run multiple iterations of the QueueLoader manually in succession. Make sure debug logging is off before starting the stress test. Does it break? Does the CPU spike?

I'll look for your db file when I return tonight.

perfessor101 · #**431** 02-14-2009, 12:58 PM

I enabled debugging last night before sending the first message this morning DOH ... and you said before it would fill the database with lots of information.
I enabled it last night to generate some log output that I thought might help.

Disabled now

Now SJQ/Jetty/SageTV is running at 1-10% CPU

Jekyll records soon ...

So I can see how things are working ... seems very smooth again.

Il uploaded the problem database to sendspace.

perfessor101 · #**432** 02-14-2009, 01:01 PM

I'll throw the old database back in tonight ... and try to get some debug output

is there anyway for me to change settings via sqlite to disable debug logging before I start SJQ/Jetty/SageTV in the old database?

perfessor101 · #**433** 02-14-2009, 02:27 PM

I put a link to the file in a message on issue 51 ...

I'll be going to sleep soon. I'll process the new recordings when I wake up ... once that is done I'll swap the old database back in and try to get some Firebug output.

When I was clicking 'load response' it would respond with a blank tab ... so I'm not sure where I was supposed to be looking for the parser errors

wayner · #**434** 02-14-2009, 06:39 PM

Slugger - Disabling debug logging seems to have fixed everything on my system. For the first time I am actually able to run jobs from a client and I was able to use comskip. I have not downloaded and used sqlite - do you still need me to do this? I am happy with the way things are now but it would be nice to see some less verbose sort of logging to troubleshoot - even something that said file1.mpg was true and was processed, file2.mpg was false so no processing. It used to show every test for every file which explains why it got so huge!

I am having one other minor problem that may have been mentioned somewhere up above in the thread.

edit - Nevermind I found posts 230-239 that deal with this problem where you have to use a batch job to set the comskip error.

My comskip jobs are showing as failed even though they seem to have run ok according to the .log file - anyone have an idea why that would be? Is it returning some other code that is causing SJQ to think it is failing?

Slugger · #**435** 02-15-2009, 12:35 PM

perfessor101: I got your db file and loaded it into my system overnight. I've been running your db file in my system since about 11pm last night using the same settings you had configured except I switched debug mode off. Everything's been running fine since 11pm and after studying your ruleset and client configs I'm now convinced it's not a problem with either of the parser classes in SJQ.

wayner: Things still running smoothly? How big is the media set you're scanning? I assume you're just scanning tv files so how many tv files are currently stored in Sage? Over 1000? I'm kind of hoping yes because I believe I've identified the underlying issue here...

I believe the problem is that the logging to the db is causing a bottleneck; this would especially be true with debug = true turned on in SJQ, but would also arise if one were running multiple tasks simultaneously and those tasks were all producing a lot of output, which was then being logged back to the SJQ server.

Based on your sigs I'll assume neither of you is running on a 64bit version of Windows? My other discovery is that I don't believe the SQLite driver I'm using includes a native DLL for Windows 64bit so 64bit users would be using the pure Java version of the SQLite db driver, which would also adversely affect performance.

With these findings, snapshot 271 is available, which hopefully fixes all the issues directly except for the 64bit issue.

A modification to the DataStore (database) layer has been made which will use the JDBC batch processing of SQL statements for the writing and purging of logs to the database. This should help relieve some of the load put on the db when logging to the database.

Reworked how log messages received from task clients are logged to the Logger layer. The task client buffers 1024 lines of output from a process before shipping it off to the logger, which saves a constant stream of HTTP requests between the client and the server when a process is running. This works well, except when I sent the 1024 lines to the server I'd log it as one massive string to the Logger layer. Well, the Logger layer itself buffers 1024 lines of messages before it writes them to the database. So quick math says that 1024 x 1024 (~1M) lines of text would be buffered by the Logger layer and written to the db at once, which certainly doesn't help anything at all. Now the server breaks up the task client log entries and sends them individually to the Logger layer, which means buffering should work as intended (at 1024 lines instead of a million). This should also alleviate some of the bottleneck issues with db transactions.

Finally, even though it doesn't appear to be relevant to the two of you, I've added a log entry that will display if you're using a native DLL implementation of the SQLite JDBC driver or if it's fallen into pure Java mode. There is a definite performance hit if using the pure java driver, however hopefully the above fixes make the pure java implementation more friendly for those who have to use it. Check the server logs after SJQ starts for a statement about the SQLite driver. You should be told that you're using the "native" driver. Can you both confirm that you are in native mode for the db driver? If for some reason you're not, that would also explain perf issues with the db layer.

Debug mode still needs to be off - it just produces too much output (for now). I'm going to take wayner's suggestion and remove the dump of everything and replace it with an input on the GUI that lets you specify a single file to test and will do the dump for one file upon request, making debug mode more appropriate to just leave on if you wanted to (when debugging other issues). This feature will be worked into the GUI rewrite, which is currently ongoing, but is still at least a few weeks away. Until then, use debug once to capture your output then turn it off. I hope the fixes I've just made prevent the locking and crashing of SJQ for you, but the kind of output being produced by debug mode today is just too massive to be left running constantly - especially for those of you with large media sets (1000+ files).

Eagerly awaiting your test results...

Slugger · #**436** 02-15-2009, 12:38 PM

Quote:

Originally Posted by perfessor101

is there anyway for me to change settings via sqlite to disable debug logging before I start SJQ/Jetty/SageTV in the old database?

Yes:

Code:

UPDATE settings SET val = 'false' WHERE var = 'Debug';
SELECT * FROM settings;

Just do the select to make sure that the setting was flipped as expected.

wayner · #**437** 02-15-2009, 12:49 PM

Things are still going well - SJQ was initializing comskip jobs through the night and all went well. Now I am Transcoding all of the Kids Shows that I like to archive using the new parameters to send the output to a different folder. It takes a while but appears to be working.

In terms of how many TV shows do I have? A few hundred - I don't think that I am in the thousand range. I think the problem could be that I was testing for a lot of conditions on each file - would that also be an issue?

One further comment - I don't know if you have addressed this before - when using the internal Transcode command SJQ marks jobs "Completed" prematurely. I had about 15 files to transcode and shortly after being put in the queue they were marked as bein completed whereas this will take several hours. But I imagine that is a shortcoming of how Sage reports Transcoder jobs?

One further question - how do I requeue jobs that have failed previously? It looks like I had a few jobs fail and now SJQ skips over these files. Is there a way to resubmit.

re: 64 bit - I am running WHS but I am not sure if it is 32 or 64 bit. I don't think there are options for WHS. But one thing with WHS is that it can address a full 4 Gig of RAM - I had to remove a 2G stick of RAM since some Hauppauge drivers don't work if you have 4 G or more in your system. So perhaps that 64bit issue affects WHS.

wayner · #**438** 02-15-2009, 01:16 PM

One other thing that would be nice would be to be able to loop through namelists where the name could be used for tests and it could be used to specify output. For example:

Code:

loop for %name% in {"Diego","Dora"}
// Process shows with a valid subtitle
	if [IsTV == "true" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.avi" && IsActivelyRecording == "false" && Filename =* %name% && Episode != ""]
	{
	   :TRANSCODEOPTS "MPEG4-Good Quality Deinterlaced AVI" "false" "\\\\standrews\\videos\\Kids Shows\\%name%\\%s%.avi"
	    _TRANSCODE
	}

// Process shows without a valid subtitle
	if [IsTV == "true" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.avi" && IsActivelyRecording == "false" && Filename =* %name% && Episode == ""]
	{
	   :TRANSCODEOPTS "MPEG4-Good Quality Deinterlaced AVI" "false" "\\\\standrews\\videos\\Kids Shows\\%name%\\%f%.avi"
	    _TRANSCODE
	}

end loop

This would be a much more elegant solution than having to have one if test for every show that I am interested in. Is this possible today or with some alterations.

perfessor101 · #**439** 02-15-2009, 01:43 PM

I've downloaded 271 and currently have it running with the latest Release Canidate of SageTV

It should have three files to process soon ;-)

I run four tasks against each recorded file with about ten files recording a day and each task is a nasty looking batch file which is a bunch of kludges to get around windows networking security and have the media files scrubbed, scanned, commercial cut and moved to a NAS.

Slugger · #**440** 02-15-2009, 02:48 PM

Quote:

Originally Posted by wayner

Things are still going well - SJQ was initializing comskip jobs through the night and all went well. Now I am Transcoding all of the Kids Shows that I like to archive using the new parameters to send the output to a different folder. It takes a while but appears to be working.

In terms of how many TV shows do I have? A few hundred - I don't think that I am in the thousand range. I think the problem could be that I was testing for a lot of conditions on each file - would that also be an issue?

Ok so media set size probably isn't an issue for you, but you did say you had debug mode enabled the whole time, which would cause the same symtoms as a larger media set. Keeping debug mode off should alleviate the problems, please keep me updated on this.

Quote:

One further comment - I don't know if you have addressed this before - when using the internal Transcode command SJQ marks jobs "Completed" prematurely. I had about 15 files to transcode and shortly after being put in the queue they were marked as bein completed whereas this will take several hours. But I imagine that is a shortcoming of how Sage reports Transcoder jobs?

This is working as designed - SJQ simply submits the job to the Sage transcode queue and if Sage tells SJQ the job was added then SJQ marks its task as completed. The API would allow me to monitor the transcode job and react on its result, but I really don't see the benefit. Once the job is put in the Sage queue then you can monitor it from there and restart it, etc. as needed from SageTV. There are other SJQ specific considerations, namely the internal task processor in SJQ is single threaded and tasks are run serially so to monitor a transcode job means other jobs are not going to get added until the first one finishes. Also other internal tasks (watched, archive, etc.) would be held up by an active transcode job if I monitored them to completion.

Quote:

One further question - how do I requeue jobs that have failed previously? It looks like I had a few jobs fail and now SJQ skips over these files. Is there a way to resubmit.

Right click on a failed job and remove it from the list then rerun the QueueLoader to get it back into the active queue. Alternatively, tell SJQ to ignore failed tasks in the settings. This will prevent SJQ from registering failed jobs, which is like an "auto-remove from the failed task list" feature.

Quote:

re: 64 bit - I am running WHS but I am not sure if it is 32 or 64 bit. I don't think there are options for WHS. But one thing with WHS is that it can address a full 4 Gig of RAM - I had to remove a 2G stick of RAM since some Hauppauge drivers don't work if you have 4 G or more in your system. So perhaps that 64bit issue affects WHS.

After loading 271, look at the server logs, it'll tell you which SQLite driver you're using, it should tell you you're using the native driver.

Quote:

Originally Posted by wayner

One other thing that would be nice would be to be able to loop through namelists where the name could be used for tests and it could be used to specify output. For example:

Code:

loop for %name% in {"Diego","Dora"}
// Process shows with a valid subtitle
	if [IsTV == "true" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.avi" && IsActivelyRecording == "false" && Filename =* %name% && Episode != ""]
	{
	   :TRANSCODEOPTS "MPEG4-Good Quality Deinterlaced AVI" "false" "\\\\standrews\\videos\\Kids Shows\\%name%\\%s%.avi"
	    _TRANSCODE
	}

// Process shows without a valid subtitle
	if [IsTV == "true" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.avi" && IsActivelyRecording == "false" && Filename =* %name% && Episode == ""]
	{
	   :TRANSCODEOPTS "MPEG4-Good Quality Deinterlaced AVI" "false" "\\\\standrews\\videos\\Kids Shows\\%name%\\%f%.avi"
	    _TRANSCODE
	}

end loop

This would be a much more elegant solution than having to have one if test for every show that I am interested in. Is this possible today or with some alterations.

The more complicated the syntax of the ruleset, the tougher it is to maintain. I managed to avoid writing a recursive decent parser for the current syntax and instead managed to write it using the StreamTokenizer class. Neither solution is particularly easy, especially since the whole area of parsers, etc. doesn't interest me at all. Introducing something like this only makes things more complicated and isn't likely to happen anytime soon (i.e. ever

) On the other hand, if someone wanted to introduce a more robust ruleset syntax I'd be more than willing to accept a proper patch.

However, with the use of the media variables, you should be able to reduce things to a single if block, something like this should work (untested):

Code:

if [IsTV == "true" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.avi" && IsActivelyRecording == "false" && Filename =% ".*(?:Dora|Diego).*" && Episode != ""] {
   :TRANSCODEOPTS "MPEG4-Good Quality Deinterlaced AVI" "false" "\\\\standrews\\videos\\Kids Shows\\%t%\\%s%.avi"
   _TRANSCODE
}

Currently, the %t% media variable includes all the spacing and punctuation in the title, so this probably won't work (or at least won't work for shows with characters that are invalid for filenames in their titles), unless the Sage transcoder modifies the received path name (which I highly doubt). But, if you were to open an issue ticket I could probably add a new media variable with the program title stripped down to a string valid for a file (or directory) name.

You'd still need two if blocks - one for shows with a subtitle and one for shows without, but you shouldn't need a separate if block for each individual show.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Plugin: Sage Job Queue (SJQ) v3	Slugger	SageTV Customizations	1355	07-25-2013 07:44 AM
Sage Job Queue (SJQ) new release notifications	Slugger	Customization Announcements	3	12-17-2009 09:59 AM
Sage Job Queue Completed tasks problem	raffmanlt	SageTV Customizations	2	08-18-2009 07:34 PM
Comskip Monitor VS Sage Job Queue SJQ	personalt	SageTV Customizations	6	03-02-2009 10:27 AM
Plugin: SJQ v1.1.0RC1 Available - Testers Needed	Slugger	SageTV Customizations	35	04-21-2008 08:12 AM

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)