Defrag Needed Even with 64KB Clusters

[jptaz]

I have read a number of threads on whether Defragmenting is needed.

Here is my situation:

I have four 80GB 7200 RPM Drives ( Average Transfer rates of 45MB and Burst speeds of 75MB and Random Access time of 12ms). All 4 of these drives are dedicated to SageTV Video Recording only.

I have a 30GB 7200 RPM Drives ( Average Transfer rates of 40MB and Burst speeds of 60MB and Random Access time of 10ms). Again Dedicated only to Video.

I have a 120GB 7200 RPm Drive ( Average Transfer Rates of 49MB and Burst Speeds of 118MB and Random Access time of 14ms). This also contains MP3s and other Data files that are rarely accessed.

I have two SageTV Clients that are pretty regularly used to watch previously recorded shows, rarely "live" shows. most of the Time that both clients are being used to watch recorded shows all 4 PVR 250 cards are being used to record shows. All shows are recorded 3.25GB/HR.

Many people on the forums have stated that with 64KB clusters Defrag does not need to be run. So I have not been using Defrag for several months. Up until two weeks ago I would getting 5 to 10 second pauses in middle of watching recorded shows on either of the client PCs with increasing frequency.
NOTE: All the drives where pretty full (6 to 10 GB free on each). The pauses seemed to get more frequent the fuller my drives got, and decrease somewhat with more free space.

Along the way of trouble shooting this problem I discovered that several of my drives were running in PIO Mode which was killing performance, which HD Tach helped me to find out. Even after fixing my Bios so that it is running in ATA133 Mode(DMA 6). I was still getting this problem. NOTE: The above performance specs are from HD Tach after fixing DMA mode. Without several drives were capped at 16.6MB which is PIO Mode 4.

Two weeks ago I downloaded a demo version of Raxco Perfect Disk and did an agressive Free Space consolidation and defrag.
After that things improved for a day then again things started pausing. I now have it scheduled to run every day when I know SageTV is not recording. Since doing this I have not seen the 5 to 10 second pause.

I am sure for some people with large drives or drives that are not so close to full, defragmentation is not and issue, but I found with my setup many of the video files had in excess of 2000 fragments. With that many fragments it means that the head has to seek to a non contiguos location every 1.8 seconds or less. While this is not a big deal if only one file on the disk is being accessed, but if several other files are being written or read at the same time it could be a bit rough. I have previously made a feature suggestion that either they not record more than one file to a hard disk if multiple drives are available as in my case, or that files are preallocated to prevent fragmentation, but at this point SageTV still has a habbit of writing all the files to one disk so the problem is magnified ( 4 files being constantly written to, while 2 files are being read all at once which creates files with 2000+ fragments each.

I am still considering moving to 1GB of memory to allow more caching, but ultimately a better solution would be to preallocated the files that recordings are going to since this would allow the OS to do its best to not fragment the file...actually it attempts to allocate the file contiguously. For now my daily defrag has mostly solved my pausing problem...So far only once in the last two weeks and it was a shorter pause than usual and later I checked and several of the files on the disk had over 2000 fragments.

Hope this helps others,
John

[kny3twalker]

Defrag Needed Even with 64KB Clusters

nope

[jptaz]

You need to read the Post...
Maybe in your configuration with No clients and only 2 PVRs it is not an issue but if you read my post you will see the only change is the daily defrag.

[Opus4]

That seems to pretty much answer my question in another thread regarding whether defragmenting 64K clusters might help when using many tuners/clients. As for preallocation, that could help, but if saving the data to the file is a function of the encoder's drivers, I wonder if SageTV could do anything about it.

Sharing the load between drives sounds like a much better option. I believe it automatically uses the drive with the most free space for the next recording... but when all drives are full, it would have to delete a recording to make room for the next one. Changing that to spread the load around would require using deleting the oldest recordings on each selected load-sharing drive.

- Andy

[kny3twalker]

You need to read the Post...

see I thought you would find that funny

[Opus4]

Originally posted by kny3twalker
see I thought you would find that funny

Not really... it seeems dismissive of what is happening in a situation you don't have with fewer encoders/clients. Edit: Unless you simply meant that you haven't needed to defragment...

- Andy

[jptaz]

[salsbst]

The exact same symptoms cropped up for me in later builds of v2. Disabling Norton Antivirus autocheck on my Sage drives seemed to completely solve the problem.

In my case has been 1 drive, 2 tuners, 2 clients.

I'll watch out for pauses going forward and take a look at the fragmentation if they crop up.

Thanks for sharing, jptaz.

[jptaz]

Originally posted by Opus4
That seems to pretty much answer my question in another thread regarding whether defragmenting 64K clusters might help when using many tuners/clients. As for preallocation, that could help, but if saving the data to the file is a function of the encoder's drivers, I wonder if SageTV could do anything about it.

Sharing the load between drives sounds like a much better option. I believe it automatically uses the drive with the most free space for the next recording... but when all drives are full, it would have to delete a recording to make room for the next one. Changing that to spread the load around would require using deleting the oldest recordings on each selected load-sharing drive.

- Andy

I actually worked on a open source program that did command line recording using the PVR250 and SageTV is actually in control of the creation of the Files. I briefly looked into making my Write Filter preallocate the file, but at about that time SageTV switched to free EPG and I bought SageTV and stopped working on WinTVCap...This was also less of a problem for me since I did load ballancing across my drives and did my best not to write more than one file to a drive.

John

[kny3twalker]

Unless you simply meant that you haven't needed to defragment...

yes of course

[jptaz]

Originally posted by salsbst
The exact same symptoms cropped up for me in later builds of v2. Disabling Norton Antivirus autocheck on my Sage drives seemed to completely solve the problem.

In my case has been 1 drive, 2 tuners, 2 clients.

I'll watch out for pauses going forward and take a look at the fragmentation if they crop up.

Thanks for sharing, jptaz.

Salsbst

I ended up doing the same thing on my server and it did make a difference.

I know there are others who do not have this issue who actually have more tuners and clients than I do, but the issue may also be tied to my multiple smaller drives getting close to capacity. I have a feeling that the amount of free space on the drive impacts the OS efficiency in managing files. It seems like NTFS does everything in its power not to use space in the upper part of the drive if any space is free in the lower.

John

[Opus4]

Originally posted by jptaz
I ended up doing the same thing on my server and it did make a difference.

salsbst's NAV comment was a very good point too. jptaz: Did this quote from you mean you had excluded the video dirs from NAV & that helped, but just not enough?

- Andy

[stanger89]

Originally posted by jptaz
I actually worked on a open source program ...WinTVCap...

John, I think you deserve a for that. WinTVCap was the greatest thing to happen to the 250, until Sage that is

FWIW, Could you provide me with a little information? Specifically how much fragmentation (%) you have/had on your drive when you were having trouble?

Also you said 4 tuners and two clients, right? Did they all have to be running to cause problems, or was something less than that sufficient?

[jptaz]

Andy,

Yes I agree that slsbst's comment is very good and actually I went one step further and turned off realtime file protection off all together. This did help, but not enough. Defrag ultimately has been what helped in my situation.

I had actually been working with George in Support and sending traces, but there was no real indication what was causing the problem. I told him about defrag helping in my situation and he said to keep him posted. I am going to continue to use my schedule and see if the problem seems to be resolved by this. The most interest aspect of the poblem was that the pause occurred and CPU dropped to nothing as well as Network. If the Disk was busy getting caught up with IO requests the CPU would be low.

Also one Notable change I made yesterday when the pausing occurred was that I changed Perfect Disk from SmartPlacment to Defrag Only. I think SmartPlacement is much better as it consolidates all the free space. This is a feature that Windows XP built in Defrag does not do, or at least not well. Tonight I am keeping all 4 tuners are busy again so we will see if the pausing does not happen now that I am back to smartplacement defrag.

John

[jptaz]

stanger89,

Thank you. We agree on both counts...Now I can't imagine living without SageTV.

I am not sure what the percentage fragmentation was, but the common factor was one or more files having 2000+ fragments each. The disk as a whole was not that fragmented, as I recall.

In terms of the load on the Server it did happen on lower loads like 3 recordings and one client, but I am not sure if this was happening before or after figuring out that my Drives were running really slowly.

My 120GB was being used heavily by sage as it had the most free space, but it was also one of the 3 Drives running at 16.6MB until I turned DMA Mode 6 on.

John

[salsbst]

How many GBs are you preserving as free space on these drives? I know you said that 6-10GBs were free, but what setting do you have in SageTV?

[jptaz]

I set it all free space. I thnk it used to be 2GB. But I think I saw a post by you that talked about this causing problems for you.

[salsbst]

Actually I have it set to 5 GB free on each of my two drives (my experience stated above w/one drive because the second one is new as of last week). I'm thinking that this might have a lot to do with it. Using up that last ounce of space may be what is causing very highly fragmented files.

[Opus4]

I think it would create fragmented files with an empty drive. I could be wrong, but from the fragmentation I've looked at, when you record multiple shows at once, each one leapfrogs the other as the file is saved, so it fragments at nearly every new cluster for each file. Note that I can't prove that, it just seems to be the case when I look at the number of fragments for shows recorded simultaneously.

- Andy

[ErsatzTom]

I've also resolved many problems by requiring it leave a certain amount of diskspace free. I don't have as many drives so I actually keep it pretty hefty but I think even a few gigs might help.

t