Should I defrag my drives?

[doc]

Just incase anyone looked at the defrag.jpg file I put up, I must stress that the Baywatch recording was a test I made when seeing if both the tuners were working

I would never intensionally record Baywatch....... honest......

[blade]

Everyone should be using 64K clusters so the absolute smallest fragments are going to be 16 times larger than the smallest possible fragments found in other environments that use the typical 4K clusters.

Also we're dealing with very large files that are being read relatively slowly. Isn't a typical SD stream well under 1MB/sec? The drives aren't being pushed to their limits and thrashing about in an attempt to transfer or read data as quickly as possible. It just seems to me the larger cluster size and low throughput requirements just aren't going to be taxing the drives all that much.

[Pumpkinhead]

Originally Posted by blade

Also we're dealing with very large files that are being read relatively slowly. Isn't a typical SD stream well under 1MB/sec? The drives aren't being pushed to their limits and thrashing about in an attempt to transfer or read data as quickly as possible. It just seems to me the larger cluster size and low throughput requirements just aren't going to be taxing the drives all that much.

Yes, less than 1MB/sec. Although if you have multiple tuners recording, plus watching back another stream, it will push things a little harder. I never did get my video drive converted to 64k clusters and never defrag it, but I only have 1 tuner, so I'm never pushing things too hard. The only issue I've had regarding drive performance was when the system suddenly and quietly switches to PIO mode due to 6 cumulative timeout errors in Win2k, although very easily resolved.

[FidgetyRat]

I use 64k cluster Raid0 with 2 drives.

The most I have done was record 2 shows, watch one natively on the server and stream 1 to the MVP. SO i guess to writes and 2 reads. Never had a problem and have never defragged the video drives.

[stevech]

Originally Posted by Pumpkinhead

Yes, less than 1MB/sec.

Megabytes/sec, right?

I see SD streams on my network using about 6 megabits/sec. That's less than 1 megabyte/sec. I measured the I/O speeds of my EIDE vs. USB2 external drives. Quite a difference: 30MBbytes/sec vs. about 6MBytes/sec on USB2. CPU = AMD1800 for those tests.

[flavius]

I got myself a defragmentation report of all the hd's - just for the fun of it. That must have been the second one in about 4 years or so.

I have a lot of fragments and fragmented files, quite a lot. But I have yet to see any performance impact. And none of my servers is or has been a killer box, believe me. I didn't adjust the cluster size on any of those partitions either.

[stevech]

Originally Posted by flavius

I got myself a defragmentation report of all the hd's - just for the fun of it. That must have been the second one in about 4 years or so.

I have a lot of fragments and fragmented files, quite a lot. But I have yet to see any performance impact. And none of my servers is or has been a killer box, believe me. I didn't adjust the cluster size on any of those partitions either.

Is the partition using 64K blocks?

[flavius]

No. 4k.

[evilpenguin]

Just thinking out loud, but would it help fragmentation if all the video metadata files (I have 5 per video right now) were stored on a separate folder on a separate drive? They're small files, but when you have 600+ of them thrown around the drive I'd imagine they get in the way of contiguous large file creation.

[AngelofDeth]

Originally Posted by stevech

Is the partition using 64K blocks?

Windows defragmentor shows 98% file fragmentation on my recording drive, though it is using 64k clusters. Never had a performance issue, although its SD only, two encoders, with only one client.

[blade]

Originally Posted by flavius

No. 4k.

Personally I'd switch to 64K if I got the chance and I'd definetly format any new drives I added to 64K before putting them into use.

You do understand why 64K is better than 4K for a recording drive? BTW I don't mean that to sound condescending. Some people really don't know what the difference is so I just wanted to check.

[flavius]

The last time I messed with the cluster size of a partition was some ext2fs based installation, probably in 1993.

I know it's a little easier today - but I'd rather play with my three sons than with my Sage system.

[Mark SS]

I suspect lots of people are missing the point with this and/or have no idea what the point actually is

[blade]

Originally Posted by flavius

The last time I messed with the cluster size of a partition was some ext2fs based installation, probably in 1993.

I know it's a little easier today - but I'd rather play with my three sons than with my Sage system.

Formatting to 64K requires no more effort than 4K. When you install a new drive you just select 64K clusters instead of 4K when you format the drive.

[blade]

Originally Posted by Mark SS

I suspect lots of people are missing the point with this and/or have no idea what the point actually is

You're probably right.

[stevech]

Isn't the whole idea of 64K clusters to significantly reduce the number of disk I/Os and seeks per gigabyte read or written?
I did it using Windows' XP's disk management tools. Easy. I presume that these large blocks reduce CPU use and lessen the occurrence of "stuttering" aka dropped frames.

[jptaz]

Just thought I would repport back...it took 2 weeks of no defragging, but now I am getting stuttering in some of my HD recordings. I think people who have no stuttering are either not recording 6 shows simultaneously (3 SD + 3 HD ) or they are not using RAID 5 with 75%+ used disk space, or there RAID 5 System is faster than mine. When my drive gets fuller not defragging causes the problems that are creeping in again. I am enabling my nightly defragger and accepting that in my case it helps and Raxco Perfect Disk 8.0 keeps my system working well and was well worth the money I spent on it.

John

[jptaz]

Also more importantly so those of you doing the math:
My SD recordings average 3.25 GB an Hour
My HD recordings average 8.83 GB an Hour
Doing the math that works out to requiring 10.3 MB a second RANDOM write performance.

Key word here is RANDOM. Lots of drives and arrays can do many times that in sequential IO performance...my array for example can sustain sequential writes of 85 MB a second, but on but has problems go much over 12 MB a second on heavy random writes...now add in random reads streaming more than one HD recording and it gets a little tight...so add in additional seek time for a fragmented file...or more important a severely fragmented file....I probably should go to a secondary drive subsystem for the reordings I don't plan on keeping, but defragging works for now.

John

[Pretzelboy]

Originally Posted by Polypro

Until recently I used Diskeeper 10's "Set it and forget it" to defrag all my drives. It did a good job and never interfered with anything. With Diskeeper 2007's "Intellisense", the CPU is constantly used for defragging...I didn't like it. I switched to Raxco Perfect Disk 8.x and like it better. With 64k clusters I guess you don't need to defrag, but I do anyway. I also use Sprinrite 6 every few months as preventative maintenance.

P

I started using Diskeeper 2007 about a month ago and I'd have to disagree. I really like the Intellisense aspect of the software. I've kept an eye on it when Comskip activated to process a show and Diskeeper was smart enough to shut down completely and wait for the process to finish. Their claims seem to be true. I've watched their process monitoring graph and and any time there's an increase of activity in processes Diskeeper sits and waits until it's safe to continue. I can't even do a manual drive Analyze when there are intensive processes running.

I don't have a 64k block drive because I already had things stored on it when I started using Sage. I would get some noticable stutters from really fragmented recordings. I haven't seen a stutter in weeks. I actually hadn't even thought about it running in the background until about a half hour ago. I recalled what the drive analysis looked like when I started. There were an average of 30+ fragments per file and now I'm down to 1. What had been a shotgun mess of defragmentation now shows as optimized for performance and I only have two large files left on the hard drive that are heavily fragmented.

I can say that I believe their claims regarding performance. It has done a great job on my media drive (300GB PATA).

[rfutscher]

I don't see where a fragmented file would have anything to do with performance, that is if you follow the advice from Sage for block size. The block size is matched to the buffer size. As the buffer for the tuner fills up one block will empty the buffer. Then the heads move to service a playback buffer. As the playback buffer approaches empty one block will fill it up. The heads have to move between record and playback. It doesn't matter if the file is fragmented or not, the heads have to move.

Originally Posted by jptaz

Also more importantly so those of you doing the math:
My SD recordings average 3.25 GB an Hour
My HD recordings average 8.83 GB an Hour
Doing the math that works out to requiring 10.3 MB a second RANDOM write performance.

Key word here is RANDOM. Lots of drives and arrays can do many times that in sequential IO performance...my array for example can sustain sequential writes of 85 MB a second, but on but has problems go much over 12 MB a second on heavy random writes...now add in random reads streaming more than one HD recording and it gets a little tight...so add in additional seek time for a fragmented file...or more important a severely fragmented file....I probably should go to a secondary drive subsystem for the reordings I don't plan on keeping, but defragging works for now.

John