Tutorial: SageTV on WHS with 64K clusters in the drive pool...

[gplasky]

You can't add/remove the disk after formatting to 64K. WHS will format the disk to 4K when you add it back in. (If you remove it from the pool you need to add it again and it will format it to 4k again.) I will be testing the method I had used before (copying all off of D drive and formatting to 64K and copying the data back on.) and documenting it.

Gerry

[Wified]

I did not try to add or remove the disk using the Console. Since my WHS only contained the primary drive, it couldn't be removed with the console. I just shut down the WHS and physically took out the primary drive.

After converting the cluster size on this drive outside of the WHS, I installed it back into the WHS, and restarted. The console started up and worked (since it is on the C: partition, which was not touched in the conversion), and I was able to log back onto the WHS. Using Remote Desktop, I was able to confirm that the cluster size of D: was 64K. But, the WHS console would not add the D: drive - it just kept reporting that it was "calculating the size" of the D: drive. The D: drive was not reformatted back to 4K cluster size - it would just not let me use the drive.

I hope that your "quick format" way of doing this can be worked out, because that would be the easiest way, especially in a new WHS setup, not requiring other software to do this. For me, trying both the "quick format" and the full format option, the result was a partition that was still 4K after the formatting attempts, and all of the files that were on D: before the format were still there - obviously the format wasn't being allowed to execute.

[calypsocowboy]

Question, that I wasn't sure if it was answered. If I get this working now. Down the road when I go to add a new hard drive. Do I need to turn duplication off again before formating the new drive?

[RagingGoblin]

I am new to the SageTV and WHS community and I have a nagging question from reading this forum. If I build a WHS box and manage to format all its drives with 64k clusters, don't the backups of my client computers grow to 16 times their size since WHS is backing up each 4k cluster onto a 64k cluster? So, for example, this could take my 50GB or so of OS and Apps and tie up 800GB on the server, right? Is this correct or am I missing something?

[gplasky]

Originally Posted by RagingGoblin

I am new to the SageTV and WHS community and I have a nagging question from reading this forum. If I build a WHS box and manage to format all its drives with 64k clusters, don't the backups of my client computers grow to 16 times their size since WHS is backing up each 4k cluster onto a 64k cluster? So, for example, this could take my 50GB or so of OS and Apps and tie up 800GB on the server, right? Is this correct or am I missing something?

It could if you just thought of backup as copying a bunch of files. WHS is more than that. And it would probably be in your best interest to read up on WHS before venturing forth. WHS Server documentation on everything

So to answer your question. No it is not the mose efficient way to store the data. But WHS has a single-instance store at the cluster level. Clusters are typically collections of data stored on the hard drive, 4 kilobytes (KB) in size. Every backup is a full backup, but the home server only stores each unique cluster once. This creates the restore-time convenience of full backups (you do not have to repeat history) with the backup time performance of incremental backups.

The Windows Home Server backup engine stores data at the cluster level. Clusters are typically 4 kilobytes (KB) in size. The backup database records include clusters and hashes of these clusters (a hash is a number that uniquely identifies a cluster based on its contents). The database also contains information on the structure of a hard disk volume (NTFS information).

During the backup process, a cluster from one home computer that is identical to a cluster from a different home computer is sent to the server exactly once and stored exactly once. Because Windows Home Server backs up multiple computers (space) every day (time) to a single database, it in essence supports “single instancing of home computer backups across space and time.”

Right now I am doing backups of 3 PCs. Default backup of C drive. C drives are 372GB, 118GB and 111GB. (These backups go back to April of last year) Currently there are about 75GB of backed up data on the WHS server. And the nice feature with WHS is when you start running out of space slap anoter drive in there and add it to the pool. (Formatting it to 64K right after you add it if you're using it for Sage.)

Gerry

[RagingGoblin]

Originally Posted by gplasky

It could if you just thought of backup as copying a bunch of files. WHS is more than that. And it would probably be in your best interest to read up on WHS before venturing forth. WHS Server documentation on everything

So to answer your question. No it is not the mose efficient way to store the data. But WHS has a single-instance store at the cluster level. Clusters are typically collections of data stored on the hard drive, 4 kilobytes (KB) in size. Every backup is a full backup, but the home server only stores each unique cluster once. This creates the restore-time convenience of full backups (you do not have to repeat history) with the backup time performance of incremental backups.

The Windows Home Server backup engine stores data at the cluster level. Clusters are typically 4 kilobytes (KB) in size. The backup database records include clusters and hashes of these clusters (a hash is a number that uniquely identifies a cluster based on its contents). The database also contains information on the structure of a hard disk volume (NTFS information).

During the backup process, a cluster from one home computer that is identical to a cluster from a different home computer is sent to the server exactly once and stored exactly once. Because Windows Home Server backs up multiple computers (space) every day (time) to a single database, it in essence supports “single instancing of home computer backups across space and time.”

Actually, I have read the WHS documentation and I THINK I understand the paradigm for its computer backups. I understand that it backs up at the cluster level and not the file level and that it keeps only one copy of each unique cluster - making it very efficient. My point was that if it needs to back up, say, 1GB of unique 4kB clusters - call that 256,000 clusters - that will translate to 256,000 clusters on the WHS disk. But now if the clusters on the WHS disk are each 64kB, then these 256,000 clusters take up 16GB of disk space. Or is it somehow smarter to pack in multiple 4kB clusters into each 64kB cluster? Just testing my understanding.

Originally Posted by gplasky

Right now I am doing backups of 3 PCs. Default backup of C drive. C drives are 372GB, 118GB and 111GB. (These backups go back to April of last year) Currently there are about 75GB of backed up data on the WHS server. And the nice feature with WHS is when you start running out of space slap anoter drive in there and add it to the pool. (Formatting it to 64K right after you add it if you're using it for Sage.)

Gerry

I guess I'm not sure I understand your numbers. Or maybe I do, I'm not sure. Are you saying that when you look at each of your client's C drives (say, in Windows Explorer), you currently see used space of 372GB, 118GB, and 111GB? And when you look at the WHS backup area, the total disk space consumed for backing up these 3 computers is only 75GB?

If that is so, it is not what I would have expected. I do understand that WHS, by default, will not back up temporary files, caches, page file, etc. - but 75GB vs. 372GB is quite a difference. The only way the backup is smaller than the original is due to temporary stuff and redundant clusters. Is there that much redundancy? Am I understanding this correctly?

Thanks for your patience with me...

[gplasky]

Originally Posted by RagingGoblin

I guess I'm not sure I understand your numbers. Or maybe I do, I'm not sure. Are you saying that when you look at each of your client's C drives (say, in Windows Explorer), you currently see used space of 372GB, 118GB, and 111GB? And when you look at the WHS backup area, the total disk space consumed for backing up these 3 computers is only 75GB?

If that is so, it is not what I would have expected. I do understand that WHS, by default, will not back up temporary files, caches, page file, etc. - but 75GB vs. 372GB is quite a difference. The only way the backup is smaller than the original is due to temporary stuff and redundant clusters. Is there that much redundancy? Am I understanding this correctly?

Thanks for your patience with me...

Bad example. Or actually bad way of making a point. Those numbers are total disk sizes for those 3 computers. I was trying to illustrate typical hard drive size to what is backed up and I should have used the disk space used. Each PC has from 35-40 GB of used disk space. So out of the 120GB of used disk space WHS is using about 75GB for backups for 3 PCs.

During the backup process, a cluster from one home computer that is identical to a cluster from a different home computer is sent to the server exactly once and stored exactly once. Because Windows Home Server backs up multiple computers (space) every day (time) to a single database, it in essence supports “single instancing of home computer backups across space and time.”

It will back up unique clusters across ALL computers. So if 3 PCs have Windows directory it will only backup (or store) the one copy for all 3 computers. This is another reason why to me it doesn't matter if the 64K cluster makes it more inefficient. Backing up and keeping 75GB of data on hand that allows me to restore any of 3 computers from scratch from a total of almost 4 TB of disk space.

From the documentation:

•Cluster files - Files that store the actual cluster data from the home computers. These are usually in 4096 byte increments. These files grow to 4 GB in size before a new file is created. These files are stored as Data.XXXX.Z.dat, where XXXX=cluster size in bytes, and Z=0,1,2,…

So WHS will roll the .DAT file over at the 4 GB mark. So yes. It is more inefficient to have it formatted with 64K blocks. But I would rather be able to add more storage easily (just throw a drive in there) and record my TV shows to the pool AND be able to restore any PC in the house by booting to a restore disk and letting it run for awhile for a complete restore.

By the same token-these backup .DAT files are stored on the D drive-the second partition of your main drive. To be more efficient just leave the D drive formatted at 4K and you can be more efficient in your backup storage strategy. The only time that recordings would spill back onto the D drive would be when the pool becomes full. WHS will warn you before that happens so you can take appriopriate measures. (So after digging more in depth regarding this matter it makes me question wanting the d drive formatted in 64K clusters. Theoretically recordings will should never end up on the D drive. And untill WHS takes advantage of the larger cluster size (if it ever does) it may not make sense to reformat the d drive. I shall take this up in the documentation I'm putting together)

Gerry

[RagingGoblin]

Originally Posted by gplasky

Bad example. Or actually bad way of making a point. Those numbers are total disk sizes for those 3 computers. I was trying to illustrate typical hard drive size to what is backed up and I should have used the disk space used. Each PC has from 35-40 GB of used disk space. So out of the 120GB of used disk space WHS is using about 75GB for backups for 3 PCs.


It will back up unique clusters across ALL computers. So if 3 PCs have Windows directory it will only backup (or store) the one copy for all 3 computers. This is another reason why to me it doesn't matter if the 64K cluster makes it more inefficient. Backing up and keeping 75GB of data on hand that allows me to restore any of 3 computers from scratch from a total of almost 4 TB of disk space.

Thank you. That's what I was after. If this has been your experience, then the "growth" that I feared with the 64k cluster solution is not bad. Like you, I can certainly live with that level of inefficiency.

Originally Posted by gplasky

By the same token-these backup .DAT files are stored on the D drive-the second partition of your main drive. To be more efficient just leave the D drive formatted at 4K and you can be more efficient in your backup storage strategy. The only time that recordings would spill back onto the D drive would be when the pool becomes full. WHS will warn you before that happens so you can take appriopriate measures. (So after digging more in depth regarding this matter it makes me question wanting the d drive formatted in 64K clusters. Theoretically recordings will should never end up on the D drive. And untill WHS takes advantage of the larger cluster size (if it ever does) it may not make sense to reformat the d drive. I shall take this up in the documentation I'm putting together)

I'm not sure you want to change your approach regarding the D drive since I don't think WHS really stores the data for the backup on the D drive, just the tombstones and other management information. Also from the documentation:

The backup database is stored entirely in the folder D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}\. The content of this folder is migrated by Windows Home Server Drive Extender from the primary data partition to a secondary data partition if the home server has more than one hard drive.

Thanks Gerry for all your help. You have cleared up this concern for me.

[Farm]

I built a WHS/SAGETV server this weekend and used the installation technique described in the first post. At this time, the WHS box only has 2 1TB drives, one of which is now C/D and the other is DATA. I formated the DATA drive to 64K and created a non-duplicated SAGETV share.

2 questions
1) Should all additional drives I add be reformatted to 64K as well?
2) If not, how do you ensure that the SAGETV share and recordings are only stored on the 64K drive?

I have also read on the forums that others create a SAGETV drive and keep it out of the WHS pool. I'm game for either, but wanted some advice first.

Thanks.

[gplasky]

If you're recording TV to the pool then any additional drives you add to the pool will need to be formatted to 64k just like you did.

If you keep your recording drive out of the pool then any additional drives for recording will need to be out of the pool and you would add it to Sage from the additional drive letter it acquires.

Gerry

[Farm]

Thanks for the reply.
I'll just keep formatting the additional drives to 64K until I notice a problem in doing so (if one ever occurs). I can understand the theory behind keeping a SageTV drive out of the pool, but if all my drives in the pool are going to be 64K, I don't see why it wouldn't work just fine using your method.
Thanks.

[scat]

In the future I want to build a Tera Server for my pc backups, Sage recordings,
DVDs, Bluray DVDs, amd misc video files to one central server location.

I have a few questions about Windows Home Server.

1 each 500GB drive for system
c: drive 100GB
d: drive 400GB+-

Drives for pool to start with
3 each 1TB drives
expandable to 20 or 24 drives.

If we have three computers and all will be backed up to the
Windows Home Server pool daily, and SageTV is recording
to the pool till the drive gets full, then how is the recording
space controlled so Sage doesn't fill up the drive and the pc's don't
have any room to put there backups?

I know we can tell Sage to reserve a certain amount of space on a drive,
but then you have to guess how much space your pc backups will take in the future?

Or when you set up a Share can you limit that share to a certain amount of space
as a quota for the Sage share?

Thanks
Scat

[Skirge01]

Originally Posted by scat

I have a few questions about Windows Home Server.

1 each 500GB drive for system
c: drive 100GB
d: drive 400GB+-

First, WHS uses the first drive for a C: and D: drive. The C: drive is 20GB and the D: drive is whatever remains on the system drive. This was originally done because D: was used as a "landing zone" for new files, but MS did away with that.

Second, any drive you add to the WHS drive pool will not be assigned a drive letter.

Drives for pool to start with
3 each 1TB drives
expandable to 20 or 24 drives.

If we have three computers and all will be backed up to the
Windows Home Server pool daily, and SageTV is recording
to the pool till the drive gets full, then how is the recording
space controlled so Sage doesn't fill up the drive and the pc's don't
have any room to put there backups?

I know we can tell Sage to reserve a certain amount of space on a drive,
but then you have to guess how much space your pc backups will take in the future?

This is how I do it. I basically reserve 1TB for PC backups by telling Sage to leave off that much drive space. I've been backing up 2 computers (admittedly, which have minimal changes day-to-day) for about 8 months and my backups are currently only taking up 342GB. I also ensured that I'm not backing up unnecessary data.

Or when you set up a Share can you limit that share to a certain amount of space
as a quota for the Sage share?

While Windows Server 2003 (what WHS is based upon) does support this, it is an unsupported functionality in WHS. I don't know of anyone who has attempted to use quotas under WHS, so it may simply break WHS and the drive pool entirely. I'd be very cautious about this.

[wayner]

Unless you already have them lying around I wouldn't bother with 1TB drives as 1.5TB are just as cheap per TB and if you expect to have lots of storage eventually you will want to minimize the number of drives. I would even consider the 2TB models. I don't know about adding 20-24 drives - that is a LOT!

One other thing to keep in mind - typically the backups don't use a lot of space since most people store their media files on the WHS box. Therefore your PCs generally just have the OS, apps and a few data files - typically far less than 100GB.

[Rogue9]

I recently upgraded my desktop/gaming box. Which means I have an extra dual-core CPU and MB, with 3 GB of RAM sitting around, that I'd love to upgrade my current server with. I've been using XP MCE on my server forever, but I thought I'd give WHS a try with the new hardware. But I do have a few questions.

My current server's MB does not have any SATA connections, so I'm using 2 IDE drives (1 X 320 GB & 1 X 250 GB). All of my media are on these drives, along with quite a few downloads and other files that I wanted to keep in a central location.

Since the new MB has SATA ports, and because I needed some more storage space, I just ordered 2 X 1 TB SATA drives. So I'm wondering if I should install WHS to one of the 1 TB drives, or if I should try and back up the data on one of my current IDE drives, and use that as the system drive, instead. I know, performance wise, its better to use the SATA drives. But space-wise, it seems like I'd be better off using one of the smaller drives for the OS, and just putting the TB drives in the Storage pool. It seems like kind of a waste to have the TB drive used as the C: and D: partitions for WHS. So just wondering what the best course of action would be here.

Also, I haven't used SageTV to record anything in a couple of years now. My server is almost exclusively used for watching ripped DVD's and movies. I use Hulu to catch the stuff that I happen to miss on TV. And I don't really plan on recording anything in the foreseeable future. So I'm assuming that I would be ok staying with the default 4k cluster size?

[Hydef4me]

Originally Posted by gplasky

Just so you know on my latest adventure to get my drives in the pool I just copied everything off the D drive (wasn't alot but did include some backups) to a USB drive I had plugged in, formatted D: with 64K blocks and then copied it all back on. I don't think WHS even knew it was gone.

Gerry

Gerry:
I am new to this and just building my first WHS, so pardon my ignorance in advance. But could you tell me "exactly" how you copied everything off of the D partition then back again after the reformat. Thanks.

Mike

[gplasky]

Simple copy and paste through the file explorer. Make sure hidden and system files can be seen. Hightlight everything in the root of the D: drive and right-click and choose copy. Paste onto a drive not in the pool. I had an external USB drive called F:. So I copied everything onto F:. In disk administrator I found the D: partition and formatted it with 64K blocks. I then highlighted and copied everything from the root of my F: drive and pasted it back into the root of D: thru file explorer. I then let it set for a while making sure all drive activity ended. Then rebooted.

Gerry

[Hydef4me]

Originally Posted by gplasky

Simple copy and paste through the file explorer. Make sure hidden and system files can be seen. Hightlight everything in the root of the D: drive and right-click and choose copy. Paste onto a drive not in the pool. I had an external USB drive called F:. So I copied everything onto F:. In disk administrator I found the D: partition and formatted it with 64K blocks. I then highlighted and copied everything from the root of my F: drive and pasted it back into the root of D: thru file explorer. I then let it set for a while making sure all drive activity ended. Then rebooted.
Gerry

Thanks Gerry

It worked for me. So far no problems.

Mike

[heatvent]

Originally Posted by Hydef4me

Thanks Gerry

It worked for me. So far no problems.

Mike

I would recomend using sync software like Synctoy or Allway Sync. I have tried the cut and paste route before and if there is an issue copying, you can't pick up where you left off easily.

[gplasky]

Originally Posted by Hydef4me

Thanks Gerry

It worked for me. So far no problems.

Mike

Originally Posted by heatvent

I would recomend using sync software like Synctoy or Allway Sync. I have tried the cut and paste route before and if there is an issue copying, you can't pick up where you left off easily.

It's a one time operation. He doesn't have to do it again. And at the time it's done there is next to no data on the drive.

Gerry