Recommended NAS?

[Peter_h]

Originally Posted by mikesm

I use showanalyzer - works great. I record directly to NAS, and another (windows) server runs showanalyzer across the network reading from the NAS. I just got VMWARE running up on the NAS itself (it is linux based), and so will move commercial processing to the VM there, which should reduce traffic and improve I/O. The NAS has a quad core CPU on it and plenty of ram, so no issue with it running there.

I see, that's the answer I was looking for. I don't think showanalyzer does h.264 yet does it?

I was just concerned about bringing the sage server to a crawl with that high of CPU useage.

My initial thought was to record directly to the NAS and use VMWARE to run comskip. So, it is good to see that somebody else does do it this way.

[mikesm]

Originally Posted by Peter_h

I see, that's the answer I was looking for. I don't think showanalyzer does h.264 yet does it?

I was just concerned about bringing the sage server to a crawl with that high of CPU useage.

My initial thought was to record directly to the NAS and use VMWARE to run comskip. So, it is good to see that somebody else does do it this way.

No, it doesn't do h.264 yet - no one does (yet).

I run Sage server on a discrete server. I don't think you can really handle the tuners well under VMware, at least the USB and PCI ones I have. So I guess I could run showanalyzer on that system too, but since the disks are on the NAS it seems somewhat more optimal.

[Peter_h]

Originally Posted by mikesm

No, it doesn't do h.264 yet - no one does (yet).

I run Sage server on a discrete server. I don't think you can really handle the tuners well under VMware, at least the USB and PCI ones I have. So I guess I could run showanalyzer on that system too, but since the disks are on the NAS it seems somewhat more optimal.

Hey Mike, check the link I posted. They have h.264 working with comskip. It's still in beta testing but it IS working.

[stevech]

This has been discussed previously, but I'll ask again anyway:

I've not found Linux based NAS that allow one to create a disk/partition/volume or whatever it's named that has big allocation blocks that are most prudent for Video files. As we do for Windows where we (should) put video in a video-only partition formatted to 64KB blocks. This makes a huge reduction in I/O's per second and head movement.

My little RAID1 for video does this. It's a pair of 500GB drives with a C: partition and a V: partition in Win XP. RAID1 on the $65 motherboard (NVIDIA). I'm OK with RAID1 as I just want to reduce the chance of a drive failure clobbering all my data - videos and photo collection. (The photos are also on one other drive since they are irreplaceable).

That AMD ASUS motherboard also has built-in gigE an a video chip that runs just fine on my 22in LCD monitor (1650 pixels wide)

[gplasky]

You usually don't need to do that based on the file system your NAS is using. For instance blocks used in files stored on XFS filesystems are managed with variable length extents where one extent describes one or more contiguous blocks. This can shorten the list considerably compared to file systems that list all blocks used by a file individually. Also many file systems manage space allocation with one or more block oriented bitmaps — in XFS these structures are replaced with an extent oriented structure consisting of a pair of B+ trees for each filesystem allocation group (AG). One of the B+ trees is indexed by the length of the free extents, while the other is indexed by the starting block of the free extents. This dual indexing scheme allows for highly efficient location of free extents for file system operations.

An extent is a contiguous area of storage in a computer file system, reserved for a file. When starting to write to a file, a whole extent is allocated. When writing to the file again, possibly after doing other write operations, the data continues where the previous write left off. This reduces or eliminates file fragmentation.

Extents are supported in:

Macintosh Hierarchical File System and HFS Plus
SGI XFS (Designed to be an extent based file system)
Reiser4 (in "extents" mode)
Universal Disk Format (UDF)
VERITAS File System (via the preallocation API and CLI).
Linux ext4 (when extents are enabled, the default since kernel 2.6.23)
OS/2 and eComStation HPFS
HP Multi-Programming Executive file system
JFS for AIX, OS/2/eComStation and Linux
BFS for BeOS, Zeta and Haiku
The SINTRAN III file system
Oracle Automatic Storage Management

XFS makes use of lazy evaluation techniques for file allocation. When a file is written to the buffer cache, rather than allocating extents for the data, XFS simply reserves the appropriate number of file system blocks for the data held in memory. The actual block allocation occurs only when the data is finally flushed to disk. This improves the chance that the file will be written in a contiguous group of blocks, reducing fragmentation problems and increasing performance.

*Info gleamed from Wikipedia

Gerry

[mikesm]

Originally Posted by stevech

This has been discussed previously, but I'll ask again anyway:

I've not found Linux based NAS that allow one to create a disk/partition/volume or whatever it's named that has big allocation blocks that are most prudent for Video files. As we do for Windows where we (should) put video in a video-only partition formatted to 64KB blocks. This makes a huge reduction in I/O's per second and head movement.

My little RAID1 for video does this. It's a pair of 500GB drives with a C: partition and a V: partition in Win XP. RAID1 on the $65 motherboard (NVIDIA). I'm OK with RAID1 as I just want to reduce the chance of a drive failure clobbering all my data - videos and photo collection. (The photos are also on one other drive since they are irreplaceable).

That AMD ASUS motherboard also has built-in gigE an a video chip that runs just fine on my 22in LCD monitor (1650 pixels wide)

Gplasky is right about this from a filesystem perspective, but you can also configure stripe size and buffering to ensure you get big chunks written to disk if you are using RAID under linux. I personally use a 128KB chunk size is raid5, so all read/writes are done in 128KB chunks. It's kind of a tie with 256 KB chunk size for best performance in video serving.

Don't insist on using windows style semantics when dealing with other OS's. Just because it's not called the same thing doesn't mean it's not there, and implemented better to boot. :-)

[heffe2001]

Originally Posted by jgourd

I do recommend a NAS for static media like music and video collections. I personally recommend the Drobo with a DroboShare for that. I recommend the Drobo for the following reasons:
1. The data is protected
2. Unlike RAID5 you can have discs of different sizes.
3. When it fills up, you take out the smallest or oldest drive and replace it with the largest one you can get for the best price.
4. The DroboShare has an open API and there are streaming apps you can get for it.
5. The DroboShare can run two Drobos at once.
6. When replacing a drive you still have access to all the data while it is working to re-protect the array.

I just recently set up an UnRaid server, and it's quite similar to the Drobo/Droboshare combo in capabilities. I use a cache disk to improve speed as well (things written to the cache drive are written without parity, so speed is much higher than writing to the cache drive, between 22-28m as opposed to around 12-14k direct to the parity-backed drives). I've got a mix of drive sizes from 250g to 640g (8 total at the moment) in my server (80g cache for now, going to move the 250 in it's place soon as my 4-in-3 drive cage comes in), and it's serving 2 MVP's, 2 HD Extenders, and a couple placeshifter/client machines. I'm only using it for storage of my DVD's & xvids at the moment (not using it for sage recordings yet, if ever). I might try at some point setting up a raid-0 cache drive for it, and see if the speed improves, if so I'll probably switch to using it for Sage recordings, but I like that it's a parity-backed system with any drive sizes (as long as your largest is parity that is). All I need now is to finally get around to ripping the last 100 or so DVD's to the server (I'm doing complete, uncompressed rips of my DVD's, so they are averaging about 7.5-8g each, but looks great on my HD Extenders).

I'd recommend this setup to anyone wanting to add storage for static data, DVD's and such that you want to be protected, it's been working great, and no proprietary hardware to contend with..

[rsw686]

Originally Posted by mikesm

This does not appear to be present in non highpoint based 88SX7042 cards like the Adaptec 1430SA, or Rosewill RC-218. For linux, you can just use the builtin sata_mv driver and avoid all the proprietary stuff.

The adaptec 1430SA is available from newegg for about $105, and under $100 wholesale, and is an X4 card. My friend is going to get one and if it works well I'll upgrade my system to it as well.

Hopefully this isn't getting too far off topic. I was going to send a PM, but that looks to be disabled, at least for members with no posts yet.

I read through the rest of the posts, but never saw a response about how the Adaptec card worked out for your friend. I've been debating between the Rosewall RC-218 and the Adaptec 1430SA card and this thread turned up in a Google search. I have read mixed reviews about the linux support for both cards. I'm not sure which distro you use, but I'm looking at using one of the cards on a CentOS 5.2 box. Any insight would be helpful.

[r00st3r]

I currently use FREENAS and have been for the past couple years. It's stable and free. I have 1TB now that I use for backing up data from the PC'S (mine, the wife's PC and the sons laptop), and also as a repository for Music, Movies, Photos, and Downloaded TV shows that I have available for Sage. If there is ever a natural disaster, this is the bad boy I'm grabbing after the wife\kid\dogs of course ;-) I eventually will be adding two more drives 1TB. It supports RAID and has a very small footprint.


I am using an old PIII with 512MB ram to run it on.
If you have access to an older machine, download it and give it a try..

..oh yeah, I need to upgrade the NIC to GB...

[mikesm]

Originally Posted by rsw686

Hopefully this isn't getting too far off topic. I was going to send a PM, but that looks to be disabled, at least for members with no posts yet.

I read through the rest of the posts, but never saw a response about how the Adaptec card worked out for your friend. I've been debating between the Rosewall RC-218 and the Adaptec 1430SA card and this thread turned up in a Google search. I have read mixed reviews about the linux support for both cards. I'm not sure which distro you use, but I'm looking at using one of the cards on a CentOS 5.2 box. Any insight would be helpful.

Both use a late model Marvell chipset. Any current kernel should have excellent support for it. The adaptec tends to be a bit higher end in quality than the rosewill, and the BIOS is better supported, so I would pick that one over the RC-218.

PMP support is there too in case you plan on running outboard sata chassis.

[stevech]

what's wrong with the following logic? ...

Goal 1: Minimize risk of data loss due to physical drive failure (head crash, bearings fail, etc)
Goal 2: Avoid catch-22 where disk formats can be read only with a certain disk controller and mating software, e.g., RAID5

Solution: Hardware RAID1, NTFS. Either disk can be independently booted. Disk can be a data drive on any windows PC.

Pitfall: NTFS corruption can corrupt both drives. But it's been years since that happened to me, i.e., NTFS seems reasonably robust.

[bastafidli]

The same can be achieved with Software Raid 1 on Linux since SWR1 can be bootable. That is the setup I am running.

[stevech]

Originally Posted by bastafidli

The same can be acchieved with Software Raid 1 on Linux since SWR1 can be bootable. That is the setup I am running.

Forgive my ignorance: SWR1 is a Linux distro?

[Taddeusz]

Originally Posted by stevech

what's wrong with the following logic? ...

Goal 1: Minimize risk of data loss due to physical drive failure (head crash, bearings fail, etc)
Goal 2: Avoid catch-22 where disk formats can be read only with a certain disk controller and mating software, e.g., RAID5

Solution: Hardware RAID1, NTFS. Either disk can be independently booted. Disk can be a data drive on any windows PC.

Pitfall: NTFS corruption can corrupt both drives. But it's been years since that happened to me, i.e., NTFS seems reasonably robust.

Goal 2 is unrealistic. With ANY RAID solution save software RAID on Linux or Windows you're going to be hooked into a particular piece of hardware. That's just the way it is. There is no such thing as a "standard RAID" disk format. It doesn't exist. RAID disks cannot ever be taken from one controller to another unless it is strictly software RAID done by the OS. But then you are tied to the OS.

Besides, the goal of RAID is to minimize downtime due to disk failure. RAID IS NOT A BACKUP. Even with RAID you should ALWAYS have a proper backup because RAID does not protect you from data corruption only disk failure.

With the proper controller or fast processor RAID 5 is the most economical RAID level to protect from disk failure. RAID 1 is ok for two drives. RAID 0+1 (or 10, whatever you want to call it) is just a waste of space with little return otherwise. RAID 5 provides the best ratio of physical drives to disk space for larger arrays.

[stanger89]

Originally Posted by Taddeusz

RAID disks cannot ever be taken from one controller to another...

Not exactly true, I know 3ware at least, you can move arrays to completely different controllers and it will be intact. In fact you can even take the array from a 7000 series controller and move it to a 8000, or 9000 series controller and your data survives.

[Taddeusz]

Originally Posted by stanger89

Not exactly true, I know 3ware at least, you can move arrays to completely different controllers and it will be intact. In fact you can even take the array from a 7000 series controller and move it to a 8000, or 9000 series controller and your data survives.

Yea, I guess I forgot to put in the exception of the possibility of being able to move arrays between different controllers from the same company.

[stanger89]

Anyway, on the "RecommendedNAS" topic, I'm really interested in the new ReadyNAS Pro. 6 bays vs the 4 on the previous models (so up to 7.5TB capacity today, but beyond that they replaced the ASIC with a Core2 Duo processor, dual-ethernet (with aggrigation or failover), and >100MB/sec transfer rates, and aparently the ability to make use of somewhat odd combinations of drive sizes. Runs Squeezecenter and a number of other streaming services.

Only downside is the price, but I'm hopeful the Pioneer edition (diskless) will be ~$1000 and keep all the really important features.

Oh, and theoretically at least, you could run SageTV on it I think.

[bastafidli]

Originally Posted by stevech

Forgive my ignorance: SWR1 is a Linux distro?

I apologize I was writing in hurry, SWR1 = software raid 1 :-)

[chrisc16]

Originally Posted by stanger89

Only downside is the price, but I'm hopeful the Pioneer edition (diskless) will be ~$1000 and keep all the really important features.

Oh, and theoretically at least, you could run SageTV on it I think.

Wow, if there was a way to run SageTV on the Pioneer model (use USB ports for tuners), I would definitely go this route. Power usage would go down, reliability would (should) go up.

-Chris

[stanger89]

Originally Posted by chrisc16

Wow, if there was a way to run SageTV on the Pioneer model (use USB ports for tuners), I would definitely go this route. Power usage would go down, reliability would (should) go up.

-Chris

Yeah, it's a very interesting thought. Like I said, it's theoretically possible. The Pro runs a C2D CPU on a (modified) linux OS. I'd think getting Sage running would be possible with minimal effort, and I'm guessing the HDHR would work. Not sure what's involved in getting USB tuners going, but the Pioneer has them so if it doesn't involve kernel compiling....