Recommended NAS?

[mtyme]

Originally Posted by mikesm

Did you format with 128kb or 256kb chunk size, and XFS for the filesystem? Did you try running my little diskopt.sh script on startup?

I used 128kb chunk size and XFS. I have your script but I'm not sure how to set it up so that it will run at startup or where I need to place it.

I verified the bios settings you mentioned, and just for giggles tried to put the drives on those ports again...and now they work. I swear I didn't change a single thing in bios so I don't know why they wouldn't work before.

Anyway they now show as sata 300, but hdparm -tT still shows 61MB/s.

Edit: I noticed that I get better results if I don't mount the array. I know I'm nowhere close to the speeds I should be getting anyway, but check this out..
/dev/sdd is a stand alone sataII WD drive partitioned with NTFS

goliath:/home/shawn # hdparm -tT /dev/evms/media_vol

/dev/evms/media_vol:
 Timing cached reads:   1956 MB in  2.00 seconds = 977.33 MB/sec
 Timing buffered disk reads:  266 MB in  3.00 seconds =  88.53 MB/sec
goliath:/home/shawn # hdparm -tT /dev/sdd

/dev/sdd:
 Timing cached reads:   1942 MB in  2.00 seconds = 971.22 MB/sec
 Timing buffered disk reads:  242 MB in  3.02 seconds =  80.13 MB/sec

goliath:/home/shawn # mount /dev/evms/media_vol /media/vol1
goliath:/home/shawn # hdparm -tT /dev/evms/media_vol

/dev/evms/media_vol:
 Timing cached reads:   1994 MB in  2.00 seconds = 996.84 MB/sec
 Timing buffered disk reads:  186 MB in  3.00 seconds =  61.93 MB/sec
goliath:/home/shawn # hdparm -tT /dev/evms/media_vol

/dev/evms/media_vol:
 Timing cached reads:   1962 MB in  2.00 seconds = 980.88 MB/sec
 Timing buffered disk reads:  186 MB in  3.01 seconds =  61.86 MB/sec
goliath:/home/shawn # umount /dev/evms/media_vol
goliath:/home/shawn # hdparm -tT /dev/evms/media_vol

/dev/evms/media_vol:
 Timing cached reads:   1958 MB in  2.00 seconds = 978.26 MB/sec
 Timing buffered disk reads:  264 MB in  3.00 seconds =  87.96 MB/sec

[johnm160]

Originally Posted by mikesm

Depends on the particular lan chipset you are using. You might be able to get 80 Mbps out of a 100 Mbps LAN on a good day, but you are seeing almost 70% utilization, so that's not that all that bad.

If the LAN is the bottleneck, even WHS can tie a good implementation for performance. :-) I would upgrade to Gigabit ethernet if you want better speeds.

Well since you said I should be able to do better......I tried

I found that by turning off QOS you can get a pretty nice speed boost.

With QOS on:
6.5 MB/s = 52 Mb/s Network utilization is 61 - 69 % during the transfer.

With QOS off:

9.8 MB/s = 78 Mb/s Network utilization is 88 - 91 %


That is a nice little improvement.

John

[mikesm]

Originally Posted by johnm160

Well since you said I should be able to do better......I tried

I found that by turning off QOS you can get a pretty nice speed boost.

With QOS on:
6.5 MB/s = 52 Mb/s Network utilization is 61 - 69 % during the transfer.

With QOS off:

9.8 MB/s = 78 Mb/s Network utilization is 88 - 91 %


That is a nice little improvement.

John

Very good work there... :-)

[mtyme]

I rebooted today and the raid volume didn't mount like it usually does. So I opened EVMS and it was there but when I tried to mount it I got an error: Broken Pipe.

What does that mean?

[ToxMox]

Originally Posted by mtyme

I rebooted today and the raid volume didn't mount like it usually does. So I opened EVMS and it was there but when I tried to mount it I got an error: Broken Pipe.

What does that mean?

If I had to guess since this happened to me (different error though) the other day. If the machine has an unclean shutdown the RAID won't mount until you run a check on it. I use JFS so I had to run jfs_fsck on the logical volume. Turned out my /etc/fstab file wasn't configured properly. Since it isn't the root volume on my system I had to change the 0 at the end to a 2 which forces a check before attempting to mount on bootup.

[mikesm]

Originally Posted by mtyme

I rebooted today and the raid volume didn't mount like it usually does. So I opened EVMS and it was there but when I tried to mount it I got an error: Broken Pipe.

What does that mean?

I agree with toxmox, though I thought EVMS set that option in the fstab when it configured the volume for mounting. Did you see any errors in the system log?

Are you still seeing the problem?

thx
mike

[mtyme]

Originally Posted by mikesm

I agree with toxmox, though I thought EVMS set that option in the fstab when it configured the volume for mounting. Did you see any errors in the system log?

Are you still seeing the problem?

thx
mike

I couldn't get xfs_check to run, it said there were errors. So I ran xfs_repair and it said something about a bad superblock, and it was trying to find the 2nd one but it never finished. I eventually just rebuilt the arrary. I didn't have any data on there I couldn't get back. The only think I can think of that I did was fix the system clock. For some reason every time I'd boot, the time/date would be different. But I don't know that that was the issue.

[mikesm]

Originally Posted by mtyme

I couldn't get xfs_check to run, it said there were errors. So I ran xfs_repair and it said something about a bad superblock, and it was trying to find the 2nd one but it never finished. I eventually just rebuilt the arrary. I didn't have any data on there I couldn't get back. The only think I can think of that I did was fix the system clock. For some reason every time I'd boot, the time/date would be different. But I don't know that that was the issue.

You didn't set clock in the "past" did you? Filesystem work needs to have a clock that always advances in time. SAMBA also will need rough clock sync with clients as well.

The clock wasn't off by exactly an hour or two right? How much variance are you seeing at boot? Do you run NTP or some other process to maintain accurate time?

[mtyme]

It was off by an hour or two after in initially installed it. I never paid any attention to it, then noticed later the date had somehow gotten moved to 2010. I noticed during a suse update there was something in there about a system date time update so I ran that and reset the time date to current.

[KJake]

Originally Posted by mikesm

Obviously, this requires another 1 TB drive purchase!

I finally have another one and new PMPs. I'm letting the new 2.7TB PV initialize over night now and then I hope I'll be able to use pvmove to move the content of my 2.3TB PV into the 2.7TB PV and remove the 2.3TB PV.

Then I think I can get rid of my 500gb AAK drives finally.

[KJake]

Originally Posted by KJake

I finally have another one and new PMPs. I'm letting the new 2.7TB PV initialize over night now and then I hope I'll be able to use pvmove to move the content of my 2.3TB PV into the 2.7TB PV and remove the 2.3TB PV.

Then I think I can get rid of my 500gb AAK drives finally.

lol, I can't catch a break. pvmove errors with a weird error and then I find:
http://archives.free.net.ph/message/...cb0b19.en.html

I'm running 2.6.22-rc6...doh. Looks like it is fixed in 2.6.22-2; I should get off of an rc kernel anyways...

[KJake]

Originally Posted by KJake

lol, I can't catch a break. pvmove errors with a weird error and then I find:
http://archives.free.net.ph/message/...cb0b19.en.html

I'm running 2.6.22-rc6...doh. Looks like it is fixed in 2.6.22-2; I should get off of an rc kernel anyways...

New kernel built - I'll try it out tonight when I'm home for the reboot.

I just found out that I messed up that disk optimization script that mikesm gave me and that a lot of the settings were not being applied. This meant that my new md array that I created last night has been syncing/initing at 1MB/sec since midnight...no wonder it isn't done yet! Also, NCQ was still on for all the drives, etc etc...here's the script with my bungling removed:

root@orion:~# cat /etc/init.d/diskopt
#!/bin/bash
#
### BEGIN INIT INFO
# Description:                  Custom disk optimization script
### END INIT INFO

. /lib/lsb/init-functions

case "$1" in
        start)


echo "Optimizing RAID Arrays..."


# This step must come first.
# See: http://www.3ware.com/KB/article.aspx?id=11050

echo "Setting max_sectors_kb to 128K for RAID5 arrays..."
for i in sda sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp
do
echo "Setting /dev/$i to 128K..."
echo 128 > /sys/block/"$i"/queue/max_sectors_kb
done

echo "Setting read-ahead to 64MB for /dev/md[0-1]"
blockdev --setra 65536 /dev/md0
blockdev --setra 65536 /dev/md1
blockdev --setra 65536 /dev/md2

echo "Setting stripe_cache_size to 16MB for /dev/md[0-1]"
echo 16384      > /sys/block/md0/md/stripe_cache_size
echo 16384      > /sys/block/md1/md/stripe_cache_size
echo 16384      > /sys/block/md2/md/stripe_cache_size

# if you use more than the default 64kb stripe with raid5
# this feature is broken so you need to limit it to 30MB/s
# neil has a patch, not sure when it will be merged.
echo "Setting minimum and maximum resync speed to 30MB/s..."
echo 30000 > /sys/block/md0/md/sync_speed_min
echo 30000 > /sys/block/md0/md/sync_speed_max
echo 30000 > /sys/block/md1/md/sync_speed_min
echo 30000 > /sys/block/md1/md/sync_speed_max
echo 30000 > /sys/block/md2/md/sync_speed_min
echo 30000 > /sys/block/md2/md/sync_speed_max


# Disable NCQ.
echo "Disabling NCQ..."
for i in sda sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp
do
echo "Disabling NCQ on $i"
echo 1 > /sys/block/"$i"/device/queue_depth
done

        ;;
        stop)
        # skip or put your own commands here
        ;;
        status)
echo "Getting max_sectors_kb from disks..."
for i in sda sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp
do
echo -n "$i: "
cat /sys/block/"$i"/queue/max_sectors_kb
done

echo "Getting read-ahead setting from /dev/md[0-2]"
echo -n "md0: "
blockdev --getra /dev/md0
echo -n "md1: "
blockdev --getra /dev/md1
echo -n "md2: "
blockdev --getra /dev/md2

echo "Getting stripe_cache_size from /dev/md[0-2]"
echo -n "md0: "
cat /sys/block/md0/md/stripe_cache_size
echo -n "md1: "
cat /sys/block/md1/md/stripe_cache_size
echo -n "md1: "
cat /sys/block/md2/md/stripe_cache_size

echo "Getting minimum and maximum resync speed from /dev/md[0-2]..."
echo -n "md0-min: "
cat /sys/block/md0/md/sync_speed_min
echo -n "md0-max: "
cat /sys/block/md0/md/sync_speed_max
echo -n "md1-min: "
cat /sys/block/md1/md/sync_speed_min
echo -n "md1-max: "
cat /sys/block/md1/md/sync_speed_max
echo -n "md2-min: "
cat /sys/block/md2/md/sync_speed_min
echo -n "md2-max: "
cat /sys/block/md2/md/sync_speed_max


echo "Getting NCQ settings..."
for i in sda sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp
do
echo -n "$i: "
cat /sys/block/"$i"/device/queue_depth
done

        ;;
esac

My md is syncing at 30MB/sec now, as it should! My real hope is that I now discover a little more performance from my system

[mikesm]

Originally Posted by KJake

New kernel built - I'll try it out tonight when I'm home for the reboot.

I just found out that I messed up that disk optimization script that mikesm gave me and that a lot of the settings were not being applied. This meant that my new md array that I created last night has been syncing/initing at 1MB/sec since midnight...no wonder it isn't done yet! Also, NCQ was still on for all the drives, etc etc...here's the script with my bungling removed:

root@orion:~# cat /etc/init.d/diskopt
#!/bin/bash
#
### BEGIN INIT INFO
# Description:                  Custom disk optimization script
### END INIT INFO

. /lib/lsb/init-functions

case "$1" in
        start)


echo "Optimizing RAID Arrays..."


# This step must come first.
# See: http://www.3ware.com/KB/article.aspx?id=11050

echo "Setting max_sectors_kb to 128K for RAID5 arrays..."
for i in sda sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp
do
echo "Setting /dev/$i to 128K..."
echo 128 > /sys/block/"$i"/queue/max_sectors_kb
done

echo "Setting read-ahead to 64MB for /dev/md[0-1]"
blockdev --setra 65536 /dev/md0
blockdev --setra 65536 /dev/md1
blockdev --setra 65536 /dev/md2

echo "Setting stripe_cache_size to 16MB for /dev/md[0-1]"
echo 16384      > /sys/block/md0/md/stripe_cache_size
echo 16384      > /sys/block/md1/md/stripe_cache_size
echo 16384      > /sys/block/md2/md/stripe_cache_size

# if you use more than the default 64kb stripe with raid5
# this feature is broken so you need to limit it to 30MB/s
# neil has a patch, not sure when it will be merged.
echo "Setting minimum and maximum resync speed to 30MB/s..."
echo 30000 > /sys/block/md0/md/sync_speed_min
echo 30000 > /sys/block/md0/md/sync_speed_max
echo 30000 > /sys/block/md1/md/sync_speed_min
echo 30000 > /sys/block/md1/md/sync_speed_max
echo 30000 > /sys/block/md2/md/sync_speed_min
echo 30000 > /sys/block/md2/md/sync_speed_max


# Disable NCQ.
echo "Disabling NCQ..."
for i in sda sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp
do
echo "Disabling NCQ on $i"
echo 1 > /sys/block/"$i"/device/queue_depth
done

        ;;
        stop)
        # skip or put your own commands here
        ;;
        status)
echo "Getting max_sectors_kb from disks..."
for i in sda sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp
do
echo -n "$i: "
cat /sys/block/"$i"/queue/max_sectors_kb
done

echo "Getting read-ahead setting from /dev/md[0-2]"
echo -n "md0: "
blockdev --getra /dev/md0
echo -n "md1: "
blockdev --getra /dev/md1
echo -n "md2: "
blockdev --getra /dev/md2

echo "Getting stripe_cache_size from /dev/md[0-2]"
echo -n "md0: "
cat /sys/block/md0/md/stripe_cache_size
echo -n "md1: "
cat /sys/block/md1/md/stripe_cache_size
echo -n "md1: "
cat /sys/block/md2/md/stripe_cache_size

echo "Getting minimum and maximum resync speed from /dev/md[0-2]..."
echo -n "md0-min: "
cat /sys/block/md0/md/sync_speed_min
echo -n "md0-max: "
cat /sys/block/md0/md/sync_speed_max
echo -n "md1-min: "
cat /sys/block/md1/md/sync_speed_min
echo -n "md1-max: "
cat /sys/block/md1/md/sync_speed_max
echo -n "md2-min: "
cat /sys/block/md2/md/sync_speed_min
echo -n "md2-max: "
cat /sys/block/md2/md/sync_speed_max


echo "Getting NCQ settings..."
for i in sda sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp
do
echo -n "$i: "
cat /sys/block/"$i"/device/queue_depth
done

        ;;
esac

My md is syncing at 30MB/sec now, as it should! My real hope is that I now discover a little more performance from my system

Yeah the out of the box settings are optimized for smaller filesystems. Using the RAM more effectively for buffering makes things go a LOT faster.

[KJake]

Originally Posted by KJake

lol, I can't catch a break. pvmove errors with a weird error and then I find:
http://archives.free.net.ph/message/...cb0b19.en.html

I'm running 2.6.22-rc6...doh. Looks like it is fixed in 2.6.22-2; I should get off of an rc kernel anyways...

It's working! Running stable 2.6.24.3 without any patches. All the devices came up clean and I just started "pvmove -v /dev/md0 /dev/md2" and it is actually moving. This is awesome. After the move, I'll actually have 14904 Free PE that I'll have to allocate and grow, so I am ending up with a bit more space (I'm taking the 6x500GB offline).

This is going to take a while, I probably should have started this on the console...I see people online saying that if it is interrupted that I will be a very unhappy camper.

[zoop]

man screen - It's your friend for such lengthy endeavors.

I commend you on your dedication, having spent the weekend fighting with 6 revs of kernels and 4 revs of ivtv before reverting, i know your pain...

/jer

[KJake]

Originally Posted by zoop

man screen - It's your friend for such lengthy endeavors.

I commend you on your dedication, having spent the weekend fighting with 6 revs of kernels and 4 revs of ivtv before reverting, i know your pain...

/jer

yeah, even better, I should have used nohup and just tail -f the output file...but I had doubts that the pvmove would work at all in the first place...

[mikesm]

Originally Posted by KJake

It's working! Running stable 2.6.24.3 without any patches. All the devices came up clean and I just started "pvmove -v /dev/md0 /dev/md2" and it is actually moving. This is awesome. After the move, I'll actually have 14904 Free PE that I'll have to allocate and grow, so I am ending up with a bit more space (I'm taking the 6x500GB offline).

This is going to take a while, I probably should have started this on the console...I see people online saying that if it is interrupted that I will be a very unhappy camper.

Great. This stuff is great isn't it? All this work and while the filesystem is online.

The pvmove will take quite awhile when the filesystem is active, but I think you will be fine.

:-)

[KJake]

Originally Posted by mikesm

Great. This stuff is great isn't it? All this work and while the filesystem is online.

The pvmove will take quite awhile when the filesystem is active, but I think you will be fine.

:-)

yeah, I really do love it
Sage is still recording away and I'm watching TV, have access to My Documents, etc !

5.3% moved

[KJake]

Originally Posted by KJake

yeah, I really do love it
Sage is still recording away and I'm watching TV, have access to My Documents, etc !

5.3% moved

29.7% moved.

I think I can now estimate this will take 30 hours (in total) to move a 2.3TB PV (21 more hours to go). My rough estimation is that it is moving about 75GB per hour. This doesn't seem bad to me knowing how much work pvmove is doing to protect each PE it moves.

[mikesm]

Originally Posted by KJake

29.7% moved.

I think I can now estimate this will take 30 hours (in total) to move a 2.3TB PV (21 more hours to go). My rough estimation is that it is moving about 75GB per hour. This doesn't seem bad to me knowing how much work pvmove is doing to protect each PE it moves.

That smells about right to me. It would be nice if it went faster, but since the volume is online and really not all that impacted performance wise, it's not that big a deal to me...