Friday, August 31, 2012

Synology DS412+ and My New Storage Network

Back in 2000 I purchased my first DV camcorder. It was a Canon Elura that used MiniDV tapes. Every tape extracted about 13GB. Even today that's still a lot. You can easily fill hard drives with that much storage. At first I only edited the extracted clips then deleted them leaving the original footage on MiniDV tape. Tape was good for archival if you didn't mind waiting for the re-extraction.

Years have passed and I've switched to CF card which can't be used for archival. I have BD burners to make archival copies but I've found that old footage is still best kept close by. Initially I built a RAID desktop for keeping just photos. Then when .5 and 1TB drives became common I started keeping video on those drives.

It was initially going to be temporary but I've found that family events like birthdays and such required me to dig out years old footage so I never went back to delete them.   I got used to it. Fast forward to today and it's time for a major upgrade. My second generation self-built NAS is an Zotac IONITX-A-U running Ubuntu and SAMBA hooked up to USB drives via a hub. It's decent but not very fast. All those drives were formatted in NTFS and maxed out at 25MB/sec to read.    That's enough for viewing a few pictures here and there.  But I wanted something closer to what direct SATA gave me.

Picking the Synology DS412+
I originally considered the Drobo, Netgear and EMC/Iomega NAS.    I read the reviews and the Synology brand kept coming up.   Two things sold me on the thing: (1) Simplicity of DSM (2) Raw performance.  It was the fastest of the more reasonably priced NAS on the market.

Here's a sample of it's performance in my current configuration.   

Robocopy Write test
Copy from internal SATA drive to Diskstation via 1Gbps port (82.5MB/sec)

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :       132       131         1         0         0         0
   Files :      9460      9460         0         0         0         1
   Bytes : 233.776 g 233.776 g         0         0         0    14.5 k
   Times :   0:48:33   0:48:22                       0:00:00   0:00:11

   Speed :            86472142 Bytes/sec.
   Speed :            4947.975 MegaBytes/min.

   Ended : Wed Aug 22 13:49:27 2012

Robocopy Read test
Copy from Diskstation to internal SATA via 1Gbps port (99.98 MB/sec)

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :       132       131         1         0         0         0
   Files :      9461      9461         0         0         0         0
   Bytes : 233.776 g 233.776 g         0         0         0         0
   Times :   0:40:03   0:39:54                       0:00:00   0:00:09

   Speed :           104837993 Bytes/sec.
   Speed :            5998.878 MegaBytes/min.

   Ended : Wed Aug 22 14:38:14 2012

Not bad is it? That's over half the bandwidth of a directly hooked up SATA drive. Given that a single gigabit ethernet port has a theoretical cap of 125MB/sec its looking pretty good.   What this allows me to do is pick out some footage, pictures and music from the NAS and output to local USB and SATA drives.   It's the not the fastest configuration but while I'm doing quick edits in Sony Vegas on my laptop, the desktop could be rendering a composition in Adobe After Effects.

My current configuration
Synology DS412+ (paid link) with 4x Seagate 3TB 7200rpm drives with LAG enabled
ProRaid+ (paid link) with 4x Western Digital 2TB 5400rpm drives attached via SATA
ioSafe SOLO G3 Fireproof and Waterproof 3TB Hard Drive (paid link) for critical backups via USB 3.0
Netgear GS108T-NAS Prosafe 8-Port Gigabit Smart Switch (paid link) attached to 2x1Gigabit Ports on DS412+
Cyberpower 485AV UPS attached via USB

Hold your horses! What's that switch and UPS for you ask? The UPS gives the NAS a chance to do a proper shutdown should there be a power failure. The Smart Switch allows you to bond two ethernet ports together on the DS412+ but keep one IP address. A plain dumb switch would allow you to use the second port as a backup. But a Smart/Managed switch allows both to be used actively with a feature called Link Aggregation (LAG). A 1GigE port has a theoretical max of 125MB/sec but there is some waste in there that brings the real throughput much less. Those individual SATA devices in our previous tests could achieve 175MB/sec. Not to mention I have the ProRaid shared simultaneously as the DS412+'s internal volumes. I can't exceed the throughput of a single port for one transfer but I can have multiple devices and transfers at once each using either available port.  Effectively you can have a ceiling of 250MB/sec on all combined transfers.

Modest Simultaneous Device Test 
Can we prove that LAG is useful?  

(a) Copy from Leonovo W520 SATA to Diskstation via Wireless-G (35.37MB/sec write)
               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :        40        38         2         0         0        12
   Files :       962       961         0         0         1         1
   Bytes : 125.429 g 125.425 g         0         0    4.39 m  226.63 m
   Times :   1:00:35   1:00:30                       0:00:00   0:00:05

   Speed :            37092186 Bytes/sec.
   Speed :            2122.431 MegaBytes/min.

   Ended : Wed Aug 29 22:31:54 2012
(b) Copy from Ubuntu IONITX-A-U NAS from USB drive via 1Gig-E (22MB/sec write)

(c) Copy from HPE-380T whole SATA HDD (74.97 MB/sec write)

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :       568       565         3         0         0         0
   Files :     13809     13809         0         0         0         0
   Bytes : 748.996 g 748.996 g         0         0         0         0
   Times :   2:51:04   2:50:30                       0:00:00   0:00:34

   Speed :            78613676 Bytes/sec.
   Speed :            4498.310 MegaBytes/min.

   Ended : Wed Aug 29 22:42:43 2012

Yes! 132MB/sec with three devices.   This is a write test which is a slower operation.  If I started multiple copies plus video playback I could probably use up more bandwidth.  Keep in mind that 125MB is a theoretical maximum for a single port.  The extra port allows other devices to access the NAS if one is fully saturated. 

A Hint on Adding a Share to your Libraries
You might be wondering how I was able to mount My Documents from the NAS.   It's a trick using symbolic links.

Start -> Command Prompt -> Right-click -> Run as Administrator

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved. 
C:\Windows\system32> mkdir \diskstation
C:\Windows\system32> cd \diskstation
C:\diskstation>mklink /d Pictures \\diskstation\Pictures
symbolic link created for Pictures <===> \\diskstation\Pictures 

From here you can go to Explorer -> Libraries -> Pictures -> Properties -> Include a Folder

Power Consumption
2.7 Watts plugged in for DS412+ with 4 ST3000DM001
17.3 watts powering up (button press)
45.9 watts powering up (spinning up)
62.2 watts peak powering up (blue flashing light)
44-46 watts powered up doing raid check
49 watts doing CrystalMark

Saturday, August 25, 2012

Mediasonic ProRaid and e-SATA without Port Multiplier

Good news everyone! The Lenovo W520 has an e-SATA port that didn't support port multiplier (tested with JBOD enclosures) but there are options for external RAID. I have on my desk a Mediasonic ProRaid+ HFR2-SU3S2FW enclosure. Unlike plain JBOD enclosures like the Thermaltake BlacX Duet or the Mediasonic ProBox series, this unit presents the drives as a single disk via built in hardware RAID.  Only if you run it on JBOD/Spanning mode do you need a port multiplier.

I have to confess, I didn't buy the ProRaid for the W520. It was meant to become an expansion unit to my NAS which didn't support port multiplier either. I'm thinking I should get another of these units or something from its family. Startech sell similar units though a bit more expensive.

I went with the Mediasonic 4-bay because it was cheaper and the good feedback from Amazon, Newegg and Synology forums.   The unit I purchased came with e-SATA, USB, FW400 and FW800 ports.   Another model skips the FW400 and FW800.

Attached to the W520's e-SATA port it is presented as "H/W RAID 5" device.   I didn't try changing the RAID type but I guess its name may vary according to the RAID selected.

Once connected it functions like any other disk which can be initialized then formatted.  Windows doesn't know or care that it is a RAID with four 3TB disks.   All it sees is an 8.353TB device.

Here is the same device attached to my SMB server.   It's hooked up to a non-port multiplier e-SATA port on a Zotac IONITX-A-U motherboard.   The operating system is Ubuntu 10.04 with support for NTFS volumes.  Disk Analyzer must be using base-10 math as the space reported is 9TB (3x3TB + 3TB parity).

Gotcha for First Initialization
Even after reading the manual it caught me unawares at first.  The unit will immediately flash the red H icon after spinup if a RAID setting has not yet been selected.   The instructions are explicit, hold down the mode button for 3 seconds until the RAID mode flashes.  Press mode several times to select the RAID type.  I chose RAID-5 in my case.  Then go to the back of the unit.  Remove the tab covering the RAID confirmation button next to the power plug.  Hold it down until it powers down.   Next time you power up the RAID mode is set.   The device will be uninitialized if the drives have never been used in that RAID mode before.  And any content will be lost once it has started up.

If the drives need to be migrated to another ProRaid unit, you're supposed to be able to use them in the same RAID mode.  The trick is to setup the RAID mode with the ProRaid empty.  As in, no drives should be inserted until it is RAID confirmed and powered down.

Performance Baseline
The ProRaid shows up as a SATA II (3.0Gbps / 384MB/sec) device.  Which is fine since the enclosure is meant for 3.5" mechanical hard drives.   Spindles will barely reach the limit of even SATA 1.0 (1.5 Gbps / 192MB/sec) bandwidth.

The above readings are with four ST3000DM001 SATA III (6.0Gbps) drives hooked up to my HPE-380T desktop.   The physical drives are hidden by the hardware RAID so S.M.A.R.T. info is rather limited.   Health status is readable for the array despite the screenshot.   What surprised me was how the RAID 5 peaked higher for sequential operation that Software RAID 0.   For all other operations types, RAID 0 still won as expected.

   Speed :            29480409 Bytes/sec.
   Speed :            1686.882 MegaBytes/min.

A Robocopy write test was much slower at 28.11 MB/sec. It seems a far cry from the Software RAID 0 setup previously tested.   But that's an apples to oranges comparison.   The CrystalMark test is still at odds at the above results.  I guess I'll take it as a low end baseline for ProRAID.  I'm not too disappointed since that's still faster than my Samba server which I use presently.   The source disk for the test contained thousands of photos, videos and sound files which were stock material for video editing.

Power Consumption
The ProRaid is supposed to power itself down when the host computer shuts down.   I tested it with my Lenovo W520, HP Elite 380T and Zotac IONITX-A-U but the unit did not power down while hooked up to either eSATA, USB 3.0 or USB 2.0.    Later I would discover that it would shutdown correctly on eSATA for a Synology DS412+.   I could have sworn the laptop and desktop were setup for AHCI mode on the BIOS of at least the HP.   This is something to research on the Mediasonic forum.

0.3 watts with power brick plugged in (Vampire Power)
28 watts average with disks at rest
36 watts average with disks operating
43 watts peak observed during spindown

I got those readings with a Kill-A-Watt EZ (paid link). My interest was in the peak reading so I could figure out which of my UPS would be able to take it. Happily my old Cyberpower 485VA (300 watt) unit had plenty of headroom for this.

Next up: Synology DS412+ and my new storage network

Tuesday, August 21, 2012

Software RAID with Seagate 3TB Disks and Windows 7

In an attempt to re-consolidate my data I've recently purchased four (4) 3000GB Seagate Barracuda drives. I use 3000GB rather carefully, that's what's written on the ST3000DM001 label. On Amazon it would say 3TB. If you're already aware of the decimal (used by marketing for Storage numbers) versus binary (what computers actually use) way of counting then you wouldn't be surprised that it formats to about 2794.39 GB for NTFS on every disk. These drives came preformatted in NTFS in the factory so it only takes a few seconds to initialize the drive in Windows as GPT then quick format with NTFS.

 Now why did I get these instead of Hitachi or Western Digital? They were the cheapest 3TB drives I could get ($149.91 at the time). These were 7200rpm. Plus they were $5 cheaper than even Western Digital Green drives. Also they were on the on Synology's compatibility list.

What about 7200rpm versus "Green" drives?
I have plenty of 5400/5900 rpm drives from several makes. These are decent enough in performance but the baseline is really the 7200rpm drive. In terms of power consumption, the slower drives don't appear to make a big impact in the numbers even I deal with. The Kill-a-Watt is reading 183-210 Watts used for my desktop plus four Barracudas running. With just the drives powered up these were showing 45-60 Watts. The UPS they are hooked up to account for 15 of those Watts. As for heat, well I'll give you that, the 7200rpm does run a bit hotter. But I'm eventually going to run them with a fan in a NAS/RAID setup. Besides, the "Green" drives, particularly WD Caviar Green, have been notorious for some problems coming back from sleep that make them go missing in some RAID configurations. WD even has disclaimer that their desktop drives are not warranted for Business Critical RAID. Which is a change now that Western Digital actually sells "Red" drives which are NAS approved. The 7200rpm models equate to the "Black" ones. And they have RE models that are Enterprise-grade.

Western Digital Caviar Green: 5400 rpm 
Western Digital Red: 5400 rpm (?)
Seagate Barracuda LP: 5900 rpm
Seagate Barracuda: 7200rpm 
Western Digital Caviar: 7200 rpm 
Hitachi Deskstar: 7200 rpm  

Are you sure these drives are reliable?
No, every drive is a gamble. What matters is whether or not you got a set of good drives in the batch. Also did your seller's carrier keep from throwing the thing around. Every manufacturer from Hitachi to Western Digital has had their share of bad history. I still remember losing 3 IBM "Deathstars" (sold to Hitachi, recently became Western Digital) some years ago. I've also had a mix of success with Maxtor (now Seagate) and Samsung (also now Seagate). And I'm very familiar with the RMA process for Western Digital, which is pretty good in convenience and turn around. They're all the same to me. I just need to have good backups and make my drives as reliable as possible. So the first thing I look at after price is comments on reliability. These ST3000DM001 actually have a firmware update released. The drives came with CC4B and the latest out there is CC4H.  

Patching the firmware
How did I know these drives came with CC4B? I read the label on the drive and checked Google for hits on "<model> firmware". That lead me to all sorts of threads, including a Synology DS412+ user that recommended upgrading to this version. The patch process is available in two ways, Windows executable and CDROM ISO. Well, that's not entirely accurate. Both of them boot up a Linux image which detects your drive through the motherboard's SATA chipset then patches them if valid. 

I've discovered the following caveats in the process:

  1. You need an Intel based CPU. It says so in Seagate's website. Also a user that tried an AMD CPU said that it didn't work for him. 
  2. The operation runs once per boot, though you could do multiple drives if they were all present on boot 
  3. It appears to ignore my JMicron based PCI e-sata card, so had to open up my desktop case and swap some drives around 
  4. Your Windows Disk 0 should either be empty, unformatted or be Windows boot device. One of my drives to be patched superceded the boot drive and I had to unformat the 3TB before Windows firmware patch stopped complaining. 

So if you have an AMD CPU, you might have to borrow another machine. If you've got a laptop, just hope that you have an eSATA port hooked up to a chip that is recognized by Seagate's software. If you're on a Mac or Linux then you should look into booting from ISO. And if you're running Itanium, Power, SPARC or PA-RISC ... well don't look at me!

Software RAID-0
Just for the heck of it, I wanted to try setting up these four drives as a Stripped Volume (Windows RAID-0).   I had a pair of Thermaltake BlacX Duet (paid link) hooked up to two separate e-SATA ports on a JMicron-based PCI card.    The disks were initialized as GPT to get a size >2TB.

Why a Stripe and not a RAID-5?   In Windows 7 you will notice that option is greyed out.  The OS-based RAID-5 is only supported in Windows Server versions.   Of course you can have hardware RAID or BIOS/driver-based RAID.

RAID-0 is supposed to be the fastest RAID setup for both read and write.  There is no fault tolerance in this setup, which is actually what I want in this case.   If any one of the drives fail the entire array fails.   I'd like to know pretty quickly that the entire set is working.

Baseline Tests
I wanted to use standard and free tools to stress the disks.   So I used Robocopy and CrystalDiskMark.

Above results from four ST3000DM001 SATA-III in a software RAID-0

This would represent the initial speeds I could reasonably expect from this array. When I use it in a RAID-5 NAS later on, I'm expecting it would be slower. This also helps me determine if I need to bother with link aggregation and a managed switch later on. The theoretical maximum of a gigabit Ethernet is 125MB/s. It should be slower than this in real use, about 111 MB/s according to a test by Tom's Hardware. So do I need < =43MB/s read or < =17MB/s write more out of this array?

One ST3000DM001 SATA-III by itself.

Tom's Hardware gives it an average sequential speed of 119.8 MB/sStorage Review tests have it in the middle of the pack.

Compared to a ST31000340AS SATA-II I've been using as a Render target

For the Robocopy test, I used a USB-3 source drive that I knew I can hook up to all my devices for comparative tests later. 85.74 MB/sec is my baseline.

   Speed :            89907202 Bytes/sec.
   Speed :            5144.531 MegaBytes/min.

Robocopy write test

The Burn In
Before feeding these drives into a hardware RAID I wanted to make sure they didn't break within the first few days.  My choice of tool was HDDScan with CrystalMarkInfo.   HDDscan would be left running overnight while I look at Load/Unload Cycle Count and temperature.   I wanted to know how hot the drives can get in operation.   Load/Unload Cycle Count is how many times the drive had to come of park.   An large (more than double-digit) increase means the drives are too aggressive in going to sleep.  This would make them unsuitable for RAID application.   Most drives are rated for a few hundred thousand in their life and park/unpark would mean a lot of wear in the drive mechanism which would lead to earlier failure.   The point of RAID is extra reliability, if your base reliability is compromised it can only do so much.

I test each drive individually in a JBOD setup. 55-degrees Celsius was the hotest any single drive ever got.   The room had an electric fan in a room in the 27-degree Celsius (81 degrees Fahrenheit) range.

Next up: Mediasonic ProRaid and e-SATA without Port Multiplier