Build your own NAS thread! ZFS! *HOWTO*

[magical]

INTRODUCTION:

After seeing what others are doing here, and not liking what is available in the consumer world, I decided to start a thread and show step by step how I build nas's for my clients. I am going to provide a video and pictures of my setup so people can understand, plus links to important sites. I also don't think people should be paying for the NAS software solution they choose, and am doing this to give people a Alternative.

I will start by going into exactly the builds I do. Firstly, since these nas's are usually holding critical data, I find it the up most importance to use the highest quality Parts for the build. Therefore, the cost of these machines isn't necessary cheaper than say buying a hardware raid/nas storage solution, but far more robust and reliable. Also since I am using standard parts, in case of a failure it should be easy to maintain and not hard to find replacements.

Lets start by why I like ZFS: ZFS is the Zetta File System, developed by Sun Microsystems. It can store up to 256 quadrillion zettabytes, I'm sure in our lifetime we probably wont exceed. It is a scaleable, reliable, and robust filesystem. It is designed from the ground up with a focus on data integrity. If used with the appropriate hardware it can protect against Silent Data Corruption, bit rot, current spikes, cosmic radiation, some bugs in disk firmware, power supplies, and other forms of corruption not usually detected by raid controllers, and or filesystems. It also has nifty features like the ability to grow your filesystem to larger capacity media, storage pools, snapshots, clones, deduplication, encryption,

Most people think ZFS and automatically think they have to use SOLARIS. But it has been ported to many different *NIX operating systems and therefore is available currently in FreeBSD, OpenSolaris, OpenIndiana, FreeNAS (freebsd kernel). I will be showing a example of Solaris and providing links to the other Options.

BASIC 10 DRIVE BUILD: (I will provide pictures of this *EXACT* Build and Step-By-Step Instructions)

The Main reasons I hate the consumer based nas solutions is:

1) SPEED. These things are SLOW!! Drives can easily sustain Gigabit speeds, when multiple drives are in a array, I expect FULL SPEED! I expect at least 75MB/Sec transfers.

2) I hate proprietary computers, if a part fails and it usually does due to cheapness (of the manufacturer), I want to replace it with hopefully something better (in the case of a proprietary NAS this usually isn't done) and something not that expensive, again, If its a commercial NAS I am usually stuck with the crap they give me.

3) Lack of fans or proper cooling, they usually are small boxes, with very limited cooling, and I don't like this. Hard Drives get hot, the more drives you get the hotter they run. Fans are a Necessity some noise is going to happen.

4) Amount of Drives it holds. Standard decent unit seems to be 4, but you pay a premium for more. 4 Drives in a RAID array regardless isn't' a sweet spot for me. It's when you have 6-10 drives.

5) Price. They usually charge a lot more than what I want to pay, especially to be locked-into hardware. This option I have for a 10 drive isn't cheap but it is also 10 drives, further on when I have time I will go into a much smaller and cheaper 6 drive solution.

HARDWARE CONSIDERATIONS:

I have spent a lot of time researching hardware for my machines, therefore I have specific intentions in mind, but I always like suggestions, please let me know if you have any input in what I am using and any thoughts on upgrades/etc.
This hardware is a SERVER GRADE machine. You put this in a corner, and forget about it, unless flooded or something major goes you are pretty much set (we will talk about ups and other solutions later). It could easily be expanded with more controllers for more drives, case upgraded to something hot-swappable.

I have chosen the integrated 8 port LSI controller this will give the machine a total of fourteen drives. Since we aren't using a expander (hot swap), having the bare SATA ports works better for us. The drawback being there are going to be a lot of cables in this machine, not really able to avoid this, but my cabling will be as clean as I can get, something I really like the 300 for.

Why did I choose the Tyan over others. Really I found only two brands to compete. The Intel's are overpriced and do not have the features I want or this is what I found. The Supermicro looks great but again, doesn't offer something with the integrated LSI controller without going to the older 1156 and spending a lot more. ASUS has a board but has no IPMI (well not necessarily true, they do but want to charge you more for it) . I see no reason not to have a headless server these days, plus the virtual filesystem, temperature monitoring.

Also I want a board with dual Intel gigabit and the 82574L works perfect. Quad Ethernet would be nice but they don't offer this with the SAS controller!

I started with this and I realize this hardware is around $1100 without the drives, but this is what is is. 10 drive capable NAS with future expansion to well over twenty drives.

MOTHERBOARD: TYAN S5512 (S5512WGM2NR) (Features: IPMI, Dual Intel 82574L gigabit ethernet, LSI SAS2008 8 port controller)
RAM: 2X Kingston ValueRAM KVR1333D3E9S/4G 4GB 1X4GB DDR3-1333 ECC 240PIN CL9 W/ Thermal Sensor Memory (Features: ECC RAM, we need this for proper error correcting and for ZFS)
CASE: Antec 300 Case (Features: Plenty of Airflow, robust, low cost, easy to work with, will also look into rackmount solutions)
POWER SUPPLY: Corsair VX550W (Features: High quality Japanese caps, reliable, good warranty)
EXTRA FANS: 4X Blade Master 120MM (Features: Low cost high CFM fans, the antec ones are crap, replace them all with these)
EXTRA FANS: 1X Noctua NF-P14FLX 140MM (Features: High Quality, low noise, replacement for the crap antec)
BAY CONVERTER: Coolermaster STB-3T4-E3 3X5.25IN (Features: Needed to get 10 drives into the 300 case)
SLOT ADAPTER: Startech S25SLOTR (Features: allows me to mount SSD inside case without taking up slot, for BOOT os)
SSD: Intel 40GB G3 SSD (Features: great speed, reliability, for booting os, and os storage)
CPU: Intel XEON E3 1230 (Features: 8 way CPU you can go cheaper tho a i3 will work)
HEATSINK: Coolermaster Hyper 212EVO (Features: the stock Intel cooler is crap)


Full Size Non-Edited Image

ASSEMBLY REQUIREMENTS:

These tools will be necessary to complete the proper assembly of the NAS:

1) Grounding Strap -- I cannot stress how much you need this, yes you can touch something grounded but, why cheap out, just get used to using one you'll save yourself a lot of headaches or possible problems.
2) Phillips #1 Screwdriver -- Anything Really just make sure the end isn't real worn out.
3) USED/OLD Credit Card/Bank Card and/OR a Mint Condition business card of someone you hate (for spreading the goop for the processor heatsink)
4) a Sledgehammer (for when things really go wrong)
5) a Joint, a Beer, or some PIZZA. Whatever is your liking. We need these things when building.

ASSEMBLY:

Step 1 -- Mainboard, Ram, CPU, and Heat-sink)

I will start by unboxing the mainboard, heat-sink, ram and CPU. I will not be using the heatsink that was provided by INTEL for my processor because, I find that they do supposedly provide adequate cooling, for the price of a better solution it is well justified, that is why I use the aftermarket hyper 212. A note, the hyper 212 will not fit in all case scenarios, because of its height, it will not work in a Rack-mount 4U design so be careful if you decide to change up some of the parts. I have taken these factors into consideration.

I now lay the motherboard, on top of its static bag it came on, this is usually a great surface to work on. I am going to start with the heat-sink first. My first step is going to be adding thermal paste to the CPU. I like to use a credit card or a clean business card (with a nice straight edge) to do this. You can use a higher quality thermal paste from another manufacturer rather than using the one that cooler master supplies but I think in this case it should be fine so I am going to use the included paste. About a grain and a half of rice equivalent on top of the CPU is all that is necessary, you want to smooth out the goop so that it is a very thin layer, maybe about 1/3rd of a mm thick. You don't want to see the CPU through the paste but you also don't want a saucy mess.

Once the thermal paste is applied to the CPU, we can work on securing the heat-sink mounting bracket..................................


to be continued.................. WORK IN PROGRESS.. SORRY

version 0.33b
by MagicaL

[Jin-n-Juice]

Engadget posted a guide last week: http://www.engadget.com/2012/02/01/h...using-freenas/

[mulambo187]

Thank you for this post, I am using a dns-320 with 2x2TB drives in jbod, its alright, would like to get something a little heavier and robust one of these days, maybe when hdd prices fall back down, I paid 69.99 each for my drives..... I really like Debian, would personally want design my NAS to run off that OS. Looking forward to your updates!

[magical]

Engadget posted a guide last week: http://www.engadget.com/2012/02/01/h...using-freenas/

Freenas is GREAT! I am going to go with a bit more advanced setup with Solaris 11 and napp-it script. but I might do a part on freenas because it is so sweet, only reason I am doin the solaris build is because you can do extras with a real working OS opposed to just having a pre-built machine.

I am also going to delve into some basic configuration for:

napp-it
iSCSI setup on solaris
and when I get my Mellanox Infiniband Cards, IPoIB, for a cheap 10 gigabit solution. (Around $100 point to point from SAN -> workstation!)

I just want to help, and get more ideas!

[JAC]

A Xeon? Good lord, man, how much data are you going to be serving up?

[Tijuana]

unRaid..

/thread

[magical]

unRaid..

/thread

I don't want to pay for the licensing cost. I don't like the fact I need a cache drive to get speed and I don't really like anything not using ZFS!!

I AM AWARE OF THE OTHER PRODUCTS/SOLTIONS out there and love suggestions, I have tried pretty much all the solutions and this is the one I chose to support!

[magical]

A Xeon? Good lord, man, how much data are you going to be serving up?

The reason for it. yes you can use a i3, we are talking very limited differences. Plus. further on in my build, I am going to work on how to set this up in conjunction with ESXi and pass through the drives to the HOST OS. You do not get vt-d with the i3... although I have a i3 in my own personal build. so I won't leave it out. Also at the point you are spending xxxx amount on server grade eq, why would you want to cheap out on the processor, but you can get a slower xeon if you so desire. The other reason for no i3, is some reports say it supports ECC while others say it doesn't.

Also this first build is a *EXAMPLE* build of something I am already building for a client, I plan on helping people and furthering this thread by offering and showing some much cheaper solutions, including micro atom builds using supermicros atom server boards and some nice small nas cases. But this way I can actually SHOW and build the machine from start to finish with this EQ!

[magical]

Engadget posted a guide last week: http://www.engadget.com/2012/02/01/h...using-freenas/

I have used freenas. It is a very well put together package. I am probably going to expand this thread with a low-cost freenas setup.

[GroceryBagHead]

unRaid..

/thread

10 years ago I'd disagree with you. I'd rather tinker with things for weeks to get it working to save 100 bucks.

But now I'm running unraid for few years now and couldn't be happier.

[Beradon]

What do you guys think of Flexraid for those who want a NAS under a windows environment?

[magical]

10 years ago I'd disagree with you. I'd rather tinker with things for weeks to get it working to save 100 bucks.

But now I'm running unraid for few years now and couldn't be happier.

You know what, I am open to suggestions, but as is I am only doing this as a "ALTERNATIVE" not dissing unraid.

Has nothing to do with the monetary price, and even unraid I think is free under a certain capacity.

NEXT. this is no tinker job. I have built over 5 working systems (SAN SYSTEMS, built Hundreds of Workstations), plus administered many other systems running ZFS, for years now. I am regularly building storage/SAN systems and thought people might want to see a ALTERNATIVE to what is available.

Please save your comments until you see my completed howto, benchmarks, and troubleshooting, as I will need input.

[magical]

What do you guys think of Flexraid for those who want a NAS under a windows environment?

Not sure, never used it, but my system will be Mac/Windows/Unix compatable.

[Luckyinfil]

SCST is pretty good

[takt]

How much did you spend for that build? What case/enclosure are you using to house the server?