Background

As a MacBook Pro user, I, like many others, suffered from the lack of internal storage in general. In particular, the rMBP that I uses daily has a merely 256 GB of SSD space, which is running short very quickly.

Ever since I started recording videos while flying, things has became even worse. With 50+ GB of videos after each flight, my poor 256 GB SSD has quickly ran out, forcing me to purchase a 1 TB external hard drive. However, after a year the external drive also started having difficulties keeping up with the amount of data that I generate, not even to mention the lack of redundancy and backup made it unsuited for putting anything you would like to keep in there.

So, at some point I decided to build a high-capacity NAS for myself and hope it will last at least a couple of years before needing another upgrade.

I wrote this post primarily as a note of what I have done in case I need to do it again. Hopefully it will be helpful for you if you are considering doing similar things.

Commercial system?

Alright, we knew what we want at this point, but what should we get for it?

First, I checked commercial solutions like Synology, which supposedly is the best consumer grade NAS system in the market. However, that come with a price. The cheapest 4-bay system will easily cost you $300+ without any hard drive included. Not even to mention the not so impressive hardware specs which makes me question it's real world performance.

That is the point when I thought: why not build a NAS server yourself?

Finding the server

First thing when building a NAS server is to find the right hardware. For this build, a used server should work perfectly fine as we do not need that much processing performance for a storage solution. But we do want to have a server that has lots of RAM, some SATA connector and good NICs. Since I will be running the server in a living environment, noise level would also be a concern.

First I tried to dig on eBay. Despite I found lots of used Dell PowerEdge R410/R210 at less than $100. Having worked in a server room before, I knew those 1U unit just makes too much noise for me to to use inside my home. In general, tower server tends to be a little quieter but unfortunately there are only a few listing of those on eBay and they are quite pricy or underpowered.

Next, I checked Craigslist, and found a gentleman selling a used HP ProLiant N40L at only $75! I knew those servers, even used, usually costs in the range of more than $300, so I emailed the seller to make sure the server is still available. Knowing it still is, I quickly drove to San Mateo and picked up the server. I was very happy when I first saw it. It has minimum wear and aside from being slightly dusty, everything seems perfectly fine.

Here are some photos I took when I first got the server:

Here is the spec of the server that I got:

CPU: AMD Turion(tm) II Neo N40L Dual-Core Processor (64-bit)

RAM: 8 GB non-ECC RAM (upgraded by previous owner)

Flash: 4 GB USB Drive

SATA Connectors: 4 + 1

NIC: 1 Gbps on-board NIC

Needless to say, despite being few years old, this server still have better spec most of the pre-built NAS system I can find on the market, especially when it comes RAM capacity. Later on I even [upgraded](#Hardware upgrade) the RAM to 16 GB ECC memory for more buffer space and better data protection.

Getting the drives

Now we have a perfectly capable system, we need to get some hard drives for use inside our home build NAS. Obviously the $75 I paid only came with the server but no hard drives (and I wasn't expecting it to).

After some quick research, I found that WD Red hard drives works best with NAS build when drives are constantly running 24x7. I picked up 4x3 TB WD Red drives from Amazon. In general you can get whatever drive you want but do make sure they are of similar capacity and speed in order to avoid weird RAID performance issue later.

Setting up the system

Now many people will use FreeNAS for their NAS build, and there is absolutely nothing wrong with using it. Despite my server being perfectly capable of running FreeNAS, I opted to use CentOS as ZFS on Linux is "production-ready" and I am more familiar with Linux server management. Plus I do not really need all the fancy UI and features FreeNAS provides, a simply RAIDZ array and AFP sharing should be sufficient for me.

Installing CentOS on the USB drive is fairly straightforward. Just make another drive as the boot device and the installation wizard will guides you from there.

Building the RAID

After I have successfully installed CentOS, I installed ZFS on Linux using steps from here.

Once that has been completed, I loaded the ZFS Kernel module:

$ sudo modprobe zfs

and create the RAIDZ1 array using the zpool command:

$ sudo zpool create data raidz1 ata-WDC_WD30EFRX-68AX9N0_WD-WMC1T0609145 ata-WDC_WD30EFRX-68AX9N0_WD-WMC1T0609146 ata-WDC_WD30EFRX-68AX9N0_WD-WMC1T0609147 ata-WDC_WD30EFRX-68AX9N0_WD-WMC1T0609148
$ sudo zpool add data log ata-SanDisk_Ultra_II_240GB_174204A06001-part5
$ sudo zpool add data cache ata-SanDisk_Ultra_II_240GB_174204A06001-part6

Notice that here I used the hard drive ID instead of their mapped names (sdx) to reduce the chance of a failed mount after boot due to the lettering of the disks changing.

I also added ZIL and L2ARC cache that is running on a seprate SSD. It is partitioned as 5GB for ZIL and the rest for L2ARC.

As of RAIDZ1, it can tolerates up to 1 disk failing. While lots of people argue you should not be using it due to the chance of a second disk failing during the RAID rebuild and potentially destroying all the data, it was chosen by me primarily because I already backs up all my important data to an offsite location regularly and failure of the entire array will only affect the availability, not the durability of my data. If you do not have good backup solutions, then using something like RAIDZ2 or RAID10 is generally a much better idea.

You can confirm your pool has been created successfully by running:

$ sudo zpool status

and

$ sudo zfs list
NAME                               USED  AVAIL  REFER  MOUNTPOINT
data                               510G  7.16T   140K  /mnt/data

By default, ZFS will mount your newly created pool directly under /, which is usually undesirable. You can change it by running:

zfs set mountpoint=/mnt/data data

From here, you can choose to create one or more datasets for storing data. I created two datasets, one for Time Machine backup and one for general file storage. I applied a quota limit of 512 GB for the Time Machine backup dataset to make sure it does not grow indefinitely.

Optimizations

zfs set compression=on data

This turns on compression support of ZFS. Compression uses minimum CPU but could dramatically improve your I/O throughput and is always recommended.

zfs set relatime=on data

This reduces number of updates to atime to reduce IOPS generated while accessing files.

By default, ZFS on Linux uses 50% of physical memory for ARC. For my use case (total number of file low), this can be safely increased to 90% as we do not run any other applications on the NAS server.

$ cat /etc/modprobe.d/zfs.conf 
options zfs zfs_arc_max=14378074112

and verified that it indeed took effect with arc_summary.py:

$ python arc_summary.py
...
ARC Size:				100.05%	11.55	GiB
	Target Size: (Adaptive)		100.00%	11.54	GiB
	Min Size (Hard Limit):		0.27%	32.00	MiB
	Max Size (High Water):		369:1	11.54	GiB
...

Setup recurring tasks

I have setted up recurring scrub systemd timers using systemd-zpool-scrub to run once a week and auto snapshot cron tasks using zfs-auto-snapshot to automatically create snapshots every 15 min, 1 hour and 1 day.

Installing Netatalk

Netatalk is an open source implementation of the AFP (Apple Filing Protocol). I followed the official installation guide for CentOS and got the RPM package built and installed in a couple of minutes.

Config tweaks

$ cat /etc/netatalk/afp.conf
[datong@Titan ~]$ cat /etc/netatalk/afp.conf 
;
; Netatalk 3.x configuration file
;

[Global]
; Global server settings
mimic model = TimeCapsule6,106

; [Homes]
; basedir regex = /home

; [My AFP Volume]
; path = /path/to/volume

; [My Time Machine Volume]
; path = /path/to/backup
; time machine = yes

[Datong's Files]
path = /mnt/data/datong
valid users = datong

[Datong's Time Machine Backups]
path = /mnt/data/datong_time_machine_backups
time machine = yes
valid users = datong

Notice that vol dbnest is a huge performance boost for me as by default Netatalk writes the CNID database to the root file system and it was really not desirable as my root system is running on a USB drive and is relatively slow. Turning on vol dbnest causes the database to be stored to the Volume root, which in this case is the ZFS pool that has relatively better performance.

Enable ports on Firewall

$ sudo firewall-cmd --permanent --zone=public --add-service=mdns
$ sudo firewall-cmd --permanent --zone=public --add-port=afpovertcp/tcp

If everything has been configured correctly, you should see your machine showing up in Finder and Time Machine should work as well.

Miscellaneous installations

S.M.A.R.T monitoring

It is a good idea to monitor the health status of your disks before failures occur.

$ sudo yum install smartmontools
$ sudo systemctl start smartd

UPS daemon

Monitors the charge of APC UPS and shuts down the system when the charge is getting critically low.

$ sudo yum install epel-release
$ sudo yum install apcupsd
$ sudo systemctl enable apcupsd

Hardware upgrade

A week after I setted up my system, I am growing more concerned about the non-ECC memory installed in my system (see Why I Chose Non-ECC RAM for my FreeNAS for a reason why you should not run ZFS with non-ECC RAM). Plus, more ram for buffering is always a good thing with ZFS. So I purchased 2xKingston DDR3 8 GB ECC RAM from Amazon for $80 each and replaced the desktop RAM installed by the previous owner. The system booted without any problem on the first try and I can confirm ECC support has indeed been enabled:

$ dmesg | grep ECC
[   10.492367] EDAC amd64: DRAM ECC enabled.

Result

The result has been amazing. I am able to constantly saturate the 1 Gbps LAN connection on the server when copying files and Time Machine works flawlessly. Overall, very happy with the setup.

Total cost

1 * HP ProLiant N40L = $75
2 * 8 GB ECC RAM = $174
4 * WD Red 3 TB HDD = $440
Total = $689

Now I call that a pretty good deal.

Updates

• 01/07/2018 - Updated zfs_arc_max size to use more of available system RAM.
• 01/07/2018 - Fixed error in command installing smartd.
• 01/07/2018 - Added ZIL and L2ARC cache configuration.