One of the banes of modern technology is how to keep your digital information safe. While modern PC's are quite reliable, often you are only one hardware failure away from losing irreplaceable documents, photos and videos. Therefore it is critical that you have some sort of backup solution.
The solution will often depend on the size of the files. Documents can often be backed up to the cloud using things like box or google drive, but cloud storage has its own issues and I would nor rely on it. While for small amounts of data it can be cheap and convenient, even with super-fast broadband backing up even moderately large files can be tedious and soon you will find yourself paying yearly subscription fees as your storage requirements increase.
The alternative then is some sort of local backup. I take a lot of photographs, and recently I have been continually hitting the limits of my disk space, which has forced me to continually prune images. It seemed like a good time therefore to look at some sort of storage expansion.
My requirements we as follows
- Easily accessible - accessible through all our computers and via internet
- Data Redundancy - loss of one component would not result in loss of data
- Expandable -i.e. capable of adding extra storage when required
- Extendable - Add extra features if required
- Low Power - does not use much energy
- Cheap - not expensive :)
Over the year I had tried a number of backup solution. I started with recordable DVD's until I found the shelf life of DVD's was only 2 or 3 years and my data requirements quickly outstripped the amount data I could store. I then moved to flash disks, but that is quite an expensive per MB way of doing it. My current solution is to back up to standalone hard disks. While this is relativity cost effective, they are not very accessible and for that reason backup rarely occur more than evey 6 months, meaning that I am living on a tightrope for the rest
The obvious solution is a NAS box (or network addressed storage). This is a box that can be connected to a Ethernet cable that allows you to transfer data as if it is a hard disk. NAS boxes allow you to have 2, 4 or even more disks in a redundant configuration.
You can get a two bay NAS box for about £130. However you then need to populate it with hard disks, which means, say a 4GB 2 bay NAS box would be about £330. Not terrible, but I felt I could do better
Build your own
The alternative is to build your own. First possibility is to re-purpose some old PC. Even a relative low power PC with two SATA interfaces will make a perfectly good NAS box. However it will be quite power hungry and noisy
The other option is to use something like the Raspberry Pi. These arevery low power, virtually silent and have good software support. However until recently, the Pi was not a good NAS server device. Firstly Pi's do not have native disk drive interfaces requiring you to connect via USB. However Pi's only supported USB2, which was relatively slow. Worse, it shard the USB bus with the Ethernet, meaning that if you are sending data and writing to disk via USB, it would cause blockages.
However recently the Pi 4 came out. This has a separate Ethernet/USB channel. Even better it supports USB 3. the difference is huge with a 10 fold increase in Ethernet throughput and a 4 fold increase in USB throughput. The Pi 4 now has a huge potential as a NAS server
A 1 Gb Pi 4 is about £30, so is good value. All we need to do is work out how to add some disk drives and add some software
Adding the disk drives
Without a native SATA interface, you are reliant on USB to connect to a drive. One solution is a USB to SATA interface. One of problems with this however is that you will need a separate power supply to power the disk. I wanted a few power supplies as possible, so I wanted to stay away from this.
The alternative is a hard disk in a USB enclosure. A 4TB USB 3.0 portable disk can be powered via USB 3.0 and strangely the price is almost exactly the same as the native disk. One downside is that the disks are likely to be slightly slower, but for the work they were to do, this was not likely to be a major issue.
Another issue is whether a Raspberry Pi would have enough power to drive two USB 3.0 hard drives. It looked marginal, so i added a powered USB 3.0 hub. The hub I chose not only powered the dives but could also power the raspberry Pi and would allow me to add more drives if required in the future. The only downside was that it was a bit larger than I wanted.
So finally I had my bits together as shown below, saving about £70 over the NAS solution.
Parts list
Raspberry Pi 4 1GB - £35
Fan £10
Pi Case £5
SD Card £10
2 WD USB 3.0 4TB disks £85
Atolla USB 3.0 Hub Powered, 11-Port USB 3.0 Hub £30
Total price £260
(The case and fan may not be needed. The PI 4 runs hotter than the Pi 3, but in this configuration it is not doing much. The fan was just a precaution, while the case was just to tidy things up. )
Getting it working
The next issue was getting it working. Here I had two solutions. Firstly set up the OS myself or install a pre-built package. I just was not in the mood to do the first, so it looked what free packages were available. It basically came down to either FreeNAS or Open Media Vault(OMV). Both would of been fine, but OMV seemed a little bit more Pi friendly and had lower hardware requirements.
The first thing was to set it up. OMV does not have a complete raspberry Pi distribution. Instead you install the Pi basic distribution and install OMV on top of it. The complete instructions are here
Installation went smoothly,, so the we moved onto the OMV setup guide.
First decision was how to format my drives. I they were to be windows drives we would use NTFS. However linux NTFS support is not as efficient as windows, and because the drives would never be attached directly to the PC, it made sense to use a native Linux format. therefore we formatted them as EXT4 drives.
It was then we hit an issue.
RAID 0 makes multiple disks look like one large disk.
RAID 1 combines 2 disks in such a way that the data written is duplicated or mirrored across both disks
However deep in the setup guide we get this.
RAID+USB= Potential Problems
Setting up RAID of any type using “USB to drive” connections is discouraged.
RAID over USB has known problems. The USB interface (there are several flavors) may filter some the characteristics of the drives, fail to pass SMART stat's and ATA drive commands, delay the assembly of a RAID array, etc. While USB may work in some RAID cases, it's not as reliable as using a standard hard drive interface. If RAID of any type is considered to be a requirement, drives should be connected with SATA or SAS ports. If users choose to use RAID over USB connections, it is done at their own risk with the potential for the total loss of stored data. RAID issues involving ARM boards, USB connected hard drives, or USB RAID enclosures are not supported on the forum.
The first thing was to set it up. OMV does not have a complete raspberry Pi distribution. Instead you install the Pi basic distribution and install OMV on top of it. The complete instructions are here
Installation went smoothly,, so the we moved onto the OMV setup guide.
First decision was how to format my drives. I they were to be windows drives we would use NTFS. However linux NTFS support is not as efficient as windows, and because the drives would never be attached directly to the PC, it made sense to use a native Linux format. therefore we formatted them as EXT4 drives.
It was then we hit an issue.
RAID
The original idea was to have my two disks in a RAID configuration. So what is RAID. RAID allows you to group multiple disks in such a way that they look like one logical unit. There are about 6 different ways you can do this, but in practice most people choose 1 of two.RAID 0 makes multiple disks look like one large disk.
RAID 1 combines 2 disks in such a way that the data written is duplicated or mirrored across both disks
However deep in the setup guide we get this.
RAID+USB= Potential Problems
Setting up RAID of any type using “USB to drive” connections is discouraged.
RAID over USB has known problems. The USB interface (there are several flavors) may filter some the characteristics of the drives, fail to pass SMART stat's and ATA drive commands, delay the assembly of a RAID array, etc. While USB may work in some RAID cases, it's not as reliable as using a standard hard drive interface. If RAID of any type is considered to be a requirement, drives should be connected with SATA or SAS ports. If users choose to use RAID over USB connections, it is done at their own risk with the potential for the total loss of stored data. RAID issues involving ARM boards, USB connected hard drives, or USB RAID enclosures are not supported on the forum.
Basically raid over USB was not fully supported. I think if I was running OMV this is information that I would put front and center and not hide it mid way through a setup document. Oh, well time for plan B
In these situations it is worth going back to first principles and see if the path you are on is where you want to be.
Clearly I wanted redundancy, but did i need RAID to do that? I chose RAID because that is what most enterprise systems use. However they have multiple users writing loads of files to disk frequently. My box on the other hand will have a few users writing infrequently to disk. In that situation RAID is not so useful and actually adds a unnecessary overhead.
So whats the alternative. Basically I want to copy files changed on one disk to another one. Linux provides a protocol to do that called RSYNC. It basically checks to see if any files have changed and copies the files to another destination. Fortunately RSYNC is supported by OMV, so I can ask the Pi to check the main disk every set period and copy changed files to the backup disk. Here is information on how to do that
This would not be as immediate as RAID and there is small window where changes may not be copied, but the likelihood of a disk failing during a RSYNC process is very small, and far lower than an issue with RAID affecting both disks.
So with RSYNC setup, I could attach to my PI via a Samba network connection. On my windows PC it looks like a new drive and i can just drop files onto it. Every 12 hours, changes to the main disk are copied to the backup disk.
So after a test and 1TB of data transfer later I have working NAS server!
So did I meet my requirements?
In terms of effort it was relatively easy, however there were some cons.
There are also other things I want to do. First I have not tried wireless transfer yet. The ability to add or retrieve via wireless devices would be useful. Secondly I would like to allow internet access. This will be a little more tricky, but the ability to access photos from anywhere would be great. There maybe other things to add in the future as I explore OMV more
Finally, at the moment it consists of 4 devices connected via Velcro. I need a better housing. There really is a market out there for a pi case with two disk slots, because I could not find one. While 3D printing could be an option, at the moment I am considering a Tupperware container, wooden box or my old standby , a cardboard box.
Clearly I wanted redundancy, but did i need RAID to do that? I chose RAID because that is what most enterprise systems use. However they have multiple users writing loads of files to disk frequently. My box on the other hand will have a few users writing infrequently to disk. In that situation RAID is not so useful and actually adds a unnecessary overhead.
Solution
So whats the alternative. Basically I want to copy files changed on one disk to another one. Linux provides a protocol to do that called RSYNC. It basically checks to see if any files have changed and copies the files to another destination. Fortunately RSYNC is supported by OMV, so I can ask the Pi to check the main disk every set period and copy changed files to the backup disk. Here is information on how to do that
This would not be as immediate as RAID and there is small window where changes may not be copied, but the likelihood of a disk failing during a RSYNC process is very small, and far lower than an issue with RAID affecting both disks.
So with RSYNC setup, I could attach to my PI via a Samba network connection. On my windows PC it looks like a new drive and i can just drop files onto it. Every 12 hours, changes to the main disk are copied to the backup disk.
So after a test and 1TB of data transfer later I have working NAS server!
So did I meet my requirements?
- Easily accessible -SMB shares mean all computers on the local LAN can access it. Internet access is still to be done.
- Data Redundancy - 2 hard drives should provide considerable redundancy
- Expandable - USB hub allows extra drives to be added
- Extendable - OMV plugins provide extra feature support
- Low Power - PI is very low energy
- Cheap - I think it is about as cheap as I can make it.
In terms of effort it was relatively easy, however there were some cons.
- Lack of decent housing - there just does not seem to be any 2 slot boxes out there of decent size of a Pi. Weird
- Cost difference not that great - I was hoping for a 50% savings, but in fact the hard drives were the biggest cost. Considering there was no hot swap drives or metal cases involved the cost saving were more modest than I hoped
- No RAID support - if I knew USB RAID was not supported when I started then I might not of gone down this route. However in hindsight RAID is not as important as I thought
On the plus side
- It was cheaper
- It is more extendable
- It was more fun
Work to done
Firstly was it worth it? Some may consider thee £70 saving over a regular NAS to be relatively modest, however that ignores that I can easily extend it by adding more or larger disks.There are also other things I want to do. First I have not tried wireless transfer yet. The ability to add or retrieve via wireless devices would be useful. Secondly I would like to allow internet access. This will be a little more tricky, but the ability to access photos from anywhere would be great. There maybe other things to add in the future as I explore OMV more
Finally, at the moment it consists of 4 devices connected via Velcro. I need a better housing. There really is a market out there for a pi case with two disk slots, because I could not find one. While 3D printing could be an option, at the moment I am considering a Tupperware container, wooden box or my old standby , a cardboard box.
