crush depth

Note: Prices aren't static, and everything written here will likely be invalid inside a year. The analysis process was fun, though, and would work regardless of how prices change.

I'm about to replace an existing backup storage server. The existing server has a simple ZFS array consisting of two mirrored 2TB disks, and was built in 2018. An analysis of my usage patterns shows that I produce around ~50GB of data per year that's significant enough to be included in backups. I only back up data that I've created and that can't be reproduced. I never back up entire operating systems and/or applications; that's wasteful and pointless - those things can easily be downloaded again later.

I generally plan backup systems around ten year forecasts, so I actually only need to expand by 500GB to provide space for the next ten years of backups. The trend line for data use seems to be getting steeper over time, however. Storage is cheap, so it seems to be a good idea to build a new system that's more than just 500GB larger. Disk prices and sizes in 2024 are drastically different to how they were in 2018 when the old system was built, so it likely makes little sense to simply double the storage capacity in the new system.

What's the most cost-effective amount of storage to buy, though?

Heterogeneity

When I buy multiple disks, I deliberately spread the purchases across as many manufacturers as possible. Why? Consider the case where you're building a RAIDZ pool. In a RAIDZ pool, a configurable number of disks are essentially configured as parity disks, and in a RAIDZN pool, N + 1 disks have to fail simultaneously in order for the pool to be lost. If failed disks are replaced before the limit is reached, no data loss occurs.

Let's assume you build a RAIDZ pool using three disks of the same model from the same manufacturer, and let's assume that those disks all come from the same manufacturing batch. Now let's assume that that particular batch from the manufacturer has some kind of contamination issue or manufacturing defect… You run the risk of that defect causing all three disks to fail at the same time.

By buying disks from different manufacturers (or perhaps even just different models of disks from the same manufacturer, if there are more disks being purchased than there are manufacturers), you effectively distribute the risk of failure. It's much less likely that two disks from completely different sources and manufacturing processes are going to fail at the same time.

Backblaze publish regular articles on disks failure rates in their arrays. The annualized failure rates for disks is still in the single-digits.

Pricing

Let's take a look at NAS hard disk prices at the time of writing from Scan. We'll analyze prices of NAS disks from each manufacturer and look at their cost per terabyte. We'll ignore any kind of "on sale" pricing, and we'll ignore open box and/or refurbished disks.

The point isn't necessarily to be able to say "this manufacturer has better pricing than this other manufacturer", but to see how individual manufacturers set prices within their own product ranges. I'm likely going to be buying from all of them regardless in order to ensure a heterogeneous pool, but there might be an obvious "best" way to spend money within a given manufacturer.

We're only interested in HDDs and not SSDs (we need large and cheap, not fast), and we'll ignore any disks smaller than 1TB. Additionally, if a manufacturer has multiple models with similar specs and the same size, we'll pick the cheapest purchase price (even if this is a higher price/TB); this is a backup system with redundant disks and no performance requirements so we don't care about fancy disk features.

Seagate

Data: seagate.csv

A chart of the pricing of Seagate disks is as follows (smaller is better):

The same chart without the purchase price, for ease of readability:

The cheapest disk is the 1TB Seagate IronWolf ST1000VN008 at £59.99. The most expensive disk is the 22TB Seagate ST22000NT001 IronWolf Pro at £439.99. However, looking at price per TB, it is most cost-effective to buy 12TB disks as this works out to around £19 per TB. Interestingly, at no point is it ever a good idea to buy a 3TB disk; the 4TB disk is both cheaper in price and cheaper in price per TB! If you don't need a 12TB disk, the 4TB disk is the most cost-effective option for small storage.

Western Digital

Data: western_digital.csv

A chart of the pricing of Western Digital disks is as follows (smaller is better):

The same chart without the purchase price, for ease of readability:

The cheapest disk is the 2TB WD Red Pro WD2002FFSX at £98.99. The most expensive disk is the 22TB WD Red PRO WD221KFGX at £575.99. However, looking at price per TB, it is most cost-effective to buy 6TB or 16TB disks as this works out to around £23 per TB. Interestingly, the 2TB and 4TB disks are priced identically, resulting in an effective 50% discount for buying the 4TB disks.

Toshiba

Data: toshiba.csv

A chart of the pricing of Toshiba disks is as follows (smaller is better):

The same chart without the purchase price, for ease of readability:

The cheapest disk is the 4TB Toshiba HDWG440UZSVA at £95.98. The most expensive disk is the 18TB Toshiba HDWG51JUZSVA at £304.98. The price per TB is straightforward for this manufacturer: Bigger is cheaper per TB, and there's far less variation in price/TB between disk sizes.

Synology

Data: synology.csv

A chart of the pricing of Synology disks is as follows (smaller is better):

The same chart without the purchase price, for ease of readability:

The cheapest disk is the 4TB Synology HAT3300-4T at £109.99. The most expensive disk is the 20TB Synology HAT5310-20T at £899.99. The price per TB is straightforward for this manufacturer: Bigger is cheaper per TB, and there's far less variation in price/TB between disk sizes. However, overall prices for these disks are far higher than for other manufacturers.

All

For fun, here are all manufacturers on a single graph. Not all manufacturers have disks for every possible size.

The cheapest possible price per TB is to buy a Toshiba 18TB disk. Synology disks are consistently the most expensive disks in both up-front cost and cost per TB. I don't know why anyone would buy a Synology disk: Are Synology NAS devices restricted from using anything other than Synology disks?

Buying anything smaller than a 4TB disk from any manufacturer is consistently a poor choice.

Bean Counting

For my requirements, eyeing the graphs, buying 6TB disks seems like an acceptable compromise across all manufacturers for price per TB vs total purchase price for all disks. Buying 8TB disks looks to be a close second, but Western Digital's odd spike in price per TB for disks at that size makes it seem like a less attractive option.

Let's see if using 6TB or 8TB disks would yield better economy.

For a single set of 6TB disks:

WD 6TB        139.99
Seagate 6TB   148.99
Toshiba 6TB   139.99
              ------
             £427.98  (£35.665 per TB of usable space [Price / 12TB])

Running in a RAIDZ1 configuration, this would yield 12TB of usable storage space (because, effectively, an entire disk's worth of storage is used for parity data). Two disks have to fail simultaneously for us to lose data, and we hedge against this by using disks from three different manufacturers.

For a set of 8TB disks:

WD 8TB        213.49
Seagate 8TB   169.99
Toshiba 8TB   159.99
              ------
             £543.47 (£33.966875 per TB of usable space [Price / 16TB])

Running in a RAIDZ1 configuration, this would yield 16TB of usable storage space.

Is the extra 4TB of storage space worth the extra cost? In other words, how does the extra 4TB of space compare in terms of cost per terabytes to our existing cost per terabytes?

£543.47 - £427.98 = £115.49
      16TB - 12TB = 4TB
    £115.49 / 4TB = £28.8725 per TB

It is fairly economical. If we buy the larger disks, we're effectively paying only ~£29 per terabyte of extra space.

What's the story if we buy six disks at 6TB or 8TB? After all, we would then be able to create a RAIDZ2 pool (meaning three disks would have to fail at the same time in order for the pool to be lost instead of only two). That would yield either 24TB or 32TB of usable space.

WD 6TB        139.99
WD 6TB        139.99
Seagate 6TB   148.99
Seagate 6TB   148.99
Toshiba 6TB   139.99
Toshiba 6TB   139.99
              ------
             £857.94  (£35.7475 per TB of usable space [Price / 24TB])

WD 8TB        213.49
WD 8TB        213.49
Seagate 8TB   169.99
Seagate 8TB   169.99
Toshiba 8TB   159.99
Toshiba 8TB   159.99
              ------
            £1086.94  (£33.966875 per TB of usable space [Price / 32TB])

Obviously, the cost per terabyte is the same as we're just multiplying prices by a constant factor.

If we assume that two disks from the same manufacturer have exactly equal likelihood of failure (which may or may not be true!), then intuitively the six disk array is safer as, even if an entire manufacturer batch catastrophically fails, we still have to lose another disk at the same time (which is guaranteed to be from a different manufacturer) before we lose data.

Is a price difference of £543.47 (or £427.98) worth the extra safety? For a backup system, I think it probably is, and the extra storage space is nice to have too.