All flash has been common for structured block storage for a while now. And 120TB QLC drives promised to drive the cost per TB down for other workloads. So it is easy to understand why all flash became the defacto expectation for many sprawling unstructured file stores.
Yet in HPC, research computing and some other verticals we continued to deploy hybrid tiered solutions, using both flash and the spinning magnetic rust hard drives.
Then of course the AI infrastructure demand gobbled up most of the world's flash supply. This makes it a great time to think why we continued to use disk storage, and if it can be useful to you.
Stands for quad level cache, giving higher density by cramming four bits per cell. They don't have the same write performance or endurance as TCL flash, but under the covers they still use wear-levelling to load-balance writes over all cells and with capacities currently up to 120TB per drive there's a lot of cells to balance over.
Hard disk drives don’t have write endurances to worry about. You can generally keep writing over and over the spinning magnetic discs until the motors give out. Worth noting the physical spinning disks and movement of the actuator arms themselves impose a mechanical limit on how much can be written so you wont be able to do one full-drive write to a 20TB hard disk drive in a single day even if you wanted too.
Tiering sounds simple in the sales pitch. Two tiers of storage. The files you use a lot go on the fast storage, the rest on the slow storage. A place for everything, and everything in it's place.
But how do you know what files to put on the fast storage?
The ones you will use, obviously. But how do you know what you will use? We only know what we've used right? And at some point we'll stop using those too which leaves us always reactively chasing a chicken & egg scenario.
If you have a petabyte of file storage, made up of say half a billion files you have an average of 2MB per file. And yes if most your files are around that average all flash is the rights solution. Afterall how do you know what files are going to be used a lot?
But in research computing the distribution can be very different. If you have half a billion files then 499,900,000 of them could very realistically be under 20KB each. Many less than 4KB. Most your files could account for very little of your actual capacity.
Far from statistical "normal", our file size distributions may look something like this:
A fair number of zero-byte files, a huge peak in "tiny" low-KB, even a good number of multi-GB. The huge number of tiny files add up to very little capacity compared to the smaller number of large files.
With this distribution of files the question stops being which file am I going to need fast access to. The question is placing files on the cheapest tier that doesn’t significantly detriment its workload.
Small files are still going to benefit from being on flash. If for example a HPC job is going to be reading numerous these then minimising the file read latency is critical to reducing the tasks runtime.
However, sequential read/write performance to larger files can be as fast from disk as from flash. Especially if we're using a parallel filesystem splitting the file and harnessing the performance of many drives in parallel. HPC jobs reading large files are going to be less sensitive to a file open latency as it's such a small portion of the jobs runtime.
Hard disk drives are still a lot cheaper than flash and when that fact changes I'll change my opinion but hybrid flash / disk remains a valid solution for those with multiple petabytes of data to manage. Who knows maybe the spinning rust will out-live flash in this specific space. Maybe we'll be storing data in genetic matter or quantum memory before disk is finally dead.