Storage is an integral part to any HPC solution. With vast experience in the most common HPC file systems - Spectrum Scale, Lustre and BeeGFS, together with a range of hardware platforms, OCF is able to offer solutions; whatever your capacity or performance requirements.
However large or small your cluster, you certainly don’t want it sitting idle waiting on IO. High parallelism is often critical in allowing many processes to write simultaneously. For that reason, OCF has expertise in a number of parallel file systems including Spectrum Scale, Lustre and BeeGFS.
Even within a discipline, IO characteristics can vary immensely. Sometimes tiering can help, or even a burst buffer, sometimes pure parallelism to hundreds of disk drives will be sufficient. Our in-depth knowledge of the strengths and weaknesses of different technologies and software can ensure your requirements can be met.
Since its introduction in 1998, IBM Spectrum Scale (formerly known as General Parallel File System or GPFS) has been field-proven on some of the world’s most powerful supercomputers, providing highly reliable and efficient use of infrastructure bandwidth.
Spectrum Scale is a distributed cluster file system in which many nodes can safely access many files concurrently. Since metadata management is delegated to Spectrum Scale, clients are able to access each file, and multiple systems can access a single file without this being a burden on the Spectrum Scale storage servers. Efficient byte-range locking is used to allow this extremely high level of concurrency.
Spectrum Scale splits files into blocks and distributes these blocks across many storage drives. This allows the performance of many drives to be yielded in parallel, potentially even for a single file to a single client.
Spectrum Scale has commercial support, enterprise features and a very strong user community.
Lustre is a well-established open-source, parallel file system used in HPC applications across the world, including a large percentage of world Top 500 installations.
OCF partner, DataDirect Networks (DDN) acquired Intel’s commercial Lustre File System business in 2018 and offers a commercially supported ExaScaler Lustre appliance utilising their high performance SFA storage systems.
OCF is also able to architect a Lustre solution utilising its range of hardware vendors to provide cost affective solutions based on you performance and capacity requirements.
Lustre is supported by the open-source community or, optionally, commercially supported through Whamcloud and DDN.
BeeGFS (formerly known as FhGFS) is an open-source, parallel file system gaining an increasing popularity in HPC due to its high performance and a distributed architecture that allows for file systems to be built from relatively inexpensive hardware.
BeeGFS also has a number of advanced features including BeeOND which is designed to integrate with HPC schedulers to create temporary parallel file system instances out of local SSDs and NVMe drives on compute nodes.
BeeGFS is supported by the open-source community or, optionally, commercially supported through OCF partner ThinkParQ.