Interconnect technologies for HPC
The network between your nodes is one of the easiest places to overspend. Here is how the two options differ, and how to tell which one your workload needs.
InfiniBand and Ethernet are the two most common high-performance solutions used to connect nodes in a High-Performance Computing (HPC) cluster.
Of the two, InfiniBand offers the lowest latency and is the most popular choice for tightly coupled workloads that run across many nodes at once. Ethernet is generally cheaper, has a broader skill-base and may well offer sufficient performance for your workloads – particularly if a significant percentage of those workloads are run on a single node.
It is worth noting that some form of ethernet network will be required for installation, management, etc of your cluster even if you choose InfiniBand for your high-performance network … so the decision isn’t really Ethernet of InfiniBand it’s if you need Ethernet and InfiniBand.
The right choice will depend on the workloads/jobs that are run across your cluster, not on which networking solution is faster on paper.
A high-speed, low-latency fabric earns its value when jobs are tightly coupled, meaning they run across many nodes at once and those nodes must talk to each other constantly. Message Passing Interface (MPI) workloads of that kind are where InfiniBand pays for itself.
If your jobs mostly run inside a single node, the nodes barely talk to each other, and the expensive fabric connecting them sits mostly unused. That is money that could have gone into compute instead.
High Performance Computing (HPC) is defined as tightly coupled workloads running in parallel over many cores and multiple nodes. Conversely, High Throughput Computing defines loosely coupled workloads being executed independently at scale.
The truth of the matter though, is that many HPC Clusters are running a mixture of HPC and HTC workloads. In a world where a single dual socket compute node can have 100’s of physical compute cores, a workload that may have been run over 10’s of compute nodes a decade ago can comfortably fit within a single compute node today. Building an HPC system is always a balancing act – trying to get the best scientific throughput per £ for the vast majority of your institution’s workloads, most of the time.
| INFINIBAND | ETHERNET | |
| Latency | Lower, built for HPC | Higher, closing the gap with RoCE |
| Cost | Higher | Lower |
| Skills in the market | Specialist | Common |
| Remote Direct Memory Access (RDMA) | Native | Available through RoCE |
| Best fit | Jobs spanning many nodes tightly coupled MPI |
Jobs mostly within a node, high throughput, budget-led |
In practice: We have had instances where a customer specified the latest highest-speed InfiniBand across an entire cluster. When working on the solution design process we reviewed their cluster utilisation using historical scheduler data. It turned out that more than 90 per cent of their jobs ran inside a single node, which meant the high-speed fabric connecting the nodes would not have been the best way for this customer to invest their budget. It was a large sum spent on a network their workload did not need.
Instead, we configured the cluster with high-performance ethernet across the entire cluster adding InfiniBand on a sub-set of their machines and set up multiple queues for users requiring that resource so they could access it, whilst being able to afford a few more compute nodes within the same budget.
The performance was there for the jobs they run, and the money saved on the fabric bought more compute nodes, which is what they were short of in the first place.
What is the difference between InfiniBand and Ethernet?
Both connect the nodes of an HPC cluster. InfiniBand offers lower latency and native remote direct memory access, and suits jobs that run across many nodes at once. Ethernet is cheaper, more familiar and often enough for jobs that run mostly within a single node.
Do I need InfiniBand for HPC?
Not always. InfiniBand is worth the cost when your jobs are tightly coupled and span many nodes. If most of your jobs run inside a single node, the fabric is largely unused, and the budget is better spent on compute. The way to know is to look at how your jobs run.
Is Ethernet fast enough for HPC?
For many workloads, yes. Ethernet has closed much of the latency gap with RoCE, and for high-throughput jobs or jobs that stay within a node it is often the sensible choice. For tightly coupled work across many nodes, InfiniBand still has advantages.
When is InfiniBand worth the cost?
When a large percentage of your jobs run across many nodes and those nodes communicate constantly, as tightly coupled MPI workloads do. The lower latency then translates into real-time saved. If that does not describe your workload, the spend is hard to justify.
What is RDMA?
Remote Direct Memory Access lets one node read from or write to another node's memory without involving the processor at each end, which lowers latency. InfiniBand supports it natively. Ethernet supports it through RoCE, RDMA over Converged Ethernet.
Are there other alternatives?
Yes, whilst InfiniBand is generally considered the market leader for High Bandwidth low latency fabrics today a number of alternatives exist:
Before you sign off a fabric, it is worth a look at how your jobs run. We can tell you quickly whether InfiniBand earns its place or whether the budget is better spent elsewhere. Simply click on the "Request a call back" button (on the top right of your screen) to request a call.
RELATED PAGES: