A High Performance Computing [HPC] cluster will often be deployed with an expected 5+ year service life. I have been in HPC long enough to see some of my earlier system's spiritual great-grand-children go into production. There are patterns of stages many systems go through which might not have been thought about early enough.
After deployment, the first stage is onboarding. At OCF, we often begin with a small initial phase of users, validating the cluster works as expected and teasing out any changes in workflows from things like newer software versions before making the system widely available.
Next comes the early production stage. If the previous HPC was indeed running for 5 years, then this is a joyous time! The new system will be more powerful with more resources. With such capacity many jobs will be allocated runtime near instantly and researchers will be able to run larger or more detailed simulations and ask deeper questions of their data. This is definitely an amazing time for an HPC user.
Skip ahead to the late production stage. Here, HPC solutions can become victims of their own success. More challenging workloads will doubtless have been developed over the years as well as new users discovering what HPC can do for them. The system becomes more contested and what was once a 1-hour job is now a 3-5 day turnaround for results.
Users have to start being more clever, splitting their jobs to fit around others to get earlier allocations instead of focusing on results. Behaviour changes to get more out of the HPC. Ideally, we'd always like our users to be optimising their research for greater efficiency but there is a risk of forcing everyone to be computer scientists rather than focusing on their own domain.
The stage I didn’t talk about was the mid-life and I think this deserves more attention. Changes we make in the mid-life stage can really take the sting out of the later stages.
If faced with growing workloads, we can and should plan to expand the cluster midway through its intended lifespan. This might well extend its lifespan past the original timeframe. Ideally this planning should start at the inception of the cluster and be part of the original design. This might begin simply by reserving network ports on core switching infrastructure to allow expansion with minimal disruption, but it should also extend to software planning. After a few years we commonly find a newer processor architecture becomes better value for money. So, a software stack with a workload manager like SLURM (which can manage heterogeneous clusters) should be a critical requirement.
If we accept that an expansion will make our cluster heterogeneous we can take that opportunity to tailor it to key workloads. "Right-sizing" a pool of new nodes to excel at particular tasks, migrating that workload to the newer nodes, subsequently enables other workloads to expand into the vacated space.
Monitoring and understanding your workload is central to this.
In my view, the earlier you plan for a mid-life upgrade, the more value you can extract from your HPC.