v1.2.5: Trivially Copyable

4 January 2025. If premature optimization is the root of all evil, we show a case where premature generalization is the root of inefficiency on the GPU.

While profiling Turbo, I identified cuda::std::tuple as a performance bottleneck. The inefficiency stemmed from the Interval class, which was originally implemented using a CartesianProduct abstraction to represent intervals as a pair of lower and upper bounds. The goal being to stay close to the mathematical definition of interval. For generality, CartesianProduct supports an arbitrary number of lattice types, storing them internally in a tuple. However, tuple is not trivially copyable. This meant that every time an interval was transferred from the CPU to GPU (and vice-versa) the constructor of each element was invoked, incurring a significant overhead.

To address this, I refactored Interval to directly store the lower and upper bounds as members, eliminating the use of CartesianProduct and tuple. As shown in the table below, this simple refactoring doubled the number of nodes explored per second, for the hybrid version with 128 blocks.

Interestingly, the full GPU architecture (-arch gpu) benefits less from this optimization.

It indicates that non-trivial copy is particularly inefficient when transferring data between CPU and GPU.

Helios (GH200 Superchip)

On Helios (GH200), however, the difference between these two versions becomes negligible. This is likely due to the unified memory architecture of SM90, which reduces the impact of memory transfer costs. The following table shows the difference with 264 blocks and hybrid architecture.

It is interesting to compare the H100 to my desktop GPU A5000. Since the A5000 has 64 SMs and H100 has 132 SMs, it should be at least twice as fast on H100. The normalized average shows indeed an increase of +125%, which is a bit more than 2 times faster, but also due to the fact that the GH200 is equipped with a 72-cores CPU, while my desktop has only a 10-cores CPU.