SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits

SOL-ExecBench is a new benchmark for evaluating AI systems that optimize GPU kernels by measuring how close they get to the theoretical maximum performance (Speed-of-Light) that the hardware can achieve, rather than just comparing against other software implementations. It includes 235 real optimization problems from actual AI models and uses a special pipeline called SOLAR to calculate what perfect performance would look like for each kernel. The benchmark includes safeguards to prevent cheating and focuses on modern NVIDIA Blackwell GPUs with various precision formats.