KernelBench

What it is

KernelBench (Ouyang, Guo, et al., Scaling Intelligence Lab, Stanford) is a benchmark for LLM-generated GPU kernels: it tasks a model with transpiling PyTorch ops into CUDA and scores correctness and speedup. Repo: github.com/ScalingIntelligence/KernelBench. Actively maintained as of 2026.

Key contributions / resources

• Four difficulty levels — single ops → operator fusion → full model architectures → HuggingFace model optimization; a graded progression of kernel-writing tasks.
• fast_p metric — scores a generation as successful only if it is both correct and faster than a threshold relative to the PyTorch baseline, so it rewards real speedups rather than merely-correct kernels.
• Well-specified problems — although the benchmark targets LLMs, its precisely-specified PyTorch-op → CUDA tasks are an excellent source of kernel problems to attempt by hand for a human learner.

Associated work

• The roadmap lists KernelBench in the competitions/benchmarks ladder as a source of practice problems: 13-gpu-engineer-roadmap (§6).
• Complements gpu-mode and its reference-kernels/KernelBot leaderboard (adversarial, community-run) with a standardized, published benchmark.
• Problem levels map onto curriculum skills: single-op and fusion kernels (flash-attention, GEMM) in gpu-systems-for-llms.