Gunjan Dhanuka — Learning Notes

Learn: The GPU Engineer's Roadmap

Properties1
tagsgpu, careers, kernels, distributed-training, inference, interviews, roadmap, capstone

23 min read

Learn: The GPU Engineer’s Roadmap

What you're getting

This is the capstone of gpu-systems-for-llms: a concrete, honest plan for turning the twelve preceding lessons into a hireable track record for a GPU researcher/engineer role at a frontier LLM lab. Not “learn CUDA and apply” — a skill tree, a portfolio, a practice ladder, an interview map, and a phased study plan. By the end you should know exactly what to build, what to read, what interviews test, and roughly how long the path takes.

1. Learning map

graph TD
    A["The 3 role flavors<br/>kernel / infra / inference"] --> B["Map skills → this curriculum<br/>(what mastery looks like)"]
    B --> C["Portfolio projects<br/>that prove skill"]
    C --> D["Practice ladder<br/>books, lectures, puzzles, comps"]
    D --> E["Interview prep<br/>what actually gets tested"]
    E --> F["Break in<br/>OSS, benchmarks, blog, track record"]
    F --> G["Phased study plan<br/>weeks → job-ready"]

    style A fill:#44a,color:#fff
    style C fill:#4a4,color:#fff
    style G fill:#4a4,color:#fff

2. Why this matters (and why it’s learnable)

GPU/ML-systems is one of the few frontier-lab specialties where you can manufacture credible proof of skill from a laptop plus a rented GPU. Unlike, say, pretraining research — which needs a cluster you don’t have — kernel and systems work is verifiable in public: a kernel is either correct and fast or it isn’t, and a profiler tells the truth. That makes the hiring signal unusually legible. A GEMM that hits 90% of cuBLAS, a Triton FlashAttention that matches a reference numerically and beats eager PyTorch, a merged PR in vLLM — these are objective artifacts a hiring manager can inspect in ten minutes.

The trap this lesson exists to prevent: tutorial hell

The single most common failure mode is consuming content — watching every GPU MODE lecture, reading PMPP cover to cover — without producing a single profiled kernel. Passive knowledge is nearly worthless as a hiring signal because everyone claims it and no one can verify it. The unit of progress in this field is a benchmarked artifact, not a finished chapter. If a week goes by where you didn’t run Nsight on something you wrote, you’re in the trap.

The second trap: a "fast" kernel you never measured

“It felt faster” is not a result. Every performance claim in your portfolio must come with: the GPU model, the problem sizes, the baseline you compared against (cuBLAS / eager PyTorch / a reference kernel), the metric (TFLOP/s, GB/s, or latency), and ideally a roofline placement. An unbenchmarked optimization is indistinguishable from a bug — see 03-performance-modeling-roofline.

3. The role, disambiguated

“GPU engineer” at a frontier lab is not one job. There are three overlapping flavors. Most postings lean toward one but expect fluency across the shared core.

graph TD
    CORE["<b>Shared core</b><br/>SIMT model · memory hierarchy · roofline<br/>profiling (Nsight) · PyTorch internals · Python+C++"]

    CORE --> K["<b>(a) Kernel / performance engineer</b><br/>write &amp; optimize CUDA/Triton kernels<br/>fusion, tensor cores, FP8, occupancy"]
    CORE --> I["<b>(b) Distributed-training / ML-infra</b><br/>DP · ZeRO/FSDP · TP/PP/EP · NCCL<br/>frameworks, MFU, cluster reliability"]
    CORE --> S["<b>(c) Inference / serving engineer</b><br/>latency &amp; throughput · paged KV cache<br/>continuous batching · speculative decoding"]

    style CORE fill:#44a,color:#fff
    style K fill:#4a4,color:#fff
    style I fill:#4a4,color:#fff
    style S fill:#4a4,color:#fff

(a) Kernel / performance engineer. Day-to-day: profile a model, find the op eating wall-clock, write or tune a CUDA/Triton kernel to beat it, verify numerics, land it. Deep on the memory hierarchy, tensor-core MMA, warp specialization, and precision (BF16/FP8). Reads SASS, lives in Nsight Compute. Curriculum backbone: lessons 01-gpu-architecture-and-simt–07-triton-and-modern-kernels.

(b) Distributed-training / ML-infra engineer. Day-to-day: make a training run scale from 8 to 8,000 GPUs at high MFU without diverging or stalling. Owns the parallelism strategy (data/tensor/pipeline/expert), the collective-communication layer, checkpointing, failure recovery, and the training framework itself (Megatron/TorchTitan/DeepSpeed-class). Curriculum backbone: 08-data-parallelism-and-collectives–11-sota-large-scale-training.

(c) Inference / serving engineer. Day-to-day: serve a model at target latency and cost — paged KV cache, continuous batching, quantization, speculative decoding, disaggregated prefill/decode. Optimizes the p50/p99 latency–throughput frontier under real traffic. Curriculum backbone: 06-memory-wall-and-flashattention and 12-llm-inference-systems.

The overlapping core is the real prerequisite

All three demand the same foundation: the SIMT execution model, the memory hierarchy, roofline reasoning, comfort in a profiler, and enough PyTorch-internals knowledge to know where a tensor actually lives and when a graph breaks. Nail the core (Tiers 1–2) and you can pivot between flavors; skip it and you’ll plateau in all three. When in doubt about which to target: kernel work has the most legible portfolio for breaking in cold, because a single fast kernel is self-contained proof.

4. The skill tree → this curriculum

Below, each competency maps to the lesson(s) that build it and a concrete “mastery” bar — the thing you can do, not recognize.

CompetencyLessonsMastery = you can…
SIMT / warps / occupancy01-gpu-architecture-and-simtexplain why a divergent branch halves throughput; reason about occupancy vs. register pressure without guessing
Memory hierarchy & coalescing02-gpu-memory-hierarchypredict whether a kernel is memory- or compute-bound before running it; spot uncoalesced access and bank conflicts in code
Roofline / arithmetic intensity / MFU03-performance-modeling-rooflineplace any kernel on a roofline from its FLOPs and bytes; compute MFU for a training run and say what’s leaving performance on the table
CUDA fundamentals04-first-cuda-kernelswrite a correct, grid-strided kernel from scratch; set up launch config sensibly
GEMM optimization ladder05-optimizing-gemmtake a naive matmul to ~90% of cuBLAS via coalescing → SMEM tiling → register blocking → vectorized loads, and explain each step’s speedup
Fusion & FlashAttention06-memory-wall-and-flashattentionderive online softmax; explain why FA is IO-bound-aware and reimplement its tiling
Triton & modern authoring07-triton-and-modern-kernelswrite an autotuned Triton kernel; know when torch.compile suffices and when to drop to CUDA
Data parallelism & collectives08-data-parallelism-and-collectivesexplain ring all-reduce bandwidth cost; overlap comms with compute; reason about NCCL/interconnect topology
Sharded DP (ZeRO/FSDP)09-zero-and-fsdpdo the 16-bytes/param memory accounting; pick ZeRO stage / FSDP config for a given model+cluster
Model parallelism (TP/PP/SP/EP)10-model-parallelismderive the pipeline bubble; reason about TP communication volume; compose 3D parallelism
SOTA training systems11-sota-large-scale-trainingreason about FP8, activation recompute, topology-aware config à la Llama 3 / DeepSeek-V3
Inference & serving12-llm-inference-systemsexplain paged attention + continuous batching; reason about the latency/throughput frontier and KV-cache economics

Two adjacent skills the table doesn't cover — pick up in passing

(1) RL-systems literacy. Post-training is increasingly RL-heavy, and RL infra (rollout generation, actor/learner splits, long-context serving inside the training loop) is a fast-growing niche that sits exactly on the training↔inference seam. If your lab does RLHF/RLVR, skim 01-mdps-and-the-rl-objective and the PPO/GRPO material so you can speak to why inference throughput is now a training bottleneck. (2) Numerics. BF16 vs FP8 vs FP4, loss scaling, and “why did my kernel match to 1e-2 but not 1e-5” come up constantly — treat numerical-tolerance reasoning as a first-class skill, not an afterthought.

5. Portfolio projects that prove skill

Ordered by difficulty. Do them in order — each assumes the previous. For every project the rule is the same: public repo + short writeup + benchmark table with hardware, sizes, baseline, and metric. A repo with a README that shows a roofline plot and a speedup table beats a repo with 10× the code and no numbers.

Package the portfolio as a story, not a pile

A single pinned GitHub profile with 4–6 of these, each with a results-first README, plus one blog post walking through your best result, is a complete cold-application package for a kernel/infra role. Recruiters and hiring managers pattern-match on exactly this.

6. The practice ladder & resources

Curated, roughly in the order you’d use them. Don’t read them all before starting — interleave reading with the projects above.

Foundations (read while doing Projects 1–2)

  • PMPP, 4th ed. — Programming Massively Parallel Processors (Hwu, Kirk, Hajj, 2022). The canonical textbook; the GPU MODE lecture series is organized around it.8
  • GPU MODE lectures (formerly CUDA MODE), by Mark Saroufim & Andreas Köpf — weekly recorded lectures + a very active Discord + a resource-stream link repo. The best single community for this path.6
  • siboehm’s SGEMM worklog — the definitive step-by-step matmul optimization writeup + repo. Do it, don’t just read it.1
  • Horace He, “Making Deep Learning Go Brrrr From First Principles” — the compute/bandwidth/overhead mental model that underlies all perf work.4

Hands-on practice (interleave)

  • GPU Puzzles (Sasha Rush) — a Colab notebook of NUMBA-based CUDA puzzles from mapping to matmul; a few-hours on-ramp before real CUDA.9
  • LeetGPU — “LeetCode for GPUs”: an in-browser CUDA/Triton practice platform with graded problems; good for drilling fundamentals.10
  • Triton tutorials — the official triton-lang tutorials (vector add → fused softmax → matmul → FlashAttention); the fastest route to writing modern kernels.3

Reference & tooling (keep open constantly)

  • CUDA C++ Programming Guide + CUDA Best Practices Guide — the primary source of truth.11
  • Nsight Systems (timeline / whole-program) and Nsight Compute (per-kernel counters, roofline) — learn these early; profiling is the job.12

Papers (the systems canon)

  • FlashAttention (and FA-2/3).2 · Megatron-LM tensor parallelism.13 · ZeRO.14 · vLLM / PagedAttention.5

Competitions & benchmarks (proof under pressure)

  • GPU MODE Leaderboard / KernelBot — Discord-run kernel competitions with real prize pools (AMD and NVIDIA-sponsored problem sets, e.g. Blackwell NVFP4); the reference-kernels repo holds the official problems. A leaderboard placement is a legible, dated signal.7
  • KernelBench (Scaling Intelligence / Stanford) — a benchmark of PyTorch-op → CUDA generation across 4 difficulty levels; excellent as a source of well-specified kernel problems to attempt yourself, even though it targets LLMs.15

Rent the right GPU, and know which one

Your numbers are meaningless without the hardware context. Results differ across A100/H100/H200/Blackwell (and consumer 4090/5090). State the GPU in every benchmark. Rent by the hour (a single modern data-center GPU for a weekend is cheap relative to the signal it buys); for multi-GPU projects (4–6) rent an 8×GPU box briefly rather than owning anything.

7. Interview preparation

GPU/systems interviews are unusually concrete: they test whether you can reason about and manipulate real hardware, live. The rough anatomy:

What they testRepresentative questionHow to prepare
CUDA fundamentals”Write a kernel that does X. What’s your launch config and why?”Project 1 + GPU Puzzles + LeetGPU until launch config is reflex
Memory-hierarchy reasoning”This kernel is slow. Is it memory- or compute-bound? How do you know?”Internalize 02-gpu-memory-hierarchy; practice predicting bound before profiling
Roofline / perf analysis”Given these FLOPs and bytes, what’s the ceiling? Where is this kernel?”Do the roofline plots in Projects 1–3 by hand (03-performance-modeling-roofline)
Live kernel writing/optimizing”Optimize this matmul / softmax with me.”The GEMM ladder from muscle memory — narrate each step’s bottleneck
Parallelism tradeoffs”You have model M on N GPUs — pick DP/TP/PP/EP and justify.”Memory accounting + bubble/comm-volume math from 09-zero-and-fsdp, 10-model-parallelism
Debugging a slow kernel”Here’s an Nsight trace / a code snippet — why is it slow?”See the self-check below; practice reading traces on your own projects
ML-systems design”Design a serving stack for a 70B model at p99 < X ms.”12-llm-inference-systems; reason out loud about KV cache, batching, TP, quantization

The meta-skill: think out loud, quantify, name the bottleneck

Interviewers care less about the final number than about your diagnostic loop: form a hypothesis about the bottleneck → predict what the profiler would show → propose the fix → estimate the speedup. The projects train exactly this loop. If you can say “this is memory-bound at ~60% of HBM bandwidth, so tiling into SMEM should help but tensor cores won’t,” you’re demonstrating the job.

8. How to break in

The market rewards visible, verifiable output. In rough order of signal strength:

  1. A merged PR in a project the lab uses or respects (vLLM, SGLang, Triton, PyTorch, Megatron/TorchTitan, Liger-Kernel). Nothing else says “can do the job in a real codebase” as directly.6
  2. A public kernel/benchmark that stands up to scrutiny — the GEMM ladder or Triton FlashAttention with honest numbers and a roofline. This is the cold-application workhorse.
  3. A competition placement — a GPU MODE / KernelBot leaderboard rank is dated, public, and adversarial.7
  4. A technical blog post / reproduction that teaches something — a worklog in the siboehm mold, or a from-scratch reproduction with a clear result. Writing well about performance is itself a differentiator, and it doubles as interview rehearsal.
  5. Being useful in the community — answering questions in the GPU MODE Discord, reviewing others’ kernels. This is how you get noticed and referred.

What actually signals hireability

Depth over breadth. One kernel taken to 90% of cuBLAS with a writeup that shows you understood why each step worked beats a dozen half-finished tutorials. Hiring managers are looking for evidence of the diagnostic loop (hypothesis → measure → fix → quantify) and taste (knowing what’s worth optimizing). Referrals matter a lot — the community route (contribute → get known → get referred) is often faster than the résumé route.

Don't fake the numbers, ever

Overstated benchmarks (cherry-picked sizes, no baseline, missing hardware, comparing your optimized kernel to a strawman) are worse than no benchmarks — a competent interviewer will catch it in minutes and it torpedoes trust. Report honestly, including where you didn’t beat the baseline. “I got to 85% of cuBLAS and here’s the 15% I couldn’t close” is a stronger signal than a suspicious 99%.

9. A concrete study plan

A phased path assuming ~10–15 focused hrs/week from a solid ML/software background. Compress or stretch to taste; the sequence matters more than the exact weeks.

timeline
    title GPU Engineer Study Plan (~6 months, part-time)
    Phase 1 — Foundations (wk 1-4) : Tier 1 lessons 01-03
                                    : PMPP ch.1-5, GPU Puzzles
                                    : first CUDA kernels, learn Nsight
    Phase 2 — Kernels (wk 5-12) : Tier 2 lessons 04-07
                                : Project 1 (GEMM ladder)
                                : Project 2 (Triton FlashAttention)
                                : Project 3 (fused kernel)
    Phase 3 — Scale (wk 13-18) : Tier 3 lessons 08-11
                               : Project 4 (DDP/FSDP + MFU)
                               : rent multi-GPU, measure scaling
    Phase 4 — Inference + break-in (wk 19-24) : Lesson 12 + Project 5
                                              : Project 6 (real OSS PR)
                                              : blog post, competition, interview drills

Phase 1 — Foundations (weeks 1–4). Lessons 01-gpu-architecture-and-simt–03-performance-modeling-roofline; PMPP early chapters; GPU Puzzles for on-ramp; write your first CUDA kernels (04-first-cuda-kernels) and — critically — learn to run Nsight this phase, not later. Exit criterion: you can predict a kernel’s bound before running it.

Phase 2 — Kernels (weeks 5–12). Lessons 05–07. Ship Projects 1–3. This is the heart of the path and where your strongest portfolio pieces come from. Exit criterion: a GEMM at ≥85% cuBLAS and a numerically-correct Triton FlashAttention, both with writeups.

Phase 3 — Scale (weeks 13–18). Lessons 08–11; rent 2–8 GPUs and ship Project 4 with an honest MFU number. Exit criterion: a DDP→FSDP scaling curve you can explain.

Phase 4 — Inference + break-in (weeks 19–24). Lesson 12 + Project 5; land Project 6 (a real merged PR); write one blog post; do a competition round; run interview drills (§7) weekly. Exit criterion: pinned portfolio, one merged PR, one blog post, one competition entry — i.e. a complete application package.

Choose your specialization in Phase 3–4

By this point you’ll know which flavor (§3) you enjoy. Weight Projects 5–6 and your OSS target accordingly — kernel folks lean vLLM/SGLang/Triton kernels, infra folks lean Megatron/TorchTitan/FSDP, inference folks lean the serving stack. But keep all portfolio pieces; the core is shared and interviews probe across it.

The plan is a scaffold, not a treadmill

Six months is a reasonable full-part-time estimate, not a promise — some pieces take longer (warptiling alone ate four weekends for siboehm1). Optimize for shipped artifacts, not weeks elapsed. Better to spend eight weeks and produce one excellent, deeply-understood GEMM writeup than to rush all six projects into shallow, unbenchmarked repos.

10. What’s next

You’ve reached the end of the gpu-systems-for-llms curriculum. From here:

  1. Build the portfolio. The lessons are done; the projects in §5 are the actual work now. Start Project 1 this week.
  2. Go deeper on your flavor. Kernel folks: SASS-level tuning, CUTLASS/CuTe, and the frontier of FP8/FP4 and Blackwell MMA. Infra folks: the internals of a real training framework and cluster reliability at scale. Inference folks: speculative decoding, disaggregated prefill/decode, and structured-output serving.
  3. Track the frontier. New hardware (each generation shifts the roofline) and new systems papers land constantly — stay plugged into the GPU MODE community6 so your mental model of “SOTA” doesn’t go stale.
  4. Adjacent thread — RL systems. If post-training/RL infra interests you, the training↔inference seam explored in 10-frontier-and-infra is one of the hottest and least-crowded niches in 2026.

Capstone of the curriculum. Filed: 2026-09-02 | Parent: gpu-systems-for-llms

References

Footnotes

  1. Simon Boehm, “How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog” (2022). https://siboehm.com/articles/22/CUDA-MMM — iteratively optimizes SGEMM from a naive kernel to ~94% of cuBLAS at FP32 via coalescing, SMEM caching, 1D/2D block+thread tiling, vectorized (float4) loads, autotuning, and warptiling; code at https://github.com/siboehm/SGEMM_CUDA. Boehm reports the first 6 kernels (~2 weekends) reached ~80% of peak and autotuning+warptiling (~4 more weekends) reached ~94%. [established] ↩ ↩2 ↩3

  2. Dao et al., “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” arXiv:2205.14135 (2022). https://arxiv.org/abs/2205.14135 — IO-aware tiled attention with online softmax; reference implementation and FA-2/FA-3 kernels at https://github.com/Dao-AILab/flash-attention. [established] ↩ ↩2

  3. Triton language and official tutorials (OpenAI / triton-lang), https://triton-lang.org/main/getting-started/tutorials/ — progression from vector add → fused softmax → matmul → FlashAttention; repo https://github.com/triton-lang/triton. [established] ↩ ↩2

  4. Horace He, “Making Deep Learning Go Brrrr From First Principles” (2022). https://horace.io/brrr_intro.html — frames performance as compute vs. memory-bandwidth vs. overhead bound, and argues optimization should start by identifying which regime you’re in. [established] ↩ ↩2 ↩3

  5. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” arXiv:2309.06180 (2023). https://arxiv.org/abs/2309.06180 — PagedAttention / vLLM; block-based KV cache that reduces fragmentation and enables high-throughput serving. [established] ↩ ↩2

  6. GPU MODE (formerly CUDA MODE), founded by Mark Saroufim and Andreas Köpf — weekly recorded GPU-programming lectures (https://www.gpumode.com/lectures, YouTube channel “GPU MODE”), an active Discord (https://discord.gg/gpumode; ~30k members), and a curated link repo resource-stream (https://github.com/gpu-mode/resource-stream). The community is organized substantially around the PMPP textbook. [recent] ↩ ↩2 ↩3 ↩4

  7. GPU MODE reference-kernels repository, https://github.com/gpu-mode/reference-kernels — official problem sets and reference kernels for the GPU MODE Leaderboard, run via “KernelBot” on Discord; hosts sponsored competitions (e.g. AMD kernel/distributed competitions with six/seven-figure prize pools, an NVIDIA Blackwell NVFP4 competition) and academic problem sets (PMPP practice, Stanford CS149, Princeton). Verified Sept 2026. [recent] ↩ ↩2 ↩3

  8. Wen-mei W. Hwu, David B. Kirk, Izzat El Hajj, Programming Massively Parallel Processors: A Hands-on Approach, 4th ed. (Morgan Kaufmann, 2022) — the canonical GPU-programming textbook; the GPU MODE lecture series follows its chapters. [established] ↩

  9. Sasha Rush, “GPU Puzzles,” https://github.com/srush/GPU-Puzzles — a Colab notebook of NUMBA-based CUDA puzzles (map/zip → threads/blocks → shared memory → pooling/conv/matmul) as a few-hours on-ramp to GPU programming. [established] ↩

  10. LeetGPU, https://leetgpu.com/ — an in-browser “GPU programming platform” offering graded CUDA/Triton practice problems (LeetCode-style) for drilling kernel fundamentals. Verified live Sept 2026; feature set beyond the tagline was not fully confirmed from the landing page. [recent] ↩

  11. NVIDIA, CUDA C++ Programming Guide (https://docs.nvidia.com/cuda/cuda-c-programming-guide/) and CUDA C++ Best Practices Guide (https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/) — primary reference for the CUDA execution and memory model. [established] ↩

  12. NVIDIA Nsight Systems (whole-application timeline profiler, https://developer.nvidia.com/nsight-systems) and Nsight Compute (per-kernel counter/roofline profiler, https://developer.nvidia.com/nsight-compute) — the standard tools for GPU performance analysis. [established] ↩

  13. Shoeybi et al., “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism,” arXiv:1909.08053 (2019). https://arxiv.org/abs/1909.08053 — the canonical tensor-parallelism formulation. [established] ↩

  14. Rajbhandari et al., “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models,” arXiv:1910.02054 (2019). https://arxiv.org/abs/1910.02054 — the sharded-optimizer-state accounting behind DeepSpeed/FSDP. [established] ↩

  15. Ouyang, Guo, et al. (Scaling Intelligence Lab, Stanford), “KernelBench,” https://github.com/ScalingIntelligence/KernelBench — a benchmark of LLM-generated GPU kernels transpiling PyTorch ops to CUDA across 4 difficulty levels (single ops → fusion → full architectures → HuggingFace model optimization), scored by a “fast_p” correct-and-faster-than-threshold metric. Actively maintained as of 2026. Useful here as a source of well-specified kernel problems to attempt by hand. [recent] ↩

Graph View

Backlinks