GPU Systems & ML Systems for LLMs — Curriculum

A progressive curriculum for becoming a GPU researcher/engineer at a frontier LLM lab: GPU architecture and the memory hierarchy → writing efficient kernels (CUDA, Triton, FlashAttention) → distributed training at scale (DP, ZeRO/FSDP, tensor/pipeline/expert parallelism) → inference systems → an interview/roadmap capstone. Each lesson is a self-contained Obsidian learning guide with a heavy practice & resources section.

How to work through this

Lessons are strictly dependency-ordered. Tier 1 (GPU fundamentals) is the bedrock for everything else — don’t skip it even if you’ve written CUDA before. The distributed and SOTA tiers assume the fundamentals and kernel tiers.

Progress tracker

How to track progress

Each lesson has a status property (unread → reading → done, or review). Set it from the Properties panel (it’s a dropdown via Metadata Menu). The table below is a live Dataview query.

TABLE WITHOUT ID
  file.link AS "Lesson",
  choice(status = "done", "✅ done",
    choice(status = "reading", "📖 reading",
      choice(status = "review", "🔁 review", "⬜ unread"))) AS "Status"
FROM "wiki/learn/gpu"
SORT file.name ASC

Overall:

TABLE WITHOUT ID
  length(filter(rows.status, (s) => s = "done")) AS "✅ Done",
  length(filter(rows.status, (s) => s = "reading")) AS "📖 Reading",
  length(filter(rows.status, (s) => s = "unread")) AS "⬜ Unread",
  length(rows) AS "Total"
FROM "wiki/learn/gpu"
GROUP BY true

Dependency map

graph TD
    subgraph "Tier 1 — GPU fundamentals"
        L1["01 · GPU architecture &<br/>the SIMT execution model"]
        L2["02 · The GPU memory hierarchy<br/>registers→SMEM→L2→HBM"]
        L3["03 · Performance modeling<br/>roofline, intensity, tensor cores"]
    end
    subgraph "Tier 2 — Writing efficient kernels"
        L4["04 · Your first CUDA kernels"]
        L5["05 · Optimizing GEMM<br/>the canonical case study"]
        L6["06 · The memory wall & fusion<br/>FlashAttention deep-dive"]
        L7["07 · Triton & modern<br/>kernel authoring"]
    end
    subgraph "Tier 3 — Distributed training at scale"
        L8["08 · Data parallelism &<br/>collective communication"]
        L9["09 · Sharded DP: ZeRO & FSDP"]
        L10["10 · Model parallelism<br/>tensor / pipeline / seq / expert"]
        L11["11 · SOTA large-scale<br/>training systems"]
    end
    subgraph "Tier 4 — Inference & the role"
        L12["12 · LLM inference &<br/>serving systems"]
        L13["13 · The GPU engineer's roadmap<br/>projects, practice, interviews"]
    end

    L1 --> L2 --> L3 --> L4 --> L5 --> L6 --> L7
    L3 --> L8
    L7 --> L8 --> L9 --> L10 --> L11
    L6 --> L12
    L11 --> L12 --> L13

    style L1 fill:#44a,color:#fff
    style L2 fill:#44a,color:#fff
    style L3 fill:#44a,color:#fff
    style L6 fill:#4a4,color:#fff
    style L11 fill:#4a4,color:#fff

Lessons

Tier 1 — GPU fundamentals ✅

• 01-gpu-architecture-and-simt — SMs, warps, the SIMT model, occupancy, tensor cores
• 02-gpu-memory-hierarchy — registers→SMEM→L2→HBM; coalescing, bank conflicts, the memory wall
• 03-performance-modeling-roofline — roofline, arithmetic intensity, MFU, precision & tensor cores

Tier 2 — Writing efficient kernels ✅

• 04-first-cuda-kernels — execution config, vector add, grid-stride, naive matmul (and why it’s slow)
• 05-optimizing-gemm — the optimization ladder: coalescing → SMEM tiling → register blocking → ~cuBLAS
• 06-memory-wall-and-flashattention — kernel fusion + FlashAttention (online softmax, IO-awareness, FA-1→4)
• 07-triton-and-modern-kernels — Triton block model, autotuning, torch.compile, when to drop to CUDA

Tier 3 — Distributed training at scale

• 08-data-parallelism-and-collectives — (queued)
• 09-zero-and-fsdp — (queued)
• 10-model-parallelism — (queued)
• 11-sota-large-scale-training — (queued)

Tier 4 — Inference & the role

• 12-llm-inference-systems — (queued)
• 13-gpu-engineer-roadmap — (queued)

Design principle

This curriculum is hands-on by design: every lesson ends with concrete exercises (write/profile a kernel, run a multi-GPU job, reproduce a result) and curated practice resources. The target is job-readiness for a GPU researcher/engineer role — you should be able to build and profile the things you learn, not just describe them.