Gunjan Dhanuka — Learning Notes

Wiki Index

8 min read

Research Hub — Index

Master catalog of all wiki pages. Updated on every ingest or /teach.

Sources

  • 2026-09-12-kimi-k3 — Kimi K3 technical report (Moonshot); KDA–MLA hybrid, NoPE, 1M context (arXiv:2607.24653)
  • 2026-09-12-qwen3.8-next — Qwen3.8-Next architecture (Qwen Team); GDN hybrid + QSA sparse attention (arXiv:2608.30320)

Concepts

Vision / multimodal

  • moonvit — Native-resolution vision encoder from Kimi-VL; the canonical GNE architecture
  • global-native-encoding — Paradigm for encoding full images at native resolution in one ViT pass
  • navit-packing — Sequence packing strategy for variable-resolution image batching without padding

Reinforcement learning — foundations

  • markov-decision-process — The formalism, returns, and the objective
  • value-function — and , their Bellman equations, and the critic/baseline role
  • advantage-function — ; the min-variance credit-assignment weight
  • policy-gradient-theorem — The score-function trick;
  • reinforce — Monte Carlo policy gradient as advantage-weighted MLE; ancestor of RLHF methods
  • generalized-advantage-estimation — ; the bias-variance knob
  • actor-critic — Joint actor + critic ; the RLHF value head
  • ppo — The clipped surrogate trust region; the four-model RLHF setup
  • trust-region — Bounding policy movement in KL space; TRPO vs PPO
  • kl-regularization-rlhf — The leash against reward hacking
  • reward-model — Learned Bradley-Terry preference scorer; the RLHF “environment”
  • rlhf — The SFT → reward-model → PPO alignment pipeline

Reinforcement learning — SOTA (preference optimization & reasoning)

  • dpo — Direct Preference Optimization: closed-form RLHF optimum as a supervised loss on preference pairs
  • preference-optimization — Umbrella for the RL-free family (DPO/IPO/KTO/ORPO/SimPO/CPO)
  • bradley-terry-model — Pairwise preference likelihood under both RM training and DPO
  • grpo — Group Relative Policy Optimization: PPO with a critic-free Monte-Carlo group baseline
  • rlvr — RL with Verifiable Rewards: a verifier as reward that dissolves overoptimization
  • process-reward-model — PRM vs ORM: step-level vs outcome supervision
  • chain-of-thought-rl — Emergent long CoT from outcome RL; test-time-compute scaling; o1/R1
  • constitutional-ai — Anthropic’s two-phase (SL critique/revise + RLAIF) method
  • rlaif — RL from AI feedback; LLM-as-judge as label source; RLAIF≈RLHF parity
  • reward-overoptimization — Goodhart on a learned proxy; Gao et al. scaling laws
  • best-of-n-sampling — BoN with analytic KL; RFT/ReST as amortization into the weights
  • agentic-rl — Multi-turn RL: tool-use credit assignment, environments/verifiers
  • rl-post-training-infra — Generation-vs-training split; disaggregated async actor-learner

GPU / ML systems

  • cuda-execution-model — grid/block/warp/thread hierarchy and its mapping to SMs and lanes
  • streaming-multiprocessor — the SM as the unit of scheduling and resource allocation
  • warp-simt — warps of 32 in lockstep; divergence serialization and predication
  • occupancy — active/max warps; register & SMEM limits; why 100% isn’t the goal
  • tensor-core — MMA units, the precision ladder, the throughput gap over CUDA cores
  • gemm — dense matmul: the canonical GPU workload and compute core of transformers
  • gpu-memory-hierarchy — registers→SMEM→L2→HBM→host; capacity/latency/bandwidth
  • memory-coalescing — combining a warp’s addresses into minimum memory transactions
  • shared-memory-bank-conflicts — the 32-bank model, conflict serialization, padding fix
  • memory-wall — compute scaling faster than bandwidth; the ridge point marching right
  • arithmetic-intensity — FLOPs/byte; the memory- vs compute-bound lever
  • roofline-model — attainable FLOP/s vs intensity; ridge point; kernel classification
  • mfu — Model FLOPs Utilization vs HFU; the rule; typical ranges
  • mixed-precision-training — FP32/TF32/FP16/BF16/FP8; master weights, loss scaling

GPU / ML systems — kernels

  • shared-memory-tiling — staging operand tiles in SMEM to cut GEMM HBM traffic
  • register-blocking — TM×TN outer-product micro-tiles; the biggest GEMM-ladder jump
  • kernel-fusion — folding op chains into one kernel to eliminate HBM round-trips
  • flash-attention — IO-aware tiled attention; O(N²)→O(N) HBM; the FA-1→4 lineage
  • online-softmax — single-pass softmax with running max/denominator + rescale
  • io-aware-algorithms — designing around HBM traffic; FLOPs cheap, bytes expensive
  • recomputation — activation checkpointing; trade FLOPs for memory
  • triton — block-level GPU language/compiler; MLIR lowering to PTX
  • torch-compile — TorchDynamo/AOTAutograd/TorchInductor; auto Triton codegen
  • kernel-autotuning — empirical selection of block sizes / num_warps / num_stages

GPU / ML systems — distributed training

  • data-parallelism — replicate + split-batch + all-reduce; DDP overlap
  • collective-communication — the six primitives; ring all-reduce’s bandwidth-optimal cost
  • nccl — NVIDIA collectives; topology-aware ring/tree/NVLS + SHARP
  • zero — DeepSpeed ZeRO stages 1/2/3; the memory↔comms tradeoff
  • fsdp — PyTorch FSDP ≡ ZeRO-3; FSDP1 FlatParameter vs FSDP2 DTensor
  • optimizer-state-memory — the 16 bytes/param mixed-precision-Adam accounting
  • tensor-parallelism — Megatron column→row pairing; one all-reduce/block
  • pipeline-parallelism — GPipe vs 1F1B; the bubble
  • sequence-parallelism — split LN/dropout/residual along the sequence
  • expert-parallelism — distribute MoE experts; all-to-all dispatch/combine
  • context-parallelism — split the sequence for long context; Ring Attention
  • 3d-parallelism — compose DP × TP × PP (× EP × CP) onto the topology
  • fp8-training — FP8 training at scale; Transformer Engine; DeepSeek-V3 recipe
  • megatron-lm — the reference large-scale training framework/techniques

GPU / ML systems — inference & serving

  • prefill-decode — the two phases: prefill compute-bound (TTFT), decode memory-bound (TPOT)
  • kv-cache — the per-sequence K/V store; memory formula; the serving bottleneck
  • paged-attention — block-based KV management (vLLM); fragmentation fix, prefix sharing
  • continuous-batching — iteration-level scheduling (Orca); admit/evict per step
  • speculative-decoding — draft+verify, distribution-preserving; Medusa/EAGLE
  • quantization-inference — weight-only (GPTQ/AWQ) vs W8A8/FP8; match to the bottleneck

Attention

  • attention-glossary — canonical vocabulary for the attention curriculum (34 terms)
  • positional-encoding — injecting order into permutation-equivariant attention
  • rope — Rotary Position Embedding; relative position via rotation; scaling
  • alibi — linear distance bias; length extrapolation
  • gqa-mqa — grouped/multi-query attention; shrink the KV cache
  • mla — Multi-head Latent Attention (DeepSeek); low-rank cached KV
  • sliding-window-attention — local windowed attention; O(s·w); receptive field ≈ L·w
  • sparse-attention — structured sparsity (Longformer, BigBird)
  • attention-sinks — the sink phenomenon; StreamingLLM streaming decode
  • linear-attention — kernel-trick + associativity → O(s); the recall gap
  • state-space-models — SSM recurrence; conv/recurrence duality; S4
  • mamba — selective SSM; input-dependent gating; constant-state inference
  • vision-transformer — ViT; self-attention over image patches; O(n_patch²)
  • cross-attention — Q from one stream, K/V from another; VLM fusion
  • perceiver-resampler — fixed learned latents compress variable-length inputs
  • vlm-fusion — cross-attention fusion vs project-and-concatenate
  • rope-scaling-yarn — extending RoPE context (PI, NTK, YaRN)
  • ring-attention — sequence-sharded attention; context parallelism
  • long-context — umbrella: positional extension + systems + evaluation
  • delta-rule — error-correcting fast-weight update; the DeltaNet primitive
  • gated-deltanet — GDN: Mamba-2 decay + delta rule (Qwen3.8-Next’s linear layer)
  • kda — Kimi Delta Attention (Kimi K3): channel-wise-gated delta rule; constant state
  • qwen-sparse-attention — QSA: compressed trainable sparse attention (O(n²/r))
  • deepseek-sparse-attention — DSA→CSA→CSA2 lineage: trainable sparse attention on MLA (V3.2/V4)
  • moba — Mixture of Block Attention (Kimi); MoE-style top-k block routing
  • hybrid-attention-ssm — few attention + many SSM/linear layers (Nemotron-H, Hymba, Jamba)
  • local-global-attention — interleaved local/global (Gemma); the KV-cache lever

Entities

  • moonshot-ai — Chinese AI lab; made Kimi-VL and MoonViT
  • gpu-mode — GPU MODE community: lectures, Discord, reference-kernels/KernelBot leaderboard
  • kernelbench — benchmark for LLM-generated GPU kernels (Scaling Intelligence / Stanford)

Topics

  • rl-for-llms — RL for frontier LLMs: a progressive curriculum (foundations → SOTA)
  • gpu-systems-for-llms — GPU/ML systems: architecture → kernels → distributed training → inference
  • attention-mechanisms — attention in LLMs & VLMs: foundations → efficiency → multimodal → SOTA

Learn

Interactive Obsidian learning guides — dependency maps, quizzes, practice problems, reading orders.

  • moonvit — MoonViT: native-res encoding, NaViT packing, dual positional embeds, GNE vs SBE

RL for LLMs curriculum → rl-for-llms

GPU systems curriculum → gpu-systems-for-llms

Attention mechanisms curriculum → attention-mechanisms

Graph View