Gunjan Dhanuka — Learning Notes

Wiki Index

5 min read

Research Hub — Index

Master catalog of all wiki pages. Updated on every ingest or /teach.

Sources

(No sources ingested yet. Drop a paper or blog into raw/ and ask me to ingest it.)

Concepts

Vision / multimodal

  • moonvit — Native-resolution vision encoder from Kimi-VL; the canonical GNE architecture
  • global-native-encoding — Paradigm for encoding full images at native resolution in one ViT pass
  • navit-packing — Sequence packing strategy for variable-resolution image batching without padding

Reinforcement learning — foundations

  • markov-decision-process — The formalism, returns, and the objective
  • value-function — and , their Bellman equations, and the critic/baseline role
  • advantage-function — ; the min-variance credit-assignment weight
  • policy-gradient-theorem — The score-function trick;
  • reinforce — Monte Carlo policy gradient as advantage-weighted MLE; ancestor of RLHF methods
  • generalized-advantage-estimation — ; the bias-variance knob
  • actor-critic — Joint actor + critic ; the RLHF value head
  • ppo — The clipped surrogate trust region; the four-model RLHF setup
  • trust-region — Bounding policy movement in KL space; TRPO vs PPO
  • kl-regularization-rlhf — The leash against reward hacking
  • reward-model — Learned Bradley-Terry preference scorer; the RLHF “environment”
  • rlhf — The SFT → reward-model → PPO alignment pipeline

Reinforcement learning — SOTA (preference optimization & reasoning)

  • dpo — Direct Preference Optimization: closed-form RLHF optimum as a supervised loss on preference pairs
  • preference-optimization — Umbrella for the RL-free family (DPO/IPO/KTO/ORPO/SimPO/CPO)
  • bradley-terry-model — Pairwise preference likelihood under both RM training and DPO
  • grpo — Group Relative Policy Optimization: PPO with a critic-free Monte-Carlo group baseline
  • rlvr — RL with Verifiable Rewards: a verifier as reward that dissolves overoptimization
  • process-reward-model — PRM vs ORM: step-level vs outcome supervision
  • chain-of-thought-rl — Emergent long CoT from outcome RL; test-time-compute scaling; o1/R1
  • constitutional-ai — Anthropic’s two-phase (SL critique/revise + RLAIF) method
  • rlaif — RL from AI feedback; LLM-as-judge as label source; RLAIF≈RLHF parity
  • reward-overoptimization — Goodhart on a learned proxy; Gao et al. scaling laws
  • best-of-n-sampling — BoN with analytic KL; RFT/ReST as amortization into the weights
  • agentic-rl — Multi-turn RL: tool-use credit assignment, environments/verifiers
  • rl-post-training-infra — Generation-vs-training split; disaggregated async actor-learner

GPU / ML systems

  • cuda-execution-model — grid/block/warp/thread hierarchy and its mapping to SMs and lanes
  • streaming-multiprocessor — the SM as the unit of scheduling and resource allocation
  • warp-simt — warps of 32 in lockstep; divergence serialization and predication
  • occupancy — active/max warps; register & SMEM limits; why 100% isn’t the goal
  • tensor-core — MMA units, the precision ladder, the throughput gap over CUDA cores
  • gemm — dense matmul: the canonical GPU workload and compute core of transformers
  • gpu-memory-hierarchy — registers→SMEM→L2→HBM→host; capacity/latency/bandwidth
  • memory-coalescing — combining a warp’s addresses into minimum memory transactions
  • shared-memory-bank-conflicts — the 32-bank model, conflict serialization, padding fix
  • memory-wall — compute scaling faster than bandwidth; the ridge point marching right
  • arithmetic-intensity — FLOPs/byte; the memory- vs compute-bound lever
  • roofline-model — attainable FLOP/s vs intensity; ridge point; kernel classification
  • mfu — Model FLOPs Utilization vs HFU; the rule; typical ranges
  • mixed-precision-training — FP32/TF32/FP16/BF16/FP8; master weights, loss scaling

GPU / ML systems — kernels

  • shared-memory-tiling — staging operand tiles in SMEM to cut GEMM HBM traffic
  • register-blocking — TM×TN outer-product micro-tiles; the biggest GEMM-ladder jump
  • kernel-fusion — folding op chains into one kernel to eliminate HBM round-trips
  • flash-attention — IO-aware tiled attention; O(N²)→O(N) HBM; the FA-1→4 lineage
  • online-softmax — single-pass softmax with running max/denominator + rescale
  • io-aware-algorithms — designing around HBM traffic; FLOPs cheap, bytes expensive
  • recomputation — activation checkpointing; trade FLOPs for memory
  • triton — block-level GPU language/compiler; MLIR lowering to PTX
  • torch-compile — TorchDynamo/AOTAutograd/TorchInductor; auto Triton codegen
  • kernel-autotuning — empirical selection of block sizes / num_warps / num_stages

Entities

  • moonshot-ai — Chinese AI lab; made Kimi-VL and MoonViT

Topics

  • rl-for-llms — RL for frontier LLMs: a progressive curriculum (foundations → SOTA)
  • gpu-systems-for-llms — GPU/ML systems: architecture → kernels → distributed training → inference

Learn

Interactive Obsidian learning guides — dependency maps, quizzes, practice problems, reading orders.

  • moonvit — MoonViT: native-res encoding, NaViT packing, dual positional embeds, GNE vs SBE

RL for LLMs curriculum → rl-for-llms

GPU systems curriculum → gpu-systems-for-llms

Graph View