Gunjan Dhanuka — Learning Notes
Search
Search
Dark mode
Light mode
Reader mode
Explorer
zero
4 items with this tag.
Sep 02, 2026
FSDP (Fully Sharded Data Parallel)
gpu
distributed-training
fsdp
fsdp2
pytorch
dtensor
sharded-data-parallelism
zero
collectives
Sep 02, 2026
Optimizer State Memory (16 bytes/param)
gpu
distributed-training
memory
optimizer-state
adam
mixed-precision
zero
fsdp
Sep 02, 2026
ZeRO (Zero Redundancy Optimizer)
gpu
distributed-training
zero
deepspeed
sharded-data-parallelism
optimizer-state
memory
offloading
Sep 02, 2026
Learn: ZeRO & FSDP
gpu
distributed-training
zero
fsdp
fsdp2
deepspeed
sharded-data-parallelism
mixed-precision
optimizer-state
offloading
collectives