Basic
Minimal multi-node inference on LeaderWorkerSet with vLLM and SGLang.
LeaderWorkerSet
LeaderWorkerSet examples, including multi-node inference, autoscaling, topology-aware placement, and native gang scheduling.
These guides deploy a distributed, multi-node inference service with
LeaderWorkerSet, spreading tensor and pipeline parallelism across the leader and
worker pods. Each guide isolates one feature and ships both a vllm.yaml and a
sglang.yaml.
- Basic: minimal multi-node deployment (no Ray).
- Autoscaling: scale replica groups with a HorizontalPodAutoscaler.
- Topology-aware scheduling: pin each group to one
topology domain with
exclusive-topology. - Gang scheduling: schedule each leader-worker replica as an all-or-nothing gang with native Kubernetes scheduling.
Autoscaling
Scale LeaderWorkerSet replica groups with a HorizontalPodAutoscaler.
Topology-aware scheduling
Pin each replica group to a single topology domain, with native placement or Kueue.
Gang scheduling
Run a CPU-only LeaderWorkerSet with one native scheduling gang per replica.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.