Basic
Minimal multi-node inference on LeaderWorkerSet with vLLM and SGLang.
LeaderWorkerSet
Multi-node inference with LeaderWorkerSet, including autoscaling and topology-aware placement.
These guides deploy a distributed, multi-node inference service with
LeaderWorkerSet, spreading tensor and pipeline parallelism across the leader and
worker pods. Each guide isolates one feature and ships both a vllm.yaml and a
sglang.yaml.
- Basic: minimal multi-node deployment (no Ray).
- Autoscaling: scale replica groups with a HorizontalPodAutoscaler.
- Topology-aware scheduling: pin each group to one
topology domain with
exclusive-topology.
Autoscaling
Scale LeaderWorkerSet replica groups with a HorizontalPodAutoscaler.
Topology-aware scheduling
Pin each replica group to a single topology domain with exclusive-topology.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.