Volcano-Kthena

Kubernetes-native LLM inference orchestration that pairs Volcano gang scheduling with KV-cache-aware, prefill-decode routing.

• Category: Orchestration & Scheduling
• CNCF maturity: Incubating
• Language: Go
• License: Apache-2.0
• Repository: volcano-sh/kthena
• Documented at commit: affd5be (2026-06-24, main HEAD)

What it is

Kthena is a subproject of Volcano that orchestrates large language model inference on Kubernetes. Volcano started as a batch scheduler for AI training; Kthena extends the same scheduling machinery to the serving side of the AI lifecycle. The Volcano community announced it on 2026-01-28.

It splits into two parts. A control plane (kthena-controller-manager) reconciles custom resources to deploy, scale, and upgrade inference replicas, and delegates gang scheduling to the Volcano scheduler. A data plane (kthena-router) is the entry point for inference traffic: it classifies each OpenAI-compatible request, applies rate limiting and traffic policy, scores candidate pods, and proxies to the chosen inference instance. The two planes can be deployed independently.

Kthena does not replace the inference engine. It sits above engines such as vLLM and SGLang and treats them as backends, adding multi-node placement, prefill-decode disaggregation, and KV-cache-aware routing on top.

When to use it

• You run LLM inference on Kubernetes and already use, or are willing to adopt, the Volcano scheduler for gang scheduling.
• You need prefill-decode disaggregation across multiple pods and want the router to keep prefill and decode endpoints paired.
• You want KV-cache and prefix-cache locality decided at an L7 routing layer rather than inside each engine.
• You serve many LoRA adapters and need adapter-aware routing without restarting inference.

It fits less well when you want a Gateway API Inference Extension native stack (see llm-d), or when you do not want a Volcano dependency. The router itself is documented as a reference implementation under active iteration.

In this deep-dive

• History: origin, milestones, and why it exists.
• Architecture: components and how requests flow.
• Adoption & Ecosystem: who runs it and what surrounds it.
• Internals: the code paths that matter, read from source.
• Getting Started: install and a first working setup.

Sources

1. volcano-sh/kthena (GitHub)
2. Introducing Kthena: LLM inference for the cloud native era (CNCF)
3. Cloud Native Batch System Volcano moves to the CNCF Incubator
4. Volcano (CNCF project page)
5. Beyond Batch: Volcano Evolves into the AI-Native Unified Scheduling Platform
6. How to choose the inference orchestration solution? AIBrix or Kthena or Dynamo? (pacoxu)
7. Kthena integration (vLLM docs)
8. Using Volcano Kthena (vllm-ascend)
9. Introducing Gateway API Inference Extension (Kubernetes)
10. Cloud-Native AI Inference using KServe and llm-d
11. AIBrix: Towards Scalable, Cost-Effective LLM Inference Infrastructure (arXiv)
12. Kthena quick-start guide