CNCF Atlas

English

日本語

English

日本語

Appearance

Architecture ​

Big picture ​

Armada is split into a control plane and a set of executors. The control plane accepts, schedules, and tracks jobs. One executor runs per worker cluster and bridges the control plane to that cluster's Kubernetes API (docs/system_overview.md:21). Clients submit jobs to the control plane, and jobs are transferred to a worker cluster only once scheduled (docs/system_overview.md:20).

The services communicate by event sourcing. Message routing goes through a log-based message broker, Apache Pulsar, shared by all services. The log is the source of truth, and each subsystem can rebuild its internal state by replaying messages (docs/system_overview.md:62-70).

Components ​

Control plane subsystems ​

The control plane is the set of components responsible for job submission and scheduling (docs/system_overview.md:54). It is made up of: job submission and control (submit, re-prioritise, cancel), job state querying, streaming job state querying, the scheduler that assigns jobs to clusters and nodes, and Lookout, the web UI showing job state (docs/system_overview.md:56-60). Each subsystem is a binary under cmd/; for example cmd/server/main.go and cmd/scheduler.

Executor ​

There is one executor per worker cluster (docs/system_overview.md:21). It is responsible for communication between the Armada control plane and the Kubernetes control plane (docs/system_overview.md:21). When the scheduler marks a job as leased, the executor creates the Kubernetes resources for the job in its cluster and binds the pod to the assigned node (docs/system_overview.md:45-46).

Dependency middleware ​

Armada relies on Apache Pulsar for message routing (docs/system_overview.md:62). The scheduler and Lookout persist to PostgreSQL, and Redis is part of the local development stack (README:81). The local stack brings up redis, postgres, and pulsar via Docker Compose (README:81).

How a request flows ​

A job submission walks through the submit server in internal/server/submit/submit.go. The Server type is the service that accepts the original Armada submit API and publishes messages to Pulsar based on those calls (internal/server/submit/submit.go:30-32).

  1. Entry point func (s *Server) SubmitJobs(...) (internal/server/submit/submit.go:72).
  2. Authorization: the caller is checked with s.authorize(ctx, req.Queue, permissions.SubmitAnyJobs, queue.PermissionVerbSubmit); failure returns codes.PermissionDenied (internal/server/submit/submit.go:76-79).
  3. Validation: validation.ValidateSubmitRequest(req, s.submissionConfig); failure returns codes.InvalidArgument (internal/server/submit/submit.go:82-84).
  4. Deduplication: s.deduplicator.GetOriginalJobIds(ctx, req.Queue, req.JobRequestItems) maps a client id to an existing job id. Deduplication is best-effort, so an error here is logged and not fatal (internal/server/submit/submit.go:88-92). A request item that is a duplicate returns the previously submitted id and is skipped (internal/server/submit/submit.go:101-106).
  5. Conversion: each non-duplicate item becomes a SubmitJob event via conversion.SubmitJobFromApiRequest(...) (internal/server/submit/submit.go:109), wrapped as an EventSequence_Event_SubmitJob (internal/server/submit/submit.go:111-116).
  6. Publish: the events are assembled into an EventSequence (internal/server/submit/submit.go:133-139) and sent with s.publisher.PublishMessages(ctx, es); failure returns codes.Internal (internal/server/submit/submit.go:141-145).
  7. After a successful publish, deduplication ids are stored with s.deduplicator.StoreOriginalJobIds(...) (internal/server/submit/submit.go:149). A comment notes that a partial Pulsar submission can result in duplicate jobs because this store only runs on success (internal/server/submit/submit.go:147-148).

The scheduler then takes over. func (s *Scheduler) Run(...) (internal/scheduler/scheduler.go:147) runs a ticker every cycle period (internal/scheduler/scheduler.go:159, :164-169). Only the leader may schedule jobs (internal/scheduler/scheduler.go:49-50) and only the leader publishes (internal/scheduler/scheduler.go:56). On becoming leader, it first catches up to all Pulsar messages with s.ensureDbUpToDate(...) before scheduling (internal/scheduler/scheduler.go:186-189). The body of one cycle is func (s *Scheduler) cycle(...) (internal/scheduler/scheduler.go:281).

Key design decisions ​

Event sourcing with Pulsar as source of truth is the central decision: subsystems update state and publish back to the log, and any subsystem can recover by replaying (docs/system_overview.md:62-70). The submit path therefore never writes jobs to a database; it only publishes to Pulsar, leaving persistence to downstream consumers (internal/server/submit/submit.go:141).

Scheduling is leader-only. A single leader schedules and publishes, which avoids two schedulers making conflicting placement decisions (internal/scheduler/scheduler.go:49-50, :56). To stay correct, a new leader blocks on catching up to Pulsar before it schedules (internal/scheduler/scheduler.go:186-189).

The scheduler decouples its hot loop from database I/O by holding all jobs in an in-memory transactional store, JobDb (internal/scheduler/jobdb/jobdb.go:68). This is covered in Internals.

Extension points ​

Armada exposes gRPC and REST submit APIs, so clients in several languages drive it without changing the server. Client libraries exist for Python, Java, and .NET (docs/client_libraries.md). An Airflow operator integrates Armada into Airflow DAGs (docs/armada_airflow_operator.md). Job scheduling behaviour is configurable through priority classes and Dominant Resource Fairness (DRF, fair allocation across multiple resource types) settings consumed by the scheduler (internal/scheduler/scheduling/fairness/fairness.go:43).