# Orloj Docs > Runtime, governance, and orchestration for agent systems. ## A2A JSON-RPC > **Stability: beta** -- A2A protocol support ships with `orloj.dev/v1`. The JSON-RPC interface follows the [A2A specification](https://github.com/a2aproject/A2A) and may evolve as the spec matures. Orloj exposes A2A functionality via JSON-RPC 2.0 over HTTP. All requests are `POST` with `Content-Type: application/json`. ### Endpoints | Endpoint | Auth | Description | | ----------------------------------- | --------------------------------- | -------------------------------------------------------------------------------------------------------- | | `POST /a2a` | Bearer token when auth is enabled | Shared JSON-RPC endpoint. Resolves target from params or defaults to the single A2A-enabled AgentSystem. | | `POST /v1/agent-systems/{name}/a2a` | Bearer token when auth is enabled | Per-system JSON-RPC endpoint. AgentSystem name in the path determines routing. | Legacy `/v1/agents/{name}/a2a` paths are accepted as aliases for AgentSystem names. ### Request Format All methods use the standard JSON-RPC 2.0 envelope: ```json { "jsonrpc": "2.0", "id": "req-1", "method": "tasks/send", "params": { ... } } ``` The `id` field can be a string or integer. If omitted, the request is treated as a notification (no response body). ### Methods #### tasks/send Create a task and wait for completion. Returns the final task state. **Params:** | Field | Type | Required | Description | | ---------- | ------------------------- | -------- | --------------------------------------------------------------------- | | `id` | string | Yes | Client-provided task ID. The server rejects requests without an `id`. | | `message` | [A2AMessage](#a2amessage) | Yes | Input message for the agent. | | `metadata` | map\[string]string | No | Arbitrary key-value pairs attached to the task. | **Result:** [A2ATask](#a2atask) **Example:** ```json { "jsonrpc": "2.0", "id": "req-1", "method": "tasks/send", "params": { "id": "task-001", "message": { "role": "user", "parts": [{"type": "text", "text": "Summarize this document"}] }, "metadata": { "source": "external-workflow" } } } ``` **Response:** ```json { "jsonrpc": "2.0", "id": "req-1", "result": { "id": "a2a-task-xyz", "status": { "state": "completed", "message": { "role": "agent", "parts": [{"type": "text", "text": "Here is the summary..."}] } }, "artifacts": [ { "name": "summary", "parts": [{"type": "text", "text": "..."}], "index": 0 } ] } } ``` #### tasks/get Retrieve the current state of an existing task. **Params:** | Field | Type | Required | Description | | ----- | ------ | -------- | -------------------- | | `id` | string | Yes | Task ID to retrieve. | **Result:** [A2ATask](#a2atask) **Example:** ```json { "jsonrpc": "2.0", "id": "req-2", "method": "tasks/get", "params": { "id": "a2a-task-xyz" } } ``` #### tasks/cancel Request cancellation of a running task. **Params:** | Field | Type | Required | Description | | -------- | ------ | -------- | -------------------- | | `id` | string | Yes | Task ID to cancel. | | `reason` | string | No | Cancellation reason. | **Result:** [A2ATask](#a2atask) with status state `canceled`. **Example:** ```json { "jsonrpc": "2.0", "id": "req-3", "method": "tasks/cancel", "params": { "id": "a2a-task-xyz", "reason": "No longer needed" } } ``` #### tasks/sendSubscribe Create a task and stream status updates via SSE. The initial HTTP response transitions to an SSE stream. **Params:** Same as [tasks/send](#taskssend). **Response:** Server-Sent Events stream with the following event types: | Event | Data | Description | | ------------- | --------------- | ------------------------------------------------------------------------------------ | | `status` | TaskResult JSON | Task state transition (full task result including `id`, `status`, `artifacts`, etc.) | | `: heartbeat` | *(comment)* | Keep-alive comment sent every 15 s; not a named event | The stream ends when the task reaches a terminal state (`completed`, `failed`, `canceled`, `rejected`). **Example SSE stream:** ``` event: status data: {"id": "a2a-task-xyz", "status": {"state": "working"}} : heartbeat event: status data: {"id": "a2a-task-xyz", "status": {"state": "completed", "message": {"role": "agent", "parts": [{"type": "text", "text": "Done."}]}}, "artifacts": []} ``` ### Error Codes JSON-RPC errors use standard codes plus A2A-specific extensions: | Code | Name | Description | | ------ | ---------------- | -------------------------------------------------------- | | -32700 | Parse error | Invalid JSON | | -32600 | Invalid request | Missing required fields | | -32601 | Method not found | Unknown method name | | -32602 | Invalid params | Invalid method parameters | | -32603 | Internal error | Server-side failure | | -32001 | Task not found | Referenced task ID does not exist | | -32002 | Task cancelled | Task has been cancelled | | -32003 | Agent not found | Target agent does not exist or A2A is not enabled for it | **Error response example:** ```json { "jsonrpc": "2.0", "id": "req-4", "error": { "code": -32001, "message": "task not found", "data": { "task_id": "a2a-task-unknown" } } } ``` ### Data Types #### A2ATask | Field | Type | Description | | ----------- | ------------------ | ---------------------------------------------------------- | | `id` | string | Unique task identifier | | `status` | A2AStatus | Current task status | | `artifacts` | \[]A2AArtifact | Output artifacts | | `history` | \[]A2AMessage | Message history (when `stateTransitionHistory` is enabled) | | `metadata` | map\[string]string | Task metadata | #### A2AStatus | Field | Type | Description | | --------- | ---------- | ----------------------------------------------------------------------------------------------- | | `state` | string | One of: `submitted`, `working`, `input-required`, `completed`, `failed`, `canceled`, `rejected` | | `message` | A2AMessage | Optional status message from the agent | #### A2AMessage | Field | Type | Description | | ------- | ---------- | --------------------- | | `role` | string | `user` or `agent` | | `parts` | \[]A2APart | Message content parts | #### A2APart | Field | Type | Description | | ---------- | ------ | ---------------------------------- | | `type` | string | Part type (e.g., `text`, `data`) | | `text` | string | Text content (when `type=text`) | | `data` | any | Structured data (when `type=data`) | | `metadata` | object | Additional part metadata | #### A2AArtifact | Field | Type | Description | | ------------- | ---------- | --------------------------- | | `name` | string | Artifact name | | `description` | string | Artifact description | | `parts` | \[]A2APart | Artifact content | | `index` | integer | Artifact index for ordering | ### Related * [A2A Interoperability](../concepts/a2a-interoperability.md) -- architecture and concepts * [Expose Agents via A2A](../guides/a2a-expose-agents.md) -- enable inbound A2A * [Use Remote A2A Agents](../guides/a2a-remote-agents.md) -- outbound A2A tools * [Agent Card](./resources/agent-card.md) -- card schema reference ## API Reference > **Stability: beta** -- This API surface ships with `orloj.dev/v1` and is suitable for production use, but may evolve with migration guidance in future minor releases. This page summarizes key HTTP endpoints and behavior contracts. ### Resource CRUD `/v1/` supports list/create and `/v1//{name}` supports get/update/delete for: * agents * agent-systems * model-endpoints * tools * secrets * sealed-secrets * memories * mcp-servers * agent-policies * agent-roles * tool-permissions * tool-approvals * task-approvals * tasks * task-schedules * task-webhooks * workers Namespace defaults to `default` and can be overridden with `?namespace=`. On create and update requests, `metadata.namespace` in the request body must match the effective request namespace (from `?namespace=` or the default). A mismatch returns `400 Bad Request`. When creating resources in a non-default namespace, pass `?namespace=` on the request URL and set the same value in the manifest body (or omit it and let the server apply the query namespace). Mutation requests (`POST`, `PUT`, `PATCH`) must use a supported `Content-Type` (`application/json`, `application/yaml`, `application/x-yaml`, or `text/yaml`). Other values return `415 Unsupported Media Type`. Requests with no `Content-Type` header are accepted for backward compatibility. `GET /v1/sealing-key/public` is cluster-scoped and returns the active public key used to create `SealedSecret` manifests. It returns `503` when no active sealing key is available. ### Capabilities * `GET /v1/capabilities` * returns deployment capability flags for feature discovery in UI/CLI integrations * extension providers may add capabilities without changing core API shape ### Authentication Endpoints * `GET /v1/auth/config` * returns auth mode and login/setup requirements; when mode is `native`, `setup_token_required` is true if the server was started with `ORLOJ_SETUP_TOKEN` (the UI shows a setup-token field; `POST /v1/auth/setup` must include `setup_token`) * `POST /v1/auth/setup` * one-time native admin bootstrap when auth mode is `native` * `POST /v1/auth/login` * local username/password login; sets session cookie * `POST /v1/auth/logout` * clears local session cookie * `GET /v1/auth/me` * returns current auth state and identity (`method`, `name`, `role`) for UI/CLI bootstrap * `POST /v1/auth/users` * admin-only native-auth endpoint; creates a local user and returns a generated password once * `GET /v1/auth/users` * admin-only native-auth endpoint; lists local users * `DELETE /v1/auth/users/{username}` * admin-only native-auth endpoint; deletes a local user (last-admin delete is blocked) * `POST /v1/auth/admin/reset-password` * admin-authenticated local password reset endpoint for a specific `username` * `POST /v1/tokens` * admin-only endpoint; creates a named API token and returns the token once * `GET /v1/tokens` * admin-only endpoint; lists store-managed tokens (`name`, `role`, `created_at`) * `DELETE /v1/tokens/{name}` * admin-only endpoint; revokes a store-managed token ### Status and Logs * `GET|PUT /v1//{name}/status` * `GET /v1/agents/{name}/logs` * `GET /v1/tasks/{name}/logs` ### Approval Decision Endpoints * `POST /v1/tool-approvals/{name}/approve` * `POST /v1/tool-approvals/{name}/deny` * `POST /v1/task-approvals/{name}/approve` * `POST /v1/task-approvals/{name}/deny` * `POST /v1/task-approvals/{name}/request-changes` Decision request bodies may include: * `decided_by`: reviewer identity * `comment`: reviewer note * `reason`: legacy alias for `comment` `TaskApproval request-changes` requires reviewer feedback via `comment` or the legacy `reason` field. It returns `409 Conflict` when the checkpoint has `allow_request_changes: false` or the approval has already reached `max_review_cycles`. `comment` is also supported on tool approval decisions for consistent reviewer audit trails. ### Watches and Events * `GET /v1/agents/watch` * `GET /v1/tasks/watch` * `GET /v1/task-schedules/watch` * `GET /v1/task-webhooks/watch` * `GET /v1/events/watch` ### Webhook Delivery * `POST /v1/webhook-deliveries/{endpoint_id}` * public ingress for `TaskWebhook` delivery * returns `202 Accepted` for accepted or duplicate deliveries * relies on webhook auth configuration for signature and idempotency validation #### Signature Profiles * `generic` * signature: HMAC-SHA256 over `timestamp + "." + rawBody` * headers: `X-Signature: sha256=`, `X-Timestamp`, `X-Event-Id` * `github` * signature: HMAC-SHA256 over raw body * headers: `X-Hub-Signature-256: sha256=`, `X-GitHub-Delivery` Both profiles support replay protection through timestamp skew and/or event-id dedupe checks. ### Memory Entries * `GET /v1/memories/{name}/entries` * query parameters: * `q` (string): search query. When provided, searches entries by keyword match (or vector similarity if the backend supports it). * `prefix` (string): filter entries by key prefix. Ignored when `q` is set. * `limit` (int): maximum number of entries to return. Defaults to `100`. * `namespace` (string): resource namespace. Defaults to `default`. * returns `{"entries": [{"key": "...", "value": "...", "score": 0.95}], "count": N}` * returns `404` if the Memory resource does not exist * returns an empty list if no persistent backend is registered for the Memory resource ### Task Observability Endpoints * `GET /v1/tasks/{name}/messages` * filters: `phase`, `from_agent`, `to_agent`, `branch_id`, `trace_id`, `limit` * `GET /v1/tasks/{name}/metrics` * includes totals and `per_agent`/`per_edge` rollups `Task.status.trace[]` may include normalized tool metadata: * `tool_contract_version` * `tool_request_id` * `tool_attempt` * `error_code` * `error_reason` * `retryable` ### Request and Response Examples #### Create a Resource ``` POST /v1/agents Content-Type: application/json ``` ```json { "apiVersion": "orloj.dev/v1", "kind": "Agent", "metadata": { "name": "research-agent", "namespace": "default" }, "spec": { "model_ref": "openai-default", "prompt": "You are a research assistant.", "tools": ["web_search"], "limits": { "max_steps": 6, "timeout": "30s" } } } ``` Response (`201 Created`): ```json { "apiVersion": "orloj.dev/v1", "kind": "Agent", "metadata": { "name": "research-agent", "namespace": "default", "resourceVersion": "1" }, "spec": { "...": "..." }, "status": { "phase": "Pending" } } ``` #### Get a Resource ``` GET /v1/agents/research-agent?namespace=default ``` Returns the full resource including `metadata`, `spec`, and `status`. #### Update a Resource ``` PUT /v1/agents/research-agent Content-Type: application/json If-Match: "1" ``` The request body must include the full resource. The `resourceVersion` (or `If-Match` header) must match the current version. Stale updates return `409 Conflict`. #### Delete a Resource ``` DELETE /v1/agents/research-agent?namespace=default ``` Returns `200 OK` on success. #### List Resources ``` GET /v1/agents?namespace=default ``` Returns an array of all resources of that type in the specified namespace. ##### Pagination All list endpoints support cursor-based pagination via query parameters: | Parameter | Description | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `limit` | Maximum number of items to return (1–1000). | | `after` | Cursor token from the previous page's `continue` field. Accepts scoped `namespace/name` tokens (preferred) or bare names (legacy; scoped to the request namespace). Returns items lexicographically after this cursor. | | `namespace` | Filter by namespace. | | `labelSelector` | Comma-separated `key=value` pairs; label filtering is applied before the page is finalized so each page contains up to `limit` matching items. | When more results are available, the response includes a `continue` field: ```json { "continue": "production/task-00042", "items": [ ... ] } ``` Pass `continue` as the `after` parameter in the next request to fetch the next page. When `continue` is absent or empty, there are no more results. Bare-name cursors from older clients (e.g. `"task-00042"`) are still accepted and scoped to the request namespace. Offset-based pagination (`?offset=N`) is supported for backward compatibility on the Tasks endpoint but is deprecated in favor of `?after=`. #### Watch Resources ``` GET /v1/agents/watch ``` Returns a server-sent event stream of resource changes. Events include the resource kind, name, and the change type (created, updated, deleted). Watch streams are bounded: * **Max duration:** 30 minutes per connection. The server sends a final `close` event and terminates the stream when the limit is reached. * **Connection limits:** concurrent watch connections are capped globally and per client IP. Excess connections receive `429 Too Many Requests` or `503 Service Unavailable`. ### Concurrency Semantics * `PUT` requires `metadata.resourceVersion` or `If-Match` * stale updates return `409 Conflict` ### A2A Endpoints AgentSystems opt in to inbound A2A with `spec.a2a.enabled: true`. Enabled systems are available through: * `GET /.well-known/agent-card.json` — root Agent Card when exactly one AgentSystem is A2A-enabled. * `GET /v1/agent-systems/{name}/.well-known/agent-card.json` — per-system Agent Card. * `POST /a2a` — shared JSON-RPC 2.0 endpoint for A2A task operations. * `POST /v1/agent-systems/{name}/a2a` — JSON-RPC endpoint scoped to one AgentSystem. * `GET /v1/a2a/agents` — registry listing A2A-enabled systems visible to the bearer token plus remote entries. See [A2A Interoperability](../concepts/a2a-interoperability.md) for protocol details. ### Related Docs * [Resource Reference](./resources/) * [CLI Reference](./cli.md) * [Tool](../concepts/tools/tool.md) * [Glossary](./glossary.md) ## CLI Reference `orlojctl` is the command-line interface for managing Orloj resources, running tasks, and inspecting system state. For server and worker daemon flags, see [Server Flags](./server-flags.md). For load-test and alert-check tools, see [Internal Tools](./internal-tools.md). > **Note:** When the [CRD operator](../deploy/kubernetes-operator.md) is deployed, `kubectl` can replace `orlojctl` for basic resource CRUD (`apply`, `get`, `delete`, `edit`, `diff`). Runtime operations (`run`, `cancel`, `approve`, `logs`, `trace`, `events`, etc.) and admin commands (`admin`, `auth`, `config`, `seal`, `eval`) remain `orlojctl`-only. See [kubectl vs orlojctl](../guides/kubectl-vs-orlojctl.md) for a detailed comparison. ### Usage patterns ```text orlojctl apply -f [--run] [--dry-run] [--namespace ] orlojctl validate -f orlojctl create secret --from-literal key=value [...] orlojctl create token --role orlojctl seal public-key orlojctl seal secret -f [--out ] [--stdout] orlojctl seal secret --from-literal key=value [...] [--out ] [--stdout] orlojctl approve tool-approval|task-approval [--decided-by ] [--comment ] orlojctl deny tool-approval|task-approval [--decided-by ] [--comment ] orlojctl request-changes task-approval --decided-by --comment orlojctl get [-w] [name] [-o table|json|yaml] orlojctl get tokens orlojctl get memory-entries [--query ] [--prefix

] [--limit ] orlojctl memory-entries [--query ] [--prefix

] [--limit ] orlojctl delete orlojctl delete token orlojctl describe orlojctl edit orlojctl diff -f [--namespace ] orlojctl wait / --for condition= [--timeout ] orlojctl cancel task [--reason ] orlojctl retry task [--with-overrides key=value ...] orlojctl top workers|tasks orlojctl run --system [key=value ...] orlojctl init [--blueprint pipeline|hierarchical|swarm-loop] orlojctl logs |task/ orlojctl trace task orlojctl graph system|task orlojctl events [filters...] orlojctl messages task/ [--agent ] [-o table|json|yaml] orlojctl metrics task/ [-o table|json|yaml] orlojctl health [-o table|json|yaml] orlojctl status [-o table|json|yaml] orlojctl completion bash|zsh|fish orlojctl auth whoami [--server URL] orlojctl auth login [-u ] [-p ] [--server URL] orlojctl admin create-user --role orlojctl admin list-users orlojctl admin delete-user orlojctl admin reset-password --username --new-password orlojctl config path|get|use |set-profile [--server URL] [--token value] [--token-env NAME] orlojctl eval run --dataset --system [--scoring ] [--concurrency ] orlojctl eval list [-o table|json|yaml] orlojctl eval get [-o table|json|yaml] orlojctl eval compare [run3 ...] orlojctl eval datasets [-o table|json|yaml] orlojctl eval export [--format csv|json] orlojctl eval annotate --sample --score [--pass] [--comment ] orlojctl eval import -f orlojctl eval finalize ``` ### Global Auth and Server Resolution * Global auth flag: `--api-token ` * Global namespace flag: `--namespace ` or `-n ` (sets the request namespace for API calls; must match `metadata.namespace` in manifest bodies on create/update) * Version command: `orlojctl version` (also `-version`, `--version`) * Token precedence: 1. `--api-token` 2. `ORLOJCTL_API_TOKEN` 3. `ORLOJ_API_TOKEN` 4. Active profile `token`, then `token_env` * Default server precedence when `--server` is omitted: 1. `ORLOJCTL_SERVER` 2. `ORLOJ_SERVER` 3. Active profile `server` 4. `http://127.0.0.1:8080` ### `orlojctl apply` | Flag | Default | Description | | ------------- | ------------------------- | ----------------------------------------------------------------------------------------------- | | `-f` | none | Path to a manifest file or directory (required). | | `--run` | `false` | Include runnable `Task` manifests when `-f` points to a directory. | | `--dry-run` | `false` | Preview create/update/no-op actions without persisting. | | `--namespace` | global namespace (if set) | Request namespace for applied manifests. Must match `metadata.namespace` in each manifest body. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | | `--server` | resolved server | API server URL. | When applying to a non-default namespace, pass `--namespace ` and ensure each manifest's `metadata.namespace` matches (or omit it and let the server apply the request namespace). A body/query mismatch returns `400 Bad Request` from the API. * **File:** applies that manifest. * **Directory:** walks recursively (skips `.git` dirs) and evaluates every `.yaml`, `.yml`, and `.json` file in sorted path order. * By default, runnable `Task` manifests (`spec.mode: run` or omitted mode) are skipped for safety. * `Task` manifests with `spec.mode: template` are always applied. * Pass `--run` to include runnable tasks during directory apply. * Failures are collected; the command exits with an error if any file failed. Behavior matrix: | Command | Runnable Task (`spec.mode: run` or omitted mode) | Template Task (`spec.mode: template`) | Other Kinds | | ------------------------------- | ------------------------------------------------ | ------------------------------------- | ----------- | | `orlojctl apply -f task.yaml` | Applied | Applied | Applied | | `orlojctl apply -f

` | Skipped | Applied | Applied | | `orlojctl apply -f --run` | Applied | Applied | Applied | ### `orlojctl validate` Parse and normalize manifests **offline** (no API server, no `orlojctl` config file required). Use in CI or before `apply` to catch schema and normalization errors early. | Flag | Default | Description | | ---- | ------- | -------------------------------------------------- | | `-f` | none | Path to a manifest file or a directory (required). | * **File:** validates that one manifest. * **Directory:** walks recursively (skips `.git` dirs) and validates every `.yaml`, `.yml`, and `.json` file. * **Exit code:** `0` if every file is valid; `1` if any file fails. Failed files are listed with path and error on stdout. Examples: ```bash orlojctl validate -f agent.yaml orlojctl validate -f ./manifests/ ``` ### `orlojctl create secret` | Flag | Default | Description | | ---------------- | --------------- | --------------------------------------------------- | | `--from-literal` | none | Repeatable `key=value` pair; at least one required. | | `--namespace` | `default` | Secret namespace. | | `-n` | `default` | Shorthand for `--namespace`. | | `--server` | resolved server | API server URL. | ### `orlojctl create token` | Flag | Default | Description | | ---------- | --------------- | ----------------------------------------------------------------- | | `--role` | none | Token role (`admin`, `writer`, `reader`, `controller`). Required. | | `--server` | resolved server | API server URL. | ### `orlojctl seal` Git-safe secret workflow commands: * `orlojctl seal public-key` -- fetch the active control-plane sealing public key from `GET /v1/sealing-key/public` * `orlojctl seal secret -f ` -- read a normal `Secret` manifest, fetch the active public key, and write `.sealed.yaml` by default * `orlojctl seal secret --from-literal key=value [...]` -- build a transient `Secret` locally, seal it, and write `.sealed.yaml` without creating an intermediate plaintext manifest `seal secret` does not talk to workers. The generated `SealedSecret` is later applied through the normal resource API. By default, `seal secret` writes YAML to a file: * with `-f secret.yaml`, the default output is `secret.sealed.yaml` next to the source file * with inline `--from-literal`, the default output is `.sealed.yaml` in the current directory Useful flags: | Flag | Default | Description | | -------------------- | ----------------------------- | --------------------------------------------------------------------------- | | `-f` | none | Path to an existing `Secret` manifest. | | `--from-literal` | none | Repeatable `key=value` pair used to build a transient `Secret` locally. | | `-o` / `--out` | auto-generated path | Explicit output path for the generated `SealedSecret` manifest. | | `--stdout` | `false` | Print the generated manifest to stdout instead of writing a file. | | `--format` | `yaml` | Output format: `yaml` or `json`. | | `--namespace` / `-n` | global namespace or `default` | Namespace override for sealed secrets generated from literals or manifests. | Examples: ```bash # Seal an existing Secret manifest into secret.sealed.yaml orlojctl seal secret -f secret.yaml # Seal literals directly into payment-gateway.sealed.yaml orlojctl seal secret payment-gateway \ --from-literal api_key=sk-prod-123 \ --from-literal org=acme # Keep stdout for scripting orlojctl seal secret -f secret.yaml --stdout ``` ### `orlojctl approve` / `orlojctl deny` Approves or denies a pending `ToolApproval` or `TaskApproval`: * `orlojctl approve tool-approval ...` * `orlojctl deny tool-approval ...` * `orlojctl approve task-approval ...` * `orlojctl deny task-approval ...` | Flag | Default | Description | | -------------- | ------------------------- | ----------------------------- | | `--server` | resolved server | API server URL. | | `--namespace` | global namespace (if set) | Optional namespace override. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | | `--decided-by` | empty | Decision actor identity. | | `--comment` | empty | Optional reviewer comment. | | `--reason` | empty | Legacy alias for `--comment`. | ### `orlojctl request-changes` Requests changes on a pending `TaskApproval` and reruns the producing agent with injected `review.*` context: * `orlojctl request-changes task-approval --decided-by reviewer@example.com --comment "Revise the disclaimer"` The command fails if the checkpoint disables `request_changes` or if the approval has already reached `max_review_cycles`. | Flag | Default | Description | | -------------- | ------------------------- | ---------------------------------------------------------------------- | | `--server` | resolved server | API server URL. | | `--namespace` | global namespace (if set) | Optional namespace override. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | | `--decided-by` | empty | Decision actor identity. | | `--comment` | empty | Required reviewer feedback unless you use the legacy `--reason` alias. | | `--reason` | empty | Legacy alias for `--comment`. | ### `orlojctl get` | Flag | Default | Description | | ------------- | ------------------------- | -------------------------------------------------- | | `--server` | resolved server | API server URL. | | `-w` | `false` | Watch mode (currently only supported for `tasks`). | | `-o` | `table` | Output format: `table`, `json`, `yaml`. | | `--namespace` | global namespace (if set) | Optional namespace override/filter. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | Supported resources: * `agents` * `agent-systems` * `model-endpoints` * `tools` * `secrets` * `sealed-secrets` * `memories` * `agent-policies` * `agent-roles` * `tool-permissions` * `tool-approvals` * `task-approvals` * `tasks` * `task-schedules` * `task-webhooks` * `workers` * `mcp-servers` * `eval-datasets` * `eval-runs` * `tokens` Notes: * `orlojctl get [name]` supports both list and single-resource fetch. * `orlojctl get memory-entries ...` delegates to memory entry inspection. Examples (MCP servers): ```bash # Apply an MCP server manifest orlojctl apply -f mcp-server.yaml # List all MCP servers orlojctl get mcp-servers # Get a specific MCP server orlojctl get mcp-server my-server # Delete an MCP server orlojctl delete mcp-server my-server ``` See the [Connect an MCP Server](../guides/connect-mcp-server.md) guide for full setup instructions. ### `orlojctl delete` | Flag | Default | Description | | ------------- | ------------------------- | ----------------------------------------------------- | | `--server` | resolved server | API server URL. | | `--namespace` | global namespace (if set) | Optional namespace override for namespaced resources. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | ### `orlojctl run` | Flag | Default | Description | | ------------- | ----------------------------------------- | ------------------------------------------------ | | `--system` | none | Target `AgentSystem` (required). | | `--server` | resolved server | API server URL. | | `--namespace` | global namespace (if set), else `default` | Task namespace. | | `-n` | global namespace (if set), else `default` | Shorthand for `--namespace`. | | `--poll` | `2s` | Poll interval while waiting for task completion. | | `--timeout` | `5m` | Max wait time for task completion. | Positional args after flags are parsed as `key=value` task input. ### `orlojctl events` | Flag | Default | Description | | ------------- | ------------------------- | --------------------------------------- | | `--server` | resolved server | API server URL. | | `--since` | `0` | Resume stream from event id. | | `--source` | empty | Filter by event source. | | `--type` | empty | Filter by event type. | | `--kind` | empty | Filter by resource kind. | | `--name` | empty | Filter by resource name. | | `--namespace` | global namespace (if set) | Filter by resource namespace. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | | `--once` | `false` | Exit after first matching event. | | `--timeout` | `0` | Max stream time (`0` means no timeout). | | `--raw` | `false` | Print raw event JSON payload. | ### `orlojctl memory-entries` Inspect stored entries for a `Memory` resource. | Flag | Default | Description | | ------------- | ------------------------- | --------------------------------------- | | `--server` | resolved server | API server URL. | | `--query` | empty | Semantic query (`q` parameter). | | `--prefix` | empty | Key prefix filter (`prefix` parameter). | | `--limit` | `100` | Max entries returned. | | `-o` | `table` | Output format: `table`, `json`, `yaml`. | | `--namespace` | global namespace (if set) | Optional namespace override. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | ### `orlojctl describe` Fetches a single resource and prints a human-readable summary plus YAML payload. | Flag | Default | Description | | ------------- | ------------------------- | --------------------------------------- | | `--server` | resolved server | API server URL. | | `--namespace` | global namespace (if set) | Optional namespace override. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | | `-o` | `table` | Output format: `table`, `json`, `yaml`. | ### `orlojctl edit` Fetches a resource, opens it in `$VISUAL`/`$EDITOR` (`vi` fallback), and applies the edited manifest on save. | Flag | Default | Description | | ------------- | ------------------------- | ---------------------------- | | `--server` | resolved server | API server URL. | | `--namespace` | global namespace (if set) | Optional namespace override. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | ### `orlojctl diff` Shows a unified diff between live state and the provided manifest(s), using normalized resource payloads (runtime status fields excluded). | Flag | Default | Description | | ------------- | ------------------------- | ---------------------------------------------------------------------------------------------- | | `-f` | none | Path to manifest file or directory (required). | | `--run` | `false` | Include runnable tasks when diffing directories. | | `--namespace` | global namespace (if set) | Request namespace for diffed manifests. Must match `metadata.namespace` in each manifest body. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | | `--server` | resolved server | API server URL. | ### `orlojctl wait` Polls a resource until a condition is met or timeout is reached. | Flag | Default | Description | | ------------- | ------------------------- | --------------------------------------------------------- | | `--for` | `condition=Complete` | Wait condition expression (`condition=`). | | `--timeout` | `5m` | Maximum wait time. | | `--interval` | `2s` | Poll interval. | | `--server` | resolved server | API server URL. | | `--namespace` | global namespace (if set) | Optional namespace override. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | Exit behavior: * Success when condition is satisfied. * Timeout exits with code `1`. * Invalid usage/request errors exit with code `2`. ### `orlojctl cancel task` Marks a non-terminal task as `Failed` through task status update. | Flag | Default | Description | | ------------- | ---------------------------- | --------------------------------------- | | `--server` | resolved server | API server URL. | | `--reason` | `task canceled via orlojctl` | Failure reason recorded on task status. | | `--namespace` | global namespace (if set) | Optional namespace override. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | ### `orlojctl retry task` Creates a new task from an existing terminal task spec. | Flag | Default | Description | | ------------------ | ------------------------- | --------------------------------------- | | `--server` | resolved server | API server URL. | | `--with-overrides` | none | Repeatable `key=value` input overrides. | | `--namespace` | global namespace (if set) | Optional namespace override. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | ### `orlojctl top` Quick operational summaries for task and worker state. | Flag | Default | Description | | ------------- | ------------------------- | --------------------------------------- | | `--server` | resolved server | API server URL. | | `-o` | `table` | Output format: `table`, `json`, `yaml`. | | `--namespace` | global namespace (if set) | Optional namespace override/filter. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | Targets: * `orlojctl top workers` * `orlojctl top tasks` ### `orlojctl messages` Inspect inter-agent task messages. | Flag | Default | Description | | ------------- | ------------------------- | ------------------------------------------------ | | `--server` | resolved server | API server URL. | | `--agent` | empty | Filter where `from_agent` or `to_agent` matches. | | `--phase` | empty | Lifecycle phase filter. | | `--limit` | `0` | Max messages returned (`0` = no limit). | | `-o` | `table` | Output format: `table`, `json`, `yaml`. | | `--namespace` | global namespace (if set) | Optional namespace override. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | Target forms: * `orlojctl messages task/` * `orlojctl messages task ` ### `orlojctl metrics` Inspect task message observability metrics. | Flag | Default | Description | | ------------- | ------------------------- | ------------------------------------------ | | `--server` | resolved server | API server URL. | | `--phase` | empty | Lifecycle phase filter. | | `--limit` | `0` | Max message samples used (`0` = no limit). | | `-o` | `table` | Output format: `table`, `json`, `yaml`. | | `--namespace` | global namespace (if set) | Optional namespace override. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | Target forms: * `orlojctl metrics task/` * `orlojctl metrics task ` ### `orlojctl health` Checks `/healthz`. | Flag | Default | Description | | ---------- | --------------- | --------------------------------------- | | `--server` | resolved server | API server URL. | | `-o` | `table` | Output format: `table`, `json`, `yaml`. | ### `orlojctl status` Composite status view using `/healthz`, `/v1/auth/config`, `/v1/capabilities`, `/v1/workers`, and `/v1/namespaces`. Table output includes `auth_mode`, `setup_required`, and `setup_token_required` (from auth config; the last is true when `ORLOJ_SETUP_TOKEN` is set on the server). JSON/YAML snapshots include `auth_setup_token_required`. | Flag | Default | Description | | ---------- | --------------- | --------------------------------------- | | `--server` | resolved server | API server URL. | | `-o` | `table` | Output format: `table`, `json`, `yaml`. | ### `orlojctl completion` Emits shell completion scripts. Usage: * `orlojctl completion bash` * `orlojctl completion zsh` * `orlojctl completion fish` ### `orlojctl logs` | Flag | Default | Description | | ---------- | --------------- | --------------- | | `--server` | resolved server | API server URL. | ### `orlojctl trace` | Flag | Default | Description | | ---------- | --------------- | --------------- | | `--server` | resolved server | API server URL. | ### `orlojctl graph` | Flag | Default | Description | | ---------- | --------------- | --------------- | | `--server` | resolved server | API server URL. | ### `orlojctl auth whoami` Returns the currently authenticated identity from `/v1/auth/me`. | Flag | Default | Description | | ---------- | --------------- | --------------- | | `--server` | resolved server | API server URL. | ### `orlojctl auth login` Authenticate against a native-mode server and save a bearer token to the active profile. Prompts interactively for username and password if not provided via flags. | Flag | Default | Description | | ------------------ | --------------- | ------------------------------------------------------------- | | `-u`, `--username` | (prompted) | Username to authenticate with. | | `-p`, `--password` | (prompted) | Password (prefer interactive prompt over flags for security). | | `--server` | resolved server | API server URL. | On success the minted token is written into the active profile's `token` field in `config.json`. Subsequent commands automatically use it. ### `orlojctl admin create-user` | Flag | Default | Description | | ---------- | --------------- | ------------------------------------------------------ | | `--role` | `reader` | User role (`admin`, `writer`, `reader`, `controller`). | | `--server` | resolved server | API server URL. | ### `orlojctl admin list-users` | Flag | Default | Description | | ---------- | --------------- | --------------- | | `--server` | resolved server | API server URL. | ### `orlojctl admin delete-user` | Flag | Default | Description | | ---------- | --------------- | --------------- | | `--server` | resolved server | API server URL. | ### `orlojctl admin reset-password` | Flag | Default | Description | | ---------------- | --------------- | --------------------------- | | `--server` | resolved server | API server URL. | | `--username` | none | Target username (required). | | `--new-password` | none | New password (required). | ### `orlojctl config set-profile` | Flag | Default | Description | | ------------- | ------- | -------------------------------------------------------- | | `--server` | empty | Profile API server URL. | | `--token` | empty | Profile bearer token (prefer `--token-env` for secrets). | | `--token-env` | empty | Env var name read at runtime for token value. | Other config subcommands: * `orlojctl config path`: print config file path * `orlojctl config get`: print current config/profile data with resolution sources (shows whether server/token come from env, profile, or defaults) * `orlojctl config use `: switch active profile (probes auth status on the target server) ### `orlojctl init` Positional argument `` is required. It sets both the output directory and the resource name prefix. | Flag | Default | Description | | ------------- | ---------- | ------------------------------------------------------------- | | `--blueprint` | `pipeline` | Blueprint topology: `pipeline`, `hierarchical`, `swarm-loop`. | ### `orlojctl eval` Evaluation framework commands for measuring and comparing agent system quality. See the [Agent Evaluation guide](../guides/run-agent-evaluation.md) for a full walkthrough. #### `orlojctl eval run` Creates an EvalRun and polls until completion. | Flag | Default | Description | | --------------- | ------------------------- | ----------------------------------------------------------------- | | `--dataset` | none | EvalDataset name (required). | | `--system` | none | AgentSystem name (required). | | `--scoring` | `exact_match` | Scoring strategy: `exact_match`, `llm_judge`, `manual`, `custom`. | | `--model-ref` | empty | Judge model for `llm_judge` scoring. | | `--rubric` | empty | Evaluation rubric for `llm_judge` scoring. | | `--concurrency` | `1` | Maximum parallel samples. | | `--timeout` | empty | Per-sample task timeout (e.g. `60s`, `5m`). | | `--server` | resolved server | API server URL. | | `--namespace` | global namespace (if set) | Namespace for the eval run. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | #### `orlojctl eval list` Lists all EvalRuns. | Flag | Default | Description | | ------------- | ------------------------- | --------------------------------------- | | `--server` | resolved server | API server URL. | | `-o` | `table` | Output format: `table`, `json`, `yaml`. | | `--namespace` | global namespace (if set) | Namespace filter. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | #### `orlojctl eval get` Gets a specific EvalRun by name. | Flag | Default | Description | | ------------- | ------------------------- | --------------------------------------- | | `--server` | resolved server | API server URL. | | `-o` | `table` | Output format: `table`, `json`, `yaml`. | | `--namespace` | global namespace (if set) | Namespace override. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | #### `orlojctl eval compare` Compares two or more completed EvalRuns side-by-side, showing pass rate, mean score, tokens, and latency. | Flag | Default | Description | | ---------- | --------------- | --------------- | | `--server` | resolved server | API server URL. | #### `orlojctl eval datasets` Lists all EvalDatasets. | Flag | Default | Description | | ------------- | ------------------------- | --------------------------------------- | | `--server` | resolved server | API server URL. | | `-o` | `table` | Output format: `table`, `json`, `yaml`. | | `--namespace` | global namespace (if set) | Namespace filter. | | `-n` | global namespace (if set) | Shorthand for `--namespace`. | #### `orlojctl eval export` Exports an EvalRun's results for review. | Flag | Default | Description | | ---------- | --------------- | ------------------------------- | | `--format` | `json` | Export format: `json` or `csv`. | | `--server` | resolved server | API server URL. | #### `orlojctl eval annotate` Annotates a single sample result in a PendingReview run. | Flag | Default | Description | | ----------- | --------------- | --------------------------------- | | `--sample` | none | Sample name (required). | | `--score` | none | Numeric score 0.0–1.0 (required). | | `--pass` | `false` | Mark the sample as passed. | | `--comment` | empty | Reviewer comment / reasoning. | | `--server` | resolved server | API server URL. | #### `orlojctl eval import` Bulk-imports annotations from a CSV file into a PendingReview run. | Flag | Default | Description | | ---------- | --------------- | ---------------------------------------------------------------------------------- | | `-f` | none | Path to CSV file (required). Columns: `sample_name`, `score`, `pass`, `reasoning`. | | `--server` | resolved server | API server URL. | #### `orlojctl eval finalize` Finalizes a PendingReview run, computing aggregate summary metrics and transitioning it to Succeeded. | Flag | Default | Description | | ---------- | --------------- | --------------- | | `--server` | resolved server | API server URL. | ### Command Discovery Use help output as the authoritative source for your current build: ```bash orlojctl help go run ./cmd/orlojctl help ``` ### Related * [Remote CLI & API Access](../deploy/remote-cli-access.md) — server setup, tokens, and profiles * [Server Flags](./server-flags.md) — orlojd and orlojworker daemon flags * [API Reference](./api.md) — REST API reference ## Glossary Canonical definitions for terms used throughout Orloj documentation. ### A **Agent** A declarative unit of work backed by a language model. Defined as a resource with a prompt, model configuration, tool bindings, role assignments, and execution limits. See [Agents and Agent Systems](../concepts/agents/agent.md). **Agent System** A composition of multiple agents wired into a directed graph. The graph defines how messages flow between agents during task execution. Supports pipeline, hierarchical, and swarm-loop topologies. See [Agents and Agent Systems](../concepts/agents/agent-system.md). **Agent Policy** A governance resource that constrains agent execution. Can restrict allowed models, block specific tools, and cap token usage. Policies may be scoped to specific systems/tasks or applied globally. See [Governance and Policies](../concepts/governance/agent-policy.md). **Agent Role** A named set of permission strings that can be bound to agents. Agents accumulate the union of permissions from all bound roles. See [Governance and Policies](../concepts/governance/agent-role.md). ### B **Blueprint** A ready-to-use template combining agents, an agent system, and a task for a specific orchestration pattern (pipeline, hierarchical, or swarm-loop). Available in `examples/blueprints/`. See [Starter Blueprints](../guides/starter-blueprints.md). ### C **Server** The management layer of Orloj, running as `orlojd`. Includes the API server, resource store, background services, and task scheduler. See [Architecture Overview](../concepts/architecture.md). **Resource Definition** A typed, declarative schema. Orloj resources use standard `apiVersion`, `kind`, `metadata`, `spec`, and `status` fields. See [Resource Reference](./resources/). ### D **Dead Letter** A terminal state for tasks or messages that have exhausted all retry attempts. Dead-lettered items require manual investigation. Tasks transition `Failed -> DeadLetter` after all retries are consumed. ### E **Edge** A directional connection between two agents in an AgentSystem graph. Edges define message routing. The `edges[]` field supports fan-out (multiple targets) and metadata annotations via labels and policy. **EvalDataset** A resource containing a list of (input, expected output) sample pairs with optional scoring rubrics. Datasets are referenced by EvalRun resources to drive agent evaluations. See [Agent Evaluation](../concepts/evaluation/). **EvalRun** A resource that executes all samples in an EvalDataset against an AgentSystem, scores the results, and produces aggregate metrics (pass rate, mean score, latency, tokens). Supports `exact_match`, `llm_judge`, `manual`, and `custom` scoring strategies. See [Agent Evaluation](../concepts/evaluation/). ### F **Fan-in** A graph pattern where multiple upstream branches converge on a single downstream node. Controlled by join gates with `wait_for_all` or `quorum` modes. See [Execution and Messaging](../concepts/execution-model.md). **Fan-out** A graph pattern where a single node routes messages to multiple downstream targets simultaneously. ### G **Governance** The authorization and policy enforcement layer. Composed of AgentPolicy, AgentRole, and ToolPermission resources. Governance is fail-closed: unauthorized actions are denied, not silently ignored. See [Governance and Policies](../concepts/governance/). ### J **Join Gate** A fan-in mechanism on an AgentSystem graph node. Modes: `wait_for_all` (wait for every upstream branch) or `quorum` (wait for a count/percentage). Configurable failure policy: `deadletter`, `skip`, or `continue_partial`. ### L **Lease** A time-bounded claim on a task held by a worker. Workers renew leases via heartbeats during execution. If a lease expires (worker crash, network partition), another worker may safely take over the task. ### M **Memory** A resource that configures a persistent memory backend for agents. Agents attach a Memory resource via `spec.memory.ref`, and may explicitly grant built-in memory operations with `spec.memory.allow` (`read`, `write`, `search`, `list`, `ingest`). Configured with a type, provider (e.g. `in-memory`, `pgvector`), and optional embedding model. Memory operates in three layers: conversation history (per-activation), task-scoped shared store (per-task), and persistent backends (cross-task). See [Memory](../concepts/memory/index.md). **Memory Tool** One of five built-in runtime tools that can be exposed when an agent both references a Memory resource and explicitly allows the corresponding operation: `memory.read`, `memory.write`, `memory.search`, `memory.list`, and `memory.ingest`. These are handled internally by the runtime without network calls. See [Memory](../concepts/memory/index.md). **Message Bus** The transport layer for agent-to-agent communication within a task. Implementations: `memory` (in-process) and `nats-jetstream` (durable). Messages carry lifecycle phase, retry state, and routing metadata. **Model Endpoint** A resource that configures a connection to a model provider. Declares the provider type, base URL, default model, provider-specific options, and auth credentials. Agents reference endpoints by name via `model_ref`. See [Model Routing](../concepts/tools/model-endpoint.md). **Model Gateway** The worker component that routes model requests to the appropriate provider based on agent configuration. Handles provider-specific request formatting and response parsing. ### N **Namespace** A scope for resource names. Defaults to `default`. Resources can reference cross-namespace targets using `namespace/name` syntax. ### R **Reconciliation** The process by which background services observe the current state of a resource and take actions to move it toward the desired state declared in `spec`. ### S **Secret** A resource for storing sensitive values (API keys, tokens). `stringData` values are base64-encoded into `data` during normalization and then cleared. The runtime reads from `data` at execution time. ### T **Task** A request to execute an AgentSystem with specific input. Tasks move through phases: `Pending -> Running -> Succeeded | Failed | DeadLetter`. See [Tasks and Scheduling](../concepts/tasks/task.md). **Task Schedule** A resource that creates tasks on a cron-based schedule from a template task. Supports timezone configuration, concurrency policy, and history limits. See [Tasks and Scheduling](../concepts/tasks/task-schedule.md). **Task Webhook** A resource that creates tasks in response to external HTTP events. Supports signature verification (generic and GitHub profiles) and idempotency-based deduplication. See [Tasks and Scheduling](../concepts/tasks/task-webhook.md). **Tool** An external capability that agents can invoke during execution. Defined as a resource with endpoint, auth, risk level, and runtime configuration (isolation, timeout, retry). See [Tools and Isolation](../concepts/tools/tool.md). **Tool Contract v1** The standardized JSON request/response envelope that all tools must implement. Defines the error taxonomy (`tool_code`, `tool_reason`, `retryable`) used by the runtime for retry decisions. See [Tool](../concepts/tools/tool.md). **Tool Permission** A governance resource that defines what permissions are required to invoke a specific tool. Checked against the agent's accumulated role permissions at execution time. See [Governance and Policies](../concepts/governance/tool-permission.md). ### W **Worker** An execution unit that claims and runs tasks. Workers register capabilities (region, GPU, supported models) and the scheduler uses these for task matching. Runs as `orlojworker`. See [Tasks and Scheduling](../concepts/infrastructure/worker.md) and [Architecture Overview](../concepts/architecture.md). ## Reference Detailed schemas and interfaces for API consumers and platform engineers. * [CLI Reference](./cli.md) * [API Reference](./api.md) * [Resource Reference](./resources/) * [Glossary](./glossary.md) ## Internal Tools Flag reference for `orloj-loadtest` (reliability/load harness) and `orloj-alertcheck` (alert profile evaluator). These are operational and CI tools, not user-facing CLIs. ### `orloj-loadtest` Print full flags: ```bash go run ./cmd/orloj-loadtest -h ``` | Flag | Default | Description | Condition / Notes | | | | ------------------------------ | -------------------------- | ------------------------------------------------------- | ------------------- | ---- | -------- | | `--base-url` | `http://127.0.0.1:8080` | Orloj API base URL. | n/a | | | | `--namespace` | `default` | Target namespace. | n/a | | | | `--tasks` | `50` | Number of tasks to create. | n/a | | | | `--create-concurrency` | `10` | Concurrent task-create workers. | n/a | | | | `--poll-concurrency` | `20` | Concurrent status-poll workers. | n/a | | | | `--poll-interval` | `500ms` | Poll interval for task status. | n/a | | | | `--run-timeout` | `5m` | Global run timeout. | n/a | | | | `--task-system` | `report-system` | AgentSystem for generated tasks. | n/a | | | | `--topic-prefix` | `loadtest-topic` | Task input topic prefix. | n/a | | | | `--task-priority` | `high` | Task priority. | n/a | | | | `--task-retry-attempts` | `3` | Generated `Task.spec.retry.max_attempts`. | n/a | | | | `--task-retry-backoff` | `2s` | Generated `Task.spec.retry.backoff`. | n/a | | | | `--message-retry-attempts` | `4` | Generated `Task.spec.message_retry.max_attempts`. | n/a | | | | `--message-retry-backoff` | `200ms` | Generated `Task.spec.message_retry.backoff`. | n/a | | | | `--message-retry-max-backoff` | `2s` | Generated `Task.spec.message_retry.max_backoff`. | n/a | | | | `--message-retry-jitter` | `full` | Generated `Task.spec.message_retry.jitter`. | \`none | full | equal\`. | | `--setup` | `true` | Apply baseline manifests before load run. | n/a | | | | `--min-ready-workers` | `2` | Minimum ready workers required before run. | `0` disables check. | | | | `--worker-ready-timeout` | `45s` | Max wait for worker readiness check. | n/a | | | | `--inject-invalid-system-rate` | `0` | Fraction routed to invalid system. | Injection control. | | | | `--invalid-system-name` | `missing-system-loadtest` | Invalid system name used for injection. | Injection control. | | | | `--inject-timeout-system-rate` | `0` | Fraction routed to timeout system. | Injection control. | | | | `--timeout-system-name` | `loadtest-timeout-system` | Timeout system name for injection. | Injection control. | | | | `--timeout-agent-name` | `loadtest-timeout-agent` | Timeout-agent name in injected system. | Injection control. | | | | `--timeout-agent-duration` | `1ms` | Timeout used by injected agent limits. | Injection control. | | | | `--inject-expired-lease-rate` | `0` | Fraction patched for expired-lease takeover simulation. | Injection control. | | | | `--expired-lease-owner` | `worker-crashed-simulated` | Synthetic owner ID used for expired-lease simulation. | Injection control. | | | | `--quality-profile` | empty | Optional JSON profile for quality gates. | n/a | | | | `--min-success-rate` | `0` | Minimum success-rate gate. | `0` disables. | | | | `--max-deadletter-rate` | `-1` | Maximum deadletter-rate gate. | `-1` disables. | | | | `--max-failed-rate` | `-1` | Maximum failed-rate gate. | `-1` disables. | | | | `--max-timed-out` | `0` | Maximum timed-out task count gate. | `-1` disables. | | | | `--min-retry-total` | `-1` | Minimum total retry-count gate. | `-1` disables. | | | | `--min-takeover-events` | `-1` | Minimum takeover-history event-count gate. | `-1` disables. | | | | `--json` | `false` | Emit machine-readable JSON report. | n/a | | | | `--verbose` | `false` | Print periodic progress. | n/a | | | *** ### `orloj-alertcheck` Print full flags: ```bash go run ./cmd/orloj-alertcheck -h ``` | Flag | Default | Description | | -------------------- | ------------------------------------------------- | --------------------------------------------------------------------- | | `--base-url` | `http://127.0.0.1:8080` | Orloj API base URL. | | `--namespace` | `default` | Target namespace. | | `--api-token` | empty | Optional bearer token for API auth (env fallback: `ORLOJ_API_TOKEN`). | | `--profile` | `monitoring/alerts/retry-deadletter-default.json` | Alert threshold profile JSON file. | | `--task-name-prefix` | empty | Optional task metadata.name prefix filter. | | `--task-system` | empty | Optional `Task.spec.system` filter. | | `--poll-concurrency` | `20` | Concurrent task metrics fetch workers. | | `--timeout` | `2m` | Global command timeout. | | `--json` | `true` | Emit JSON output. | | `--verbose` | `false` | Emit verbose progress logs. | ### Command Discovery Use help output as the authoritative source for your current build: ```bash go run ./cmd/orloj-loadtest -h go run ./cmd/orloj-alertcheck -h ``` ### Related * [Monitoring & Alerts](../operations/monitoring-alerts.md) — alert profiles and dashboards * [Server Flags](./server-flags.md) — orlojd and orlojworker daemon flags ## Server Flags Flag reference for the `orlojd` (API server) and `orlojworker` (task worker) daemon binaries. Both binaries share the same flag groups for **tool isolation**, **model secret resolution**, and **message bus** configuration. Flags that differ between the two are noted in the Condition / Notes column. See [Configuration](../operations/configuration.md) for the full environment-variable matrix and precedence rules. ### `orlojd` Print full flags: ```bash go run ./cmd/orlojd -h ``` #### Core, auth, and storage | Flag | Default | Description | Condition / Notes | | | | | ------------------------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ----------------------------------------------- | --------------------------------- | | `--version` | `false` | Print version and exit. | n/a | | | | | `--log-level` | `info` | Minimum log level. | \`debug | info | warn | error`; env `ORLOJ\_LOG\_LEVEL\`. | | `--debug` | `false` | Enable debug logging. | Equivalent to `--log-level=debug`; takes precedence over `--log-level`. | | | | | `--addr` | `:8080` | Server listen address. | n/a | | | | | `--ui-path` | `/` | Base URL path for the web console. | Env fallback: `ORLOJ_UI_PATH`. Set to a subpath (e.g. `/console/`) when sharing a hostname via reverse proxy. | | | | | `--cors-allowed-origins` | empty | Comma-separated CORS allowed origins. | Env fallback: `ORLOJ_CORS_ALLOWED_ORIGINS`. Empty means same-origin only. | | | | | `--tls-cert-file` | empty | TLS certificate file for HTTPS. | Env fallback: `ORLOJ_TLS_CERT_FILE`. Requires `--tls-key-file`. | | | | | `--tls-key-file` | empty | TLS private key file for HTTPS. | Env fallback: `ORLOJ_TLS_KEY_FILE`. Requires `--tls-cert-file`. | | | | | `--api-key` | empty | Bearer token auth key. | Env fallback: `ORLOJ_API_TOKEN`; see also `ORLOJ_API_TOKENS`. Prefer env over flag (flag values are visible in process listings). | | | | | `--auth-mode` | `off` | API auth mode. | \`off | native | sso` (`sso\` unavailable in this distribution). | | | `--auth-session-ttl` | `24h` | Session TTL for local auth mode. | Env fallback: `ORLOJ_AUTH_SESSION_TTL`. | | | | | `--auth-reset-admin-username` | empty | One-shot admin reset username. | Env fallback: `ORLOJ_AUTH_RESET_ADMIN_USERNAME`. | | | | | `--auth-reset-admin-password` | empty | One-shot admin reset password and exit. | Env fallback: `ORLOJ_AUTH_RESET_ADMIN_PASSWORD`. Prefer env over flag. | | | | | `--trusted-proxies` | empty | Comma-separated CIDRs of reverse proxies whose `X-Forwarded-For` / `X-Real-IP` headers are trusted for client IP extraction. | Env fallback: `ORLOJ_TRUSTED_PROXIES`. Required for correct per-client auth rate limiting behind a proxy. See [Security — Trusted proxy configuration](../operations/security.md#trusted-proxy-configuration). | | | | | `--secret-encryption-key` | empty | AES-256-GCM key for Secret encryption at rest. | Env fallback: `ORLOJ_SECRET_ENCRYPTION_KEY`. Prefer env over flag. On `orlojd`, also wraps the DB-stored `SealedSecret` private key. | | | | | `--storage-backend` | `memory` | State backend. | \`memory | postgres\`. | | | | `--postgres-dsn` | empty | Postgres DSN. | Required when `--storage-backend=postgres`; env `ORLOJ_POSTGRES_DSN`. | | | | | `--sql-driver` | `pgx` | `database/sql` driver for Postgres backend. | Postgres backend only. | | | | | `--postgres-max-open-conns` | `20` | Max open Postgres connections. | Postgres backend only. | | | | | `--postgres-max-idle-conns` | `10` | Max idle Postgres connections. | Postgres backend only. | | | | | `--postgres-conn-max-lifetime` | `30m` | Max Postgres connection lifetime. | Postgres backend only. | | | | #### A2A protocol | Flag | Default | Description | Condition / Notes | | -------------------------------- | ------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | `--a2a-public-base-url` | empty | Public base URL for Agent Card `url` fields. | Env `ORLOJ_A2A_PUBLIC_BASE_URL`. Required for externally-reachable Agent Cards. | | `--a2a-protocol-version` | empty | A2A protocol version to advertise. | Env `ORLOJ_A2A_PROTOCOL_VERSION`. | | `--a2a-card-cache-ttl` | `5m` | TTL for cached remote Agent Cards. | Env `ORLOJ_A2A_CARD_CACHE_TTL`. | | `--a2a-allow-private-endpoints` | `false` | Allow outbound A2A requests to private/loopback IPs. | Env `ORLOJ_A2A_ALLOW_PRIVATE_ENDPOINTS`. See [Security — A2A Security](../operations/security.md#a2a-security). | | `--a2a-remote-agents` | empty | JSON-encoded list of static remote A2A agents. | Env `ORLOJ_A2A_REMOTE_AGENTS`. | | `--a2a-rate-limit-enabled` | `true` | Enable per-IP rate limiting for A2A endpoints. | Env `ORLOJ_A2A_RATE_LIMIT_ENABLED`. | | `--a2a-rate-limit-rpm` | `30` | Max A2A JSON-RPC requests per minute per IP. | Env `ORLOJ_A2A_RATE_LIMIT_RPM`. | | `--a2a-rate-limit-max-subscribe` | `10` | Max concurrent SSE subscribe connections globally (server-wide). | Env `ORLOJ_A2A_RATE_LIMIT_MAX_SUBSCRIBE`. | #### CRD conflict policy | Flag | Default | Description | Condition / Notes | | | | ----------------------- | ------- | -------------------------------------------------------------- | ----------------- | ---- | -------------------------------------------------------------------------------------------------- | | `--crd-conflict-policy` | `warn` | How `orlojd` handles REST API writes to CRD-managed resources. | \`off | warn | reject`; env `ORLOJ\_CRD\_CONFLICT\_POLICY\`. Only relevant when the CRD operator is also running. | Modes: * **`off`** — No conflict detection. REST writes proceed normally even if the resource is CRD-managed. * **`warn`** (default) — REST writes succeed, but `orlojd` logs a warning and sets the `X-Orloj-CRD-Managed: true` response header. The operator will overwrite the change on its next reconcile. * **`reject`** — REST writes to CRD-managed resources return `409 Conflict` with a message directing the user to update via `kubectl apply` or Git. See [Kubernetes CRD Operator](../deploy/kubernetes-operator.md) for full operator documentation. #### Task execution and embedded worker | Flag | Default | Description | Condition / Notes | | | ---------------------------------------- | ----------------- | ----------------------------------------- | ----------------------------------------------------------- | ----------------------------------------------------- | | `--reconcile-interval` | `2s` | Agent reconcile interval. | n/a | | | `--task-execution-mode` | `sequential` | Task execution mode. | \`sequential | message-driven`; env `ORLOJ\_TASK\_EXECUTION\_MODE\`. | | `--run-task-worker` | `false` | Run embedded task worker in `orlojd`. | Alias exists: `--embedded-worker`. | | | `--embedded-worker` | `false` | Alias for `--run-task-worker`. | n/a | | | `--task-worker-id` | `embedded-worker` | Embedded worker identity. | n/a | | | `--task-worker-region` | `default` | Embedded worker region. | Env fallback: `ORLOJ_TASK_WORKER_REGION`. | | | `--embedded-worker-max-concurrent-tasks` | `1` | Embedded worker max concurrent tasks. | Env fallback: `ORLOJ_EMBEDDED_WORKER_MAX_CONCURRENT_TASKS`. | | | `--task-lease-duration` | `30s` | Embedded worker task lease duration. | Embedded worker only. | | | `--task-heartbeat-interval` | `10s` | Embedded worker lease heartbeat interval. | Embedded worker only. | | #### Event bus and runtime message bus | Flag | Default | Description | Condition / Notes | | | | -------------------------------- | ----------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------------ | ----------------------------------------- | ------------------------------------------------------------ | | `--event-bus-backend` | `memory` | Control-plane event bus backend. | \`memory | nats`; env `ORLOJ\_EVENT\_BUS\_BACKEND\`. | | | `--nats-url` | `nats://127.0.0.1:4222` | NATS URL for control-plane event bus. | Used when `--event-bus-backend=nats`; env `ORLOJ_NATS_URL`. | | | | `--nats-subject-prefix` | `orloj.controlplane` | NATS subject prefix for control-plane events. | NATS event bus only; env `ORLOJ_NATS_SUBJECT_PREFIX`. | | | | `--agent-message-bus-backend` | `none` | Runtime agent message bus backend. | \`none | memory | nats-jetstream`; env `ORLOJ\_AGENT\_MESSAGE\_BUS\_BACKEND\`. | | `--agent-message-nats-url` | `nats://127.0.0.1:4222` | NATS URL for runtime agent messages. | Used when `nats-jetstream`; env `ORLOJ_AGENT_MESSAGE_NATS_URL` (falls back to `ORLOJ_NATS_URL`). | | | | `--agent-message-subject-prefix` | `orloj.agentmsg` | Subject prefix for runtime agent messages. | Env `ORLOJ_AGENT_MESSAGE_SUBJECT_PREFIX`. | | | | `--agent-message-stream-name` | `ORLOJ_AGENT_MESSAGES` | JetStream stream name for runtime messages. | Env `ORLOJ_AGENT_MESSAGE_STREAM`. | | | | `--agent-message-history-max` | `2048` | In-memory runtime message history capacity. | In-memory runtime message backend behavior. | | | | `--agent-message-dedupe-window` | `2m` | In-memory runtime message dedupe window. | In-memory runtime message backend behavior. | | | #### Model secret resolution | Flag | Default | Description | Condition / Notes | | --------------------------- | --------------- | -------------------------------------------- | ---------------------------------------------- | | `--model-secret-env-prefix` | `ORLOJ_SECRET_` | Env prefix for model `secretRef` resolution. | Env fallback: `ORLOJ_MODEL_SECRET_ENV_PREFIX`. | Model routing (provider, base URL, default model, API key, timeout) is configured exclusively via **ModelEndpoint** resources. Agents reference endpoints through `spec.model_ref`. See [Configure Model Routing](../guides/configure-model-routing.md). #### Tool isolation runtime | Flag | Default | Description | Condition / Notes | | | --------------------------------- | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | --------------------------------------------------- | | `--tool-isolation-backend` | `none` | Container isolation backend for tool sandboxing. | \`none | container`; env `ORLOJ\_TOOL\_ISOLATION\_BACKEND\`. | | `--tool-container-runtime` | `docker` | Container runtime binary. | Container backend; env `ORLOJ_TOOL_CONTAINER_RUNTIME`. | | | `--tool-container-image` | `curlimages/curl:8.8.0` | Container image for isolated tool calls. | Container backend; env `ORLOJ_TOOL_CONTAINER_IMAGE`. | | | `--tool-container-network` | `none` | Container network mode. | Container backend; env `ORLOJ_TOOL_CONTAINER_NETWORK`. | | | `--tool-container-memory` | `128m` | Default container memory limit. Per-tool `spec.cli.resources.memory` and per-McpServer `spec.resources.memory` take precedence when set. | Container backend; env `ORLOJ_TOOL_CONTAINER_MEMORY`. | | | `--tool-container-cpus` | `0.50` | Default container CPU limit. Per-tool `spec.cli.resources.cpus` and per-McpServer `spec.resources.cpus` take precedence when set. | Container backend; env `ORLOJ_TOOL_CONTAINER_CPUS`. | | | `--tool-container-pids-limit` | `64` | Default container PID limit. Per-tool `spec.cli.resources.pids_limit` and per-McpServer `spec.resources.pids_limit` take precedence when set. | Container backend; env `ORLOJ_TOOL_CONTAINER_PIDS_LIMIT`. | | | `--tool-container-user` | `65532:65532` | Container user. | Container backend; env `ORLOJ_TOOL_CONTAINER_USER`. | | | `--tool-container-max-memory` | empty | Operator ceiling for per-tool/McpServer `resources.memory`. Empty means unbounded. Manifests exceeding this are rejected at apply time. | `orlojd` only; env `ORLOJ_TOOL_CONTAINER_MAX_MEMORY`. | | | `--tool-container-max-cpus` | empty | Operator ceiling for per-tool/McpServer `resources.cpus`. Empty means unbounded. | `orlojd` only; env `ORLOJ_TOOL_CONTAINER_MAX_CPUS`. | | | `--tool-container-max-pids-limit` | `0` | Operator ceiling for per-tool/McpServer `resources.pids_limit`. 0 means unbounded. | `orlojd` only; env `ORLOJ_TOOL_CONTAINER_MAX_PIDS_LIMIT`. | | | `--tool-secret-env-prefix` | `ORLOJ_SECRET_` | Env prefix for tool `secretRef` resolution. | Env fallback: `ORLOJ_TOOL_SECRET_ENV_PREFIX`. | | | `--tool-wasm-module` | empty | Default WASM module path (per-tool `spec.wasm.module` takes precedence). | Always available; env `ORLOJ_TOOL_WASM_MODULE`. | | | `--tool-wasm-entrypoint` | `run` | Default WASM entrypoint function. | Always available; env `ORLOJ_TOOL_WASM_ENTRYPOINT`. | | | `--tool-wasm-memory-bytes` | `67108864` | Default max WASM memory bytes. | Always available; env `ORLOJ_TOOL_WASM_MEMORY_BYTES`. | | | `--tool-wasm-fuel` | `1000000` | Default WASM execution fuel limit. | Always available; env `ORLOJ_TOOL_WASM_FUEL`. | | | `--tool-wasm-wasi` | `true` | Default: enable WASI host functions. | Always available; env `ORLOJ_TOOL_WASM_WASI`. | | | `--tool-wasm-cache-dir` | `~/.orloj/wasm-cache` | Disk cache directory for remote WASM modules (HTTPS/OCI). | Always available; env `ORLOJ_TOOL_WASM_CACHE_DIR`. | | | `--tool-k8s-enabled` | `false` | Enable Kubernetes tool isolation runtime. | Env `ORLOJ_TOOL_K8S_ENABLED`. When true, tools with `isolation_mode: kubernetes` run as K8s Jobs. | | | `--tool-k8s-namespace` | pod namespace or `default` | Namespace for tool Jobs. | Env `ORLOJ_TOOL_K8S_NAMESPACE`. | | | `--tool-k8s-service-account` | empty | Service account for tool Pods. | Env `ORLOJ_TOOL_K8S_SERVICE_ACCOUNT`. | | | `--tool-k8s-job-ttl` | `300` | TTL seconds after Job finishes (`ttlSecondsAfterFinished`). | Env `ORLOJ_TOOL_K8S_JOB_TTL`. | | | `--tool-k8s-default-image` | `curlimages/curl:8.8.0` | Fallback image for HTTP tools without an explicit image. | Env `ORLOJ_TOOL_K8S_DEFAULT_IMAGE`. | | #### Agent Kubernetes execution | Flag | Default | Description | Condition / Notes | | ----------------------------- | -------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | `--agent-k8s-enabled` | `false` | Run agents as ephemeral K8s Jobs. | Env `ORLOJ_AGENT_K8S_ENABLED`. Agents with Docker-dependent tools fall back to in-process. | | `--agent-k8s-namespace` | pod namespace or `default` | Namespace for agent Jobs. | Env `ORLOJ_AGENT_K8S_NAMESPACE`. | | `--agent-k8s-service-account` | empty | Service account for agent Pods. | Env `ORLOJ_AGENT_K8S_SERVICE_ACCOUNT`. | | `--agent-k8s-image` | own image | Container image for agent Jobs. | Env `ORLOJ_AGENT_K8S_IMAGE`. Defaults to the running binary's own image. | | `--agent-k8s-job-ttl` | `600` | TTL seconds after Job finishes (`ttlSecondsAfterFinished`). | Env `ORLOJ_AGENT_K8S_JOB_TTL`. | | `--agent-k8s-default-memory` | `512Mi` | Default memory limit for agent Pods. | Env `ORLOJ_AGENT_K8S_DEFAULT_MEMORY`. | | `--agent-k8s-default-cpu` | `500m` | Default CPU limit for agent Pods. | Env `ORLOJ_AGENT_K8S_DEFAULT_CPU`. | *** ### `orlojworker` Print full flags: ```bash go run ./cmd/orlojworker -h ``` #### Core, storage, and identity | Flag | Default | Description | Condition / Notes | | | | | ------------------------------ | ---------- | ---------------------------------------------- | ----------------------------------------------------------------------------------------------- | --------- | ---- | --------------------------------- | | `--version` | `false` | Print version and exit. | n/a | | | | | `--log-level` | `info` | Minimum log level. | \`debug | info | warn | error`; env `ORLOJ\_LOG\_LEVEL\`. | | `--debug` | `false` | Enable debug logging. | Equivalent to `--log-level=debug`; takes precedence over `--log-level`. | | | | | `--worker-id` | `worker-1` | Worker identity. | n/a | | | | | `--healthz-addr` | empty | Optional `/healthz` listener address. | Empty disables; env `ORLOJ_WORKER_HEALTHZ_ADDR`. | | | | | `--region` | `default` | Worker region. | n/a | | | | | `--gpu` | `false` | Declare GPU capability. | n/a | | | | | `--supported-models` | empty | Comma-separated supported model IDs. | n/a | | | | | `--max-concurrent-tasks` | `1` | Worker concurrency capacity. | n/a | | | | | `--storage-backend` | `postgres` | State backend. | \`postgres | memory\`. | | | | `--postgres-dsn` | empty | Postgres DSN. | Required when `--storage-backend=postgres`; env `ORLOJ_POSTGRES_DSN`. | | | | | `--sql-driver` | `pgx` | `database/sql` driver for Postgres backend. | Postgres backend only. | | | | | `--postgres-max-open-conns` | `20` | Max open Postgres connections. | Postgres backend only. | | | | | `--postgres-max-idle-conns` | `10` | Max idle Postgres connections. | Postgres backend only. | | | | | `--postgres-conn-max-lifetime` | `30m` | Max Postgres connection lifetime. | Postgres backend only. | | | | | `--secret-encryption-key` | empty | AES-256-GCM key for Secret encryption at rest. | Env fallback: `ORLOJ_SECRET_ENCRYPTION_KEY`. Workers do not use the `SealedSecret` private key. | | | | #### Task execution and runtime inbox consumers | Flag | Default | Description | Condition / Notes | | | | ---------------------------------------- | ----------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------ | | `--reconcile-interval` | `1s` | Claim/reconcile interval. | n/a | | | | `--lease-duration` | `30s` | Task lease duration. | n/a | | | | `--heartbeat-interval` | `10s` | Lease heartbeat interval. | n/a | | | | `--task-execution-mode` | `sequential` | Task execution mode. | \`sequential | message-driven`; env `ORLOJ\_TASK\_EXECUTION\_MODE\`. | | | `--agent-message-bus-backend` | `none` | Runtime agent message bus backend. | \`none | memory | nats-jetstream`; env `ORLOJ\_AGENT\_MESSAGE\_BUS\_BACKEND\`. | | `--agent-message-nats-url` | `nats://127.0.0.1:4222` | NATS URL for runtime agent messages. | Used when `nats-jetstream`; env `ORLOJ_AGENT_MESSAGE_NATS_URL` (fallback `ORLOJ_NATS_URL`). | | | | `--agent-message-subject-prefix` | `orloj.agentmsg` | Subject prefix for runtime messages. | Env `ORLOJ_AGENT_MESSAGE_SUBJECT_PREFIX`. | | | | `--agent-message-stream-name` | `ORLOJ_AGENT_MESSAGES` | JetStream stream name for runtime messages. | Env `ORLOJ_AGENT_MESSAGE_STREAM`. | | | | `--agent-message-history-max` | `2048` | In-memory runtime message history capacity. | In-memory runtime message backend behavior. | | | | `--agent-message-dedupe-window` | `2m` | In-memory runtime message dedupe window. | In-memory runtime message backend behavior. | | | | `--agent-message-consume` | `false` | Enable runtime inbox consumers in worker. | Env fallback: `ORLOJ_AGENT_MESSAGE_CONSUME`. | | | | `--agent-message-consumer-namespace` | empty | Namespace filter for runtime inbox consumers. | Env fallback: `ORLOJ_AGENT_MESSAGE_CONSUMER_NAMESPACE`. | | | | `--agent-message-consumer-refresh` | `10s` | Consumer reconciliation interval. | n/a | | | | `--agent-message-consumer-dedupe-window` | `10m` | Inbox processing dedupe window. | n/a | | | #### Model secret resolution | Flag | Default | Description | Condition / Notes | | --------------------------- | --------------- | -------------------------------------------- | ---------------------------------------------- | | `--model-secret-env-prefix` | `ORLOJ_SECRET_` | Env prefix for model `secretRef` resolution. | Env fallback: `ORLOJ_MODEL_SECRET_ENV_PREFIX`. | Model routing (provider, base URL, default model, API key, timeout) is configured exclusively via **ModelEndpoint** resources. Agents reference endpoints through `spec.model_ref`. See [Configure Model Routing](../guides/configure-model-routing.md). #### Tool isolation runtime | Flag | Default | Description | Condition / Notes | | | ----------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------- | --------------------------------------------------- | | `--tool-isolation-backend` | `none` | Container isolation backend for tool sandboxing. | \`none | container`; env `ORLOJ\_TOOL\_ISOLATION\_BACKEND\`. | | `--tool-container-runtime` | `docker` | Container runtime binary. | Container backend; env `ORLOJ_TOOL_CONTAINER_RUNTIME`. | | | `--tool-container-image` | `curlimages/curl:8.8.0` | Container image for isolated tool calls. | Container backend; env `ORLOJ_TOOL_CONTAINER_IMAGE`. | | | `--tool-container-network` | `none` | Container network mode. | Container backend; env `ORLOJ_TOOL_CONTAINER_NETWORK`. | | | `--tool-container-memory` | `128m` | Default container memory limit. Per-tool `spec.cli.resources.memory` takes precedence when set. | Container backend; env `ORLOJ_TOOL_CONTAINER_MEMORY`. | | | `--tool-container-cpus` | `0.50` | Default container CPU limit. Per-tool `spec.cli.resources.cpus` takes precedence when set. | Container backend; env `ORLOJ_TOOL_CONTAINER_CPUS`. | | | `--tool-container-pids-limit` | `64` | Default container PID limit. Per-tool `spec.cli.resources.pids_limit` takes precedence when set. | Container backend; env `ORLOJ_TOOL_CONTAINER_PIDS_LIMIT`. | | | `--tool-container-user` | `65532:65532` | Container user. | Container backend; env `ORLOJ_TOOL_CONTAINER_USER`. | | | `--tool-secret-env-prefix` | `ORLOJ_SECRET_` | Env prefix for tool `secretRef` resolution. | Env fallback: `ORLOJ_TOOL_SECRET_ENV_PREFIX`. | | | `--tool-wasm-module` | empty | Default WASM module path (per-tool `spec.wasm.module` takes precedence). | Always available; env `ORLOJ_TOOL_WASM_MODULE`. | | | `--tool-wasm-entrypoint` | `run` | Default WASM entrypoint function. | Always available; env `ORLOJ_TOOL_WASM_ENTRYPOINT`. | | | `--tool-wasm-memory-bytes` | `67108864` | Default max WASM memory bytes. | Always available; env `ORLOJ_TOOL_WASM_MEMORY_BYTES`. | | | `--tool-wasm-fuel` | `1000000` | Default WASM execution fuel limit. | Always available; env `ORLOJ_TOOL_WASM_FUEL`. | | | `--tool-wasm-wasi` | `true` | Default: enable WASI host functions. | Always available; env `ORLOJ_TOOL_WASM_WASI`. | | | `--tool-wasm-cache-dir` | `~/.orloj/wasm-cache` | Disk cache directory for remote WASM modules (HTTPS/OCI). | Always available; env `ORLOJ_TOOL_WASM_CACHE_DIR`. | | | `--tool-k8s-enabled` | `false` | Enable Kubernetes tool isolation runtime. | Env `ORLOJ_TOOL_K8S_ENABLED`. When true, tools with `isolation_mode: kubernetes` run as K8s Jobs. | | | `--tool-k8s-namespace` | pod namespace or `default` | Namespace for tool Jobs. | Env `ORLOJ_TOOL_K8S_NAMESPACE`. | | | `--tool-k8s-service-account` | empty | Service account for tool Pods. | Env `ORLOJ_TOOL_K8S_SERVICE_ACCOUNT`. | | | `--tool-k8s-job-ttl` | `300` | TTL seconds after Job finishes (`ttlSecondsAfterFinished`). | Env `ORLOJ_TOOL_K8S_JOB_TTL`. | | | `--tool-k8s-default-image` | `curlimages/curl:8.8.0` | Fallback image for HTTP tools without an explicit image. | Env `ORLOJ_TOOL_K8S_DEFAULT_IMAGE`. | | #### Agent Kubernetes execution | Flag | Default | Description | Condition / Notes | | ----------------------------- | -------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | `--agent-k8s-enabled` | `false` | Run agents as ephemeral K8s Jobs. | Env `ORLOJ_AGENT_K8S_ENABLED`. Agents with Docker-dependent tools fall back to in-process. | | `--agent-k8s-namespace` | pod namespace or `default` | Namespace for agent Jobs. | Env `ORLOJ_AGENT_K8S_NAMESPACE`. | | `--agent-k8s-service-account` | empty | Service account for agent Pods. | Env `ORLOJ_AGENT_K8S_SERVICE_ACCOUNT`. | | `--agent-k8s-image` | own image | Container image for agent Jobs. | Env `ORLOJ_AGENT_K8S_IMAGE`. Defaults to the running binary's own image. | | `--agent-k8s-job-ttl` | `600` | TTL seconds after Job finishes (`ttlSecondsAfterFinished`). | Env `ORLOJ_AGENT_K8S_JOB_TTL`. | | `--agent-k8s-default-memory` | `512Mi` | Default memory limit for agent Pods. | Env `ORLOJ_AGENT_K8S_DEFAULT_MEMORY`. | | `--agent-k8s-default-cpu` | `500m` | Default CPU limit for agent Pods. | Env `ORLOJ_AGENT_K8S_DEFAULT_CPU`. | | `--single-agent` | `false` | Run a single agent execution (used by K8s agent Jobs). | Internal flag; not for manual use. | | `--task-id` | empty | Task ID for single-agent mode. | Used with `--single-agent`. | | `--agent-name` | empty | Agent name for single-agent mode. | Used with `--single-agent`. | | `--attempt` | `0` | Attempt number for single-agent mode. | Used with `--single-agent`. | | `--message-id` | empty | Message ID for single-agent mode. | Used with `--single-agent`. | ### Command Discovery Use help output as the authoritative source for your current build: ```bash go run ./cmd/orlojd -h go run ./cmd/orlojworker -h ``` ### Related * [Configuration](../operations/configuration.md) — full env-variable matrix and precedence rules * [CLI (orlojctl)](./cli.md) — user-facing CLI reference * [Deployment](../deploy/) — deployment guides for all targets ## Agent Card > **Stability: beta** -- Agent Cards are generated resources that follow the [A2A specification](https://github.com/a2aproject/A2A). The schema may evolve as the A2A spec matures. An Agent Card is a JSON document that describes an Orloj AgentSystem's A2A identity, capabilities, skills, and endpoint. Cards are auto-generated from AgentSystem metadata and its agents' tools. Agent Cards are not stored resources -- they are computed on request from the agent's current state. ### Discovery URLs | URL | Description | | ---------------------------------------------------------- | ----------------------------------------------------- | | `GET /.well-known/agent-card.json` | Root card when exactly one AgentSystem is A2A-enabled | | `GET /v1/agent-systems/{name}/.well-known/agent-card.json` | Card for a specific AgentSystem | Both endpoints are public (no authentication required). Cards contain only metadata, not secrets. ### Schema | Field | Type | Required | Description | | ----------------- | ------------ | -------- | -------------------------------------------------------------------------------------------- | | `name` | string | Yes | Human-readable agent name. Derived from `metadata.name`. | | `description` | string | No | Agent description. From `metadata.annotations["orloj.dev/description"]` or a prompt summary. | | `url` | string (URI) | Yes | A2A JSON-RPC endpoint URL. Constructed from `a2a.publicBaseURL` + AgentSystem path. | | `version` | string | No | Agent version. From `metadata.labels["orloj.dev/version"]` if present. | | `protocolVersion` | string | No | A2A protocol version. From `a2a.protocolVersion` config. | | `capabilities` | object | No | See [Capabilities](#capabilities). | | `skills` | \[]object | No | See [Skills](#skills). | | `authentication` | object | No | See [Authentication](#authentication). | | `provider` | object | No | See [Provider](#provider). | #### Capabilities | Field | Type | Description | | ------------------------ | ------- | -------------------------------------------------------------------------------------- | | `streaming` | boolean | Whether the agent supports `tasks/sendSubscribe`. Derived from task trace SSE support. | | `pushNotifications` | boolean | Whether push notifications are available. Derived from TaskWebhook support. | | `stateTransitionHistory` | boolean | Whether task history is available. | #### Skills Each skill entry represents a tool available to the agent: | Field | Type | Required | Description | | ------------- | --------- | -------- | ----------------------------------------------------------------------------- | | `id` | string | Yes | Skill identifier. From `Tool.metadata.name`. | | `name` | string | Yes | Display name. From `Tool.metadata.name`. | | `description` | string | No | From `Tool.spec.description`. Omitted if the tool has no description. | | `inputSchema` | object | No | From `Tool.spec.input_schema`. Omitted (not empty) if the tool has no schema. | | `tags` | \[]string | No | Derived from tool capabilities and operation classes. | #### Authentication | Field | Type | Description | | --------- | --------- | -------------------------------------------------------------------------------------------------------- | | `schemes` | \[]string | Auth schemes accepted by the JSON-RPC endpoint. Reflects server auth configuration (e.g., `["bearer"]`). | #### Provider | Field | Type | Description | | -------------- | ------------ | --------------------------------------------- | | `organization` | string | Organization name. From server configuration. | | `url` | string (URI) | Organization URL. | ### Field Mapping How Orloj resources map to Agent Card fields: | Card Field | Source | | -------------------------------- | ------------------------------------------------------------------------ | | `name` | `Agent.metadata.name` | | `description` | `Agent.metadata.annotations["orloj.dev/description"]`, or prompt excerpt | | `url` | `a2a.publicBaseURL` + `/v1/agent-systems/{name}/a2a` | | `protocolVersion` | `a2a.protocolVersion` server config | | `capabilities.streaming` | Server SSE support enabled | | `capabilities.pushNotifications` | TaskWebhook controller active | | `skills[].id` | `Tool.metadata.name` (for each tool in `Agent.spec.tools`) | | `skills[].description` | `Tool.spec.description` | | `skills[].inputSchema` | `Tool.spec.input_schema` | | `authentication.schemes` | Server auth mode | ### Example Card For an AgentSystem defined as: ```yaml apiVersion: orloj.dev/v1 kind: AgentSystem metadata: name: research-system annotations: orloj.dev/description: "AI research assistant for academic papers" spec: agents: - research-agent a2a: enabled: true ``` The generated card (at `GET /v1/agent-systems/research-system/.well-known/agent-card.json`): ```json { "name": "research-system", "description": "AI research assistant for academic papers", "url": "https://orloj.example.com/v1/agent-systems/research-system/a2a", "protocolVersion": "0.2", "capabilities": { "streaming": true, "pushNotifications": false, "stateTransitionHistory": true }, "skills": [ { "id": "web_search", "name": "web_search", "description": "Search the web for information", "inputSchema": { "type": "object", "properties": { "query": { "type": "string" } }, "required": ["query"] }, "tags": ["search"] }, { "id": "arxiv_search", "name": "arxiv_search", "description": "Search arXiv for academic papers", "inputSchema": { "type": "object", "properties": { "query": { "type": "string" }, "max_results": { "type": "integer" } }, "required": ["query"] }, "tags": ["search", "academic"] } ], "authentication": { "schemes": ["bearer"] }, "provider": { "organization": "Orloj" } } ``` ### Related * [A2A Interoperability](../../concepts/a2a-interoperability.md) -- architecture and concepts * [A2A JSON-RPC](../a2a-jsonrpc.md) -- protocol reference * [Agent](./agent.md) -- agent resource schema ## AgentPolicy > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. ### spec * `max_tokens_per_run` (int) * `allowed_models` (\[]string) * `blocked_tools` (\[]string) * `apply_mode` (string): `scoped` or `global` * `target_systems` (\[]string) * `target_tasks` (\[]string) * `target_agents` (\[]string): when set, only listed agents are subject to this policy's constraints (model checks, blocked tools, token budget). Agents not in the list are unaffected. When empty, the policy applies to all agents in the matched systems/tasks. ### Defaults and Validation * `apply_mode` defaults to `scoped`. * `apply_mode` must be `scoped` or `global`. ### status * `phase`, `lastError`, `observedGeneration` Example: `examples/resources/agent-policies/cost_policy.yaml` See also: [Agent policy concepts](../../concepts/governance/agent-policy.md). ## AgentRole > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. ### spec * `description` (string) * `permissions` (\[]string): normalized permission strings. ### Defaults and Validation * `permissions` are trimmed and deduplicated (case-insensitive). ### status * `phase`, `lastError`, `observedGeneration` Examples: `examples/resources/agent-roles/*.yaml` See also: [Agent role concepts](../../concepts/governance/agent-role.md). ## AgentSystem > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. ### spec * `agents` (\[]string): participating agent names. * `graph` (map\[string]GraphEdge): per-node routing. * `context_adapter` (string): optional reference to a [ContextAdapter](./context-adapter.md) resource. When set, the adapter's tool sanitizes raw task input before the first agent runs. * `completion_review` (ReviewCheckpoint): optional final human review before the task is marked `Succeeded`. * `a2a.enabled` (bool): expose this AgentSystem through inbound A2A Agent Card discovery and JSON-RPC invocation. Omitted or `false` keeps the system off the A2A surface. * `a2a.auth` (string: `"public"` | `"bearer"`): controls whether A2A invoke requires authentication for this system. `"public"` allows unauthenticated callers; `"bearer"` (default when omitted) requires a valid API token when instance-wide auth is enabled. Public systems' Agent Cards omit `authentication.schemes` so A2A clients know not to send tokens. `GraphEdge` fields: * `next` (string): legacy single-hop route. * `edges` (\[]GraphRoute): fan-out routes. * `to` (string) * `labels` (map\[string]string) * `policy` (map\[string]string) * `join` (GraphJoin): fan-in behavior. * `mode`: `wait_for_all` or `quorum` * `quorum_count` (int, >= 0) * `quorum_percent` (int, 0-100) * `on_failure`: `deadletter`, `skip`, `continue_partial` * `review` (ReviewCheckpoint): optional human review checkpoint for this node's output. `ReviewCheckpoint` fields: * `checkpoint_id` (string, required) * `display_name` (string) * `reason` (string) * `ttl` (duration string, defaults to `10m`) * `allow_request_changes` (bool, defaults to `true`) * `max_review_cycles` (int, defaults to `3`) If `allow_request_changes` is `false`, reviewers can only approve or deny that checkpoint. Once `max_review_cycles` is reached, additional `request_changes` attempts are rejected. ### Defaults and Validation * `graph[*].next` and `graph[*].edges[].to` are trimmed. * Route targets are normalized/deduplicated for execution. * `join` normalization defaults: * `mode` -> `wait_for_all` * `on_failure` -> `deadletter` * `quorum_percent` clamped to `0..100` * invalid values are coerced to safe defaults in graph normalization. * Runtime task validation additionally checks: * graph nodes/edges must reference agents in `spec.agents` * cyclic graphs require `Task.spec.max_turns > 0` * non-cyclic graphs require at least one entrypoint (zero indegree node) * review `checkpoint_id` values must be unique within the system ### status * `phase`, `lastError`, `observedGeneration` Example: [`examples/resources/agent-systems/`](https://github.com/OrlojHQ/orloj/tree/main/examples/resources/agent-systems) See also: [Agent system concept](../../concepts/agents/agent-system.md) ## Agent > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. ### spec * `model_ref` (string): reference to a `ModelEndpoint` (`name` or `namespace/name`). * `prompt` (string): agent instruction prompt. * `tools` (\[]string): tool names available to the agent. * `allowed_tools` (\[]string): tools pre-authorized without RBAC. Bypasses AgentRole/ToolPermission checks for listed tools. * `roles` (\[]string): bound `AgentRole` names. * `memory` (object): * `ref` (string): reference to a `Memory` resource. This attaches the memory backend to the agent. See [Memory](../../concepts/memory/index.md). * `allow` (\[]string): explicit built-in memory operations allowed for the agent: `read`, `write`, `search`, `list`, `ingest`. * `type` (string) * `provider` (string) * `limits` (object): * `max_steps` (int) * `timeout` (string duration) * `execution` (object): optional per-agent execution contract. * `profile` (string): `dynamic` (default) or `contract`. * `tool_sequence` (\[]string): required tool names when `profile=contract`. Tracked as a set (order-independent). * `required_output_markers` (\[]string): strings that should appear in final model output when `profile=contract`. Treated as best-effort: missing markers at `max_steps` produce a warning, not a hard failure, when all tools completed. * `duplicate_tool_call_policy` (string): `short_circuit` (default) or `deny`. In `short_circuit` mode, duplicate tool calls reuse cached results and inject a completion hint. This applies to **all profiles**, not just `contract`. * `on_contract_violation` (string): `observe` or `non_retryable_error` (default). In `observe` mode, violations are logged as telemetry events but do not stop execution or deadletter the task. * `tool_use_behavior` (string): Controls what happens after a tool call succeeds. See [Tool Use Behavior](#tool-use-behavior) below. #### Tool Use Behavior The `tool_use_behavior` field controls whether the model gets another turn after a successful tool call. This is the primary lever for optimizing token usage in tool-calling agents. | Value | Model calls | When to use | | ------------------------- | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | `run_llm_again` (default) | Tool call + follow-up model call to process the result | The agent needs to **interpret, format, or synthesize** the tool output before handing off. Most agents need this. | | `stop_on_first_tool` | Tool call only -- tool output becomes the agent's final output directly | The agent is a **relay** that calls a tool and passes raw data to the next agent in the pipeline. No interpretation needed. | **Example: `run_llm_again` (default)** An analyst agent calls an API tool, then needs to produce labeled output from the raw response: ```yaml kind: Agent metadata: name: analyst-agent spec: prompt: "Call the API, then return SUMMARY: and EVIDENCE: labels." tools: - external-api-tool # tool_use_behavior defaults to run_llm_again -- agent gets a # second model call to read the tool result and produce labels. ``` Step 1: model calls `external-api-tool` → Step 2: model reads tool result, produces labeled output → done (2 model calls). **Example: `stop_on_first_tool`** A fetcher agent's only job is to call a tool and pass the raw result downstream: ```yaml kind: Agent metadata: name: fetcher-agent spec: prompt: "Fetch the latest data from the API." tools: - external-api-tool execution: tool_use_behavior: stop_on_first_tool # Agent exits immediately after the tool returns. # Raw tool output becomes the agent's output -- no extra model call. ``` Step 1: model calls `external-api-tool` → done (1 model call). The next agent in the pipeline receives the raw tool response as context. **When NOT to use `stop_on_first_tool`:** * The agent needs to produce structured/labeled output from the tool result. * The agent has multiple tools and may need to call more than one. * The agent needs to reason about the tool result before responding. ### Defaults and Validation * `model_ref` is required. * `roles` are trimmed and deduplicated (case-insensitive). * `memory.allow` is trimmed, normalized, and deduplicated. It requires `memory.ref`. * `limits.max_steps` defaults to `10` when `<= 0`. * `execution.profile` defaults to `dynamic`. * `execution.duplicate_tool_call_policy` defaults to `short_circuit`. Applies to all profiles. * `execution.on_contract_violation` defaults to `non_retryable_error`. Set to `observe` for safe production rollout. * `execution.tool_use_behavior` defaults to `run_llm_again`. * `execution.tool_sequence` and `execution.required_output_markers` are trimmed and deduplicated. * When `execution.profile=contract`, `execution.tool_sequence` is required. * Tool sequence is tracked as a set: tools may be called in any order. * When all tools in `tool_sequence` complete but `required_output_markers` are not satisfied at `max_steps`, the task completes with a `contract_warning` event instead of deadlettering. **Structured tool protocol:** Tool results are sent to the model using the provider's native structured tool calling protocol (OpenAI `role: "tool"` with `tool_call_id`, Anthropic `tool_result` content blocks). This gives the model structured evidence that a tool was already called, preventing unnecessary repeat calls. **Scaling ladder for cost control:** 1. `profile: dynamic` (default): structured tool protocol prevents repeat calls. Succeeded tools are filtered from the available tools list. No YAML changes needed. 2. `tool_use_behavior: stop_on_first_tool`: for pipeline stages that pass raw data, eliminates all extra model calls (1 model call + 1 tool call total). 3. `profile: contract` + `on_contract_violation: observe`: adds guaranteed early completion when all tools succeed plus telemetry for contract deviations. 4. `profile: contract` + `on_contract_violation: non_retryable_error`: hard enforcement for critical pipeline stages. Violations deadletter the task. ### status * `phase`, `lastError`, `observedGeneration` Example: [`examples/resources/agents/`](https://github.com/OrlojHQ/orloj/tree/main/examples/resources/agents) See also: [Agent concept](../../concepts/agents/agent.md) ## ContextAdapter > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. A ContextAdapter configures a pre-agent sanitization step that transforms raw task input before any agent sees it. It references a [Tool](./tool.md) that receives the input as JSON and returns sanitized JSON. The adapter enforces the handoff contract and error policy but leaves all sanitization logic to the tool. This is useful for workflows that ingest sensitive data (PII, financial records, medical information) where the AI agent needs to reason about the data but should never have access to the raw values. ### spec * `tool_ref` (string, required): name of a Tool resource. The tool receives the task's `spec.input` as a `map[string]string` JSON object and must return a `map[string]string` JSON object with sanitized values. * `on_error` (string): behavior when the tool call fails or returns invalid output. * `reject` (default): abort the task with an error. No raw data reaches any agent. * `passthrough`: log a warning and pass the original unmodified input to the agent. Useful for development or non-critical paths. ### How it works The ContextAdapter is declared on an [AgentSystem](./agent-system.md) via `spec.context_adapter`, not on individual tasks. This ensures every task that runs against the system is automatically protected. ```yaml apiVersion: orloj.dev/v1 kind: ContextAdapter metadata: name: tx-sanitizer spec: tool_ref: tx-sanitize-tool on_error: reject --- apiVersion: orloj.dev/v1 kind: AgentSystem metadata: name: fraud-detection spec: context_adapter: tx-sanitizer agents: - tx-analyst ``` At runtime, the adapter fires after a task is created but before the first agent executes: 1. Raw task input (`task.spec.input`) is JSON-encoded and sent to the tool. 2. The tool performs sanitization (masking, tokenization, scrubbing, etc.) and returns a JSON object. 3. The sanitized map replaces the original input for agent execution. 4. If the tool fails and `on_error` is `reject`, the task aborts. If `passthrough`, the raw input is used with a logged warning. The adapter runs once per task, before the first agent only. On task resume (e.g. after a human review checkpoint), the adapter does not re-run. ### Tool contract The referenced tool receives a JSON object: ```json { "account_number": "4111-1111-1111-1111", "ssn": "123-45-6789", "amount": "9800.00", "memo": "wire transfer" } ``` It must return a JSON object with the same or modified keys: ```json { "account_number": "ACCT_7x3k9m", "ssn": "XXX-XX-XXXX", "amount": "9800.00", "memo": "wire transfer" } ``` The tool decides what to sanitize and how. WASM tools (via Wazero) are recommended for handling PII because they run fully sandboxed with no filesystem or network access, but any tool runtime works (container, CLI, HTTP, gRPC). ### Defaults and Validation * `spec.tool_ref` is required. Normalization trims whitespace. * `spec.on_error` defaults to `reject` when omitted or empty. * `spec.on_error` must be `reject` or `passthrough`. * `status.phase` defaults to `Pending`. ### status * `phase`: `Pending` or `Ready`. * `message`: description of the current state. See also: [AgentSystem](./agent-system.md), [Tool](./tool.md), [Build a WASM Tool](../../guides/build-wasm-tool.md) ## EvalDataset > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. An EvalDataset is a declarative list of (input, expected output) pairs, optionally with per-sample scoring rubrics. Datasets are applied like any other resource and referenced by [EvalRun](./eval-run.md) resources to drive evaluations. ### Manifest ```yaml apiVersion: orloj.dev/v1 kind: EvalDataset metadata: name: support-triage-golden namespace: default spec: description: "Golden set for the support triage agent" samples: - name: billing-question input: prompt: "I was charged twice for my subscription" expected: output_contains: "billing" output_json_path: "$.category" equals: "billing" - name: refund-request input: prompt: "I want a refund for order #12345" expected: output_contains: "refund" scoring: strategy: llm_judge model_ref: gpt-4o-judge rubric: "The response should identify this as a refund request and extract the order number." ``` ### spec * `description` (string, optional): human-readable description of the dataset. * `samples` (\[]EvalSample, required, min 1): the evaluation cases. #### EvalSample | Field | Type | Description | | ---------- | ---------------------------- | ----------------------------------------------------------------------------------- | | `name` | string, required | Unique name within the dataset (case-insensitive uniqueness). | | `input` | map\[string]string, required | Input passed to the agent system as `task.spec.input`. Must be non-empty. | | `expected` | EvalExpected, optional | Expected output criteria for `exact_match` scoring. | | `scoring` | EvalScoringConfig, optional | Per-sample scoring override. When set, takes precedence over the run-level default. | #### EvalExpected Mirrors [EdgeCondition](./agent-system.md) semantics. All non-empty fields must match (logical AND). | Field | Type | Description | | --------------------- | ------ | --------------------------------------------------------------------------- | | `output_contains` | string | Output must contain this substring (case-insensitive). | | `output_not_contains` | string | Output must NOT contain this substring (case-insensitive). | | `output_matches` | string | Output must match this regex. Validated during normalization. | | `output_json_path` | string | Dot-notation JSON path (e.g. `$.category`). Requires a comparison operator. | | `equals` | string | JSON path value must equal this string. | | `not_equals` | string | JSON path value must NOT equal this string. | | `contains` | string | JSON path value (string or array) must contain this value. | | `greater_than` | string | JSON path numeric value must be greater than this threshold. | | `less_than` | string | JSON path numeric value must be less than this threshold. | #### EvalScoringConfig | Field | Type | Description | | ----------- | ------ | ---------------------------------------------------------------------------- | | `strategy` | string | One of: `exact_match`, `llm_judge`, `manual`, `custom`. | | `model_ref` | string | Required for `llm_judge`. References a [ModelEndpoint](./model-endpoint.md). | | `rubric` | string | Evaluation rubric for `llm_judge`. | | `tool_ref` | string | Required for `custom`. References a [Tool](./tool.md). | ### Defaults and Validation * `apiVersion` defaults to `orloj.dev/v1`. * `kind` defaults to `EvalDataset`. * `metadata.namespace` defaults to `default`. * `status.phase` defaults to `Ready`. * Sample names must be unique within a dataset (case-insensitive). * Each sample must have at least one input entry. * `output_matches` is validated as a valid regex during normalization. * `llm_judge` strategy requires `model_ref`. * `custom` strategy requires `tool_ref`. ### status * `phase`: `Ready` (datasets are config-only; no controller moves them through phases). ### API Endpoints | Method | Path | Description | | -------- | -------------------------- | --------------------------------------------------------------------------- | | `GET` | `/v1/eval-datasets` | List all datasets (supports `namespace`, `limit`, `continue` query params). | | `POST` | `/v1/eval-datasets` | Create or update a dataset. | | `GET` | `/v1/eval-datasets/{name}` | Get a dataset by name. | | `PUT` | `/v1/eval-datasets/{name}` | Update a dataset. | | `DELETE` | `/v1/eval-datasets/{name}` | Delete a dataset. | ### Related * [EvalRun](./eval-run.md) -- runs a dataset against an agent system * [Agent Evaluation (concept)](../../concepts/evaluation/) -- overview of the evaluation framework * [Guide: Run Your First Agent Evaluation](../../guides/run-agent-evaluation.md) ## EvalRun > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. An EvalRun executes all samples in an [EvalDataset](./eval-dataset.md) against an [AgentSystem](./agent-system.md), scores the results, and produces aggregate metrics. Runs track per-sample scores, latency, token usage, and pass/fail verdicts. ### Manifest ```yaml apiVersion: orloj.dev/v1 kind: EvalRun metadata: name: triage-gpt4o-20260510 namespace: default spec: dataset_ref: support-triage-golden system: support-triage-system scoring: strategy: llm_judge model_ref: gpt-4o-judge rubric: "Rate the accuracy and helpfulness of the triage response." concurrency: 5 timeout: 120s agent_overrides: triage-agent: prompt: "You are a support triage bot. Classify and route." model_ref: gpt-4o-mini ``` ### spec | Field | Type | Description | | ----------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dataset_ref` | string, required | Name of the [EvalDataset](./eval-dataset.md) to evaluate against. | | `system` | string, required | Name of the [AgentSystem](./agent-system.md) to evaluate. | | `scoring` | EvalScoringConfig | Default scoring strategy for all samples. Per-sample overrides in the dataset take precedence. | | `concurrency` | int | Maximum parallel tasks (samples) to execute. Defaults to 5. Minimum 1. | | `timeout` | duration string | Per-sample task timeout (e.g. `120s`, `5m`). Must be a valid Go `time.Duration`. | | `agent_overrides` | map\[string]AgentOverride | Ephemeral overrides for agent configuration within this run. Keys are agent names. | | `labels` | map\[string]string | Arbitrary labels for filtering and comparison. | | `suspended` | bool | When true, the controller will not execute this run. Defaults to `true` when created via `apply` (use `--run` to override or `orlojctl eval start` to trigger later). Defaults to `false` when created via `orlojctl eval run`. | #### AgentOverride Used for A/B testing models, prompts, or parameters without modifying the base agent resource. The map key is the name of the agent to override. | Field | Type | Description | | ----------- | ------ | ------------------------------------ | | `prompt` | string | Override the agent's system prompt. | | `model_ref` | string | Override the agent's model endpoint. | #### EvalScoringConfig See [EvalDataset](./eval-dataset.md#evalscoringconfig) for the full field list. ### Defaults and Validation * `apiVersion` defaults to `orloj.dev/v1`. * `kind` defaults to `EvalRun`. * `metadata.namespace` defaults to `default`. * `status.phase` defaults to `Pending`. * `spec.suspended` defaults to `true` when created via `POST /v1/eval-runs` (unless `?run=true` is set). `orlojctl eval run` sets `?run=true` automatically. * `spec.concurrency` defaults to 5; must be >= 1. * `spec.dataset_ref` and `spec.system` are required. * `spec.timeout` must be a valid Go duration string when set. * `llm_judge` scoring requires `model_ref`. * `custom` scoring requires `tool_ref`. * Agent override names must be unique. ### status | Field | Type | Description | | ------------------- | ------------------- | ------------------------------------------ | | `phase` | string | Current lifecycle phase (see below). | | `message` | string | Human-readable status message. | | `results` | \[]EvalSampleResult | Per-sample results. | | `summary` | EvalSummary | Aggregate metrics computed after scoring. | | `total_samples` | int | Total number of samples in the dataset. | | `completed_samples` | int | Number of samples with completed tasks. | | `scored_samples` | int | Number of samples that have been scored. | | `errored_samples` | int | Number of samples that encountered errors. | #### EvalSampleResult | Field | Type | Description | | ------------- | --------- | ------------------------------------------------------- | | `sample_name` | string | Name matching the dataset sample. | | `task_name` | string | Name of the task created for this sample. | | `output` | string | Raw output from the agent system. | | `score` | \*float64 | Numeric score (0.0–1.0). Nil for unscored/manual. | | `pass` | \*bool | Pass/fail verdict. Nil for unscored. | | `reasoning` | string | Explanation from the scorer (e.g. LLM judge reasoning). | | `latency_ms` | int64 | Execution time in milliseconds. | | `tokens_used` | int | Total tokens consumed by the task. | | `error` | string | Non-empty if the sample errored. | #### EvalSummary | Field | Type | Description | | ----------------- | ------- | ------------------------------------------------- | | `pass_rate` | float64 | Fraction of scored samples that passed (0.0–1.0). | | `mean_score` | float64 | Average score across all scored samples. | | `total_tokens` | int | Sum of tokens across all samples. | | `mean_latency_ms` | int64 | Average latency across all completed samples. | ### Lifecycle Phases ``` Pending ──► Running ──► Scoring ──► Succeeded └──► PendingReview ──► Succeeded └──► Failed └──► Cancelled ``` | Phase | Meaning | | --------------- | ------------------------------------------------------------------------------------------------------------------- | | `Pending` | Created, waiting for the controller. If `spec.suspended` is true, the controller skips the run until it is started. | | `Running` | Tasks are being created and executed (up to `concurrency` in parallel). | | `Scoring` | All tasks completed; scoring pipeline is evaluating results. | | `PendingReview` | Manual scoring; awaiting human annotations before finalization. | | `Succeeded` | Scoring complete; `status.summary` is final. | | `Failed` | Fatal error during the run (e.g. missing dataset or system). | | `Cancelled` | User-initiated cancellation. In-flight tasks are also cancelled. | ### API Endpoints | Method | Path | Description | | -------- | --------------------------------------- | --------------------------------------------------- | | `GET` | `/v1/eval-runs` | List all runs. | | `POST` | `/v1/eval-runs` | Create a new eval run. | | `GET` | `/v1/eval-runs/{name}` | Get a run by name. | | `PUT` | `/v1/eval-runs/{name}` | Update a run. | | `DELETE` | `/v1/eval-runs/{name}` | Delete a run. | | `GET` | `/v1/eval-runs/{name}/export` | Export results as JSON (or CSV with `?format=csv`). | | `PUT` | `/v1/eval-runs/{name}/results/{sample}` | Annotate a single sample (manual review). | | `POST` | `/v1/eval-runs/{name}/results` | Bulk import sample annotations. | | `POST` | `/v1/eval-runs/{name}/start` | Start a suspended eval run. | | `POST` | `/v1/eval-runs/{name}/finalize` | Finalize a PendingReview run (computes summary). | | `POST` | `/v1/eval-runs/{name}/cancel` | Cancel a running eval. | | `GET` | `/v1/eval-runs/compare?names=a,b,c` | Compare multiple runs side-by-side. | ### CLI Quick Reference ```bash orlojctl eval run --dataset golden --system my-system # Create and start a run orlojctl eval start my-run # Start a suspended run orlojctl eval list # List all runs orlojctl eval get my-run # Get run detail orlojctl eval export my-run --format csv # Export for review orlojctl eval annotate my-run --sample s1 --score 0.9 # Annotate sample orlojctl eval import my-run -f reviewed.csv # Bulk import orlojctl eval finalize my-run # Finalize manual run orlojctl eval compare run-a run-b # Compare runs orlojctl eval datasets # List datasets orlojctl apply -f eval-run.yaml # Apply (suspended by default) orlojctl apply -f eval-run.yaml --run # Apply and start immediately ``` ### Related * [EvalDataset](./eval-dataset.md) -- define golden test data * [Agent Evaluation (concept)](../../concepts/evaluation/) -- overview and workflow * [Guide: Run Your First Agent Evaluation](../../guides/run-agent-evaluation.md) * [AgentSystem](./agent-system.md) -- the systems being evaluated ## Resources This section documents selected resource kinds in `orloj.dev/v1`, based on the runtime types and normalization logic in: * `resources/agent.go` * `resources/model_endpoint.go` * `resources/resource_types.go` * `resources/graph.go` Each kind has a dedicated page; see [Resource reference pages](#resource-reference-pages) below. ### Common Conventions * Every resource uses standard top-level fields: `apiVersion`, `kind`, `metadata`, `spec`, `status`. * `metadata.name` is required for all resources. * `metadata.namespace` defaults to `default` when omitted. * Most resources default `status.phase` to `Pending` during normalization. ### Resource reference pages * [Agent](./agent.md) * [AgentSystem](./agent-system.md) * [Task](./task.md) * [TaskSchedule](./task-schedule.md) * [TaskWebhook](./task-webhook.md) * [Tool](./tool.md) * [ModelEndpoint](./model-endpoint.md) * [McpServer](./mcp-server.md) * [Memory](./memory.md) * [ContextAdapter](./context-adapter.md) * [Secret](./secret.md) * [SealedSecret](./sealed-secret.md) * [AgentPolicy](./agent-policy.md) * [AgentRole](./agent-role.md) * [ToolPermission](./tool-permission.md) * [ToolApproval](./tool-approval.md) * [TaskApproval](./task-approval.md) * [Worker](./worker.md) * [EvalDataset](./eval-dataset.md) * [EvalRun](./eval-run.md) ## McpServer > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. Represents a connection to an external MCP (Model Context Protocol) server. The McpServer controller discovers tools via `tools/list` and auto-generates `Tool` resources (type=mcp) for each. ### spec * `transport` (string): **required**. `stdio` or `http`. * `command` (string): stdio transport: command to spawn the MCP server process. Required unless `image` is set. * `args` (\[]string): stdio transport: command arguments. * `env` (\[]object): stdio transport: environment variables for the child process. Each entry has: * `name` (string): environment variable name. * `value` (string): literal value. * `secretRef` (string): resolve value from a Secret resource. Mutually exclusive with `value`. * `mountPath` (string): absolute path inside the container where the resolved value is written as a file. Only valid when `image` is set. The env var is set to the mount path so the MCP server can locate the file. * `image` (string): stdio transport: container image. When set, the MCP server runs inside a Docker container (`docker run --rm -i`) with sandboxing. * `image_pull_secret` (string): name of a Secret containing registry credentials for pulling `spec.image`. The Secret must contain either a `.dockerconfigjson` key with a complete Docker config JSON, or `registry`, `username`, and `password` keys. Requires `image` to be set. * `idle_timeout` (duration string): duration after which an idle session is shut down (e.g. `5m`). Default `0` means never evict. * `endpoint` (string): http transport: the MCP server URL. * `auth` (object): http transport: authentication configuration. * `secretRef` (string): secret reference for auth. * `profile` (string): `bearer` or `api_key_header`. Defaults to `bearer`. * `tool_filter` (object): optional tool import filtering. * `include` (\[]string): allowlist of MCP tool names. When set, only listed tools are generated. When empty, all discovered tools are generated. * `reconnect` (object): reconnection policy. * `max_attempts` (int): max reconnection attempts. Defaults to 3. * `backoff` (duration string): backoff between attempts. Defaults to `2s`. * `default_tool_runtime` (object): default runtime policy inherited by all generated Tool resources. When set, each tool synced from this server receives this policy as its `spec.runtime`. * `timeout` (duration string): max tool call execution time (e.g. `30s`, `2m`). * `isolation_mode` (string): isolation mode for tool execution. * `retry` (object): retry policy (see [Tool resource](./tool.md)). * `allowPrivate` (boolean): http transport only. When `true`, permits this MCP server's HTTP transport to connect to RFC 1918 / ULA / CGNAT addresses (e.g. in-cluster Services like `http://mcp.internal.svc.cluster.local:8000`). Loopback, link-local, cloud metadata, and unspecified addresses remain blocked regardless. Defaults to `false`; set `true` only for trusted internal MCP servers. Has no effect on `stdio` transport. ### Defaults and Validation * `transport` is required. Must be `stdio` or `http`. * `command` or `image` is required when `transport=stdio`. * `endpoint` is required when `transport=http`. * `image` is only valid with `transport=stdio`. * `image_pull_secret` requires `image` to be set. * `env[].secretRef` and `env[].value` are mutually exclusive. * `env[].mountPath` requires `image` to be set and must be an absolute path. * `idle_timeout` defaults to `0` (never evict). * `reconnect.max_attempts` defaults to `3`. * `reconnect.backoff` defaults to `2s`. ### status * `phase`: `Pending`, `Connecting`, `Ready`, `Error`. * `discoveredTools` (\[]string): all tool names from the MCP server's `tools/list` response. * `generatedTools` (\[]string): names of the `Tool` resources actually created. * `lastSyncedAt` (timestamp): last successful tool sync. * `lastError` (string): last error message. Guide: [Connect an MCP Server](../../guides/connect-mcp-server.md) Examples: [`examples/resources/mcp-servers/`](https://github.com/OrlojHQ/orloj/tree/main/examples/resources/mcp-servers) See also: [MCP server concepts](../../concepts/tools/mcp-server.md). ## Memory > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. A Memory resource configures a persistent memory backend that agents can read from and write to using built-in memory tools. See [Memory Concepts](../../concepts/memory/index.md) for a full overview. ### spec * `type` (string): categorization of the memory use case (e.g. `vector`, `kv`). Informational in v1. * `provider` (string): backend implementation. Built-in values: * `in-memory` (default): in-process key-value store. No endpoint needed. Data is lost on restart. * `pgvector`: PostgreSQL with the pgvector extension. Full vector-similarity search. Requires `endpoint` (Postgres DSN) and `embedding_model` (ModelEndpoint reference). See [pgvector](../../concepts/memory/providers.md#pgvector). * `http`: delegates to an external HTTP service. Requires `endpoint`. See [HTTP Adapter](../../concepts/memory/providers.md#http-adapter). * Custom providers can be registered via the Go provider registry. * `embedding_model` (string): reference to a ModelEndpoint resource that provides an OpenAI-compatible `/embeddings` API. Required for vector providers like `pgvector`. The endpoint's `base_url`, `auth`, and `default_model` are used to generate embeddings. Resolved in the same namespace by default; use `namespace/name` for cross-namespace references. * `endpoint` (string): connection string or URL. For `pgvector`, a Postgres DSN (e.g. `postgres://user@host:5432/db`). For `http`, the adapter service URL. Not needed for `in-memory`. Mutually exclusive with `endpoint_secret_ref`. * `endpoint_secret_ref` (string): reference to a Secret resource whose first data value contains the full endpoint connection string or URL (including credentials when applicable). Use this instead of `endpoint` when the connection string contains sensitive information (hostnames, internal network topology, passwords). When using a full DSN with embedded password, `auth.secretRef` is not needed. Mutually exclusive with `endpoint`. The controller resolves the Secret and uses the decoded value as the endpoint. * `auth` (object): * `secretRef` (string): reference to a Secret resource containing credentials. For `http`, used as a bearer token. For `pgvector`, injected as the Postgres password into the DSN. Not needed when `endpoint_secret_ref` points to a DSN that already includes the password. #### Built-in Memory Tools When an Agent references a Memory resource via `spec.memory.ref` and explicitly grants operations with `spec.memory.allow`, the runtime exposes the following built-in tools: | Tool | Description | | --------------- | ---------------------------------------------------------- | | `memory.read` | Retrieve a value by key. | | `memory.write` | Store a key-value pair. | | `memory.search` | Search entries by keyword (or vector similarity). | | `memory.list` | List entries, optionally filtered by key prefix. | | `memory.ingest` | Chunk a document into overlapping segments and store them. | These tools do not need to be listed in the agent's `spec.tools` -- they are injected automatically. ### Defaults and Validation * `provider` defaults to `in-memory` when omitted or empty. * `endpoint` or `endpoint_secret_ref` is required when `provider` is `pgvector`, `http`, or any cloud-hosted built-in provider. If both are set, `endpoint_secret_ref` takes precedence. * `embedding_model` is required when `provider` is `pgvector`. It must reference a valid ModelEndpoint. * When `auth.secretRef` is set, the controller resolves the Secret and passes the token to the provider. * The Memory controller validates the provider, resolves auth, and performs a connectivity check (`Ping`). Unsupported providers, missing secrets, or failed connectivity moves the resource to `Error` phase. ### status * `phase`: `Pending`, `Ready`, or `Error`. * `lastError`: description of the most recent error (e.g. unsupported provider, connectivity failure). * `observedGeneration` Example: `examples/resources/memories/research_memory.yaml` See also: [Memory concepts](../../concepts/memory/). ## ModelEndpoint > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. ### spec * `provider` (string, required): provider id (`openai`, `anthropic`, `azure-openai`, `ollama`, `openai-compatible`, `mock`, or registry-added providers). * `base_url` (string) * `default_model` (string, required): the model identifier sent in API requests. * `options` (map\[string]string): provider-specific options. * `auth.secretRef` (string): namespaced reference to a `Secret`. * `allowPrivate` (boolean): for model gateways only, permits trusted local/private model endpoints, including loopback and RFC 1918 / ULA / CGNAT addresses. Cloud metadata, link-local, and unspecified addresses remain blocked. ### Defaults and Validation * `provider` defaults to `openai` and is normalized to lowercase. * `default_model` is required. Validation fails if omitted. * `base_url` defaults by provider: * `openai` -> `https://api.openai.com/v1` * `anthropic` -> `https://api.anthropic.com/v1` * `ollama` -> `http://127.0.0.1:11434` * `openai-compatible` -> (no default; must be set explicitly) * `options` keys are normalized to lowercase; keys/values are trimmed. * `allowPrivate` defaults to `true` for `ollama` and `false` for all other providers. Set it to `true` for local/private `openai-compatible` servers such as vLLM, LM Studio, LocalAI, LiteLLM, or Ollama's `/v1` endpoint. * auth behavior by provider: * `openai`, `anthropic`, `azure-openai`: `auth.secretRef` is required. * `openai-compatible`: `auth.secretRef` is optional. * `ollama`: `auth.secretRef` is optional and usually omitted. * Anthropic credential types (same `auth.secretRef`): * Standard API key (`sk-ant-api...`): sent as `x-api-key`. * OAuth access token (`sk-ant-oat...`, or an explicit `Bearer ` prefix): sent as `Authorization: Bearer`. ### status * `phase`, `lastError`, `observedGeneration` Example: `examples/resources/model-endpoints/*.yaml` See also: [Model endpoint concepts](../../concepts/tools/model-endpoint.md). ## SealedSecret > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. `SealedSecret` is the git-safe counterpart to `Secret`. It stores encrypted secret entries that only `orlojd` can decrypt, then reconciles them into a normal `Secret` with the same name and namespace. ### spec * `encryptedData` (map\[string]object): encrypted secret entries keyed by final secret key name. * `keyId` (string, required): active sealing key identifier used to encrypt this entry. * `wrappedKey` (string, required): base64 RSA-OAEP wrapped AES data key. * `ciphertext` (string, required): base64 `nonce || aes_gcm_ciphertext`. * `template.labels` (map\[string]string): labels copied onto the generated `Secret`. * `template.annotations` (map\[string]string): annotations copied onto the generated `Secret`. In v1, the generated `Secret` always uses the same `metadata.name` and `metadata.namespace` as the `SealedSecret`. ### status * `phase` (string): `Pending`, `Ready`, or `Error`. * `lastError` (string): controller-visible decrypt, key, or ownership conflict error. * `observedGeneration` (int64): latest generation processed by the controller. ### Controller behavior * `orlojd` decrypts `spec.encryptedData` using the active sealing private key. * The resulting `Secret` is written through the normal `Secret` store path, so existing consumers and worker secret resolution do not change. * Generated Secrets are annotated with `orloj.dev/sealedsecret-owner=/`. * If a target `Secret` already exists without that ownership annotation, reconcile fails closed and `status.phase` becomes `Error`. * A background orphan cleanup pass removes generated Secrets whose source `SealedSecret` no longer exists. ### API Endpoints * `POST /v1/sealed-secrets` * `GET /v1/sealed-secrets` * `GET /v1/sealed-secrets/{name}` * `PUT /v1/sealed-secrets/{name}` * `DELETE /v1/sealed-secrets/{name}` * `GET /v1/sealing-key/public` Public key response: ```json { "keyId": "4d8e4f1f7c2b8b27d6f2e2f8d1fef3c5", "algorithm": "rsa-oaep-sha256+aes-256-gcm", "publicKeyPEM": "-----BEGIN PUBLIC KEY-----\n...\n-----END PUBLIC KEY-----\n" } ``` ### CLI Workflow Fetch the active public key: ```bash orlojctl seal public-key ``` Seal a normal `Secret` manifest into a `SealedSecret` manifest: ```bash orlojctl seal secret -f secret.yaml ``` Seal directly from literals without creating `secret.yaml` first: ```bash orlojctl seal secret openai-api-key \ --from-literal value=sk-prod-123 \ --out secrets/openai-api-key.sealed.yaml ``` Then apply the sealed manifest as usual: ```bash orlojctl apply -f secret.sealed.yaml ``` For key generation, storage, crypto details, and a comparison with Bitnami's renewal model, see [Sealing Key Security Model](../../operations/security.md#sealing-key-security-model). See also: * [Secret](./secret.md) * [Secret Handling and production guidance](../../operations/security.md#secret-handling) * [CLI Reference](../cli.md) * [API Reference](../api.md) ## Secret > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. ### spec * `data` (map\[string]string): base64-encoded values. * `stringData` (map\[string]string): write-only plaintext convenience input. ### Defaults and Validation * `stringData` entries are merged into `data` as base64 during normalization. * Every `data` value must be non-empty valid base64. * `stringData` is cleared after normalization (write-only behavior). ### status * `phase`, `lastError`, `observedGeneration` Examples: `examples/resources/secrets/*.yaml` See also: * [Secret concepts](../../concepts/tools/secret.md) * [SealedSecret](./sealed-secret.md) ## TaskApproval > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. Captures a pending human review checkpoint for agent output or final task output. ### spec * `task_ref` (string, required): task waiting for review. * `checkpoint_id` (string, required): stable checkpoint identifier from the `AgentSystem`. * `checkpoint_type` (string): `agent_output` or `task_output`. * `agent` (string): producing agent. * `reason` (string): reviewer-facing instructions. * `ttl` (duration string): time-to-live before expiry. Defaults to `10m`. * `allow_request_changes` (bool): whether reviewers may send the output back for revision. Defaults to `true`. * `max_review_cycles` (int): maximum number of review loops permitted for this checkpoint. Defaults to `3`. * `review_cycle` (int): review iteration number. Defaults to `1`. * `supersedes` (string): previous `TaskApproval` name when this is a re-review cycle. * `output` (string or object): frozen output snapshot presented to the reviewer. * `output_format` (string): `text` or `json`. * `resume_context` (object): runtime-owned context used to resume deterministically after review. ### status * `phase` (string): `Pending`, `Approved`, `Denied`, `ChangesRequested`, `Expired`. * `decision` (string): `approved`, `denied`, or `request_changes`. * `decided_by` (string): reviewer identity. * `decided_at` (string): RFC3339 timestamp of the decision. * `comment` (string): optional reviewer comment. Required by the API for `request_changes`. * `expires_at` (string): RFC3339 expiry timestamp. ### API Endpoints * `POST /v1/task-approvals` * `GET /v1/task-approvals` * `GET /v1/task-approvals/{name}` * `DELETE /v1/task-approvals/{name}` * `POST /v1/task-approvals/{name}/approve` * `POST /v1/task-approvals/{name}/deny` * `POST /v1/task-approvals/{name}/request-changes` Decision body: ```json { "decided_by": "reviewer@example.com", "comment": "Tighten the medical disclaimer and regenerate." } ``` `POST /v1/task-approvals/{name}/request-changes` requires `comment` or the legacy `reason` alias. It returns `409 Conflict` when `allow_request_changes` is `false` or the approval has already reached `max_review_cycles`. See also: * [TaskApproval concept](../../concepts/governance/task-approval.md) * [Task](./task.md) * [AgentSystem](./agent-system.md) * [Human Review Checkpoints guide](../../guides/human-review-checkpoints.md) ## TaskSchedule > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. ### spec * `task_ref` (string): task template reference (`name` or `namespace/name`). * `schedule` (string): 5-field cron expression. * `time_zone` (string): IANA timezone. * `suspend` (bool): stop triggering when `true`. * `starting_deadline_seconds` (int): max lateness window for catch-up. * `concurrency_policy` (string): `forbid` (v1). * `successful_history_limit` (int): retained successful run count. * `failed_history_limit` (int): retained failed/deadletter run count. ### Defaults and Validation * `task_ref` is required and must be `name` or `namespace/name`. * `schedule` is required and must be a valid 5-field cron. * `time_zone` defaults to `UTC`. * `starting_deadline_seconds` defaults to `300`. * `concurrency_policy` defaults to `forbid`. * `successful_history_limit` defaults to `10`. * `failed_history_limit` defaults to `3`. ### status * `phase`, `lastError`, `observedGeneration` * `lastScheduleTime`, `lastSuccessfulTime`, `nextScheduleTime` * `lastTriggeredTask`, `activeRuns` Example: [`examples/resources/task-schedules/`](https://github.com/OrlojHQ/orloj/tree/main/examples/resources/task-schedules) See also: [Task schedule concept](../../concepts/tasks/task-schedule.md) ## TaskWebhook > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. ### spec * `task_ref` (string): template task reference (`name` or `namespace/name`). Mutually exclusive with `task_template`. * `task_template` (object): inline task spec used instead of a separate template Task. Mutually exclusive with `task_ref`. Fields: `system` (required), `priority`, `input`, `max_turns`, `retry`, `message_retry`. * `suspend` (bool): rejects deliveries when `true`. * `auth` (object): * `profile` (string): `generic` (default), `github`, `hmac`, or `shared_token`. * `secret_ref` (string): required secret reference (`name` or `namespace/name`). * `signature_header` (string) * `signature_prefix` (string) * `timestamp_header` (string): used by `generic` and `hmac` with plain header format. * `max_skew_seconds` (int): timestamp tolerance (default `300`). * `algorithm` (string): HMAC hash algorithm -- `sha256` (default), `sha1`, `sha512`. Used with `hmac` profile. * `payload_format` (string): HMAC signing input -- `body`, `timestamp_dot_body`, or `prefix_timestamp_body`. Used with `hmac` profile. * `payload_prefix` (string): literal prefix for `prefix_timestamp_body` format. * `payload_separator` (string): separator between parts (default `.`). Used with `prefix_timestamp_body`. * `signature_encoding` (string): `hex` (default) or `base64`. Used with `hmac` profile. * `header_format` (string): `plain` (default) or `kv_pairs`. Used with `hmac` profile. * `signature_key` (string): key for signature in `kv_pairs` header (e.g., `v1` for Stripe). * `timestamp_key` (string): key for timestamp in `kv_pairs` header (e.g., `t` for Stripe). * `idempotency` (object): * `event_id_header` (string): header containing unique delivery id. * `dedupe_window_seconds` (int): dedupe TTL. * `payload` (object): * `mode` (string): `raw` (v1 only). * `input_key` (string): generated task input key for raw payload. ### Defaults and Validation * Exactly one of `task_ref` or `task_template` must be set. * `task_ref` must be `name` or `namespace/name`. * When `task_template` is set, `system` is required; `priority` defaults to `normal`; retry/message\_retry defaults mirror Task defaults. * `auth.secret_ref` is required. * `auth.profile` defaults to `generic`; supported values: `generic`, `github`, `hmac`, `shared_token`. * profile defaults: * `generic`: * `signature_header` -> `X-Signature` * `signature_prefix` -> `sha256=` * `timestamp_header` -> `X-Timestamp` * `idempotency.event_id_header` -> `X-Event-Id` * `github`: * `signature_header` -> `X-Hub-Signature-256` * `signature_prefix` -> `sha256=` * `idempotency.event_id_header` -> `X-GitHub-Delivery` * `hmac`: `algorithm` -> `sha256`, `payload_format` -> `body`, `signature_encoding` -> `hex`, `header_format` -> `plain`, `payload_separator` -> `.`, `idempotency.event_id_header` -> `X-Event-Id`. `signature_header` is required. `timestamp_header` required when `payload_format` uses a timestamp and `header_format` is `plain`. `signature_key` required when `header_format` is `kv_pairs`. * `shared_token`: `signature_header` is required (the header containing the static token). `idempotency.event_id_header` -> `X-Event-Id`. * `auth.max_skew_seconds` defaults to `300` and must be `>= 0`. * `idempotency.dedupe_window_seconds` must be `>= 0`. Defaults to `259200` (72 hours) for `github` profile or `86400` (24 hours) for `generic` profile. * `payload.mode` defaults to `raw` and only `raw` is allowed in v1. * `payload.input_key` defaults to `webhook_payload`. ### status * `phase`, `lastError`, `observedGeneration` * `endpointID`, `endpointPath` * `lastDeliveryTime`, `lastEventID`, `lastTriggeredTask` * `acceptedCount`, `duplicateCount`, `rejectedCount` Example: `examples/resources/task-webhooks/*.yaml` See also: [Task webhook concepts](../../concepts/tasks/task-webhook.md). ## Task > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. ### spec * `system` (string): target `AgentSystem` name. * `mode` (string): `run` (default) or `template`. * `input` (map\[string]string): task payload. * `priority` (string) * `max_turns` (int, >= 0): required for cyclic graph traversal. * `retry` (object): * `max_attempts` (int) * `backoff` (duration string) * `message_retry` (object): * `max_attempts` (int) * `backoff` (duration string) * `max_backoff` (duration string) * `jitter`: `none`, `full`, `equal` * `non_retryable` (\[]string) * `requirements` (object): * `region` (string) * `gpu` (bool) * `model` (string) ### Defaults and Validation * `input` defaults to `{}`. * `priority` defaults to `normal`. * `mode` defaults to `run`. * `mode=template` marks a task as non-executable template for schedules. * `max_turns` must be `>= 0`. * `retry` defaults: * `max_attempts` -> `1` * `backoff` -> `0s` * `message_retry` defaults: * `max_attempts` -> `retry.max_attempts` * `backoff` -> `retry.backoff` * `max_backoff` -> `24h` * `jitter` -> `full` * `retry.backoff`, `message_retry.backoff`, and `message_retry.max_backoff` must parse as durations. ### status Primary fields: * `phase`: `Pending`, `Running`, `WaitingApproval`, `Succeeded`, `Failed`, `DeadLetter`. * `lastError`, `startedAt`, `completedAt`, `nextAttemptAt`, `attempts` * `output`, `assignedWorker`, `claimedBy`, `leaseUntil`, `lastHeartbeat` * `blocked_on`: exact approval resource currently pausing the task (`kind`, `name`, `reason`) * `observedGeneration` The `WaitingApproval` phase indicates the task is paused pending either a `ToolApproval` or `TaskApproval`. `Task.status.blocked_on` identifies the exact blocker so resume logic is deterministic. Approved reviews transition the task back to `Running` or `Succeeded` depending on the checkpoint. Denied or expired approvals transition the task to `Failed`. Observability arrays: * `trace[]`: detailed execution/tool-call events. * `history[]`: lifecycle transitions. * `messages[]`: message bus records. * `message_idempotency[]`: message idempotency state. * `join_states[]`: fan-in join activation state. * `delegation_states[]`: delegation-gate activation state. Example: [`examples/resources/tasks/`](https://github.com/OrlojHQ/orloj/tree/main/examples/resources/tasks) See also: [Task concept](../../concepts/tasks/task.md) ## ToolApproval > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. Captures a pending human/system approval request for a tool invocation that was flagged by a `ToolPermission` `operation_rules` verdict of `approval_required`. Use `ToolApproval` for "may this tool call happen?" and [TaskApproval](./task-approval.md) for "is this output acceptable to continue?" ### spec * `task_ref` (string, required): name of the Task resource waiting for approval. * `tool` (string, required): tool name that triggered the approval request. * `operation_class` (string): the operation class that requires approval. * `agent` (string): agent that attempted the tool call. * `input` (string): tool input payload (for audit context). * `reason` (string): human-readable reason for the approval request. * `ttl` (duration string): time-to-live before auto-expiry. Defaults to `10m`. ### Defaults and Validation ### status * `phase` (string): `Pending`, `Approved`, `Denied`, `Expired`. Defaults to `Pending`. * `decision` (string): `approved` or `denied`. * `decided_by` (string): identity of the approver/denier. * `decided_at` (string): RFC3339 timestamp of the decision. * `comment` (string): optional reviewer comment. * `expires_at` (string): RFC3339 timestamp when the approval expires. ### API Endpoints * `POST /v1/tool-approvals` -- create an approval request. * `GET /v1/tool-approvals` -- list approval requests (supports namespace and label filters). * `GET /v1/tool-approvals/{name}` -- get a specific approval. * `DELETE /v1/tool-approvals/{name}` -- delete an approval. * `POST /v1/tool-approvals/{name}/approve` -- approve a pending request. Body: `{"decided_by": "...", "comment": "..."}` (`comment` optional; `reason` is still accepted as a compatibility alias). * `POST /v1/tool-approvals/{name}/deny` -- deny a pending request. Body: `{"decided_by": "...", "comment": "..."}` (`comment` optional; `reason` is still accepted as a compatibility alias). See also: [Tool approval concepts](../../concepts/governance/tool-approval.md). ## ToolPermission > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. ### spec * `tool_ref` (string): tool name reference. * `action` (string): action name (commonly `invoke`). * `required_permissions` (\[]string) * `match_mode` (string): `all` or `any` * `apply_mode` (string): `global` or `scoped` * `target_agents` (\[]string): required when `apply_mode=scoped` * `operation_rules` (\[]object): per-operation-class policy verdicts. * `operation_class` (string): `read`, `write`, `delete`, `admin`, or `*` (wildcard). Defaults to `*`. * `verdict` (string): `allow`, `deny`, or `approval_required`. Defaults to `allow`. ### Defaults and Validation * `tool_ref` defaults to `metadata.name` when omitted. * `action` defaults to `invoke`. * `match_mode` defaults to `all`. * `apply_mode` defaults to `global`. * `required_permissions` and `target_agents` are trimmed and deduplicated. * `target_agents` must be non-empty when `apply_mode=scoped`. * `operation_rules` values are trimmed and lowercased. Invalid `operation_class` or `verdict` values are rejected. * When `operation_rules` is present, the authorizer evaluates the tool's `operation_classes` against the rules. The most restrictive matching verdict wins (`deny` > `approval_required` > `allow`). * When `operation_rules` is empty, behavior is unchanged (backward-compatible binary allow/deny). ### status * `phase`, `lastError`, `observedGeneration` Examples: `examples/resources/tool-permissions/*.yaml` See also: [Tool permission concepts](../../concepts/governance/tool-permission.md). ## Tool > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. ### spec * `type` (string): tool type. Allowed values: `http`, `external`, `grpc`, `webhook-callback`, `mcp`, `wasm`, `cli`, `a2a`. Unknown values are rejected at apply time. * `endpoint` (string): tool endpoint URL (or `host:port` for gRPC). * `description` (string): human-readable description of the tool. Passed to model gateways for richer tool definitions. Auto-populated for MCP-generated tools. * `input_schema` (object): JSON Schema for tool parameters. Passed to model gateways for structured parameter definitions. Auto-populated for MCP-generated tools. * `mcp_server_ref` (string): name of the McpServer that provides this tool. Required when `type=mcp`. * `mcp_tool_name` (string): the tool name as reported by the MCP server's `tools/list`. Required when `type=mcp`. * `cli` (object): CLI tool configuration. Required when `type=cli`. * `command` (string): binary path or name to execute. Required. * `args` (\[]string): argument templates. Each entry is evaluated as a Go `text/template` with the parsed JSON input as data context. Each template produces one argv entry. * `image` (string): container image containing the binary. Required when `isolation_mode` is not `none`. * `image_pull_secret` (string): name of a Secret containing registry credentials for pulling `cli.image`. The Secret must contain either a `.dockerconfigjson` key with a complete Docker config JSON, or `registry`, `username`, and `password` keys. Requires `image` to be set. * `network` (string): Docker network mode for the container. Defaults to the operator's `--tool-container-network` setting (`none` by default). Set to `bridge` (or another mode) when the binary needs outbound network access. * `stdin_from_input` (bool): if `true`, pipe the raw model input to the process's stdin. * `output` (string): which streams to return. Allowed values: `stdout` (default), `stderr`, `both`. * `working_dir` (string): working directory inside the container or on the host. * `env` (map\[string]string): literal environment variables. * `env_from` (\[]object): environment variables resolved from secrets. * `name` (string): env var name. Required. * `secretRef` (string): Orloj secret reference. Required. * `key` (string): key within the secret (default: `value`). * `capabilities` (\[]string): declared operations. * `operation_classes` (\[]string): operation class annotations. Allowed values: `read`, `write`, `delete`, `admin`. Used by `ToolPermission.operation_rules` for per-class policy verdicts. * `risk_level` (string): `low`, `medium`, `high`, `critical`. * `runtime` (object): * `timeout` (duration string) * `isolation_mode`: `none`, `sandboxed`, `container`, `kubernetes`, `wasm` * `retry.max_attempts` (int) * `retry.backoff` (duration string) * `retry.max_backoff` (duration string) * `retry.jitter`: `none`, `full`, `equal` * `auth` (object): * `profile` (string): auth profile. Allowed values: `bearer`, `api_key_header`, `basic`, `oauth2_client_credentials`. Defaults to `bearer` when `secretRef` is set. * `secretRef` (string): namespaced secret reference. Required when `profile` is set. * `headerName` (string): custom header name. Required when `profile=api_key_header`. * `tokenURL` (string): OAuth2 token endpoint. Required when `profile=oauth2_client_credentials`. * `scopes` (\[]string): OAuth2 scopes. ### Defaults and Validation * `type` defaults to `http`. Unknown types are rejected with a validation error. `mcp` type tools are typically auto-generated by the McpServer controller; see [Connect an MCP Server](../../guides/connect-mcp-server.md). `cli` type tools invoke local binaries via execve; see [CLI Tool Guide](../../concepts/tools/cli-tool.md). `a2a` type tools delegate execution to remote A2A-compatible agents; see below. * `cli.command` is required when `type=cli`. `cli.image` is required when `isolation_mode` is not `none`. `cli.image_pull_secret` requires `cli.image`. * `cli.output` defaults to `stdout`. When `cli.network` is omitted, the container uses the operator's `--tool-container-network` default (`none`). * `auth` is not supported for `type=cli` tools. Use `cli.env_from` for credential injection. * `auth.profile` defaults to `bearer` when `secretRef` is set. Unknown profiles are rejected. * `auth.headerName` is required when `profile=api_key_header`. * `auth.tokenURL` is required when `profile=oauth2_client_credentials`. * `capabilities` are trimmed and deduplicated (case-insensitive). * `operation_classes` are trimmed, lowercased, and deduplicated. Invalid values are rejected. Defaults to `["read"]` for `low`/`medium` risk, `["write"]` for `high`/`critical` risk. * `risk_level` defaults to `low`. * `runtime.timeout` defaults to `30s` and must parse as duration. * `runtime.isolation_mode` defaults to: * `container` for `type=cli` (regardless of risk level) * `sandboxed` for `high`/`critical` risk (non-CLI) * `none` for `low`/`medium` risk (non-CLI) * `kubernetes` must be set explicitly; requires `--tool-k8s-enabled=true` * `runtime.retry` defaults: * `max_attempts` -> `1` * `backoff` -> `0s` * `max_backoff` -> `30s` * `jitter` -> `none` #### type: a2a A2A tools delegate execution to remote A2A-compatible agents. | Field | Required | Description | | --------------------------- | -------- | ------------------------------------- | | `spec.a2a.agent_url` | Yes | Remote agent A2A endpoint URL | | `spec.a2a.protocol_version` | No | Protocol version override | | `spec.a2a.prefer_streaming` | No | Use streaming when remote supports it | Example: ```yaml apiVersion: orloj.dev/v1 kind: Tool metadata: name: remote-research-agent spec: type: a2a a2a: agent_url: https://remote.example.com/a2a protocol_version: "0.2.1" prefer_streaming: true description: Delegates research tasks to a remote A2A agent. risk_level: medium runtime: timeout: 120s ``` ### status * `phase`, `lastError`, `observedGeneration` Examples: * `examples/resources/tools/*.yaml` * `examples/resources/tools/wasm-reference/wasm_echo_tool.yaml` * `examples/resources/tools/cli_kubectl_tool.yaml` * `examples/resources/tools/cli_gh_tool.yaml` See also: [Tool concepts](../../concepts/tools/tool.md). ## Worker > **Stability: beta** -- This resource kind ships with `orloj.dev/v1` and is suitable for production use, but its schema may evolve with migration guidance in future minor releases. ### spec * `region` (string) * `capabilities.gpu` (bool) * `capabilities.supported_models` (\[]string) * `max_concurrent_tasks` (int) ### Defaults and Validation * `max_concurrent_tasks` defaults to `1` when `<= 0`. ### status * `phase`, `lastError`, `lastHeartbeat`, `observedGeneration`, `currentTasks` Example: `examples/resources/workers/worker_a.yaml` See also: [Worker concepts](../../concepts/infrastructure/worker.md). ## Backup and Restore This guide covers backup and restore procedures for Orloj deployments using the Postgres storage backend. Memory-backend deployments are ephemeral and do not require backup. ### What to Back Up | Component | Location | Required | | --------------------------- | --------------------------------------------- | ---------------------- | | Postgres database | `ORLOJ_POSTGRES_DSN` target | Yes | | Secret encryption key | `ORLOJ_SECRET_ENCRYPTION_KEY` env var or flag | Yes (if secrets exist) | | Server/worker configuration | Flags, env vars, Kubernetes manifests | Recommended | | Monitoring profiles | `monitoring/` directory | Recommended | The secret encryption key is critical. Without it, encrypted `Secret` resource values cannot be decrypted after restore, and `orlojd` cannot unwrap the stored `SealedSecret` private key. Store it separately from the database backup in a secure vault. ### Postgres Backup #### Full Dump ```bash pg_dump "$ORLOJ_POSTGRES_DSN" \ --format=custom \ --file=orloj-backup-$(date +%Y%m%d-%H%M%S).dump ``` `--format=custom` produces a compressed archive that supports selective restore and parallel jobs. #### Automated Scheduled Backup For production, schedule backups with cron or your orchestrator's job scheduler: ```bash # Daily backup with 7-day retention 0 2 * * * pg_dump "$ORLOJ_POSTGRES_DSN" --format=custom \ --file=/backups/orloj-$(date +\%Y\%m\%d).dump \ && find /backups -name "orloj-*.dump" -mtime +7 -delete ``` #### Cloud-Managed Databases If using a managed Postgres service (RDS, Cloud SQL, Azure Database), use the provider's automated backup and point-in-time recovery features instead of `pg_dump`. Ensure the retention window meets your recovery objectives. ### Restore Procedure #### 1. Stop Orloj Services Stop `orlojd` and all `orlojworker` instances to prevent writes during restore. #### 2. Restore the Database Restore to a fresh database or the existing one: ```bash # Create fresh database (recommended) createdb orloj_restored # Restore from dump pg_restore --dbname=orloj_restored \ --clean --if-exists \ --no-owner \ orloj-backup-20260317-020000.dump ``` If restoring to the existing database: ```bash pg_restore --dbname="$ORLOJ_POSTGRES_DSN" \ --clean --if-exists \ --no-owner \ orloj-backup-20260317-020000.dump ``` #### 3. Update DSN (if restored to a new database) Point `ORLOJ_POSTGRES_DSN` to the restored database before restarting services. #### 4. Verify the Encryption Key Ensure `ORLOJ_SECRET_ENCRYPTION_KEY` matches the key that was active when the backup was taken. A mismatched key will cause Secret resource decryption failures at runtime and prevent `SealedSecret` reconciliation. #### 5. Restart and Validate ```bash # Start orlojd ./orlojd --storage-backend=postgres ... # Verify health curl -sf http://127.0.0.1:8080/healthz | jq . # Verify resources are accessible go run ./cmd/orlojctl get agents go run ./cmd/orlojctl get tasks go run ./cmd/orlojctl get workers # Run a smoke load test go run ./cmd/orloj-loadtest \ --base-url=http://127.0.0.1:8080 \ --tasks=10 \ --quality-profile=monitoring/loadtest/quality-default.json ``` ### Point-in-Time Recovery For Postgres deployments with WAL archiving enabled, you can recover to a specific point in time. This requires: 1. A base backup taken before the target recovery point. 2. Continuous WAL archiving to a durable location. 3. Postgres `recovery_target_time` configuration. Refer to the [PostgreSQL PITR documentation](https://www.postgresql.org/docs/current/continuous-archiving.html) for setup details. Cloud-managed databases typically expose PITR as a built-in feature. ### Upgrade Safety Before any Orloj version upgrade: 1. Take a full Postgres backup. 2. Record the current `ORLOJ_SECRET_ENCRYPTION_KEY`. 3. Record the current binary versions and configuration. 4. Proceed with the upgrade per the [Upgrades and Rollbacks](upgrades.md) guide. If the upgrade fails, restore from the backup and revert to the previous binary version. ### Disaster Recovery Checklist * [ ] Postgres backups run on a schedule and are verified periodically. * [ ] Secret encryption key is stored in a secure vault, separate from backups. * [ ] Backup retention meets your recovery point objective (RPO). * [ ] Restore procedure has been tested in a non-production environment. * [ ] Monitoring alerts cover backup job failures. ## Configuration This page is the canonical reference for runtime environment variables and flag-to-env precedence for `orlojd`, `orlojworker`, and `orlojctl`. See also [CLI reference](../reference/cli.md) for exhaustive flag definitions. ### Precedence 1. CLI flags 2. Environment variable fallback 3. Code defaults Example: * `--task-execution-mode` overrides `ORLOJ_TASK_EXECUTION_MODE`. * If neither is set, code defaults apply. ### Runtime Environment Matrix | Variable | Used By | Flag Overrides | Purpose / Conditions | | -------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `ORLOJ_POSTGRES_DSN` | `orlojd`, `orlojworker` | `--postgres-dsn` | Postgres DSN when `--storage-backend=postgres`. | | `ORLOJ_TASK_EXECUTION_MODE` | `orlojd`, `orlojworker` | `--task-execution-mode` | Task execution mode: `sequential` or `message-driven`. | | `ORLOJ_EMBEDDED_WORKER_MAX_CONCURRENT_TASKS` | `orlojd` | `--embedded-worker-max-concurrent-tasks` | Embedded worker default concurrency. | | `ORLOJ_TASK_WORKER_REGION` | `orlojd` | `--task-worker-region` | Region for embedded worker registration. | | `ORLOJ_WORKER_HEALTHZ_ADDR` | `orlojworker` | `--healthz-addr` | Optional worker liveness endpoint bind address. | | `ORLOJ_MODEL_SECRET_ENV_PREFIX` | `orlojd`, `orlojworker` | `--model-secret-env-prefix` | Env prefix for model endpoint `secretRef` lookups. | | `ORLOJ_TOOL_ISOLATION_BACKEND` | `orlojd`, `orlojworker` | `--tool-isolation-backend` | Container isolation backend: `none` or `container`. WASM tools run independently. | | `ORLOJ_TOOL_CONTAINER_RUNTIME` | `orlojd`, `orlojworker` | `--tool-container-runtime` | Container runtime binary for tool isolation. | | `ORLOJ_TOOL_CONTAINER_IMAGE` | `orlojd`, `orlojworker` | `--tool-container-image` | Container image used by isolated tool execution. | | `ORLOJ_TOOL_CONTAINER_NETWORK` | `orlojd`, `orlojworker` | `--tool-container-network` | Container network mode for isolated tools. | | `ORLOJ_TOOL_CONTAINER_MEMORY` | `orlojd`, `orlojworker` | `--tool-container-memory` | Container memory limit for isolated tools. | | `ORLOJ_TOOL_CONTAINER_CPUS` | `orlojd`, `orlojworker` | `--tool-container-cpus` | Container CPU limit for isolated tools. | | `ORLOJ_TOOL_CONTAINER_PIDS_LIMIT` | `orlojworker` | `--tool-container-pids-limit` | Container PID limit for isolated tools. | | `ORLOJ_TOOL_CONTAINER_USER` | `orlojd`, `orlojworker` | `--tool-container-user` | Container user/group for isolated tools. | | `ORLOJ_TOOL_SECRET_ENV_PREFIX` | `orlojd`, `orlojworker` | `--tool-secret-env-prefix` | Env prefix for tool `secretRef` lookups. | | `ORLOJ_TOOL_WASM_MODULE` | `orlojd`, `orlojworker` | `--tool-wasm-module` | Default WASM module path (per-tool `spec.wasm.module` takes precedence). | | `ORLOJ_TOOL_WASM_ENTRYPOINT` | `orlojd`, `orlojworker` | `--tool-wasm-entrypoint` | Default WASM entrypoint function name. | | `ORLOJ_TOOL_WASM_MEMORY_BYTES` | `orlojd`, `orlojworker` | `--tool-wasm-memory-bytes` | Default max memory bytes for WASM runtime. | | `ORLOJ_TOOL_WASM_FUEL` | `orlojd`, `orlojworker` | `--tool-wasm-fuel` | Default WASM execution fuel limit. | | `ORLOJ_TOOL_WASM_WASI` | `orlojd`, `orlojworker` | `--tool-wasm-wasi` | Default: enable WASI host functions for WASM tools. | | `ORLOJ_TOOL_WASM_CACHE_DIR` | `orlojd`, `orlojworker` | `--tool-wasm-cache-dir` | Disk cache directory for remote WASM modules (HTTPS/OCI). Default: `~/.orloj/wasm-cache`. | | `ORLOJ_TOOL_K8S_ENABLED` | `orlojd`, `orlojworker` | `--tool-k8s-enabled` | Enable Kubernetes tool isolation runtime. Default: `false`. | | `ORLOJ_TOOL_K8S_NAMESPACE` | `orlojd`, `orlojworker` | `--tool-k8s-namespace` | Namespace for tool Jobs. Default: current pod namespace or `default`. | | `ORLOJ_TOOL_K8S_SERVICE_ACCOUNT` | `orlojd`, `orlojworker` | `--tool-k8s-service-account` | Service account for tool Pods. | | `ORLOJ_TOOL_K8S_JOB_TTL` | `orlojd`, `orlojworker` | `--tool-k8s-job-ttl` | TTL seconds after Job finishes. Default: `300`. | | `ORLOJ_TOOL_K8S_DEFAULT_IMAGE` | `orlojd`, `orlojworker` | `--tool-k8s-default-image` | Fallback image for HTTP tools without an explicit image. Default: `curlimages/curl:8.8.0`. | | `ORLOJ_EVENT_BUS_BACKEND` | `orlojd` | `--event-bus-backend` | Control-plane event bus backend: `memory` or `nats`. | | `ORLOJ_NATS_URL` | `orlojd`, `orlojworker` | `--nats-url` (server), `--agent-message-nats-url` (runtime bus) | Base NATS URL; also fallback for runtime message bus URL. | | `ORLOJ_NATS_SUBJECT_PREFIX` | `orlojd` | `--nats-subject-prefix` | Subject prefix used for control-plane NATS event bus. | | `ORLOJ_AGENT_MESSAGE_BUS_BACKEND` | `orlojd`, `orlojworker` | `--agent-message-bus-backend` | Runtime message bus backend: `none`, `memory`, `nats-jetstream`. | | `ORLOJ_AGENT_MESSAGE_NATS_URL` | `orlojd`, `orlojworker` | `--agent-message-nats-url` | NATS URL used when runtime bus backend is `nats-jetstream`. | | `ORLOJ_AGENT_MESSAGE_SUBJECT_PREFIX` | `orlojd`, `orlojworker` | `--agent-message-subject-prefix` | Subject prefix for runtime agent messages. | | `ORLOJ_AGENT_MESSAGE_STREAM` | `orlojd`, `orlojworker` | `--agent-message-stream-name` | JetStream stream name for runtime agent messages. | | `ORLOJ_AGENT_MESSAGE_CONSUME` | `orlojworker` | `--agent-message-consume` | Enables worker-side runtime inbox consumers. | | `ORLOJ_AGENT_MESSAGE_CONSUMER_NAMESPACE` | `orlojworker` | `--agent-message-consumer-namespace` | Optional namespace filter for runtime inbox consumers. | | `ORLOJ_API_TOKEN` | `orlojd`, `orlojctl`, `orloj-alertcheck` | `--api-key` (server), `--api-token` (client/checker) | Bearer token fallback for API auth. | | `ORLOJ_API_TOKENS` | `orlojd` | none | Multi-token auth map (`name:token:role` entries; legacy `token:role` supported). | | `ORLOJ_UI_PATH` | `orlojd` | `--ui-path` | Base URL path for the web console (default `/`). | | `ORLOJ_CORS_ALLOWED_ORIGINS` | `orlojd` | `--cors-allowed-origins` | Comma-separated CORS allowed origins. Empty means same-origin only. | | `ORLOJ_TLS_CERT_FILE` | `orlojd` | `--tls-cert-file` | TLS certificate file for native HTTPS. Requires `ORLOJ_TLS_KEY_FILE`. | | `ORLOJ_TLS_KEY_FILE` | `orlojd` | `--tls-key-file` | TLS private key file for native HTTPS. Requires `ORLOJ_TLS_CERT_FILE`. | | `ORLOJ_AUTH_MODE` | `orlojd` | `--auth-mode` | API auth mode (`off`, `native`, `sso`; `sso` unavailable in this distribution). | | `ORLOJ_AUTH_SESSION_TTL` | `orlojd` | `--auth-session-ttl` | Session TTL for native auth mode. | | `ORLOJ_AUTH_RESET_ADMIN_USERNAME` | `orlojd` | `--auth-reset-admin-username` | One-shot local admin reset username. | | `ORLOJ_AUTH_RESET_ADMIN_PASSWORD` | `orlojd` | `--auth-reset-admin-password` | One-shot local admin reset password and exit. | | `ORLOJ_SETUP_TOKEN` | `orlojd` | none | Protects `/v1/auth/setup`; required request value for initial setup when set. | | `ORLOJ_SECRET_ENCRYPTION_KEY` | `orlojd`, `orlojworker` | `--secret-encryption-key` | AES key for encrypting Secret resource data at rest. On `orlojd`, it also wraps the stored `SealedSecret` private key. | | `ORLOJ_SECRET_` | `orlojd`, `orlojworker` | `--model-secret-env-prefix`, `--tool-secret-env-prefix` | Dynamic secret lookup fallback for `secretRef` resolution. | | `ORLOJ_SERVER` | `orlojctl` | `--server` | Default API base URL after `ORLOJCTL_SERVER`. | | `ORLOJCTL_SERVER` | `orlojctl` | `--server` | Highest-precedence env default API base URL. | | `ORLOJCTL_API_TOKEN` | `orlojctl` | `--api-token` | Bearer token for CLI API calls. | | `OTEL_EXPORTER_OTLP_ENDPOINT` | `orlojd`, `orlojworker` | none | OTLP gRPC endpoint for OpenTelemetry traces. Empty disables export. | | `OTEL_EXPORTER_OTLP_INSECURE` | `orlojd`, `orlojworker` | none | Set `true` for non-TLS OTLP in development. | | `ORLOJ_LOG_LEVEL` | `orlojd`, `orlojworker` | `--log-level`, `--debug` | Minimum log level: `debug`, `info` (default), `warn`, or `error`. `--debug` is equivalent to `--log-level=debug` and takes precedence. | | `ORLOJ_LOG_FORMAT` | `orlojd`, `orlojworker` | none | Log format: `json` (default) or `text`. | ### A2A Protocol | Variable | Used By | Flag Overrides | Purpose / Conditions | | ------------------------------------ | ----------------------- | -------------------------------- | ----------------------------------------------------------------------------------------- | | `ORLOJ_A2A_PUBLIC_BASE_URL` | `orlojd` | `--a2a-public-base-url` | Public base URL used in Agent Card `url` fields. Required for externally-reachable cards. | | `ORLOJ_A2A_PROTOCOL_VERSION` | `orlojd` | `--a2a-protocol-version` | A2A protocol version string to advertise in Agent Cards. | | `ORLOJ_A2A_CARD_CACHE_TTL` | `orlojd`, `orlojworker` | `--a2a-card-cache-ttl` | Cache TTL for fetched remote Agent Cards. Default: `5m`. | | `ORLOJ_A2A_ALLOW_PRIVATE_ENDPOINTS` | `orlojd`, `orlojworker` | `--a2a-allow-private-endpoints` | Allow outbound A2A requests to private/loopback IPs. Default: `false`. | | `ORLOJ_A2A_REMOTE_AGENTS` | `orlojd` | `--a2a-remote-agents` | JSON-encoded list of static remote A2A agents to register. | | `ORLOJ_A2A_RATE_LIMIT_ENABLED` | `orlojd` | `--a2a-rate-limit-enabled` | Enable per-IP rate limiting on A2A endpoints. Default: `true`. | | `ORLOJ_A2A_RATE_LIMIT_RPM` | `orlojd` | `--a2a-rate-limit-rpm` | Max JSON-RPC requests per minute per IP when rate limiting is enabled. Default: `30`. | | `ORLOJ_A2A_RATE_LIMIT_MAX_SUBSCRIBE` | `orlojd` | `--a2a-rate-limit-max-subscribe` | Max concurrent SSE subscribe connections globally. Default: `10`. | ### Server and Worker Flags Use [CLI reference](../reference/cli.md) as the exhaustive list for all flags and defaults. Quick grouping: * Server (`orlojd`): auth, storage, embedded worker, control-plane event bus, runtime message bus, model secret resolution, tool isolation. * Worker (`orlojworker`): identity/capacity, storage, runtime inbox consumers, model secret resolution, tool isolation. ### Web Console Path By default, `orlojd` serves the built-in web console at the root path (`/`). The REST API lives under `/v1/...`, `/healthz`, and `/metrics`, so there is no collision. To mount the console at a subpath instead (useful when multiple services share a single reverse proxy hostname): ```bash # Serve the console at https://tools.example.com/orloj/ orlojd --ui-path=/orloj/ # or ORLOJ_UI_PATH=/orloj/ orlojd ``` | Setting | Console URL | API URL | | ----------------------- | ---------------------------------- | ---------------------------------- | | `--ui-path=/` (default) | `https://example.com/` | `https://example.com/v1/...` | | `--ui-path=/console/` | `https://example.com/console/` | `https://example.com/v1/...` | | `--ui-path=/orloj/` | `https://tools.example.com/orloj/` | `https://tools.example.com/v1/...` | The value is normalized to always have a leading and trailing `/`. Client-side routes (e.g. `/tasks/my-task`) are served via SPA fallback so browser refreshes work at any depth. When using a custom DNS (e.g. `console.example.com`), you typically do **not** need to set `ORLOJ_UI_PATH` — the default `/` means the console is at `https://console.example.com/`. Point your DNS and reverse proxy at `orlojd` and everything works. ### Secret Resolution Model endpoints and tools resolve `secretRef` values in this order: 1. Secret resources in the control-plane store. 2. Environment variables with configurable prefixes (`ORLOJ_SECRET_` by default). #### Encryption at Rest Set `--secret-encryption-key` (or `ORLOJ_SECRET_ENCRYPTION_KEY`) on every process sharing the same backing store. * Use one consistent key for all `orlojd`/`orlojworker` processes against the same database. * On `orlojd`, the same key also protects the persisted `SealedSecret` private key. * Rotating keys requires a migration procedure (see security/upgrade runbooks). ### Postgres Tuning #### Connection Pool (main store) The main Postgres pool is configured via CLI flags: | Flag | Default | Description | | ------------------------------ | ------- | ------------------------------------------------- | | `--postgres-max-open-conns` | 20 | Maximum open connections | | `--postgres-max-idle-conns` | 10 | Maximum idle connections kept warm | | `--postgres-conn-max-lifetime` | 30m | Maximum lifetime of a connection before recycling | Idle connections are evicted after 5 minutes to reduce stale TCP connection risk behind firewalls/load balancers. #### Connection Pool (pgvector memory backend) The pgvector backend uses a separate `pgxpool` created from the Memory resource `spec.endpoint` DSN. Tune it with DSN params: ```text postgres://user:pass@host:5432/db?pool_max_conns=10&pool_min_conns=2&pool_max_conn_idle_time=5m&pool_health_check_period=1m ``` | Parameter | Default | Description | | -------------------------- | -------------- | ------------------------------------------ | | `pool_max_conns` | max(4, NumCPU) | Maximum pool size | | `pool_min_conns` | 0 | Minimum warm connections | | `pool_max_conn_lifetime` | 1h | Recycle connections after this duration | | `pool_max_conn_idle_time` | 30m | Close idle connections after this duration | | `pool_health_check_period` | 1m | How often to ping idle connections | #### Statement Timeout Neither the main store nor pgvector backend sets `statement_timeout` by default. Add it via DSN `options`: ```bash # Main store (30-second statement timeout) --postgres-dsn="postgres://user:pass@host:5432/db?options=-c%20statement_timeout%3D30000" # pgvector memory endpoint postgres://user:pass@host:5432/db?options=-c%20statement_timeout%3D30000&pool_max_conns=10 ``` ### Recommended Production Baseline * `orlojd`: `--storage-backend=postgres`, `--task-execution-mode=message-driven`, `--agent-message-bus-backend=nats-jetstream` * `orlojworker`: `--storage-backend=postgres`, `--task-execution-mode=message-driven`, `--agent-message-consume` * Enable `--secret-encryption-key` on all processes when using Secret resources * Configure model/tool credentials via `ORLOJ_SECRET_` or external secret management * Set `OTEL_EXPORTER_OTLP_ENDPOINT` for distributed tracing * See [Observability](./observability.md) for tracing, metrics, and logs setup ### Verification ```bash curl -s http://127.0.0.1:8080/healthz | jq . go run ./cmd/orlojctl get workers go run ./cmd/orlojctl get tasks ``` ## Monitoring and Alerts Use `orloj-alertcheck` and dashboard contracts to validate runtime reliability signals. > For Prometheus metrics, OpenTelemetry tracing, structured logging, and trace visualization, see [Observability](./observability.md). ### Purpose This guide defines repeatable checks for retry storms, dead-letter growth, and latency saturation. ### Artifacts * Alert profile (default): `monitoring/alerts/retry-deadletter-default.json` * Alert profile (CI): `monitoring/alerts/retry-deadletter-ci.json` * Dashboard contract: `monitoring/dashboards/retry-deadletter-overview.json` * Alert check command: `cmd/orloj-alertcheck` The CI profile uses a lower `min_tasks` floor and a higher latency ceiling to accommodate CI runner variability. It is used by the `reliability` job in `.github/workflows/ci.yml`. ### Alert Check Command ```bash go run ./cmd/orloj-alertcheck \ --base-url=http://127.0.0.1:8080 \ --namespace=default \ --profile=monitoring/alerts/retry-deadletter-default.json \ --json=true ``` #### `orloj-alertcheck` Flags | Flag | Default | Description | | -------------------- | ------------------------------------------------- | --------------------------------------------------------------------- | | `--base-url` | `http://127.0.0.1:8080` | Orloj API base URL. | | `--namespace` | `default` | Target namespace for task queries. | | `--api-token` | empty | Optional bearer token for API auth (env fallback: `ORLOJ_API_TOKEN`). | | `--profile` | `monitoring/alerts/retry-deadletter-default.json` | Alert threshold profile JSON file. | | `--task-name-prefix` | empty | Optional filter by task name prefix. | | `--task-system` | empty | Optional filter by `Task.spec.system`. | | `--poll-concurrency` | `20` | Concurrent task metrics fetch workers. | | `--timeout` | `2m` | Global command timeout. | | `--json` | `true` | Emit machine-readable JSON output. | | `--verbose` | `false` | Emit verbose progress logs. | For authoritative defaults and full CLI context, see [CLI reference](../reference/cli.md#orloj-alertcheck). ### Loadtest Reliability Gates and Injection Controls Use `orloj-loadtest` to validate system behavior under expected and fault-injected load patterns. Key reliability gate controls: * `--quality-profile` * `--min-success-rate` * `--max-deadletter-rate` * `--max-failed-rate` * `--max-timed-out` * `--min-retry-total` * `--min-takeover-events` Key injection controls: * `--inject-invalid-system-rate`, `--invalid-system-name` * `--inject-timeout-system-rate`, `--timeout-system-name`, `--timeout-agent-name`, `--timeout-agent-duration` * `--inject-expired-lease-rate`, `--expired-lease-owner` Worker readiness and pacing controls: * `--min-ready-workers`, `--worker-ready-timeout` * `--poll-concurrency`, `--poll-interval`, `--run-timeout` For exhaustive loadtest flags and defaults, see [CLI reference](../reference/cli.md#orloj-loadtest). ### Exit Behavior * `0`: no violations * `2`: one or more alert violations found * `1`: command/config/API failure ### Default Threshold Profile The default profile checks: * retry storm absolute total and per-task rate * dead-letter absolute total and dead-letter task rate * in-flight saturation ceiling * max p95 latency ceiling (complement with `orloj_agent_step_duration_seconds` Prometheus histogram for live percentile queries) * optional `require_any_task_succeeded` ### Dashboard Contract `monitoring/dashboards/retry-deadletter-overview.json` defines backend-agnostic panel expectations for: * retry totals * dead-letter totals * dead-letter task rate * in-flight totals * max p95 latency ## Observability Orloj provides built-in observability through OpenTelemetry tracing, Prometheus metrics, structured logging, and an in-app trace visualization UI. These features work out of the box in OSS deployments and integrate with standard observability backends. ### Trace Visualization (Web Console) The web console includes a **Trace** tab on every task detail page. It renders the `TaskTraceEvent` data that the runtime already records during execution. To view a task trace: 1. Open the web console at `http:///`. 2. Navigate to a task and click into its detail page. 3. Click the **Trace** tab. The trace view shows: * **Summary bar** -- total events, cumulative latency, token count, tool calls, and error count. * **Waterfall timeline** -- each row is one trace event (agent start/end, tool call, model call, error, dead-letter). The horizontal bar shows time offset from task start and duration. * **Filters** -- filter by agent or branch when the task fans out across multiple agents. * **Expandable detail rows** -- click any row to see step ID, attempt, branch, tool name, tokens, error code/reason, and the full message. The trace data comes from `GET /v1/tasks/{name}` (the `status.trace` field). No additional backend is required -- trace events are stored alongside the task resource. ### OpenTelemetry Tracing Orloj emits OpenTelemetry spans for task execution, agent steps, and message processing. Spans are exported via OTLP gRPC to any compatible backend (Jaeger, Grafana Tempo, Datadog, Honeycomb, etc.). #### Enabling OTel Export Set the OTLP endpoint via environment variable: ```bash export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 ``` Or for non-TLS backends in development: ```bash export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 export OTEL_EXPORTER_OTLP_INSECURE=true ``` Both `orlojd` and `orlojworker` initialize the OTel trace provider on startup. When no endpoint is configured, a no-op provider is installed and tracing has zero overhead. #### Span Hierarchy Spans follow the task execution structure: ``` task.execute (root span) ├── agent.execute (one per agent step) │ ├── model.call (model gateway invocations) │ └── tool.execute (tool runtime calls) └── ... ``` For message-driven execution, each message consumption creates a `message.process` span with a nested `agent.execute` span. #### Span Attributes All spans carry `orloj.*` attributes: | Attribute | Description | | ------------------------ | ----------------------------------------------- | | `orloj.task` | Task resource name | | `orloj.system` | AgentSystem resource name | | `orloj.namespace` | Resource namespace | | `orloj.agent` | Agent resource name | | `orloj.step_id` | Step identifier (e.g. `a1.s3`) | | `orloj.attempt` | Current attempt number | | `orloj.tokens.used` | Tokens consumed by this step | | `orloj.tokens.estimated` | Estimated tokens (when exact count unavailable) | | `orloj.tool_calls` | Number of tool invocations | | `orloj.latency_ms` | Step duration in milliseconds | | `orloj.message_id` | Message ID (message-driven mode) | | `orloj.from_agent` | Source agent for message handoff | | `orloj.to_agent` | Destination agent for message handoff | | `orloj.branch_id` | Branch ID for fan-out tracking | | `orloj.tool` | Tool name | | `orloj.tool.attempt` | Tool retry attempt | | `orloj.model` | Model identifier | #### W3C Trace Context Orloj propagates `traceparent` and `tracestate` headers using the W3C Trace Context standard. This means external tools that support W3C propagation will automatically appear as child spans in your traces. #### Dual Write OTel spans are emitted in parallel with the internal `Task.status.trace` events. The internal trace powers the web console trace tab, while OTel spans flow to your external tracing backend. Both views are consistent. ### Prometheus Metrics Orloj exposes a standard Prometheus scrape endpoint at `/metrics` on the `orlojd` HTTP server. The endpoint is unauthenticated (like `/healthz`) so Prometheus can scrape it without API tokens. #### Available Metrics | Metric | Type | Labels | Description | | ----------------------------------- | --------- | ------------------------------- | ------------------------------------------------ | | `orloj_task_duration_seconds` | histogram | `namespace`, `system`, `status` | End-to-end task duration | | `orloj_agent_step_duration_seconds` | histogram | `agent`, `step_type` | Duration of a single agent step | | `orloj_tokens_used_total` | counter | `agent`, `model`, `type` | Tokens consumed (`type` = `used` or `estimated`) | | `orloj_messages_total` | counter | `phase`, `agent` | Message lifecycle transitions | | `orloj_deadletters_total` | counter | `agent` | Messages moved to dead-letter | | `orloj_retries_total` | counter | `agent` | Message retry count | | `orloj_inflight_messages` | gauge | `agent` | Currently in-flight messages | #### Prometheus Scrape Configuration ```yaml scrape_configs: - job_name: orloj static_configs: - targets: ['orlojd:8080'] metrics_path: /metrics scrape_interval: 15s ``` #### Example Queries Task success rate over the last hour: ```promql sum(rate(orloj_task_duration_seconds_count{status="succeeded"}[1h])) / sum(rate(orloj_task_duration_seconds_count[1h])) ``` Token consumption by agent: ```promql sum by (agent) (rate(orloj_tokens_used_total{type="used"}[5m])) ``` Dead-letter rate by agent: ```promql sum by (agent) (rate(orloj_deadletters_total[5m])) ``` P95 agent step latency: ```promql histogram_quantile(0.95, sum by (le, agent) (rate(orloj_agent_step_duration_seconds_bucket[5m]))) ``` ### Structured Logging Both `orlojd` and `orlojworker` emit structured JSON logs by default. Log output can be configured via the `ORLOJ_LOG_FORMAT` environment variable, and log verbosity can be configured with `ORLOJ_LOG_LEVEL`, `--log-level`, or the `--debug` shortcut. #### Configuration | Variable | Values | Default | Description | | ------------------ | -------------------------------- | ------- | --------------------------------------------------------------------------------------------------------- | | `ORLOJ_LOG_LEVEL` | `debug`, `info`, `warn`, `error` | `info` | Minimum log level. Use `debug` when investigating scheduling, worker, message bus, and runtime decisions. | | `ORLOJ_LOG_FORMAT` | `json`, `text` | `json` | Log output format. Use `text` for local development. | Local debugging example: ```bash ORLOJ_LOG_FORMAT=text go run ./cmd/orlojd --debug --storage-backend=memory --embedded-worker ``` Kubernetes/Helm deployments should usually set the environment variable instead: ```yaml runtimeConfig: ORLOJ_LOG_LEVEL: debug ``` #### Log Fields All log entries include a `service` field (`orlojd` or `orlojworker`). When processing HTTP requests, entries also include: * `request_id` -- unique ID for the request (propagated from `X-Request-ID` header or auto-generated) When OpenTelemetry is enabled, log entries from traced code paths include: * `trace_id` -- OTel trace ID for correlation with spans * `span_id` -- OTel span ID #### Request ID Propagation The HTTP server automatically generates a request ID for each incoming request and returns it in the `X-Request-ID` response header. If the client sends an `X-Request-ID` header, it is reused. This enables end-to-end request correlation across services. #### Correlating Logs with Traces In Grafana, you can use the `trace_id` field to link from a log entry directly to the corresponding trace in Tempo or Jaeger. The trace ID in logs matches the OTel trace ID in exported spans. ### Docker Compose Example To run Orloj with Jaeger and Prometheus in a local development stack: ```yaml services: jaeger: image: jaegertracing/jaeger:2 ports: - "16686:16686" # Jaeger UI - "4317:4317" # OTLP gRPC prometheus: image: prom/prometheus:latest volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml ports: - "9090:9090" orlojd: image: orloj:latest command: > orlojd --embedded-worker environment: OTEL_EXPORTER_OTLP_ENDPOINT: jaeger:4317 OTEL_EXPORTER_OTLP_INSECURE: "true" ORLOJ_LOG_FORMAT: json ports: - "8080:8080" ``` With the corresponding `prometheus.yml`: ```yaml scrape_configs: - job_name: orloj static_configs: - targets: ['orlojd:8080'] scrape_interval: 15s ``` ### CLI Trace Inspection For operators who prefer the CLI, `orlojctl trace task` prints the full trace timeline: ```bash go run ./cmd/orlojctl trace task my-task ``` This is useful for quick debugging without opening the web console or an external tracing backend. ### Related Docs * [Monitoring and Alerts](./monitoring-alerts.md) -- `orloj-alertcheck` threshold profiles and dashboard contracts * [Configuration](./configuration.md) -- all environment variables and CLI flags * [Troubleshooting](./troubleshooting.md) -- diagnosis workflows * [Runbook](./runbook.md) -- production operations * [API Reference](../reference/api.md) ## Operations Runbook Use this runbook for baseline production operation and incident response. ### Reference Topology 1. `orlojd` server 2. Postgres state backend 3. NATS JetStream for message-driven execution 4. multiple `orlojworker` instances ### Startup Procedure 1. Start Postgres and NATS. 2. Start `orlojd` with `--storage-backend=postgres` and `--task-execution-mode=message-driven`. 3. Start at least two workers with `--agent-message-consume`. 4. Configure model provider and credentials. 5. Apply required resources (`ModelEndpoint`, `Tool`, `Agent`, `AgentSystem`, `Task`, governance resources). ### Verification ```bash curl -s http://127.0.0.1:8080/healthz | jq . go run ./cmd/orlojctl get workers go run ./cmd/orlojctl get tasks curl -s http://127.0.0.1:8080/metrics | head -20 ``` Expected result: * API health endpoint reports healthy. * Workers report `Ready` and heartbeat updates. * Tasks transition through expected lifecycle. * `/metrics` returns Prometheus text output with `orloj_*` metrics. ### Failure and Recovery Expectations * Worker crash: lease expires and another worker can claim. * Retry behavior: delayed requeue until success or dead-letter. * Policy/graph validation failures: non-retryable, deterministic dead-letter. * Tool runtime denials/errors: normalized metadata in trace/log paths. ### Observability * Configure `OTEL_EXPORTER_OTLP_ENDPOINT` on both `orlojd` and `orlojworker` for distributed tracing. * Prometheus scrapes `/metrics` on the `orlojd` HTTP port -- add the target to your Prometheus scrape config. * Logs are structured JSON by default (`ORLOJ_LOG_FORMAT=json`) with `request_id` and `trace_id` fields. * The web console Trace tab shows task execution waterfall without any external backend. * See [Observability](./observability.md) for full setup details. ### Reliability Operations * Run `go run ./cmd/orloj-loadtest` for repeatable load/failure validation. * Run `go run ./cmd/orloj-alertcheck` to validate retry/dead-letter thresholds. * Keep alert and load profile thresholds aligned with SLO targets. ### Related Docs * [Observability](./observability.md) * [Deployment Overview](../deploy/) * [VPS Deployment](../deploy/vps.md) * [Kubernetes Deployment](../deploy/kubernetes.md) * [Configuration](./configuration.md) * [Troubleshooting](./troubleshooting.md) * [Upgrades and Rollbacks](./upgrades.md) ## Security and Isolation This page describes current runtime security controls and expected operator practices. ### Current Controls * `AgentPolicy` gates model/tool/token usage. * `AgentRole` and `ToolPermission` enforce per-tool authorization. * `ToolApproval` and `TaskApproval` pause risky actions or sensitive outputs for explicit human review. * Tool runtime enforces timeout/retry/isolation policy from `Tool.spec.runtime`. * Unsupported tools and disallowed runtime requests fail closed. * Permission denials are terminal for the current execution path. For regulated environments, `TaskApproval` checkpoints add a second control plane beyond tool authorization: a human can review and approve, deny, or request changes on sensitive agent output before the workflow continues. ### Namespace Isolation Namespaces are an **organizational boundary**, not a security boundary. Any authenticated user with the correct role (e.g., `reader`, `writer`, `admin`) can access resources in any namespace. There is no per-namespace access control by default. For deployments that require per-namespace or per-resource authorization, the server exposes a `ResourceAuthorizer` extension point (see `ServerOptions.ResourceAuthorizer` in `api/auth_context.go`). A custom authorization layer can implement this interface to enforce fine-grained policies based on the caller's identity, the target namespace, resource type, and HTTP method. This hook is nil by default and all requests that pass the role check are permitted. #### Multi-tenant authorization (Enterprise) Fine-grained, per-namespace tenant isolation — mapping principals to the namespaces they may access, with separation of duties between tenants — is provided by **Orloj Enterprise**, which ships a managed `ResourceAuthorizer` implementation along with SSO/SCIM identity mapping and audit integration. Contact the Orloj team for access. Self-hosters on the open-source build can implement the `ResourceAuthorizer` interface themselves. The hook receives the caller's identity, HTTP method, resource type, namespace, and name, and returns an allow/deny decision: ```go type ResourceAuthorizer interface { AuthorizeResource(r *http.Request, method, resourceType, namespace, name string) (allowed bool, statusCode int, message string) } ``` A minimal policy reads the authenticated identity (`api.AuthIdentityFromRequest`), permits `admin`-role callers cluster-wide, and otherwise checks the caller against an allowed-namespace set. Wire your implementation in via `ServerOptions.ResourceAuthorizer`; it is nil by default, so all requests that pass the built-in role check are permitted. For true multi-tenant isolation, combine namespace authorization with separate API tokens per tenant, network policies between workloads, and per-tenant secret scoping. ### Control plane API tokens The HTTP API (including `orlojctl`) authenticates automation with **`Authorization: Bearer `** when you enable token validation on the server. Orloj **does not** mint or email API keys: the **operator** chooses a secret string, configures it on `orlojd`, and distributes the **same** value to people and CI that need API access. **See also:** [Remote CLI and API access](../deploy/remote-cli-access.md) — end-to-end flow for self-hosters (env vars, `orlojctl config`, `config.json` lifecycle). This is separate from **native UI sign-in** (`--auth-mode=native`), which uses an admin username/password and **session cookies** in the browser. The CLI does not use that password for API calls; use a bearer token as below (or run with auth disabled in trusted dev environments only). #### 1. Generate a token Use a cryptographically random value (length is flexible; treat it like a password): ```bash # Hex (64 characters); easy to paste into env files openssl rand -hex 32 # Or base64 (~44 characters) openssl rand -base64 32 ``` Store the output in your secrets manager, Kubernetes Secret, or password manager—**not** in git. #### 2. Configure the server Pick **one** of these (same token string you generated). **Prefer environment variables** over CLI flags — values passed via `--api-key`, `--secret-encryption-key`, or `--auth-reset-admin-password` are visible in process listings (`ps`) and the server logs a warning when a secret flag is used. * **Environment (recommended):** `ORLOJ_API_TOKEN=''` * **Flag:** `orlojd --api-key=''` (env fallback when unset; see server help) For **multiple** distinct tokens with different roles (reader vs admin-style access), use: ```bash export ORLOJ_API_TOKENS='reader-bot:reader-token-here:reader,automation-bot:automation-token-here:admin' ``` Format is comma-separated `name:token:role` entries. Legacy `token:role` entries are still accepted for backward compatibility. A2A invoke-only tokens use `name:token:a2a:namespace/system|other-system` and can invoke only those A2A-enabled AgentSystems. When `ORLOJ_API_TOKENS` is set, it populates the token map and a single `ORLOJ_API_TOKEN` is only used if that list is empty (see `loadAuthConfig` in `api/authz.go`). For runtime-managed tokens (no server restart required), use: ```bash orlojctl create token --role orlojctl get tokens orlojctl delete token ``` When creating an `a2a` role token through the API, include `a2a_agent_systems` with the allowed AgentSystem refs. Native-auth browser sessions are not accepted for A2A JSON-RPC; external A2A callers must use bearer tokens. #### 3. Configure clients (`orlojctl` and automation) Use the **same** token the server expects: * **Environment:** `ORLOJ_API_TOKEN` or `ORLOJCTL_API_TOKEN` * **Flag:** `orlojctl --api-token '' ...` * **Profile:** `orlojctl config set-profile ... --token-env VAR` so the token stays in the environment, not on disk See [Remote CLI and API access](../deploy/remote-cli-access.md) for client precedence, default `--server` resolution, and profiles. #### 4. Native auth mode and APIs If you use `--auth-mode=native`, the UI still requires a bearer token (or session cookie) for protected API routes. Configure `ORLOJ_API_TOKEN` / `--api-key` on the server so `orlojctl` and other API clients can authenticate with `Authorization: Bearer`—the admin password alone is not used for programmatic access. #### 5. Initial setup protection When deploying with `--auth-mode=native` on a network-exposed instance, set `ORLOJ_SETUP_TOKEN` to prevent unauthorized admin account creation. When this variable is set, the `/v1/auth/setup` endpoint requires a matching `setup_token` field in the JSON request body: ```json { "username": "admin", "password": "...", "setup_token": "your-setup-token-here" } ``` The comparison uses constant-time comparison to prevent timing side-channels. Without `ORLOJ_SETUP_TOKEN`, the setup endpoint is open to the first caller (protected only by rate limiting). #### 6. Authentication rate limiting Authentication endpoints (`/v1/auth/login`, `/v1/auth/setup`, `/v1/auth/change-password`, `/v1/auth/admin/reset-password`) are rate-limited per client IP address. The default policy allows 10 requests per minute sustained with a burst of 20 to accommodate legitimate multi-step flows. Requests that exceed the limit receive HTTP 429. ##### Trusted proxy configuration By default, the rate limiter ignores `X-Forwarded-For` and `X-Real-IP` headers and uses the TCP peer address (`RemoteAddr`) to identify clients. This prevents attackers from bypassing rate limits by rotating spoofed forwarding headers. If Orloj runs behind a reverse proxy or load balancer, configure `--trusted-proxies` (env: `ORLOJ_TRUSTED_PROXIES`) with the CIDR(s) of your proxy so the server can extract the real client IP from forwarding headers: ```bash # Single proxy orlojd --trusted-proxies='10.0.0.0/8' # Multiple proxies orlojd --trusted-proxies='10.0.0.0/8,172.16.0.0/12' # Single IP (treated as /32) export ORLOJ_TRUSTED_PROXIES='192.168.1.50' ``` When trusted proxies are configured, `X-Forwarded-For` is parsed right-to-left: entries added by trusted proxies are skipped, and the first untrusted entry is used as the client IP. If the immediate peer is not in the trusted set, forwarding headers are ignored regardless of their content. **Without `--trusted-proxies`**, all requests arriving through a proxy share a single rate-limit bucket (the proxy's IP). The server logs a warning when it detects forwarding headers but has no trusted proxies configured. The same trust gate applies to `X-Forwarded-Proto` for session cookie security: the `Secure` flag is only set based on the forwarding header when the peer is a trusted proxy. ### A2A Security When A2A protocol support is enabled, additional security considerations apply. #### SSRF Protection Outbound A2A requests (fetching remote Agent Cards and sending JSON-RPC calls) use the same `SafeHTTPClient` and `ValidateEndpointURL` checks described in [SSRF Protection](#ssrf-protection). By default, loopback, link-local, cloud metadata, and private network addresses are blocked. #### Auth Enforcement Inbound A2A JSON-RPC requests are subject to authentication based on the target AgentSystem's `spec.a2a.auth` policy: * **`bearer`** (default): requires a valid bearer token with `a2a`, `writer`, or `admin` role. Scoped `a2a` tokens can only invoke systems listed in their `a2a_agent_systems`. This is the same enforcement as other protected API endpoints. * **`public`**: allows unauthenticated A2A invoke for that specific system, even when instance-wide auth is configured. Control-plane APIs (`/v1/agents`, `/v1/tools`, etc.) remain protected. Invalid tokens are still rejected (401) — only missing tokens are permitted. This enables a common production pattern: admin secret for `orlojctl` / control-plane APIs, plus public unauthenticated A2A invoke for selected systems, all on the same `orlojd` instance. Agent Card discovery (GET) is always public regardless of `spec.a2a.auth`. Public systems' Agent Cards omit `authentication.schemes` so A2A clients know not to send tokens. The A2A registry (`GET /v1/a2a/agents`) shows only public systems to unauthenticated callers and all accessible systems to authenticated callers. #### Private Endpoint Risks Setting `--a2a-allow-private-endpoints=true` (env: `ORLOJ_A2A_ALLOW_PRIVATE_ENDPOINTS`) permits outbound A2A requests to private and loopback IPs. This weakens SSRF protection and should only be enabled in trusted network environments (e.g., when remote A2A agents run on the same private network). Cloud metadata endpoints remain blocked regardless of this setting. #### Production Recommendations * Keep `allowPrivateEndpoints` disabled unless remote agents are on a trusted private network. * Use TLS for all A2A endpoints to protect task payloads in transit. * Configure `a2a.rateLimit` to prevent abuse of inbound A2A JSON-RPC endpoints. * Review remote agent URLs before adding them to `--a2a-remote-agents` or the Helm `a2a.remoteAgents` list. * Monitor A2A request metrics for anomalous traffic patterns. ### Tool Types All tool types (`http`, `external`, `grpc`, `webhook-callback`, `mcp`, `cli`, `wasm`, `a2a`) flow through the governed runtime pipeline, so policy enforcement, retry, timeout, and error handling behave identically regardless of transport. See [Tools](../concepts/tools/tool.md) for type details. #### gRPC TLS gRPC tool connections require TLS (minimum TLS 1.2) by default. Plaintext gRPC is available as an opt-in for development environments only. Production deployments should always use the default TLS transport. #### SSRF Protection Outbound HTTP, gRPC, and MCP connections validate the target endpoint twice: once at call time (URL parsing and scheme allowlist) and again at dial time via a `net.Dialer.Control` hook that inspects the actual IP the kernel is about to connect to. Dial-time enforcement closes the hostname-bypass and DNS-rebinding gaps that a URL-only check cannot catch. For generic tool and MCP egress, the following destinations are blocked regardless of configuration: * Loopback addresses (`127.0.0.0/8`, `::1`, and IPv4-mapped IPv6 equivalents like `::ffff:127.0.0.1`) * Link-local addresses (`169.254.0.0/16`, `fe80::/10`) * Cloud metadata endpoints (`169.254.169.254` for AWS/GCP/Azure IMDS, `fd00:ec2::254` for AWS IMDSv2 IPv6) * Unspecified addresses (`0.0.0.0`, `::`) Private network addresses (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `fc00::/7`) and RFC 6598 carrier-grade NAT (`100.64.0.0/10`) are also blocked unless the caller explicitly opts in. `ModelEndpoint` resources use a model-gateway-specific safe client. Set `spec.allowPrivate: true` only for trusted local/private model servers; it permits loopback plus private/CGNAT addresses for model traffic while still blocking cloud metadata, link-local, and unspecified addresses. The default is `false` for all providers except `ollama`, which defaults to `true` because Ollama is a local-first runtime. **Upgrading from earlier versions:** if you run an OpenAI-compatible server (vLLM, LM Studio, LocalAI, LiteLLM proxy, TGI, etc.) on localhost or a private network under `provider: openai-compatible`, add `spec.allowPrivate: true` to those `ModelEndpoint` resources before upgrading, or the gateway will fail at dial time with an error that names the resolved IP and the exact field to change. #### MCP Server Security `McpServer` resources connect to external MCP (Model Context Protocol) servers that expose tools for agent use. Security considerations vary by transport: * **stdio** (`transport: stdio`): The MCP server runs as a subprocess managed by Orloj. The `command` and `args` fields control exactly what binary is executed. The subprocess inherits only the environment variables explicitly listed in `spec.env` and resolved `spec.env[].secretRef` values -- no host environment leaks into the child process. * **HTTP** (`transport: http`): The MCP server is a remote endpoint. SSRF validation (above) applies to the `spec.endpoint` URL, blocking loopback, link-local, and private-network targets by default. Use `spec.auth` to attach bearer or API-key credentials to outbound requests. **Tool scoping:** Use `spec.tool_filter.include` to restrict which tools the MCP server exposes. Without a filter, all tools reported by `tools/list` are generated as `Tool` resources. In production, prefer an explicit allowlist to minimize attack surface. **Credential injection:** Secrets referenced via `spec.env[].secretRef` follow the same [secret resolution chain](#secret-handling) as other resources. Avoid placing credentials in `spec.env[].value` plaintext fields outside of development. **Governed runtime:** Tools discovered from MCP servers are generated as standard `Tool` resources with `spec.type: mcp`. They flow through the same governed runtime pipeline (policy enforcement, retry, auth injection, approvals) as all other tool types. See [MCP Server concept](../concepts/tools/mcp-server.md) and the [Connect an MCP Server](../guides/connect-mcp-server.md) guide for setup details. ### Isolation Modes * `none` -- direct execution with real HTTP/gRPC calls (no isolation boundary) * `sandboxed` -- restricted container with secure defaults (see below) * `container` -- per-invocation isolated container * `kubernetes` -- ephemeral Kubernetes Job (see below) * `wasm` -- WebAssembly module with host-guest stdin/stdout boundary Container backend supports constrained execution for high-risk paths. WASM backend uses executor-factory boundaries and command-backed runtime execution (default runtime binary `wasmtime`). Invalid wasm runtime configuration fails closed with deterministic non-retryable policy errors. #### Sandboxed Container Defaults When `isolation_mode=sandboxed` (the default for `high`/`critical` risk tools), the container backend enforces these security constraints: | Control | Value | | -------------------- | ---------------------------------- | | Filesystem | `--read-only` | | Linux capabilities | `--cap-drop=ALL` | | Privilege escalation | `--security-opt no-new-privileges` | | Network | `--network none` | | User | `65532:65532` (non-root) | | Memory | `128m` | | CPU | `0.50` cores | | Process limit | `64` PIDs | These defaults are enforced by `SandboxedContainerDefaults()` in the runtime and validated by conformance tests. Override with `--tool-container-*` flags only when necessary. #### Kubernetes Isolation When `isolation_mode: kubernetes`, tool invocations run as ephemeral Kubernetes Jobs. This provides cluster-native isolation without requiring a Docker socket on worker nodes. #### Agent Execution on Kubernetes When `--agent-k8s-enabled=true`, each agent in a multi-agent task runs as an ephemeral Kubernetes Job. This isolates agent execution at the pod level. **Security properties:** * **Ephemeral Pods**: Each agent execution creates a dedicated Job and Pod. No state persists between executions. * **Configurable service accounts**: Agent Pods run under a dedicated service account (`--agent-k8s-service-account`), separate from the orchestrator's service account. * **Resource limits**: Default memory and CPU limits are enforced on every agent Pod (`--agent-k8s-default-memory`, `--agent-k8s-default-cpu`). * **Timeout enforcement**: Agent-level `spec.limits.timeout` sets `activeDeadlineSeconds` on the Job. * **Automatic cleanup**: Completed Jobs are garbage-collected via `ttlSecondsAfterFinished` (default: 600s). * **Deterministic naming**: Job names are derived from the task, agent, and attempt, enabling crash recovery without orphaned resources. * **Transparent fallback**: Agents with Docker-dependent tools (container isolation or stdio MCP servers with images) fall back to in-process execution automatically. * **No privilege escalation**: Agent Pods do not mount the orchestrator's filesystem, Docker socket, or service account token. **Operational guidance:** * Use a dedicated namespace (`--agent-k8s-namespace`) to isolate agent Jobs. * Apply NetworkPolicies to restrict agent Pod egress. * Set resource quotas on the agent namespace to bound total resource consumption. * Monitor Job completion and cleanup for stuck or leaked Pods. **Security properties:** * **Ephemeral Pods**: Each tool invocation creates a dedicated Job and Pod. No state persists between invocations. * **Configurable service accounts**: Tool Pods run under a dedicated service account (`--tool-k8s-service-account`), enabling least-privilege RBAC for the tool workload itself. * **Resource limits**: `spec.cli.resources` (memory, CPU) map to Kubernetes resource requests and limits on the Pod spec, enforced by the kubelet. * **Timeout enforcement**: `spec.runtime.timeout` sets `activeDeadlineSeconds` on the Job, ensuring runaway tools are killed by the cluster. * **Automatic cleanup**: Completed Jobs are garbage-collected via `ttlSecondsAfterFinished` (default: 300s, configurable with `--tool-k8s-job-ttl`). * **Network isolation**: Kubernetes NetworkPolicies can restrict tool Pod egress/ingress independently of worker Pod network rules. This replaces the Docker `--network` flag. * **No privilege escalation**: Tool Pods do not mount the worker's filesystem, Docker socket, or service account token. **Operational guidance:** * Apply NetworkPolicies to the tool Job namespace to restrict egress to only required endpoints. * Use a dedicated namespace (`--tool-k8s-namespace`) to isolate tool Jobs from application workloads. * Monitor Job completion rates and cleanup to detect stuck or leaked Pods. * Set resource quotas on the tool namespace to bound total resource consumption from tool invocations. #### CLI Tool Isolation CLI tools (`spec.type: cli`) default to `container` isolation regardless of risk level. Container network mode inherits the operator's `--tool-container-network` setting, which defaults to **`none`** (same as HTTP container tools). This keeps CLI tools network-isolated unless the operator or tool spec opts in. Tools that call external APIs (e.g., `kubectl`, `gh`, `aws`) must set `spec.cli.network: bridge` (or another appropriate mode) explicitly. **Security properties:** * **No shell**: invocations use `exec.CommandContext` (execve-style argv). There is no `sh -c` path and no opt-in for shell mode. * **Arg templates are per-entry**: each Go template produces exactly one argv element. No shell splitting or word expansion occurs. * **Secrets via env\_from only**: process environment is constructed exclusively from `spec.cli.env` (literals) and `spec.cli.env_from` (resolved secrets). No host environment variables leak into the container. * **Binary allowlist** (optional): `--cli-tool-allowed-commands` rejects commands not on the list before exec. * **Argv length limit**: `--cli-tool-max-argv-length` (default 4096 bytes) prevents oversized argument lists. * **`spec.auth` rejected**: CLI tools must use `env_from` for credentials; setting `spec.auth` produces a validation error to prevent silent misconfiguration. Set `spec.cli.network: bridge` (or another mode) when a CLI tool needs outbound network access. Leave it unset or set `none` for tools that operate on stdin/stdout only (e.g., `jq`, `yq`). ### Secret Handling Orloj resolves secrets referenced by `secretRef` fields (on ModelEndpoint and Tool resources) using a chain of resolvers, tried in order: 1. **Resource Store** -- looks up a `Secret` resource by name and reads the base64-encoded value from `spec.data`. 2. **Environment Variables** -- looks up `ORLOJ_SECRET_` (configurable prefix via `--model-secret-env-prefix` / `--tool-secret-env-prefix`). The first resolver that returns a value wins. #### Development Use `Secret` resources for local development. The fastest way is the imperative CLI command -- no YAML file needed: ```bash orlojctl create secret openai-api-key --from-literal value=sk-your-key-here ``` Or with a YAML manifest: ```yaml apiVersion: orloj.dev/v1 kind: Secret metadata: name: openai-api-key spec: stringData: value: sk-your-key-here ``` #### Encryption at Rest When using the Postgres storage backend, enable encryption at rest for `Secret` resources by passing a 256-bit AES key to both `orlojd` and `orlojworker`: ```bash # Generate a key (hex-encoded, 64 characters) openssl rand -hex 32 # Pass via environment variable (recommended) export ORLOJ_SECRET_ENCRYPTION_KEY= orlojd ... orlojworker ... # Or via flag (logs a warning; prefer env in production) orlojd --secret-encryption-key= ... orlojworker --secret-encryption-key= ... ``` When enabled, all `Secret.spec.data` values are encrypted with AES-256-GCM before being written to the database and decrypted transparently on read. This protects secrets against direct database access, backup exposure, and log/dump leaks. The key must be identical across all server and worker processes that share the same database. Both hex-encoded (64 characters) and base64-encoded (44 characters) formats are accepted. **Without** an encryption key, `Secret` data is stored as base64-encoded plaintext in the JSONB payload -- suitable for development but not for production. On `orlojd`, the same `--secret-encryption-key` / `ORLOJ_SECRET_ENCRYPTION_KEY` setting also wraps the private key used for `SealedSecret` decryption when sealing is enabled. If no encryption key is configured, `SealedSecret` resources remain storable but reconcile to `Error`, and `GET /v1/sealing-key/public` returns `503`. #### Git-safe Sealed Secrets `Secret` resources protect values in the API and optionally at rest in Postgres, but they are still plaintext manifests before apply. Use `SealedSecret` when you need to commit encrypted secret manifests to git. The workflow is: 1. `orlojd` creates or loads one active sealing keypair in the control plane. 2. Clients fetch the public key from `GET /v1/sealing-key/public` or `orlojctl seal public-key`. 3. Clients convert a normal `Secret` manifest into a `SealedSecret` manifest locally with `orlojctl seal secret -f secret.yaml`, or generate one directly from literals with `orlojctl seal secret --from-literal key=value`. 4. `orlojd` decrypts the `SealedSecret` and writes a normal `Secret` through the existing secret store path. 5. Workers continue to read the generated `Secret` exactly as they do for manually created secrets. `SealedSecret` and the generated `Secret` use the same name and namespace in v1. Generated Secrets are marked with `orloj.dev/sealedsecret-owner=/`. If a Secret with that name already exists and is not owned by the same `SealedSecret`, reconcile fails closed instead of overwriting user-managed data. Examples: ```bash # Seal an existing Secret manifest into secret.sealed.yaml orlojctl seal secret -f secret.yaml # Generate a SealedSecret file directly from literals orlojctl seal secret openai-api-key \ --from-literal value=sk-prod-123 \ --out secrets/openai-api-key.sealed.yaml ``` #### Sealing Key Security Model Orloj v1 uses one active control-plane sealing keypair per backing store. * `orlojd` only generates a sealing key if no active key exists and `ORLOJ_SECRET_ENCRYPTION_KEY` is set. Startup loads an existing active key when present; it does not generate a new key on every restart. * The generated sealing keypair is RSA-4096. * The sealing private key is stored in Postgres encrypted with AES-256-GCM under `ORLOJ_SECRET_ENCRYPTION_KEY`. * Each `SealedSecret` entry uses a fresh random 32-byte AES data key. The entry plaintext is encrypted with AES-256-GCM, and the AES data key is wrapped with RSA-OAEP-SHA256. * The AES-GCM authenticated data binds the ciphertext to ``, ``, and the secret entry key. A ciphertext copied to a different secret identity will fail to decrypt. Operationally, this means: * A committed `SealedSecret` manifest is safe to store in git as long as the control-plane private key remains protected. * If an attacker gets both the database and `ORLOJ_SECRET_ENCRYPTION_KEY`, they can recover the stored sealing private key. * If an attacker gets code execution on `orlojd`, they can unseal secrets. * Losing `ORLOJ_SECRET_ENCRYPTION_KEY` makes both encrypted `Secret` data and the stored sealing private key unrecoverable. * Orloj v1 does not rotate sealing keys automatically yet; it keeps one active key until a future manual rotation flow is introduced. #### Production For production, choose one or both of the following approaches: **1. Encrypted Secret resources** -- enable `--secret-encryption-key` and continue using `Secret` resources as in development. This is the simplest upgrade path. **2. SealedSecret manifests** -- keep declarative secret manifests in git without exposing plaintext. This works well when you want resource-driven configuration and reviewable manifests, but do not want plaintext `Secret` YAML in the repository. **3. Environment variables** -- bypass `Secret` resources entirely by injecting provider keys into the runtime environment: ```bash export ORLOJ_SECRET_openai_api_key="sk-prod-key" ``` The resolver normalizes the secret name: a `secretRef: openai-api-key` looks up `ORLOJ_SECRET_openai_api_key` (hyphens become underscores). **4. External secret managers** -- inject secrets as environment variables using your platform's native mechanism: * **Kubernetes**: Use [external-secrets-operator](https://external-secrets.io/) or the CSI secrets driver to sync Vault/AWS Secrets Manager/GCP Secret Manager values into pod env vars. * **HashiCorp Vault**: Use [Vault Agent](https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent) sidecar to render secrets into env or files. * **Cloud providers**: Use AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault with their respective injection mechanisms. Approaches 3 and 4 do not require `Secret` resources -- the env-var resolver handles resolution directly. #### API Redaction The REST API never returns plaintext secret data. All `GET` responses for `Secret` resources replace every value in `spec.data` with `"***"`. This applies to both individual resource fetches and list responses. Secret data is write-only through the API; to verify a secret value, use the resource it references (e.g., test a model endpoint or tool that depends on it). Event bus messages for secret create/update operations are also redacted before publication. `SealedSecret` resources are returned as ciphertext blobs. The API never exposes the control-plane private key. #### Security Requirements * Raw secret values must not appear in logs or trace payloads. * Store the encryption key itself in a secure location (e.g., a KMS, Vault, or hardware security module). Do not commit it to version control. * Validate redaction behavior during incident drills. * Back up `ORLOJ_SECRET_ENCRYPTION_KEY` separately from the database. Losing it prevents decrypting encrypted `Secret` values and the stored `SealedSecret` private key. ### Tool Auth Profiles Tools can authenticate using one of four profiles via `spec.auth.profile`: | Profile | Suitable for | Notes | | --------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------ | | `bearer` (default) | API tokens, service keys | Injected as `Authorization: Bearer ` | | `api_key_header` | APIs using custom header auth (e.g., `X-Api-Key`) | Requires `auth.headerName` | | `basic` | Legacy HTTP basic auth | Secret must be `username:password` | | `oauth2_client_credentials` | Machine-to-machine OAuth2 | Requires `auth.tokenURL`; uses multi-key secret with `client_id` and `client_secret` | #### Auth in Container Isolation For container-isolated tools, auth is injected as environment variables rather than HTTP headers. The container's entrypoint script maps these to the appropriate `curl` headers: | Env Var | Auth Profile | | -------------------------------------------------- | ------------------------------------- | | `TOOL_AUTH_BEARER` | `bearer`, `oauth2_client_credentials` | | `TOOL_AUTH_BASIC` | `basic` | | `TOOL_AUTH_HEADER_NAME` + `TOOL_AUTH_HEADER_VALUE` | `api_key_header` | #### Auth Error Handling Auth failures produce distinct error codes (`auth_invalid` for HTTP 401, `auth_forbidden` for HTTP 403) that are non-retryable. For `oauth2_client_credentials`, a 401 triggers automatic token cache eviction and one retry with a fresh token. #### Auth Audit Trail Every tool invocation records `tool_auth_profile` and `tool_auth_secret_ref` (the secret name, not the resolved value) in the task trace. Use these fields for audit queries and compliance reporting. ### Audit Logging Orloj emits normalized audit events for security-relevant operations — admin token/user CRUD, approval decisions (`approved`, `denied`, `expired`, `changes_requested`) with reviewer identity, and other governed runtime actions. Each event carries a timestamp, component, action, outcome, principal, and the affected resource. **Audit logging is off by default.** The runtime uses a no-op `AuditSink`, so unless you wire a sink, these events are produced but not persisted anywhere durable. This is intentional — Orloj does not assume a particular log destination — but it means **operators must opt in to retain an audit trail**. #### Reference: structured audit sink A reference sink, `SlogAuditSink` (`runtime/audit_sink_slog.go`), writes audit events as structured JSON via `log/slog`. Wire it through `Extensions`: ```go ext := agentruntime.Extensions{ Audit: agentruntime.NewSlogAuditSink(nil), // JSON to stdout } server := api.NewServer(api.ServerOptions{ Extensions: ext, // ... other options }) ``` Passing a custom `*slog.Logger` lets you direct records to a file or a collector. For production, forward these records to durable, append-only, access-controlled storage (e.g., a SIEM or log pipeline) rather than relying on container stdout alone. #### Retention and integrity guidance * **Retain** audit records for at least as long as your compliance regime requires (commonly 1 year; longer for regulated environments). * **Protect integrity:** ship audit records off-host to write-once or append-only storage so a compromised node cannot rewrite its own trail. * **Restrict access** to audit storage to a separate role from the operators whose actions are being logged (separation of duties). * **Validate redaction during drills:** confirm raw secret values never appear in audit records or traces. * **Alert** on anomalous patterns (spikes in denials, approval overrides, admin token creation). ### Risk-Tier Routing and Approvals Tools can declare operation classes (`read`, `write`, `delete`, `admin`) via `spec.operation_classes`. Policy rules in `ToolPermission.spec.operation_rules` define per-class verdicts: `allow`, `deny`, or `approval_required`. When a tool call triggers `approval_required`: * The task enters `WaitingApproval` phase. * A `ToolApproval` resource is created for the pending decision. * An operator approves or denies via the REST API. * Approval outcomes produce `approval_pending`, `approval_denied`, or `approval_timeout` error codes. All approval-related outcomes are non-retryable and do not consume retry budget. #### Operational Guidance * Use `operation_rules` with `verdict: approval_required` for destructive operations (`delete`, `admin`) in production environments. * Set appropriate TTLs on `ToolApproval` resources (default: 10 minutes) to prevent tasks from waiting indefinitely. * Monitor `WaitingApproval` task counts and approval latencies to detect bottlenecks. ### Operational Requirements * Enforce least-privilege tool permissions. * Monitor denial and runtime policy error trends. * Monitor auth failure rates by profile for early detection of expired credentials. * Monitor approval request volume and response latency for `WaitingApproval` tasks. ### Related Docs * [Tool](../concepts/tools/tool.md) * [MCP Server](../concepts/tools/mcp-server.md) * [Connect an MCP Server](../guides/connect-mcp-server.md) ## Task Scheduling (Cron) Use `TaskSchedule` to create recurring run tasks from a task template. ### Purpose `TaskSchedule` evaluates a 5-field cron expression and creates a new `Task` run from a template task (`spec.mode=template`). ### Before You Begin * `orlojd` is running (scheduler/controller active). * At least one worker is available for execution. * The target `Task` template exists and sets `spec.mode: template`. ### 1. Apply a Task Template ```bash go run ./cmd/orlojctl apply -f examples/resources/tasks/weekly_report_template_task.yaml ``` Template reference used by schedules: * `metadata.name`: `weekly-report-template` * `spec.mode`: `template` ### 2. Apply a Schedule Example schedule resource: * [`examples/resources/task-schedules/weekly_report_schedule.yaml`](https://github.com/OrlojHQ/orloj/tree/main/examples/resources/task-schedules/weekly_report_schedule.yaml) Apply it: ```bash go run ./cmd/orlojctl apply -f examples/resources/task-schedules/weekly_report_schedule.yaml ``` Key fields: * `spec.task_ref`: template task name (`name` or `namespace/name`) * `spec.schedule`: 5-field cron expression (for example, `0 9 * * 1`) * `spec.time_zone`: IANA timezone (for example, `America/Chicago`) * `spec.concurrency_policy`: v1 supports `forbid` * `spec.starting_deadline_seconds`: lateness window before a missed slot is skipped ### 3. Verify Schedule State List schedules: ```bash go run ./cmd/orlojctl get task-schedules ``` Inspect schedule status directly: ```bash curl -s "http://127.0.0.1:8080/v1/task-schedules/weekly-report?namespace=default" | jq .status ``` Important status fields: * `nextScheduleTime` * `lastScheduleTime` * `lastTriggeredTask` * `activeRuns` ### 4. Verify Triggered Run Tasks ```bash go run ./cmd/orlojctl get tasks ``` Generated run tasks are labeled with schedule metadata: * `orloj.dev/task-schedule` * `orloj.dev/task-schedule-namespace` * `orloj.dev/task-schedule-slot` ### Common Controls * Pause scheduling: set `spec.suspend: true` * Resume scheduling: set `spec.suspend: false` * Retention: tune `successful_history_limit` and `failed_history_limit` ### Troubleshooting * If no tasks are created, verify `spec.task_ref` points to an existing template task. * If schedule status is `Error`, inspect `.status.lastError` and timezone/cron syntax. * If runs are skipped, check `starting_deadline_seconds` and `concurrency_policy` behavior. ### Related Docs * [Resource Reference (`TaskSchedule`)](../reference/resources/task-schedule.md) * [API Reference](../reference/api.md) * [Troubleshooting](./troubleshooting.md) ## Threat Model This page consolidates Orloj's security design: trust boundaries, the attacker model it defends against, the controls at each boundary, and the residual risks operators must own. It is a companion to [Security and Isolation](./security.md), which documents each control in depth. This is a self-assessment of the open-source platform as built. It is not a certification or a substitute for an environment-specific risk assessment. ### Scope and assumptions * Covers the Orloj control plane (`orlojd`), workers (`orlojworker`), the Kubernetes operator (`orloj-operator`), the web console, and the governed tool/agent runtime. * Assumes a self-hosted deployment. Operators are responsible for the host OS, container runtime, network, TLS termination, secret-key custody, and backups. * Assumes the threat actor does **not** have the control-plane host's root access or the `ORLOJ_SECRET_ENCRYPTION_KEY`. Controls that depend on those secrets are explicitly noted as out of scope below. ### Trust boundaries ``` untrusted callers / agents authored by users │ ┌─────────────────▼──────────────────┐ Boundary 1: API edge │ orlojd (control plane API + UI) │ authn, authz, rate limit └─────────────────┬──────────────────┘ │ governed runtime pipeline Boundary 2: governance ┌─────────────────▼──────────────────┐ policy, RBAC, approvals │ orlojworker (task/agent execution) │ └───────┬───────────────────┬────────┘ │ │ Boundary 3: │ │ Boundary 4: secret custody tool sandbox ▼ ▼ encryption at rest, redaction ┌───────────────────┐ ┌────────────────────┐ │ isolated tool/ │ │ secret store / │ │ agent execution │ │ sealing keypair │ └─────────┬─────────┘ └────────────────────┘ │ Boundary 5: outbound egress (SSRF) ▼ external HTTP / gRPC / MCP / A2A endpoints ``` **Boundary 1 — API edge.** Separates untrusted network callers from the control plane. Enforced by bearer-token / native-session authentication, role checks (reader/writer/admin/a2a), per-IP authentication rate limiting with trusted-proxy handling, and setup-token protection for first-admin creation. **Boundary 2 — Governance.** Separates an agent's *intent* (which tool it tries to call) from *authorized action*. Enforced fail-closed by `AgentPolicy`, `AgentRole`, `ToolPermission`, and human `ToolApproval`/`TaskApproval` gates with risk-tier routing. **Boundary 3 — Tool/agent sandbox.** Separates tool/agent code from the worker host. Enforced by isolation modes (sandboxed container defaults: read-only FS, `cap-drop=ALL`, `no-new-privileges`, `network none`, non-root, resource limits; ephemeral Kubernetes Jobs; shell-free CLI exec). **Boundary 4 — Secret custody.** Separates secret material from operators, the database, logs, and git. Enforced by AES-256-GCM encryption at rest, RSA-4096 SealedSecrets with per-entry data keys and AAD binding, API/event-bus redaction, and write-only secret APIs. **Boundary 5 — Outbound egress.** Separates Orloj-initiated outbound calls from internal network resources. Enforced by two-stage SSRF validation including a dial-time IP check that blocks loopback, link-local, cloud-metadata, and (by default) private ranges. ### Attacker model Threats Orloj is designed to resist: | Attacker | Example | Primary control | | ----------------------------------------- | --------------------------------------- | ------------------------------------------------------ | | Unauthenticated network caller | Hits the API/UI directly | Boundary 1: authn + rate limit | | Authenticated but under-privileged caller | Reader tries to mutate resources | Boundary 1: role check | | Malicious/compromised agent | Agent tries to call a tool it shouldn't | Boundary 2: fail-closed governance | | Hostile tool/MCP code | Tool tries to read host FS or pivot | Boundary 3: sandbox isolation | | SSRF via tool/agent input | Tool URL points at cloud metadata | Boundary 5: dial-time IP block | | Secret exfiltration via API/logs | Reads secret values back out | Boundary 4: redaction + encryption at rest | | Secrets committed to git | Plaintext `Secret` YAML in a repo | SealedSecret workflow | | Supply-chain tampering of releases | Modified image pulled by users | Cosign signing, SBOM, build provenance, image scanning | | Credential brute force | Password/token guessing | Argon2id hashing, auth rate limiting | ### Residual risks (operator-owned) These are **known and accepted** in the current design. Operators must mitigate them at the deployment layer. 1. **Namespaces are not a security boundary by default.** Any authenticated caller with the right role can access any namespace. Per-namespace tenant isolation is available in **Orloj Enterprise**, or self-hosters can implement the `ResourceAuthorizer` hook themselves (see [Multi-tenant authorization](./security.md#multi-tenant-authorization-enterprise)). 2. **Audit logging is off by default.** The runtime emits audit events but persists them only if an `AuditSink` is wired. Enable the reference `SlogAuditSink` and forward to durable storage. See [Audit Logging](./security.md#audit-logging). 3. **Control-plane TLS is the operator's responsibility.** `orlojd`/`orlojworker` serve plain HTTP; terminate TLS at a reverse proxy/load balancer and configure `--trusted-proxies` accordingly. gRPC *tool* connections require TLS 1.2+ by default. 4. **Host root or `ORLOJ_SECRET_ENCRYPTION_KEY` compromise is out of scope.** An attacker with both the database and the encryption key, or with code execution on `orlojd`, can recover secrets. Protect the key in a KMS/HSM and restrict host access. 5. **Sealing keys do not rotate automatically yet.** One active control-plane sealing keypair is used until a manual rotation flow is introduced. Plan periodic manual rotation and protect the key accordingly. 6. **Private-endpoint and public-A2A opt-ins weaken defaults.** `--a2a-allow-private-endpoints`, `ModelEndpoint spec.allowPrivate`, and `a2a.auth: public` are deliberate trade-offs; enable them only in trusted network contexts. 7. **GitHub Actions / dependency trust.** CI now runs dependency, SAST, secret, and image scanning, but a compromised upstream dependency or action can still introduce risk. Keep Dependabot current and review third-party action updates. ### Verifying the model The controls above are exercised by CI (build/test/vet, govulncheck, CodeQL, gitleaks, Trivy) and by conformance tests for sandbox defaults and SSRF enforcement. Re-run these scans when the architecture changes. ### Related docs * [Security and Isolation](./security.md) * [Governance and Policies](../concepts/governance/index.md) * [Worker](../concepts/infrastructure/worker.md) ## Troubleshooting Use this page for deterministic diagnosis and remediation of common failures. ### First Checks ```bash curl -s http://127.0.0.1:8080/healthz | jq . go run ./cmd/orlojctl get workers go run ./cmd/orlojctl get tasks ``` If these checks fail, inspect `orlojd` and `orlojworker` logs first. ### Common Issues #### `postgres backend selected but --postgres-dsn is empty` Cause: * `--storage-backend=postgres` is set without DSN. Fix: ```bash export ORLOJ_POSTGRES_DSN='postgres://orloj:orloj@127.0.0.1:5432/orloj?sslmode=disable' ``` #### Unsupported backend values Cause: * invalid value for storage/event/message/tool-isolation backend flags. Fix: * storage: `memory|postgres` * event bus (`orlojd`): `memory|nats` * runtime message bus: `none|memory|nats-jetstream` * tool isolation: `none|container|wasm` #### Workers never claim tasks Checks: * worker is `Ready` and heartbeating * execution mode matches deployment mode * model provider/auth is valid * task requirements (`region`, `gpu`, `model`) match worker capabilities Commands: ```bash go run ./cmd/orlojctl get workers go run ./cmd/orlojctl get tasks go run ./cmd/orlojctl trace task ``` #### Message-driven flow not progressing Cause: * worker consumer is not enabled. Fix: * set `--agent-message-consume` * set non-`none` `--agent-message-bus-backend` #### Tool calls fail with permission denials Cause: * governance policy denies requested action. Fix: * validate `Agent.spec.roles`, `AgentRole`, and `ToolPermission`. * inspect `tool_code`, `tool_reason`, and `retryable` in trace metadata. #### Model provider auth failures Cause: * missing or invalid API key on the ModelEndpoint resource. * Anthropic OAuth access token (`sk-ant-oat...`) used against a build that only sends `x-api-key` (pre-OAuth support), or a standard API key stored with an accidental `Bearer ` prefix. Fix: * verify `auth.secretRef` is set for providers that require auth (`openai`, `anthropic`, `azure-openai`). * for `openai-compatible`, auth is optional, but if `auth.secretRef` is set, verify that Secret exists and is valid. * if you use env-based secret resolution, set `ORLOJ_SECRET_` (or your configured prefix) to match the `secretRef` value. * for Anthropic: store either a standard API key (`sk-ant-api...`) or an OAuth access token (`sk-ant-oat...`) in the secret. Orloj selects `x-api-key` vs `Authorization: Bearer` from the token prefix — do not mix prefixes. #### Wasm/container runtime errors Cause: * missing runtime binary/module path or invalid runtime configuration. Fix: * verify backend-specific settings (container runtime settings or wasm module/runtime configuration). ### Observability Diagnostics #### Logs are unstructured or missing request IDs Cause: * `ORLOJ_LOG_FORMAT` is not set or binary predates the structured logging migration. Fix: * Set `ORLOJ_LOG_FORMAT=json` (default) to emit structured JSON logs with `request_id`, `trace_id`, and `span_id` fields. * Set `ORLOJ_LOG_FORMAT=text` for human-readable output during local development. #### Traces not appearing in Jaeger/Tempo Cause: * `OTEL_EXPORTER_OTLP_ENDPOINT` is not set or the backend is unreachable. Fix: ```bash export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 export OTEL_EXPORTER_OTLP_INSECURE=true # for non-TLS dev backends ``` Restart `orlojd` and `orlojworker`. Verify spans appear in the backend UI. #### Prometheus `/metrics` returning 404 Cause: * Running a build that predates the metrics endpoint addition. Fix: * Rebuild from the latest source and verify `curl http://127.0.0.1:8080/metrics` returns Prometheus text output. #### Correlating a log entry with a trace Use the `trace_id` field from a JSON log entry to search in your tracing backend: ```bash # Find trace ID in logs grep '"trace_id"' /var/log/orlojd.log | head -5 ``` Then search for that trace ID in Jaeger, Tempo, or the web console Trace tab. ### Operator #### CRD applied but resource not appearing in Orloj Cause: * The operator failed to upsert the resource into Postgres. Checks: * Inspect CRD status: `kubectl get agent my-agent -o jsonpath='{.status}'` * Look for `phase: SyncError` and read `syncError`. * Check operator logs: `kubectl -n orloj logs deploy/orloj-operator --tail=200` * Verify the operator has connectivity to Postgres (same DSN as `orlojd`). Fix: * Resolve the validation or connectivity error. The operator will retry automatically. #### Namespace stuck in Terminating (stale finalizer) Cause: * The operator was uninstalled or is down while CRD resources with the `orloj.dev/sync` finalizer still exist. Kubernetes cannot delete the namespace because the finalizer is never removed. Fix: * Redeploy the operator so it can process deletions and remove finalizers, or manually patch the finalizer off each stuck resource: ```bash kubectl patch agent my-agent -p '{"metadata":{"finalizers":null}}' --type=merge ``` #### REST API returns `X-Orloj-CRD-Managed` header warning Cause: * You are updating a resource via the REST API (or `orlojctl apply`) that was originally created by the CRD operator. The `--crd-conflict-policy=warn` mode is active. Fix: * Update the resource via `kubectl apply` or your Git repo instead. The operator will overwrite REST changes on its next reconcile. * To enforce this, set `--crd-conflict-policy=reject` on `orlojd`. #### Resource adopted unexpectedly Cause: * You `kubectl apply`'d a CRD manifest whose `metadata.name` and `metadata.namespace` match an existing REST-created resource. The operator upserted it, adding the `orloj.dev/managed-by: crd-sync` annotation. Fix: * This is expected behavior. Once a CRD with a matching name is applied, the operator takes ownership. Delete the CRD to return to REST-only management, or keep the CRD as the source of truth. #### CRD status stuck on old generation Cause: * The status writer runs on a periodic interval (`--status-sync-interval`, default 5s). Immediately after apply, the status may lag. Checks: * Wait for one status sync interval and re-check: `kubectl get agent my-agent -o jsonpath='{.status.observedGeneration}'` * Verify the operator pod is running and the leader election lease is held. Fix: * If status never updates, check operator logs for errors writing the status subresource. ### Escalation Workflow 1. Capture failing command and exact error text. 2. Capture task trace: ```bash go run ./cmd/orlojctl trace task ``` 3. Capture recent events: ```bash go run ./cmd/orlojctl events --once --timeout=30s --raw ``` 4. Capture relevant Prometheus metrics (if applicable): ```bash curl -s http://127.0.0.1:8080/metrics | grep orloj_ ``` 5. File an issue with logs, trace, metrics, and manifest snippets. ## Upgrades and Rollbacks This guide defines safe upgrade and rollback procedures for the Orloj server and workers. ### Principles * prefer staged rollouts over full replacement * take Postgres backups before upgrades * validate reliability gates before production promotion * couple release behavior with contract documentation ### Pre-Upgrade Checklist * [ ] Read release notes and migration notes. * [ ] Take a full Postgres backup per the [Backup and Restore](backup-restore.md) guide. * [ ] Record the current `ORLOJ_SECRET_ENCRYPTION_KEY`. * [ ] Verify baseline health (`/healthz`, workers, task flow). * [ ] Run smoke checks in staging. ### Upgrade Procedure 1. Upgrade `orlojd` in staging. 2. Verify API health and resource status. 3. Upgrade one worker (canary). 4. Validate task execution paths used by your deployment. 5. Upgrade remaining workers. 6. Run reliability checks: * `orloj-loadtest` * `orloj-alertcheck` ### Production Rollout * canary one server instance and one worker first * monitor task success/dead-letter ratio, retry volume, p95 latency, heartbeat stability ### Rollback Triggers * server health degradation * retry/dead-letter rates exceed SLO thresholds * unexpected increase in non-retryable runtime/policy failures ### Rollback Procedure 1. Revert server and worker binaries to previous release. 2. Restore previous configuration values. 3. Restore Postgres from backup if required (see [Backup and Restore](backup-restore.md)). 4. Re-run smoke checks before resuming rollout. ### Compatibility Guidance * keep compatibility checks green for pinned downstream consumers * avoid unversioned breaking changes on public contracts * treat contract graduation and lifecycle changes as release events ### Validation Commands ```bash curl -s http://127.0.0.1:8080/healthz | jq . go run ./cmd/orlojctl get workers go run ./cmd/orlojctl get tasks go run ./cmd/orloj-loadtest --quality-profile=monitoring/loadtest/quality-default.json --tasks=50 go run ./cmd/orloj-alertcheck --profile=monitoring/alerts/retry-deadletter-default.json ``` For complete reliability and alert flag coverage (including gate/injection controls and verbose/debug options), see: * [CLI reference](../reference/cli.md#orloj-loadtest) * [CLI reference](../reference/cli.md#orloj-alertcheck) * [Monitoring and Alerts](./monitoring-alerts.md) ## Webhook Triggers Use `TaskWebhook` to trigger task runs from signed external HTTP events. ### Purpose `TaskWebhook` receives inbound deliveries on a generated endpoint, validates signature/auth and idempotency, and creates a run task from a template task. ### Before You Begin * `orlojd` is running. * A template task exists with `spec.mode: template`. * A secret exists for signing or token verification. ### 1. Apply Prerequisites Apply template task: ```bash go run ./cmd/orlojctl apply -f examples/resources/tasks/weekly_report_template_task.yaml ``` Apply webhook signing secret: ```bash go run ./cmd/orlojctl apply -f examples/resources/secrets/webhook_shared_secret.yaml ``` ### 2. Apply a Webhook Resource Generic profile example: * [`examples/resources/task-webhooks/generic_webhook.yaml`](https://github.com/OrlojHQ/orloj/tree/main/examples/resources/task-webhooks/generic_webhook.yaml) GitHub profile example: * [`examples/resources/task-webhooks/github_push_webhook.yaml`](https://github.com/OrlojHQ/orloj/tree/main/examples/resources/task-webhooks/github_push_webhook.yaml) Apply one: ```bash go run ./cmd/orlojctl apply -f examples/resources/task-webhooks/generic_webhook.yaml ``` ### 3. Get the Delivery Endpoint ```bash curl -s "http://127.0.0.1:8080/v1/task-webhooks/report-generic-webhook?namespace=default" | jq -r '.status.endpointPath' ``` `endpointPath` maps to: * `POST /v1/webhook-deliveries/{endpoint_id}` ### 4. Send a Signed Test Delivery (Generic Profile) ```bash BODY='{"event":"report.trigger","topic":"AI startups"}' TS="$(date +%s)" SECRET='replace-me' SIG_HEX="$(printf '%s' "${TS}.${BODY}" | openssl dgst -sha256 -hmac "$SECRET" -binary | xxd -p -c 256)" curl -i -X POST "http://127.0.0.1:8080$(curl -s "http://127.0.0.1:8080/v1/task-webhooks/report-generic-webhook?namespace=default" | jq -r '.status.endpointPath')" \ -H "Content-Type: application/json" \ -H "X-Timestamp: ${TS}" \ -H "X-Event-Id: evt-001" \ -H "X-Signature: sha256=${SIG_HEX}" \ --data "$BODY" ``` Expected response: * HTTP `202 Accepted` * JSON with `accepted: true` * `duplicate: false` on first delivery ### 4b. Send a Signed Test Delivery (GitHub Profile) ```bash BODY='{"ref":"refs/heads/main","repository":{"full_name":"acme/repo"}}' SECRET='replace-me' SIG_HEX="$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -binary | xxd -p -c 256)" curl -i -X POST "http://127.0.0.1:8080$(curl -s \"http://127.0.0.1:8080/v1/task-webhooks/report-github-push?namespace=default\" | jq -r '.status.endpointPath')" \ -H "Content-Type: application/json" \ -H "X-GitHub-Delivery: gh-evt-001" \ -H "X-Hub-Signature-256: sha256=${SIG_HEX}" \ --data "$BODY" ``` ### 5. Verify Task Creation ```bash go run ./cmd/orlojctl get tasks ``` Webhook-triggered run tasks include: * `webhook_payload` * `webhook_event_id` * `webhook_received_at` * `webhook_source` ### 4c. Shared Token Delivery (Telegram-style) Telegram and similar services send a static secret token in a header. No HMAC computation is needed. ```bash BODY='{"update_id":12345,"message":{"text":"hello"}}' TOKEN='your-telegram-bot-secret' curl -i -X POST "http://127.0.0.1:8080" \ -H "Content-Type: application/json" \ -H "X-Telegram-Bot-Api-Secret-Token: ${TOKEN}" \ --data "$BODY" ``` No `X-Event-Id` header is needed when `event_id_from_body` is configured (e.g. `update_id` for Telegram). ### Profile Notes * `generic`: signs `timestamp + "." + rawBody` and checks timestamp skew. Default dedup window is 24 hours. * `github`: signs raw body and uses GitHub delivery id header defaults. Default dedup window is **72 hours** (vs 24h for generic) because GitHub webhooks do not include a timestamp in the HMAC payload, so replay protection relies entirely on event ID deduplication. The 72-hour window matches GitHub's maximum retry window. * `hmac`: fully configurable HMAC verification. Supports `sha256`, `sha1`, `sha512` algorithms; `body`, `timestamp_dot_body`, and `prefix_timestamp_body` payload formats; `hex` and `base64` signature encoding; and `plain` or `kv_pairs` header parsing. See [TaskWebhook concepts](../concepts/tasks/task-webhook.md) for field details and examples for Stripe and Slack. * `shared_token`: constant-time comparison of a static token in a header. No HMAC. Suitable for Telegram and similar services. ### Event ID Extraction By default, the event ID for deduplication is read from an HTTP header (`event_id_header`). For services like Telegram that put the deduplication key in the JSON body, use `event_id_from_body` instead: ```yaml idempotency: event_id_from_body: update_id # top-level JSON field # or nested: data.event_id # dot-separated path ``` When `event_id_from_body` is set, the header is not required. Both options can coexist — the header is checked first, then the body field as fallback. ### Rotation and Operations * Secret rotation: update referenced `Secret`; keep webhook resource unchanged. * Endpoint rotation: recreate webhook with a new `metadata.name` and update sender URL. * Duplicate deliveries return `202` with `duplicate: true`. ### Troubleshooting * `401 signature verification failed`: verify signature algorithm, prefix (`sha256=`), and secret. * `400 missing event id`: include configured event id header (`X-Event-Id` or `X-GitHub-Delivery`), or set `event_id_from_body` to extract it from the JSON body. * `400 webhook task creation failed`: the webhook was authenticated and deduplicated successfully, but task creation failed (e.g., the referenced task is not a template, or validation failed). The HTTP response returns a generic message; the detailed error is recorded in `status.lastError` on the `TaskWebhook` resource. Inspect it with `orlojctl get task-webhook `. * `404 webhook endpoint not found`: verify current `.status.endpointPath`. ### Related Docs * [Task Webhook Examples](https://github.com/OrlojHQ/orloj/tree/main/examples/resources/task-webhooks) * [Resource Reference (`TaskWebhook`)](../reference/resources/task-webhook.md) * [API Reference (Webhook Delivery)](../reference/api.md) ## Expose AgentSystems via A2A This guide walks through exposing selected Orloj AgentSystems to external A2A clients. ### Prerequisites * Orloj server (`orlojd`) running with `--embedded-worker` * `orlojctl` available * At least one agent and model endpoint configured If you have not set up Orloj yet, follow the [Install](../getting-started/install.md) and [Quickstart](../getting-started/quickstart.md) guides first. ### Step 1: Expose an AgentSystem Add `spec.a2a.enabled: true` to each AgentSystem that should be reachable through A2A. Systems without this block remain internal. ```yaml apiVersion: orloj.dev/v1 kind: AgentSystem metadata: name: research-system spec: agents: - research-agent a2a: enabled: true ``` Set the public base URL so generated Agent Cards point at the externally reachable host: ```bash orlojd --a2a-public-base-url https://orloj.example.com ``` ### Step 2: Verify the Default Agent Card If exactly one AgentSystem is A2A-enabled, the root well-known URL returns its card: ```bash curl -s http://localhost:8080/.well-known/agent-card.json | jq . ``` Expected output: ```json { "name": "research-system", "description": "Research assistant with web search capabilities", "url": "https://orloj.example.com/v1/agent-systems/research-system/a2a", "protocolVersion": "0.2", "capabilities": { "streaming": true, "pushNotifications": false, "stateTransitionHistory": true }, "skills": [ { "id": "web-search", "name": "web_search", "description": "Search the web for information", "tags": ["search", "web"] } ], "authentication": { "schemes": ["bearer"] } } ``` For a specific AgentSystem: ```bash curl -s http://localhost:8080/v1/agent-systems/research-system/.well-known/agent-card.json | jq . ``` ### Step 3: Test Inbound Task Creation Send an A2A `tasks/send` request to create a task: ```bash curl -s -X POST http://localhost:8080/v1/agent-systems/research-system/a2a \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $ORLOJ_TOKEN" \ -d '{ "jsonrpc": "2.0", "id": "req-1", "method": "tasks/send", "params": { "id": "task-001", "message": { "role": "user", "parts": [{"type": "text", "text": "Summarize recent AI news"}] } } }' | jq . ``` The response contains an A2A task with a status: ```json { "jsonrpc": "2.0", "id": "req-1", "result": { "id": "a2a-task-abc123", "status": { "state": "completed", "message": { "role": "agent", "parts": [{"type": "text", "text": "Here is a summary of recent AI news..."}] } }, "artifacts": [ { "name": "summary", "parts": [{"type": "text", "text": "..."}] } ] } } ``` ### Step 4: Multiple Systems For deployments with multiple exposed systems, use per-system cards and endpoints: ```bash # Discovery curl -s http://localhost:8080/v1/agent-systems/research-system/.well-known/agent-card.json # Task submission curl -s -X POST http://localhost:8080/v1/agent-systems/research-system/a2a \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $ORLOJ_TOKEN" \ -d '{ "jsonrpc": "2.0", "id": "req-2", "method": "tasks/send", "params": { "id": "task-002", "message": { "role": "user", "parts": [{"type": "text", "text": "Find papers on transformer architecture"}] } } }' ``` The per-system card's `url` field points to `https://orloj.example.com/v1/agent-systems/research-system/a2a`, so A2A clients that discover the card know where to send requests. ### Step 5: Streaming Subscribe For long-running tasks, use `tasks/sendSubscribe` to receive streaming updates via SSE: ```bash curl -s -N -X POST http://localhost:8080/v1/agent-systems/research-system/a2a \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $ORLOJ_TOKEN" \ -d '{ "jsonrpc": "2.0", "id": "req-3", "method": "tasks/sendSubscribe", "params": { "id": "task-003", "message": { "role": "user", "parts": [{"type": "text", "text": "Write a detailed report on quantum computing"}] } } }' ``` The server responds with a stream of SSE events containing status updates and artifact chunks as the agent works. ### How It Works When an A2A request arrives: 1. The JSON-RPC method is parsed and validated. 2. The target AgentSystem is resolved from the URL path or request params (shared endpoint). 3. An Orloj Task is created with A2A metadata labels. 4. The AgentSystem executes the task using the normal Orloj pipeline. 5. Task status transitions are mapped to A2A states and returned in the response. 6. For `tasks/sendSubscribe`, the task's trace/watch SSE stream is converted to A2A streaming events. ### Per-System Auth Policy By default, A2A invoke requires a bearer token when instance-wide auth is configured. To allow unauthenticated callers to invoke a specific system, set `spec.a2a.auth: public`: ```yaml apiVersion: orloj.dev/v1 kind: AgentSystem metadata: name: public-assistant spec: agents: - assistant-agent a2a: enabled: true auth: public ``` This is useful when you want admin tokens for the control plane (`/v1/agents`, `/v1/tools`, etc.) but need a public-facing A2A endpoint for external clients. Invalid tokens are still rejected — only missing tokens are permitted on public systems. Public systems' Agent Cards omit `authentication.schemes`, so A2A clients that discover the card know not to send tokens. The A2A registry (`GET /v1/a2a/agents`) shows public systems to unauthenticated callers. To require auth (the default), omit `auth` or set `spec.a2a.auth: bearer`. ### Agent Card Customization Agent Cards are auto-generated from the AgentSystem and its agents, but you can influence the output with annotations: ```yaml apiVersion: orloj.dev/v1 kind: AgentSystem metadata: name: research-system annotations: orloj.dev/description: "AI research assistant specializing in academic papers" spec: agents: - research-agent a2a: enabled: true ``` The `orloj.dev/description` annotation overrides the description in the generated card. ### Next Steps * [Use Remote A2A Agents](./a2a-remote-agents.md) -- call external A2A agents from your Orloj pipelines * [A2A Interoperability](../concepts/a2a-interoperability.md) -- concept deep-dive * [A2A JSON-RPC Reference](../reference/a2a-jsonrpc.md) -- per-method documentation * [Agent Card Reference](../reference/resources/agent-card.md) -- full card schema ## Use Remote A2A Agents This guide walks through configuring Orloj to call external A2A agents as tools. Remote A2A agents appear as regular tools in your agent's toolset -- the A2A protocol details are handled transparently by the runtime. ### Prerequisites * Orloj server (`orlojd`) running, with any inbound A2A exposure enabled per AgentSystem when needed * `orlojctl` available * A remote A2A agent endpoint (any A2A-compliant agent) ### Step 1: Create a type:a2a Tool Define a tool that points to the remote A2A agent: ```yaml apiVersion: orloj.dev/v1 kind: Tool metadata: name: external-analyst spec: type: a2a description: "Remote analyst agent that produces market research reports" a2a: agent_url: https://analyst.example.com protocol_version: "0.2" prefer_streaming: true ``` The `agent_url` is the base URL of the remote A2A agent. The runtime fetches `{agent_url}/.well-known/agent-card.json` to discover capabilities and the JSON-RPC endpoint. Apply: ```bash orlojctl apply -f external-analyst-tool.yaml ``` ### Step 2: Configure Authentication If the remote agent requires authentication, use the standard `spec.auth` field: ```yaml apiVersion: orloj.dev/v1 kind: Tool metadata: name: external-analyst spec: type: a2a description: "Remote analyst agent" a2a: agent_url: https://analyst.example.com prefer_streaming: true auth: profile: bearer secretRef: analyst-api-key ``` Create the secret: ```yaml apiVersion: orloj.dev/v1 kind: Secret metadata: name: analyst-api-key spec: stringData: value: sk-remote-agent-token ``` All four auth profiles work with A2A tools: `bearer`, `api_key_header`, `basic`, and `oauth2_client_credentials`. ### Step 3: Attach to an Agent Reference the A2A tool in your agent's `spec.tools`, just like any other tool: ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: coordinator spec: model_ref: openai-default prompt: | You coordinate research tasks. When the user asks for market analysis, delegate to the external-analyst tool. tools: - external-analyst - web_search limits: max_steps: 10 timeout: 120s ``` Apply and test: ```bash orlojctl apply -f coordinator-agent.yaml orlojctl task submit --agent coordinator --input "Analyze the AI chip market" ``` ### Step 4: Test the Invocation When the coordinator agent decides to call `external-analyst`, the runtime: 1. Fetches the remote Agent Card from `https://analyst.example.com/.well-known/agent-card.json`. 2. Sends a JSON-RPC `tasks/send` (or `tasks/sendSubscribe` if `prefer_streaming` is true and the remote supports it) to the card's `url`. 3. Maps the A2A response back to a tool result. Check the task messages to see the A2A interaction: ```bash orlojctl get tasks -o json | jq '.status' ``` ### Streaming vs Polling The `prefer_streaming` field controls how Orloj communicates with the remote agent: | `prefer_streaming` | Remote supports streaming | Behavior | | ------------------ | ------------------------- | ---------------------------------------------- | | `true` (default) | Yes | `tasks/sendSubscribe` -- real-time SSE updates | | `true` | No | Falls back to `tasks/send` | | `false` | Any | Always uses `tasks/send` | Streaming is recommended for long-running remote agents. The A2A tool runtime converts streaming status updates into progress events visible in the task trace. ### Error Handling A2A tool invocations follow the standard tool error taxonomy: | Scenario | Error code | Retryable | | ----------------------------- | --------------------------------- | --------------------- | | Remote agent unreachable | `connection_error` | Yes | | Remote returns JSON-RPC error | `a2a_rpc_error` | Depends on error code | | Remote task fails | `a2a_task_failed` | No | | Remote task times out | `timeout` | Yes | | Auth failure (401/403) | `auth_invalid` / `auth_forbidden` | No | | Card fetch fails | `a2a_discovery_error` | Yes | The tool's `runtime.retry` policy applies to retryable errors: ```yaml spec: type: a2a a2a: agent_url: https://analyst.example.com runtime: timeout: 60s retry: max_attempts: 3 backoff: 2s max_backoff: 30s jitter: full ``` ### Registry: List All A2A Agents View locally exposed agents and configured remote agents: ```bash curl -s http://localhost:8080/v1/a2a/agents \ -H "Authorization: Bearer $ORLOJ_TOKEN" | jq . ``` Response: ```json { "localAgents": [ { "name": "research-agent", "url": "https://orloj.example.com/v1/agent-systems/research-system/a2a", "capabilities": { "streaming": true } } ], "remoteAgents": [ { "name": "external-analyst", "url": "https://analyst.example.com", "cacheStatus": "ok", "lastRefreshed": "2025-06-01T10:30:00Z", "card": { "..." } } ] } ``` ### Governance A2A tools participate in the full governance pipeline: * **AgentRole**: bind roles to agents that restrict which A2A tools they can call. * **ToolPermission**: define per-operation-class rules (e.g., require approval for `write`-class A2A calls). * **AgentPolicy**: enforce cost, rate, or content policies on A2A tool invocations. ```yaml apiVersion: orloj.dev/v1 kind: ToolPermission metadata: name: allow-external-analyst spec: toolRef: external-analyst operation_rules: - operations: ["read"] verdict: allow - operations: ["write"] verdict: approval_required ``` ### Next Steps * [Expose Agents via A2A](./a2a-expose-agents.md) -- make your Orloj agents discoverable * [A2A Interoperability](../concepts/a2a-interoperability.md) -- concept deep-dive * [A2A JSON-RPC Reference](../reference/a2a-jsonrpc.md) -- protocol details * [Build a Custom Tool](./build-custom-tool.md) -- for non-A2A tool integrations ## Build a Custom Tool This guide is for developers who need to extend agent capabilities by implementing a custom tool. You will implement the Tool Contract v1, register the tool as a resource, configure isolation and retry, and validate it with the conformance harness. ### Prerequisites * Orloj server (`orlojd`) and at least one worker running * `orlojctl` available * Familiarity with the [Tools and Isolation](../concepts/tools/tool.md) concepts ### What You Will Build A custom HTTP tool that agents can invoke during execution, registered with Orloj and configured with appropriate runtime controls. ### Step 1: Implement the Tool Contract Every tool must accept a JSON request envelope and return a JSON response envelope. **Request** (sent by the Orloj runtime to your tool): ```json { "request_id": "req-abc-123", "tool": "my-custom-tool", "action": "invoke", "parameters": { "query": "example input" }, "auth": { "type": "bearer", "token": "sk-..." }, "context": { "task": "weekly-report", "agent": "research-agent", "attempt": 1 } } ``` **Success response** (returned by your tool): ```json { "request_id": "req-abc-123", "status": "success", "result": { "data": "your tool output here" } } ``` **Error response** (for retryable failures): ```json { "request_id": "req-abc-123", "status": "error", "error": { "tool_code": "rate_limited", "tool_reason": "API rate limit exceeded", "retryable": true } } ``` The error taxonomy includes `tool_code` (machine-readable), `tool_reason` (human-readable), and `retryable` (boolean). The runtime uses `retryable` to decide whether to retry or move to dead-letter. ### Step 2: Register the Tool Create a Tool resource manifest: ```yaml apiVersion: orloj.dev/v1 kind: Tool metadata: name: my-custom-tool spec: type: http endpoint: https://your-tool-service.internal/invoke capabilities: - custom.query.invoke operation_classes: - read - write risk_level: medium runtime: timeout: 10s retry: max_attempts: 3 backoff: 1s max_backoff: 10s jitter: full auth: secretRef: my-tool-api-key ``` Apply: ```bash orlojctl apply -f my-custom-tool.yaml ``` #### Field Choices **`risk_level`** -- Determines the default isolation mode: * `low` / `medium`: defaults to `none` (direct execution) * `high` / `critical`: defaults to `sandboxed` **`operation_classes`** -- Declares the types of operations this tool performs. Valid values: `read`, `write`, `delete`, `admin`. Policy rules in `ToolPermission.operation_rules` can define per-class verdicts (`allow`, `deny`, `approval_required`). When omitted, defaults to `["read"]` for low/medium risk or `["write"]` for high/critical risk. **`runtime.timeout`** -- How long the runtime waits for your tool to respond before treating the invocation as failed. Choose based on your tool's expected latency. **`runtime.retry`** -- Configure retry behavior for transient failures. The `jitter: full` setting randomizes backoff intervals to prevent thundering herd effects when multiple agents hit the same tool. ### Step 3: Create a Secret and Configure Auth If your tool requires authentication, create a Secret and set the auth profile on the Tool. Orloj supports four auth profiles: #### Bearer token (default) ```yaml apiVersion: orloj.dev/v1 kind: Secret metadata: name: my-tool-api-key spec: stringData: value: your-api-key-here ``` ```yaml spec: auth: secretRef: my-tool-api-key ``` When `profile` is omitted, it defaults to `bearer`. The runtime injects `Authorization: Bearer `. #### API key header ```yaml spec: auth: profile: api_key_header secretRef: my-tool-api-key headerName: X-Api-Key ``` The runtime injects the secret value as `X-Api-Key: `. #### Basic auth ```yaml apiVersion: orloj.dev/v1 kind: Secret metadata: name: my-basic-creds spec: stringData: value: "username:password" ``` ```yaml spec: auth: profile: basic secretRef: my-basic-creds ``` The secret must contain `username:password`. The runtime base64-encodes it and injects `Authorization: Basic `. #### OAuth2 client credentials ```yaml apiVersion: orloj.dev/v1 kind: Secret metadata: name: my-oauth-creds spec: stringData: client_id: your-client-id client_secret: your-client-secret ``` ```yaml spec: auth: profile: oauth2_client_credentials secretRef: my-oauth-creds tokenURL: https://auth.provider.com/oauth/token scopes: - read - write ``` The runtime exchanges client credentials for an access token, caches it with TTL, and injects `Authorization: Bearer `. Tokens are refreshed automatically on expiry or HTTP 401. Apply: ```bash orlojctl apply -f my-tool-secret.yaml orlojctl apply -f my-custom-tool.yaml ``` ### Step 4: Grant Agent Access Add the tool to an agent's `tools` list: ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: research-agent spec: model_ref: openai-default tools: - web_search - my-custom-tool limits: max_steps: 6 timeout: 30s ``` If governance is enabled, you also need a ToolPermission and an AgentRole that grants the required permissions. See the [governance guide](./setup-governance.md) for details. ### Step 5: Choose a Tool Type The examples above use `type: http` (the default). If your tool uses a different transport or execution model, set `spec.type` accordingly. The tool type and isolation mode are independent -- any type can run under any isolation mode. #### External (standalone service) For tools that need the full Orloj execution context (task, agent, namespace, attempt): ```yaml spec: type: external endpoint: https://your-tool-service.internal/execute runtime: timeout: 30s ``` Your service receives the complete `ToolExecutionRequest` JSON envelope and must return a `ToolExecutionResponse`. This is the right choice when your tool is a dedicated microservice that makes decisions based on who called it and why. #### gRPC For tools that expose a gRPC service: ```yaml spec: type: grpc endpoint: your-grpc-service:50051 runtime: timeout: 15s ``` Implement the `orloj.tool.v1.ToolService/Execute` unary method. Payloads are the same `ToolExecutionRequest` / `ToolExecutionResponse` envelopes as `external`, marshaled as JSON over gRPC (no protobuf compilation needed). #### Webhook-Callback (async / long-running) For tools that take seconds-to-minutes to complete: ```yaml spec: type: webhook-callback endpoint: https://your-async-tool.internal/submit runtime: timeout: 120s ``` Execution flow: 1. Orloj POSTs a `ToolExecutionRequest` to your endpoint. 2. Your tool returns `202 Accepted` to acknowledge receipt. 3. Orloj polls `{endpoint}/{request_id}` at intervals until your tool returns a `ToolExecutionResponse` with a terminal status, or the timeout expires. 4. Alternatively, your tool can push the result to Orloj's callback delivery API instead of waiting for a poll. Use this for batch processing, CI triggers, human approval workflows, or any tool where the response isn't immediate. ### Step 6: Configure Isolation (Optional) For tools that run untrusted code or interact with sensitive resources, set an explicit isolation mode. This is independent of tool type. **Container isolation:** ```yaml spec: runtime: isolation_mode: container timeout: 15s ``` **WASM isolation** (for tools compiled to WebAssembly): ```yaml spec: type: wasm wasm: module: my-tool.wasm enable_wasi: true runtime: isolation_mode: wasm timeout: 5s ``` WASM tools communicate over stdin/stdout using a JSON contract and run in the embedded wazero runtime. See [Build a WASM Tool](./build-wasm-tool.md) for the full contract specification and authoring guide. **Sandboxed isolation** (secure-by-default container): ```yaml spec: risk_level: high runtime: isolation_mode: sandboxed ``` Sandboxed mode runs tools in a locked-down container: read-only filesystem, no capabilities, no privilege escalation, no network, non-root user, and strict memory/CPU/pids limits. This is the default for `high` and `critical` risk tools. ### Step 7: Validate with the Conformance Harness Orloj provides a tool runtime conformance harness that tests your tool against the contract specification. The harness covers eight test groups: 1. **Contract** -- request/response envelope validation 2. **Timeout** -- tool respects configured timeouts 3. **Retry** -- retryable errors trigger retry; non-retryable errors do not 4. **Auth** -- credentials are passed correctly 5. **Policy** -- governance denials are handled properly 6. **Isolation** -- isolation backends enforce boundaries 7. **Observability** -- trace metadata is propagated 8. **Determinism** -- identical inputs produce consistent outputs ### Next Steps * [Tool](../concepts/tools/tool.md) -- tool types, isolation, and contract details * [Build a WASM Tool](./build-wasm-tool.md) -- authoring WebAssembly tools with the stdin/stdout contract * [Connect an MCP Server](./connect-mcp-server.md) -- for MCP-compatible tool servers instead of custom implementations ## Build a WASM Tool This guide walks through building and deploying a WebAssembly tool that agents can invoke through Orloj's embedded wazero runtime. WASM tools execute in-process with host-enforced resource limits -- no external runtime binary required. ### Prerequisites * Orloj server (`orlojd`) and at least one worker running * `orlojctl` available * A language toolchain that can compile to WebAssembly (Go, Rust, C/C++, Zig, AssemblyScript, etc.) * Familiarity with [Tools and Isolation](../concepts/tools/tool.md) concepts ### How WASM Tools Work WASM tools communicate with the host over **stdin/stdout** using a JSON contract (v1): 1. Orloj writes a JSON request to the module's **stdin**. 2. The module reads the request, does its work, and writes a JSON response to **stdout**. 3. The host reads the response and passes the result back to the agent. The module runs inside an embedded [wazero](https://wazero.io) runtime (pure Go, zero CGO). Memory, CPU fuel, and I/O access are enforced by the host, not by the guest. The module cannot escape its sandbox. ### The WASM Tool Contract v1 #### Request (host writes to stdin) The host sends a single JSON object: ```json { "contract_version": "v1", "namespace": "production", "tool": "my-wasm-tool", "input": "{\"query\": \"search term\"}", "capabilities": ["wasm.my-tool.invoke"], "risk_level": "low", "runtime": { "entrypoint": "run", "max_memory_bytes": 67108864, "fuel": 1000000, "enable_wasi": true }, "auth": { "profile": "bearer", "headers": { "Authorization": "Bearer sk-..." } } } ``` | Field | Type | Description | | ------------------ | --------- | ----------------------------------------------------------------------------------------- | | `contract_version` | string | Always `"v1"`. | | `namespace` | string | The Orloj namespace the tool is running in. | | `tool` | string | The tool resource name. | | `input` | string | The agent's tool input, serialized as a JSON string. | | `capabilities` | string\[] | Declared capabilities from the Tool manifest. | | `risk_level` | string | `low`, `medium`, `high`, or `critical`. | | `runtime` | object | Resource limits and entrypoint (informational; enforced by host). | | `auth` | object | Auth profile and resolved headers. Only present if `spec.auth` is configured on the Tool. | The `input` field is a **string** containing serialized JSON from the agent. Your module should parse it to extract parameters. #### Response (module writes to stdout) The module must write exactly one JSON object to stdout. **Success:** ```json { "contract_version": "v1", "status": "ok", "output": "The result of the tool invocation." } ``` **Error (retryable):** ```json { "contract_version": "v1", "status": "error", "error": { "code": "rate_limited", "reason": "upstream API throttled", "message": "try again in 5s", "retryable": true } } ``` **Denied:** ```json { "contract_version": "v1", "status": "denied", "error": { "code": "permission_denied", "reason": "insufficient scope", "message": "tool requires admin access", "retryable": false } } ``` | Field | Type | Required | Description | | ------------------ | ------ | --------------- | ----------------------------------------------------------------------------------------------------------------- | | `contract_version` | string | yes | Must be `"v1"`. | | `status` | string | yes | `"ok"`, `"error"`, or `"denied"`. | | `output` | string | on success | The tool result returned to the agent. | | `error` | object | on error/denied | Structured error with `code`, `reason`, `message`, `retryable`, and optional `details` (map of string to string). | The `error.code` and `error.retryable` fields drive the runtime's retry and dead-letter behavior. Use the [error taxonomy](../concepts/tools/tool.md#error-taxonomy) codes when applicable. #### WASI and `proc_exit` When WASI is enabled (`enable_wasi: true`), the guest has access to stdin, stdout, and stderr. Go and Rust WASI guests typically call `proc_exit(0)` on normal termination. The host treats `proc_exit(0)` as success -- only non-zero exit codes are treated as failures. ### Step 1: Write a Guest Module Any language that compiles to WASM can produce a guest. The simplest path is Go targeting `wasip1`. #### Go ```go package main import ( "encoding/json" "io" "os" ) type request struct { ContractVersion string `json:"contract_version"` Tool string `json:"tool"` Input string `json:"input"` } type response struct { ContractVersion string `json:"contract_version"` Status string `json:"status"` Output string `json:"output"` } type errorResponse struct { ContractVersion string `json:"contract_version"` Status string `json:"status"` Error errorDetail `json:"error"` } type errorDetail struct { Code string `json:"code"` Message string `json:"message"` } func main() { data, err := io.ReadAll(os.Stdin) if err != nil { writeError("guest_error", "failed to read stdin: "+err.Error()) return } var req request if err := json.Unmarshal(data, &req); err != nil { writeError("guest_error", "failed to parse request: "+err.Error()) return } // --- Your tool logic here --- result := "processed: " + req.Input resp := response{ ContractVersion: "v1", Status: "ok", Output: result, } _ = json.NewEncoder(os.Stdout).Encode(resp) } func writeError(code, msg string) { resp := errorResponse{ ContractVersion: "v1", Status: "error", Error: errorDetail{Code: code, Message: msg}, } _ = json.NewEncoder(os.Stdout).Encode(resp) } ``` Build: ```bash GOOS=wasip1 GOARCH=wasm go build -o my-tool.wasm my_tool.go ``` #### Rust ```rust use std::io::{self, Read}; fn main() { let mut input = String::new(); io::stdin().read_to_string(&mut input).unwrap(); // Parse JSON (use serde_json in real code) // ... your tool logic ... println!(r#"{{"contract_version":"v1","status":"ok","output":"result here"}}"#); } ``` Build with the WASI target: ```bash cargo build --target wasm32-wasip1 --release cp target/wasm32-wasip1/release/my_tool.wasm . ``` #### Other Languages Any language with a WASI compilation target works: C/C++ (via wasi-sdk or Emscripten), Zig (`-target wasm32-wasi`), AssemblyScript, etc. The only requirement is reading JSON from stdin and writing JSON to stdout. ### Step 2: Register the Tool Create a Tool manifest with `type: wasm` and a `spec.wasm` block: ```yaml apiVersion: orloj.dev/v1 kind: Tool metadata: name: my-wasm-tool spec: type: wasm wasm: module: my-tool.wasm entrypoint: run max_memory_bytes: 67108864 # 64 MB (default) fuel: 1000000 # Execution step limit (default: 1M) enable_wasi: true # Required for stdin/stdout capabilities: - wasm.my-tool.invoke risk_level: low runtime: isolation_mode: wasm timeout: 5s ``` Apply: ```bash orlojctl apply -f my-wasm-tool.yaml ``` #### `spec.wasm` Fields | Field | Default | Description | | ------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | | `module` | *(required)* | Relative path under `--tool-wasm-cache-dir`, HTTPS URL, or OCI artifact reference (`oci://...`) to the `.wasm` module. | | `entrypoint` | `run` | Exported function name to invoke. | | `max_memory_bytes` | `67108864` (64 MB) | Maximum WASM linear memory. Host-enforced. | | `fuel` | `1000000` (1M) | Execution fuel limit. Prevents runaway modules. Host-enforced. | | `enable_wasi` | `false` | Enable WASI (stdin/stdout/stderr). Most tools need this set to `true`. | | `image_pull_secret` | *(optional)* | Name of a Secret containing registry credentials for pulling OCI-referenced modules. The Secret must have `username` and `password` keys. | ##### Module reference formats The `module` field accepts three formats: * **Local path** (relative): `my-tool.wasm` or `tools/echo.wasm` — resolved under `--tool-wasm-cache-dir` (default `~/.orloj/wasm-cache`). Absolute paths, `..` segments, and paths outside the cache directory are rejected. * **HTTPS URL**: `https://artifacts.example.com/tools/echo-v1.2.wasm` — plain `http://` URLs are not supported. * **OCI reference**: `oci://ghcr.io/orloj-tools/echo:v1.2` Remote modules (HTTPS and OCI) are fetched once and cached on disk in `--tool-wasm-cache-dir`, keyed by SHA-256 of the reference. Subsequent invocations use the cached copy. For local paths, copy or mount the `.wasm` file into `--tool-wasm-cache-dir` (or a subdirectory). In a containerized deployment, mount the cache directory or place modules under it at startup. ##### Private OCI registries For private OCI registries, set `image_pull_secret` to reference a Secret with `username` and `password` keys: ```yaml spec: wasm: module: oci://ghcr.io/my-org/private-tool:v1 image_pull_secret: ghcr-creds ``` ### Step 3: Grant Agent Access Add the tool to an agent's `tools` list: ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: research-agent spec: model_ref: openai-default tools: - web_search - my-wasm-tool limits: max_steps: 10 timeout: 60s ``` If governance is enabled, you also need a `ToolPermission` and an `AgentRole`. See the [governance guide](./setup-governance.md). ### Step 4: Test Locally You can test the guest contract without running Orloj by piping JSON to the WASM binary. Using Go's WASI runner: ```bash echo '{"contract_version":"v1","tool":"my-wasm-tool","input":"hello"}' | \ GOOS=wasip1 GOARCH=wasm go run my_tool.go ``` Using wasmtime (if installed): ```bash echo '{"contract_version":"v1","tool":"my-wasm-tool","input":"hello"}' | \ wasmtime run my-tool.wasm ``` Expected output: ```json {"contract_version":"v1","status":"ok","output":"processed: hello"} ``` ### Resource Limits All limits are enforced by the **host**, not the guest module. A malicious or buggy guest cannot override them. | Limit | What it controls | Default | | ------------------ | ------------------------------------------------------------- | --------- | | `max_memory_bytes` | WASM linear memory ceiling | 64 MB | | `fuel` | Execution step budget (prevents infinite loops) | 1,000,000 | | `runtime.timeout` | Wall-clock timeout for the entire invocation | 30s | | `enable_wasi` | When `false`, the module has no access to stdin/stdout/stderr | `false` | If fuel is exhausted before the module completes, the host terminates execution and returns an error to the agent. ### Coexistence with Container Tools WASM tools run on a dedicated runtime slot, independent of the `--tool-isolation-backend` flag. You can mix WASM tools and container-isolated tools (including MCP servers, CLI tools, etc.) in the same agent system: ```yaml spec: tools: - my-wasm-tool # Runs in wazero (always available) - kubectl-get # Runs in container (requires --tool-isolation-backend=container) - web_search # Runs as HTTP (no isolation) ``` ### Error Handling Best Practices 1. **Always write a response.** If your module exits without writing to stdout, the host treats it as a contract violation. 2. **Use `contract_version: "v1"` and a valid `status`.** Missing or unsupported values cause a contract error. 3. **Prefer structured errors over panics.** Write an error response JSON instead of crashing. Panics produce an opaque host-level error with no retry information. 4. **Set `retryable` accurately.** The runtime uses this field to decide whether to retry or dead-letter the invocation. ### Scaffold a New Tool Use `orlojctl tool scaffold` to generate a ready-to-build project: ```bash orlojctl tool scaffold my-echo --lang go ``` This creates a `my-echo/` directory with a contract-compliant guest module, Makefile, tool manifest, test fixtures, and a README. Supported languages: `go`, `rust`. ### Test a Tool Use `orlojctl tool test` to validate a WASM module against fixture files: ```bash orlojctl tool test my-echo.wasm --fixtures fixtures/ ``` Each fixture is a JSON file specifying input, expected status, and expected output: ```json { "name": "echo hello", "input": "{\"query\": \"hello\"}", "expected_status": "ok", "expected_output": "processed: {\"query\": \"hello\"}", "timeout": "5s" } ``` The test runner validates the contract (v1, valid status), asserts expected output, and reports pass/fail with timing. Options: | Flag | Default | Description | | ----------------- | ----------- | ---------------------------------------- | | `--fixtures` | `fixtures/` | Directory containing JSON fixture files. | | `--fuel-budget` | `1000000` | Maximum fuel per fixture run. | | `--memory-budget` | `67108864` | Maximum memory bytes per fixture run. | ### Observability WASM tool execution emits Prometheus metrics automatically: | Metric | Type | Labels | Description | | ------------------------------------------- | --------- | ------------------------ | --------------------------------------- | | `orloj_tool_execution_duration_seconds` | Histogram | `tool`, `type`, `status` | Duration of tool execution (all types). | | `orloj_wasm_fuel_consumed` | Counter | `tool` | Total fuel consumed. | | `orloj_wasm_compilation_cache_hits_total` | Counter | `tool` | Module compilation cache hits. | | `orloj_wasm_compilation_cache_misses_total` | Counter | `tool` | Module compilation cache misses. | | `orloj_wasm_module_fetch_duration_seconds` | Histogram | `source` | Remote module fetch duration. | ### Reference Example A complete working example lives in the repository: * **Guest source**: `examples/resources/tools/wasm-reference/echo_guest.go` * **Tool manifest**: `examples/resources/tools/wasm-reference/wasm_echo_tool.yaml` * **README with build instructions**: `examples/resources/tools/wasm-reference/README.md` ### Next Steps * [Tool Concepts](../concepts/tools/tool.md) -- tool types, isolation modes, and the error taxonomy * [Build a Custom Tool](./build-custom-tool.md) -- HTTP, gRPC, external, and webhook-callback tool types * [Connect an MCP Server](./connect-mcp-server.md) -- auto-discover tools from MCP servers ## Configure Model Routing This guide is for platform engineers who need to route agents to different model providers. You will set up ModelEndpoints for multiple providers, bind agents to endpoints by reference, and verify that requests route correctly. ### Prerequisites * Orloj server (`orlojd`) and at least one worker running * API keys for the providers you want to configure * `orlojctl` available ### What You Will Build A multi-provider setup where different agents route to different model providers: * A research agent using OpenAI's GPT-4o * A writer agent using Anthropic's Claude ### Step 1: Create Secrets for API Keys Each provider needs a Secret resource to hold its API key. The fastest way is the CLI -- no YAML file needed: ```bash orlojctl create secret openai-api-key --from-literal value=sk-your-openai-key-here orlojctl create secret anthropic-api-key --from-literal value=sk-ant-your-anthropic-key-here # Anthropic OAuth access tokens (sk-ant-oat...) also work in the same secret — # Orloj sends Authorization: Bearer for those, and x-api-key for API keys. ``` Alternatively, use YAML manifests with `orlojctl apply -f`: ```yaml apiVersion: orloj.dev/v1 kind: Secret metadata: name: openai-api-key spec: stringData: value: sk-your-openai-key-here ``` > **Production note:** Enable `--secret-encryption-key` on `orlojd` and `orlojworker` to encrypt secret data at rest in the database, or use environment variables (`ORLOJ_SECRET_openai_api_key`) / an external secret manager. See [Security and Isolation](../operations/security.md#secret-handling) for details. ### Step 2: Create Model Endpoints **OpenAI endpoint:** ```yaml apiVersion: orloj.dev/v1 kind: ModelEndpoint metadata: name: openai-default spec: provider: openai base_url: https://api.openai.com/v1 default_model: gpt-4o-mini auth: secretRef: openai-api-key ``` **Anthropic endpoint:** ```yaml apiVersion: orloj.dev/v1 kind: ModelEndpoint metadata: name: anthropic-default spec: provider: anthropic base_url: https://api.anthropic.com/v1 default_model: claude-3-5-sonnet-latest options: anthropic_version: "2023-06-01" max_tokens: "1024" auth: secretRef: anthropic-api-key ``` Apply both: ```bash orlojctl apply -f openai_default.yaml orlojctl apply -f anthropic_default.yaml ``` Verify they are ready: ```bash orlojctl get model-endpoints ``` ### Step 3: Bind Agents to Endpoints Use `spec.model_ref` to point each agent at its ModelEndpoint: ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: research-agent spec: model_ref: openai-default prompt: | You are a research assistant. Produce concise evidence-backed answers. limits: max_steps: 6 timeout: 30s ``` ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: writer-agent spec: model_ref: anthropic-default prompt: | You are a writing agent. Produce clear, concise final output from provided research. limits: max_steps: 4 timeout: 20s ``` Apply: ```bash orlojctl apply -f research-agent.yaml orlojctl apply -f writer-agent.yaml ``` When these agents execute, the model gateway resolves their `model_ref` to the corresponding ModelEndpoint, then constructs provider-specific API requests using the endpoint's `base_url`, `default_model`, `options`, and auth credentials. ### Step 4: Verify Routing Submit a task that uses these agents and check the logs: ```bash orlojctl apply -f task.yaml orlojctl logs task/your-task-name ``` In the task trace, you should see model requests routing to the appropriate providers based on each agent's `model_ref`. ### Adding Azure OpenAI Azure OpenAI requires an explicit `base_url` and an `api_version` option: ```yaml apiVersion: orloj.dev/v1 kind: ModelEndpoint metadata: name: azure-openai-default spec: provider: azure-openai base_url: https://YOUR_RESOURCE_NAME.openai.azure.com default_model: gpt-4o-deployment options: api_version: "2024-10-21" auth: secretRef: azure-openai-api-key ``` ### Adding AWS Bedrock Bedrock uses IAM credentials instead of a simple API key. For development, store explicit credentials as a JSON blob in a Secret: ```bash orlojctl create secret aws-credentials --from-literal value='{"access_key_id":"AKIA...","secret_access_key":"..."}' ``` Then create the ModelEndpoint: ```yaml apiVersion: orloj.dev/v1 kind: ModelEndpoint metadata: name: bedrock-claude spec: provider: bedrock default_model: anthropic.claude-sonnet-4-20250514-v1:0 options: region: us-east-1 max_tokens: "4096" auth: secretRef: aws-credentials ``` For production deployments on AWS infrastructure (EC2, ECS, Lambda), you can omit `auth.secretRef` entirely and let the AWS SDK resolve credentials from the instance's IAM role: ```yaml apiVersion: orloj.dev/v1 kind: ModelEndpoint metadata: name: bedrock-production spec: provider: bedrock default_model: anthropic.claude-sonnet-4-20250514-v1:0 options: region: us-east-1 ``` Cross-region inference profiles (e.g. `us.anthropic.claude-sonnet-4-20250514-v1:0`) work transparently -- use the profile ID as `default_model`. ### Adding Ollama (Local Models) For local model inference with no API key required: ```yaml apiVersion: orloj.dev/v1 kind: ModelEndpoint metadata: name: ollama-default spec: provider: ollama base_url: http://127.0.0.1:11434 default_model: llama3.1 ``` Use the Ollama server root as `base_url` for `provider: ollama`. Do not append `/v1` in this mode. ### Using OpenAI-Compatible Providers The `openai-compatible` provider lets you connect to any service that speaks the OpenAI Chat Completions protocol. Set `base_url` to the provider's API base. `auth.secretRef` is optional. This covers a wide range of providers: Groq, Together AI, Fireworks AI, Mistral AI, DeepSeek, xAI (Grok), Google Gemini, Perplexity, OpenRouter, Cerebras, SambaNova, vLLM, text-generation-inference, LM Studio, and LiteLLM. See [ModelEndpoint > OpenAI-Compatible Providers](../concepts/tools/model-endpoint.md#openai-compatible-providers) for the full list with `base_url` values and example models. **Groq:** ```yaml apiVersion: orloj.dev/v1 kind: ModelEndpoint metadata: name: groq-default spec: provider: openai-compatible base_url: https://api.groq.com/openai/v1 default_model: llama-3.3-70b-versatile auth: secretRef: groq-api-key ``` **Mistral AI:** ```yaml apiVersion: orloj.dev/v1 kind: ModelEndpoint metadata: name: mistral-default spec: provider: openai-compatible base_url: https://api.mistral.ai/v1 default_model: mistral-large-latest auth: secretRef: mistral-api-key ``` **Google Gemini:** ```yaml apiVersion: orloj.dev/v1 kind: ModelEndpoint metadata: name: gemini-default spec: provider: openai-compatible base_url: https://generativelanguage.googleapis.com/v1beta/openai default_model: gemini-2.5-pro auth: secretRef: gemini-api-key ``` **Ollama via OpenAI-compatible endpoint:** ```yaml apiVersion: orloj.dev/v1 kind: ModelEndpoint metadata: name: ollama-openai spec: provider: openai-compatible base_url: http://127.0.0.1:11434/v1 default_model: llama3.1 allowPrivate: true ``` ### Agent Requirement Agents must set `spec.model_ref` to a valid ModelEndpoint. ### Constraining Models with Policy To restrict which models agents can use, create an AgentPolicy with `allowed_models`: ```yaml apiVersion: orloj.dev/v1 kind: AgentPolicy metadata: name: cost-policy spec: allowed_models: - gpt-4o - claude-3-5-sonnet-latest max_tokens_per_run: 50000 ``` Agents configured with models not on this list will be denied at execution time. ### Next Steps * [Model Routing](../concepts/tools/model-endpoint.md) -- deeper dive into ModelEndpoint configuration * [Configuration](../operations/configuration.md) -- environment variables and flags for model gateway setup * [Build a Custom Tool](./build-custom-tool.md) -- extend agent capabilities with external tools ## Connect an MCP Server This guide is for platform engineers who want to connect external MCP (Model Context Protocol) servers to Orloj. You will register an MCP server, verify tool discovery, selectively import tools, and assign them to agents. ### Prerequisites * Orloj server (`orlojd`) running with `--embedded-worker` * `orlojctl` available (or `go run ./cmd/orlojctl`) * An MCP server to connect (stdio-based or remote HTTP) If you have not set up Orloj yet, follow the [Install](../getting-started/install.md) and [Quickstart](../getting-started/quickstart.md) guides first. ### Background MCP servers are external processes or services that expose tools via the [Model Context Protocol](https://modelcontextprotocol.io/). Unlike regular Orloj tools (which are 1:1 -- one resource, one capability), a single MCP server can provide many tools. Orloj bridges this gap with the `McpServer` resource kind. When you register an MCP server, the controller: 1. Connects using the configured transport (stdio or Streamable HTTP) 2. Calls `tools/list` to discover available tools 3. Auto-generates a `Tool` resource (type=mcp) for each discovered tool 4. Keeps tools in sync on every reconcile cycle Generated tools are first-class `Tool` resources. Agents reference them by name just like any other tool. ### Step 1: Register an MCP Server (stdio) Stdio MCP servers run as child processes. Orloj spawns the process, communicates via stdin/stdout using JSON-RPC 2.0, and manages its lifecycle. Create a manifest (`github-mcp.yaml`): ```yaml apiVersion: orloj.dev/v1 kind: McpServer metadata: name: github-mcp spec: transport: stdio command: npx @github/mcp-server args: - "--token-from-env" env: - name: GITHUB_TOKEN secretRef: github-token ``` Create the secret for the token: ```yaml apiVersion: orloj.dev/v1 kind: Secret metadata: name: github-token spec: stringData: value: ghp_your_github_token_here ``` Apply both: ```bash orlojctl apply -f github-token-secret.yaml orlojctl apply -f github-mcp.yaml ``` ### Step 2: Register an MCP Server (Docker with file-based secrets) Some MCP servers require file-based credentials (OAuth JSON keys, service account files, TLS certificates) instead of environment variables. Use `spec.image` to run the server in a container and `mountPath` to deliver secrets as files. Create the secret with the credential file contents: ```yaml apiVersion: orloj.dev/v1 kind: Secret metadata: name: gmail-creds spec: stringData: oauth_keys: | {"installed":{"client_id":"...","client_secret":"..."}} credentials: | {"refresh_token":"...","token_type":"bearer"} ``` Create the MCP server manifest (`gmail-mcp.yaml`): ```yaml apiVersion: orloj.dev/v1 kind: McpServer metadata: name: gmail spec: transport: stdio image: mcp/gmail idle_timeout: 5m env: - name: GMAIL_OAUTH_PATH secretRef: gmail-creds/oauth_keys mountPath: /secrets/gcp-oauth.keys.json - name: GMAIL_CREDENTIALS_PATH secretRef: gmail-creds/credentials mountPath: /secrets/credentials.json ``` When `mountPath` is set, the resolved secret value is written to an ephemeral host file and bind-mounted read-only into the container at that path. The env var (`GMAIL_OAUTH_PATH`) is set to the mount path so the MCP server can locate the file. The files are automatically cleaned up when the session ends. Apply both: ```bash orlojctl apply -f gmail-creds-secret.yaml orlojctl apply -f gmail-mcp.yaml ``` ### Step 2b: Register an MCP Server (HTTP) Remote MCP servers communicate over HTTP using the Streamable HTTP transport. Use this for MCP servers running as hosted services. ```yaml apiVersion: orloj.dev/v1 kind: McpServer metadata: name: remote-mcp spec: transport: http endpoint: https://mcp.example.com/rpc auth: secretRef: mcp-api-key profile: bearer ``` Apply: ```bash orlojctl apply -f remote-mcp.yaml ``` ### Step 3: Verify Tool Discovery After applying, check the McpServer status: ```bash orlojctl get mcp-servers ``` ``` NAME TRANSPORT STATUS TOOLS LAST_SYNCED github-mcp stdio Ready 12 2025-03-18T14:30:00Z remote-mcp http Ready 5 2025-03-18T14:30:05Z ``` List the auto-generated tools: ```bash orlojctl get tools ``` Each generated tool follows the naming convention `{server}--{mcp-tool-name}`: ``` NAME TYPE STATUS github-mcp--create-issue mcp Ready github-mcp--search-repos mcp Ready github-mcp--list-prs mcp Ready ... ``` Inspect a specific generated tool to see its rich schema: ```bash orlojctl get tools github-mcp--create-issue -o json ``` The tool's `spec.input_schema` is populated directly from the MCP server's `tools/list` response. The model gateway uses this schema when formatting tool definitions for the LLM, giving it structured parameter information instead of the generic `{input: string}` fallback. ### Step 4: Filter Tools (Optional) By default, all tools discovered from an MCP server are imported. If a server exposes many tools and you only need a subset, use `spec.tool_filter.include`: ```yaml apiVersion: orloj.dev/v1 kind: McpServer metadata: name: github-mcp spec: transport: stdio command: npx @github/mcp-server args: - "--token-from-env" env: - name: GITHUB_TOKEN secretRef: github-token tool_filter: include: - create_issue - search_repos ``` Only the listed tools will be generated as `Tool` resources. Tools not in the allowlist are still discovered (visible in `status.discoveredTools`) but are not imported. Re-apply the manifest to update: ```bash orlojctl apply -f github-mcp.yaml ``` Tools that were previously generated but are no longer in the allowlist are automatically deleted on the next reconcile cycle. ### Step 5: Assign Tools to an Agent Generated MCP tools are referenced by name in `agent.spec.tools`, exactly like any other tool: ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: github-agent spec: model_ref: openai-default prompt: | You are a GitHub assistant. Use your tools to help the user manage issues and search repositories. tools: - github-mcp--create-issue - github-mcp--search-repos limits: max_steps: 8 timeout: 60s ``` Apply and submit a task: ```bash orlojctl apply -f github-agent.yaml ``` When the agent runs, the LLM sees the rich tool schemas from the MCP server and can call tools with structured arguments. The `GovernedToolRuntime` automatically routes `type=mcp` tools through the `MCPToolRuntime`, which sends `tools/call` to the MCP server via the session manager. ### Step 6: Configure Reconnection (Optional) MCP server connections can drop. The `reconnect` policy controls how aggressively Orloj retries: ```yaml spec: reconnect: max_attempts: 5 backoff: 2s ``` Defaults: 3 attempts with 2s backoff. If all attempts fail, the McpServer enters the `Error` phase. The controller retries on the next reconcile cycle. ### How It Works The data flow for an MCP tool call: ``` Agent step → GovernedToolRuntime (policy, timeout, retry) → MCPToolRuntime (resolves mcp_server_ref) → McpSessionManager (connection pool) → McpTransport (stdio or HTTP) → MCP Server (tools/call JSON-RPC 2.0) ``` Key implementation details: * **Session pooling**: One session per McpServer. Sessions are reused across tool calls and reconcile cycles. * **Schema propagation**: `spec.input_schema` and `spec.description` from tool discovery flow through to the model gateway, so the LLM gets rich parameter definitions. * **Garbage collection**: Generated tools carry an `orloj.dev/mcp-server` label. When an MCP server is deleted, all its generated tools are cleaned up. * **Governance**: MCP tools participate in the full governance pipeline. You can create `ToolPermission` and `AgentRole` resources for them, same as any other tool. ### McpServer Spec Reference | Field | Description | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------- | | `transport` | **Required**. `stdio` or `http`. | | `command` | stdio: command to spawn the MCP server process. Required unless `image` is set. | | `args` | stdio: command arguments. | | `env` | stdio: environment variables. Each entry has `name`, `value` (literal), or `secretRef` (resolved from Secret resource). | | `env[].mountPath` | Absolute path inside the container where the resolved value is written as a file. Only valid with `image`. | | `image` | stdio: container image. When set, the MCP server runs inside a Docker container. | | `idle_timeout` | Duration after which an idle session is shut down (e.g. `5m`). Default `0` means never evict. | | `endpoint` | http: the MCP server URL. | | `auth.secretRef` | http: secret for authentication. | | `auth.profile` | http: auth profile (`bearer`, `api_key_header`). Defaults to `bearer`. | | `tool_filter.include` | Optional allowlist of MCP tool names to import. When empty, all tools are imported. | | `reconnect.max_attempts` | Max reconnection attempts. Defaults to 3. | | `reconnect.backoff` | Backoff duration between attempts. Defaults to `2s`. | #### Status Fields | Field | Description | | ----------------- | ------------------------------------------------------- | | `phase` | `Pending`, `Connecting`, `Ready`, or `Error`. | | `discoveredTools` | All tool names from `tools/list`, regardless of filter. | | `generatedTools` | Tool resource names actually created. | | `lastSyncedAt` | Timestamp of last successful reconcile. | | `lastError` | Last error message, if any. | ### Next Steps * [Tools and Isolation](../concepts/tools/tool.md) -- concept deep-dive on tool types and isolation modes * [Build a Custom Tool](./build-custom-tool.md) -- for non-MCP tools that need custom implementation * [Set Up Multi-Agent Governance](./setup-governance.md) -- enforce authorization on MCP tools * [Resource Reference](../reference/resources/) -- full spec for all resource kinds ## Deploy Your First Pipeline This guide is for platform engineers who want to run a multi-agent pipeline end-to-end. You will define three agents, wire them into a sequential graph, submit a task, and inspect the results. ### Prerequisites * Orloj server (`orlojd`) running (sequential mode with `--embedded-worker` is fine for this guide) * `orlojctl` available (or `go run ./cmd/orlojctl`) If you have not set up Orloj yet, follow the [Install](../getting-started/install.md) and [Quickstart](../getting-started/quickstart.md) guides first. ### What You Will Build A three-stage pipeline where each agent hands off to the next: ``` planner ──► researcher ──► writer ``` The planner breaks the task into research requirements, the researcher gathers evidence, and the writer produces the final output. ### Step 1: Define the Agents Create three agent manifests. Each agent has a model, a system prompt, and execution limits. **Planner agent** (`planner-agent.yaml`): ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: bp-pipeline-planner-agent spec: model_ref: openai-default prompt: | You are the planning stage. Break the task into concrete research and writing requirements. limits: max_steps: 4 timeout: 20s ``` **Research agent** (`research-agent.yaml`): ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: bp-pipeline-research-agent spec: model_ref: openai-default prompt: | You are the research stage. Produce concise, verifiable findings for the writer. limits: max_steps: 6 timeout: 30s ``` **Writer agent** (`writer-agent.yaml`): ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: bp-pipeline-writer-agent spec: model_ref: openai-default prompt: | You are the writing stage. Synthesize prior handoffs into a polished final output. limits: max_steps: 4 timeout: 20s ``` Apply all three: ```bash orlojctl apply -f planner-agent.yaml orlojctl apply -f research-agent.yaml orlojctl apply -f writer-agent.yaml ``` ### Step 2: Define the Agent System The AgentSystem wires the agents into a pipeline graph: ```yaml apiVersion: orloj.dev/v1 kind: AgentSystem metadata: name: bp-pipeline-system labels: orloj.dev/pattern: pipeline spec: agents: - bp-pipeline-planner-agent - bp-pipeline-research-agent - bp-pipeline-writer-agent graph: bp-pipeline-planner-agent: edges: - to: bp-pipeline-research-agent bp-pipeline-research-agent: edges: - to: bp-pipeline-writer-agent ``` The `graph` field defines a directed acyclic graph. Each node lists its outbound edges. The planner routes to the researcher, who routes to the writer. The writer has no outbound edges, making it the terminal node. Apply the system: ```bash orlojctl apply -f agent-system.yaml ``` ### Step 3: Submit a Task Create a task that targets the pipeline system: ```yaml apiVersion: orloj.dev/v1 kind: Task metadata: name: bp-pipeline-task spec: system: bp-pipeline-system input: topic: state of enterprise AI copilots priority: high retry: max_attempts: 2 backoff: 2s message_retry: max_attempts: 2 backoff: 250ms max_backoff: 2s jitter: full ``` Apply the task: ```bash orlojctl apply -f task.yaml ``` ### Step 4: Monitor Execution Watch the task progress: ```bash orlojctl get tasks -w ``` View agent logs: ```bash orlojctl logs task/bp-pipeline-task ``` Trace the execution path through the graph: ```bash orlojctl trace task bp-pipeline-task ``` Visualize the system graph: ```bash orlojctl graph system bp-pipeline-system ``` ### What Happens at Runtime 1. The scheduler assigns `bp-pipeline-task` to an available worker. 2. The worker claims the task and acquires a lease. 3. The planner agent runs first (entry node -- zero indegree in the graph). 4. The planner's output is routed as a message to the research agent. 5. The research agent processes the message and routes its output to the writer. 6. The writer produces the final output. With no further edges, the task transitions to `Succeeded`. If any agent fails, the message-level retry configuration kicks in. After `max_attempts` exhaustion, the message moves to `deadletter`. If the task-level retry is also exhausted, the task itself transitions to `DeadLetter`. ### Using the Pre-Built Blueprint The complete pipeline blueprint is available in the repository: ```bash orlojctl apply -f examples/blueprints/pipeline/agents/ orlojctl apply -f examples/blueprints/pipeline/agent-system.yaml orlojctl apply -f examples/blueprints/pipeline/task.yaml ``` ### Next Steps * [Starter Blueprints](./starter-blueprints.md) -- explore hierarchical and swarm-loop topologies * [Set Up Multi-Agent Governance](./setup-governance.md) -- add policies and permissions to your pipeline * [Tasks and Scheduling](../concepts/tasks/task.md) -- understand the full task lifecycle ## Your First Agent System in 5 Minutes This tutorial walks you through a three-agent **pipeline**: planner → research → writer. You will go from an empty directory to a running task with real model calls, using the web console to watch progress. If you prefer to work from the repository’s checked-in examples instead of scaffolding, see [Quickstart](../getting-started/quickstart.md). ### Prerequisites * **orlojctl** installed — [Homebrew](../getting-started/install.md#homebrew-macos--linux) (`brew tap OrlojHQ/orloj && brew install orlojctl`) or the [install script](../getting-started/install.md#install-script-all-binaries) * **orlojd** — same install script, [release binaries](https://github.com/OrlojHQ/orloj/releases), or `go run ./cmd/orlojd` from a clone * An **OpenAI API key** (or change the scaffolded `model-endpoint.yaml` to another [supported provider](../guides/configure-model-routing.md)) ### 1. Start the server In one terminal, run the API server with an embedded worker and in-memory storage (no database required): ```bash orlojd --storage-backend=memory --embedded-worker ``` The default task execution mode is **sequential**, which is ideal for this walkthrough. Open the web console at [http://127.0.0.1:8080/](http://127.0.0.1:8080/) — you should see the dashboard. ### 2. Scaffold a pipeline In another terminal, scaffold the manifests: ```bash orlojctl init demo ``` This creates a `demo/` directory containing: * `agents/planner_agent.yaml`, `agents/research_agent.yaml`, `agents/writer_agent.yaml` * `agent-system.yaml` — graph wiring the three agents in order * `model-endpoint.yaml` — OpenAI endpoint named `openai-default` referencing a secret * `task.yaml` — a sample task (you will use `orlojctl run` instead in the next steps) The **AgentSystem** resource is named **`demo-system`** (prefix `demo` plus `-system`). You will pass that name to `orlojctl run`. ### 3. Add your API key The scaffold expects a Secret named **`openai-api-key`** with the default key **`value`** (this is what the model gateway reads): ```bash orlojctl create secret openai-api-key --from-literal value=sk-your-key-here ``` You do not need to edit `model-endpoint.yaml` unless you want a different secret name or provider. See [Configure Model Routing](./configure-model-routing.md) for Anthropic, Azure OpenAI, Ollama, and more. ### 4. Apply manifests Apply every manifest in the scaffolded directory at once: ```bash orlojctl apply -f demo/ --run ``` Or apply resources individually: ```bash orlojctl apply -f demo/agents/ orlojctl apply -f demo/model-endpoint.yaml orlojctl apply -f demo/agent-system.yaml ``` The first form also applies `task.yaml`, which creates a sample **Task** named `demo-task`. If you want to apply the directory without runnable tasks, omit `--run`. Either way, `orlojctl run` in the next step still creates a **new** task with your topic. ### 5. Run the pipeline Submit a task with input for the agents (the scaffold uses a `topic` field): ```bash orlojctl run --system demo-system topic="The future of open source AI" ``` The CLI prints a line like `task run-demo-system-… created, watching…`, waits until the task finishes, and prints the final **output** on success. ### 6. Watch execution * **Web console:** Open the task from the UI to see topology, status, and streaming detail. * **Logs snapshot:** After `run` reports the task name, fetch stored log lines (no live follow flag): ```bash orlojctl logs task/ ``` Use the exact task name printed when the task was created (for example `run-demo-system-1730000000000`). ### What just happened? Orloj stored your **Agent**, **AgentSystem**, **ModelEndpoint**, and **Secret** resources, then created a **Task** that references `demo-system`. The embedded worker **claimed** the task, executed the graph in order (planner, then research, then writer), routed each step through the **model gateway** using `openai-default`, and recorded status and output. Handoffs between agents follow the **execution and messaging** rules for your mode (here, sequential). For a deeper picture of components and scaling, read [Architecture](../concepts/architecture.md) and [Execution & Messaging](../concepts/execution-model.md). ### Next steps * [Build a Custom Tool](./build-custom-tool.md) — give agents callable tools * [Set Up Multi-Agent Governance](./setup-governance.md) — policies, roles, and tool permissions * [Deploy to a VPS](../deploy/vps.md) — Postgres, TLS, and production-like operation * [Starter Blueprints](./starter-blueprints.md) — try `orlojctl init myproject --blueprint hierarchical` or `--blueprint swarm-loop` ## Human Review Checkpoints This guide shows how to pause an Orloj workflow for human review of agent output or final task output. Example manifests in the repo: * [`examples/resources/agent-systems/review_sequential_system.yaml`](https://github.com/OrlojHQ/orloj/tree/main/examples/resources/agent-systems/review_sequential_system.yaml) * [`examples/resources/agent-systems/review_message_driven_system.yaml`](https://github.com/OrlojHQ/orloj/tree/main/examples/resources/agent-systems/review_message_driven_system.yaml) * [`examples/resources/tasks/review_sequential_task.yaml`](https://github.com/OrlojHQ/orloj/tree/main/examples/resources/tasks/review_sequential_task.yaml) * [`examples/resources/tasks/review_message_driven_task.yaml`](https://github.com/OrlojHQ/orloj/tree/main/examples/resources/tasks/review_message_driven_task.yaml) ### Sequential Example ```yaml apiVersion: orloj.dev/v1 kind: AgentSystem metadata: name: seq-review-system spec: agents: - draft-agent - publish-agent graph: draft-agent: review: checkpoint_id: draft-review reason: Editor must approve the draft before publication. next: publish-agent completion_review: checkpoint_id: final-review reason: Final signoff before success. ``` When `draft-agent` finishes, the task pauses in `WaitingApproval` and a `TaskApproval` is created. If the reviewer requests changes, Orloj reruns `draft-agent` with `review.*` fields in its input. Use `allow_request_changes: false` on a checkpoint when reviewers should only approve or deny. Use `max_review_cycles` to cap how many revision loops a single checkpoint can trigger. ### Message-Driven Example ```yaml apiVersion: orloj.dev/v1 kind: AgentSystem metadata: name: msg-review-system spec: agents: - intake-agent - decision-agent graph: intake-agent: review: checkpoint_id: intake-review reason: Human reviewer must approve intake output before routing. edges: - to: decision-agent ``` In message-driven mode, Orloj freezes the downstream messages, creates a `TaskApproval`, and only publishes those messages after approval. ### Reviewer Commands ```bash orlojctl get task-approvals orlojctl approve task-approval my-approval --decided-by reviewer@example.com --comment "Looks good" orlojctl deny task-approval my-approval --decided-by reviewer@example.com --comment "Do not send this" orlojctl request-changes task-approval my-approval --decided-by reviewer@example.com --comment "Add a compliance disclaimer" ``` `request-changes` requires reviewer feedback via `--comment` or the legacy `--reason` alias. The API and CLI return a conflict if that checkpoint disables `request_changes` or if it has already reached `max_review_cycles`. ### What To Watch * `Task.status.phase = WaitingApproval` * `Task.status.blocked_on` for the exact approval resource * `TaskApproval.spec.review_cycle` and `spec.supersedes` for re-review loops See also: * [TaskApproval concept](../concepts/governance/task-approval.md) * [TaskApproval resource reference](../reference/resources/task-approval.md) ## Guides Step-by-step tutorials for common Orloj workflows. **Start here:** **[Your First Agent System in 5 Minutes](./five-minute-tutorial.md)** — install `orlojctl` and `orlojd`, scaffold a pipeline with `orlojctl init`, add an API key, apply manifests, and run a task end-to-end with the web console. Other guides use real manifests from the `examples/` directory or walk through authoring resources by hand. For **ready-made scenario folders** (full YAML sets you can copy into your environment), see [examples/use-cases/](https://github.com/OrlojHQ/orloj/tree/main/examples/use-cases). If you have not installed Orloj yet, begin with [Install](../getting-started/install.md). Developers building from a clone may prefer the [Quickstart](../getting-started/quickstart.md) (`go run` + checked-in blueprints). ### Available Guides **[Your First Agent System in 5 Minutes](./five-minute-tutorial.md)** *The fastest path from zero to a running multi-agent pipeline with a real model.* **[Deploy Your First Pipeline](./deploy-pipeline.md)** *For platform engineers who want to run a multi-agent pipeline end-to-end.* Walk through the pipeline blueprint: define three agents (planner, researcher, writer), wire them into a sequential graph, submit a task, and inspect the results. **[Set Up Multi-Agent Governance](./setup-governance.md)** *For platform engineers who need to enforce tool authorization and model constraints.* Create policies, roles, and tool permissions. Deploy a governed agent system and verify that unauthorized tool calls are denied. **[Configure Model Routing](./configure-model-routing.md)** *For platform engineers who need to route agents to different model providers.* Set up ModelEndpoints for OpenAI and Anthropic, bind agents to endpoints by reference, and verify that requests route correctly. **[Connect an MCP Server](./connect-mcp-server.md)** *For platform engineers who want to integrate MCP-compatible tool servers.* Register an MCP server (stdio or HTTP), verify tool discovery, filter imported tools, and assign them to agents. **[Build a Custom Tool](./build-custom-tool.md)** *For developers who need to extend agent capabilities with external tools.* Implement the Tool Contract v1, register the tool as a resource, configure isolation and retry, and validate with the conformance harness. **[Capture README Orloj in Action Media](./readme-media-capture.md)** *For maintainers refreshing repository branding assets.* Generate reproducible frontend screenshots and lifecycle GIF media for the README Orloj in Action section. **[Run Your First Agent Evaluation](./run-agent-evaluation.md)** *For platform engineers and ML teams who want to measure agent quality.* Create a golden dataset, run evaluations with multiple scoring strategies, compare models side-by-side, and set up human review workflows. ## kubectl vs orlojctl — Which Tool Do I Use? When the [CRD operator](../deploy/kubernetes-operator.md) is deployed, Orloj resources live as real Kubernetes CRDs. This means `kubectl` can handle basic CRUD — but `orlojctl` remains the primary tool for operations, runtime control, and admin tasks. ### Resource Lifecycle (CRUD) | Action | `kubectl` (with operator) | `orlojctl` (always works) | | --------------- | ---------------------------------------------- | ---------------------------------- | | Create / Update | `kubectl apply -f agent.yaml` | `orlojctl apply -f agent.yaml` | | List | `kubectl get agents` | `orlojctl get agents` | | Describe | `kubectl describe agent my-agent` | `orlojctl describe agent my-agent` | | Delete | `kubectl delete agent my-agent` | `orlojctl delete agent my-agent` | | Diff | `kubectl diff -f agent.yaml` | `orlojctl diff -f agent.yaml` | | Edit | `kubectl edit agent my-agent` | `orlojctl edit agent my-agent` | | Validate | `kubectl apply --dry-run=server -f agent.yaml` | `orlojctl validate -f agent.yaml` | | Watch | `kubectl get agents -w` | `orlojctl get agents -w` | Both tools work for CRUD when the operator is running. Choose based on your workflow: * **GitOps / CI pipeline** → `kubectl apply` (or let Argo CD / Flux do it) * **Interactive / ad-hoc** → `orlojctl apply` (talks directly to the Orloj API, no operator required) ### Operations (orlojctl only) These commands interact with the Orloj runtime and have no `kubectl` equivalent: | Command | Purpose | | -------------------------------- | ----------------------------------- | | `orlojctl run --system ` | Create and execute a task | | `orlojctl cancel task ` | Cancel a running task | | `orlojctl retry task ` | Retry a terminal task | | `orlojctl approve / deny` | Approve or deny tool/task approvals | | `orlojctl logs ` | Stream agent execution logs | | `orlojctl trace task ` | Inspect task execution trace | | `orlojctl graph system ` | Render agent system topology | | `orlojctl events` | Stream control-plane events | | `orlojctl get tasks -w` | Watch task lifecycle changes | | `orlojctl messages task/` | Inspect inter-agent messages | | `orlojctl metrics task/` | View task message metrics | | `orlojctl memory-entries ` | Query memory store entries | | `orlojctl top workers` | Worker utilization overview | | `orlojctl top tasks` | Task status overview | | `orlojctl wait task/` | Block until a task condition is met | ### Admin (orlojctl only) | Command | Purpose | | ------------------------------- | --------------------------------------- | | `orlojctl admin create-user` | Create a local user account | | `orlojctl admin list-users` | List all user accounts | | `orlojctl admin delete-user` | Remove a user account | | `orlojctl admin reset-password` | Reset a user's password | | `orlojctl auth whoami` | Show current identity | | `orlojctl create token` | Create an API bearer token | | `orlojctl get tokens` | List API tokens | | `orlojctl config set-profile` | Configure CLI connection profiles | | `orlojctl seal secret` | Encrypt secrets for git-safe storage | | `orlojctl validate -f` | Offline manifest validation (no server) | | `orlojctl eval run` | Run agent evaluations | | `orlojctl tool test` | Test WASM tool modules | | `orlojctl tool scaffold` | Scaffold a new WASM tool project | | `orlojctl init` | Scaffold a new agent system | ### Bottom Line * **`kubectl`** handles resource CRUD when the operator is deployed — ideal for GitOps and teams that already use `kubectl` for everything. * **`orlojctl`** is required for runtime operations (run, cancel, approve, logs, trace) and admin tasks (users, tokens, secrets, eval). It also works for CRUD without the operator. * You don't have to choose one exclusively. Most teams use `kubectl apply` in CI for configuration and `orlojctl` interactively for operations. ### Related Docs * [Kubernetes CRD Operator](../deploy/kubernetes-operator.md) * [CLI Reference](../reference/cli.md) ## Capture README Orloj in Action Media This guide documents the reproducible workflow for generating the README media assets used in the **Orloj in Action** section. The capture pipeline targets the **frontend web console** (not docs pages) and outputs assets to `docs/public/readme/`. ### Prerequisites * Repository root as current working directory * Built binaries (`orlojd`, `orlojctl`) or Go toolchain available for fallback build * Python packages: * `playwright` * `Pillow` * Playwright Chromium runtime: ```bash python3 -m playwright install chromium ``` ### One-Command Capture From repo root: ```bash scripts/capture_readme_media.sh ``` What the script does: 1. Starts `orlojd` in deterministic local mode (`memory`, `sequential`, embedded worker). 2. Applies `testing/scenarios-real/01-pipeline` by default. 3. Waits for: * `agent-systems/` to reach `Ready` * `tasks/` to reach `Succeeded` 4. Captures fixed UI routes via Playwright and writes: * `docs/public/readme/dashboard-overview.png` * `docs/public/readme/system-topology.png` * `docs/public/readme/task-detail-graph.png` * `docs/public/readme/task-trace-logs.png` * `docs/public/readme/task-run-lifecycle.gif` ### Capture Targets and Sizing * Screenshot viewport: `1728x1080` (desktop-oriented) * GIF output size: `960x540` * Theme: light mode * Locale: `en-US` The scripts intentionally keep names stable so README links do not need to change. Default capture inputs are controlled by env vars in `scripts/capture_readme_media.sh`: * `ORLOJ_CAPTURE_MANIFEST_DIR` (default `testing/scenarios-real/01-pipeline`) * `ORLOJ_CAPTURE_NAMESPACE` (default `rr-real-pipeline`) * `ORLOJ_CAPTURE_SYSTEM` (default `rr-real-pipeline-system`) * `ORLOJ_CAPTURE_REFERENCE_TASK` (default `rr-real-pipeline-task`) * `ORLOJ_CAPTURE_READY_TIMEOUT` (default `2m`) * `ORLOJ_CAPTURE_TASK_TIMEOUT` (default `3m`) ### Naming Convention * `dashboard-overview.png` : Control-plane overview * `system-topology.png` : AgentSystem resource tree/topology view * `task-detail-graph.png` : Task detail with graph tab * `task-trace-logs.png` : Task detail with logs/trace context * `task-run-lifecycle.gif` : Task lifecycle (created -> running -> succeeded) ### Redaction and Safety Rules * Use safe demo task input only. * Do not capture personal usernames, hostnames, or secrets. * Keep auth in `off` mode for capture runs. * If any sensitive text appears, regenerate after clearing local state. ### Quality Gate Checklist Before committing refreshed media: 1. Confirm all five media files were regenerated in `docs/public/readme/`. 2. Confirm README images render in GitHub markdown preview. 3. Confirm status and UI text are legible on desktop and mobile widths. 4. Keep combined media payload reasonably small (target under \~8 MB total). ## Run Your First Agent Evaluation This guide walks you through creating a golden dataset, running an evaluation against an agent system, and comparing results across different configurations. By the end, you will have a repeatable evaluation workflow for measuring agent quality. ### Prerequisites * Orloj server (`orlojd`) running (sequential mode with `--embedded-worker` is fine for this guide) * `orlojctl` available (or `go run ./cmd/orlojctl`) * At least one [AgentSystem](../concepts/agents/agent-system.md) and [ModelEndpoint](../reference/resources/model-endpoint.md) already applied If you have not set up Orloj yet, follow the [Install](../getting-started/install.md) and [Quickstart](../getting-started/quickstart.md) guides first. ### What You Will Build A golden dataset that tests a support triage agent, two eval runs (one per model), and a comparison showing which model performs better. ``` Dataset ──► EvalRun (gpt-4o) ──► ┐ ├──► Compare Dataset ──► EvalRun (claude) ──► ┘ ``` ### Step 1: Define a Golden Dataset A dataset contains sample inputs and expected outputs. Create `triage-dataset.yaml`: ```yaml apiVersion: orloj.dev/v1 kind: EvalDataset metadata: name: triage-golden spec: description: "Golden set for the support triage agent" samples: - name: billing-question input: prompt: "I was charged twice for my subscription last month" expected: output_contains: "billing" - name: password-reset input: prompt: "I can't log in, I forgot my password" expected: output_contains: "password" output_not_contains: "billing" - name: refund-request input: prompt: "I want a refund for order #12345, the item arrived broken" expected: output_contains: "refund" - name: feature-request input: prompt: "It would be great if you could add dark mode" expected: output_contains: "feature" ``` Apply it: ```bash orlojctl apply -f triage-dataset.yaml ``` Verify: ```bash orlojctl eval datasets ``` ``` NAME NAMESPACE SAMPLES DESCRIPTION triage-golden default 4 Golden set for the support triage agent ``` ### Step 2: Run an Evaluation with exact\_match The simplest scoring strategy is `exact_match`, which checks expected output fields against the agent's actual output. Start a run: ```bash orlojctl eval run \ --dataset triage-golden \ --system support-triage-system ``` The CLI creates an EvalRun resource and polls until completion. You will see output like: ``` EvalRun triage-golden-run-a1b2c3 created (Pending) Phase: Running (2/4 samples completed) Phase: Scoring... EvalRun triage-golden-run-a1b2c3: Succeeded Pass Rate: 75.0% Mean Score: 0.750 Tokens: 2340 Latency: 1.8s SAMPLE PASS SCORE LATENCY billing-question ✓ 1.000 1.2s password-reset ✓ 1.000 1.5s refund-request ✓ 1.000 2.1s feature-request ✗ 0.000 2.4s ``` You can also create runs declaratively: ```yaml apiVersion: orloj.dev/v1 kind: EvalRun metadata: name: triage-eval-exact spec: dataset_ref: triage-golden system: support-triage-system scoring: strategy: exact_match concurrency: 2 timeout: 60s ``` ```bash orlojctl apply -f eval-run.yaml # creates the run in suspended state orlojctl eval start triage-eval-exact # start it when ready # or apply and start immediately: orlojctl apply -f eval-run.yaml --run ``` ### Step 3: Run with LLM-as-Judge For subjective quality assessment, use `llm_judge`. The judge model evaluates each sample against a rubric: ```bash orlojctl eval run \ --dataset triage-golden \ --system support-triage-system \ --scoring llm_judge \ --model-ref gpt-4o-judge \ --rubric "Rate accuracy and helpfulness of the triage classification (0-1)." ``` Or declaratively: ```yaml apiVersion: orloj.dev/v1 kind: EvalRun metadata: name: triage-eval-llm spec: dataset_ref: triage-golden system: support-triage-system scoring: strategy: llm_judge model_ref: gpt-4o-judge rubric: "Rate accuracy and helpfulness of the triage classification (0-1)." concurrency: 4 timeout: 120s ``` The judge model must be configured as a [ModelEndpoint](../reference/resources/model-endpoint.md) like any other model. ### Step 4: A/B Test with Agent Overrides Compare two models without modifying your agents. Use `agent_overrides` to swap the model for a specific agent: ```yaml apiVersion: orloj.dev/v1 kind: EvalRun metadata: name: triage-eval-claude spec: dataset_ref: triage-golden system: support-triage-system scoring: strategy: llm_judge model_ref: gpt-4o-judge rubric: "Rate accuracy and helpfulness of the triage classification (0-1)." concurrency: 4 agent_overrides: triage-agent: model_ref: claude-sonnet ``` Apply and start: ```bash orlojctl apply -f eval-claude.yaml --run orlojctl eval get triage-eval-claude -w ``` ### Step 5: Compare Runs Compare any number of completed runs side-by-side: ```bash orlojctl eval compare triage-eval-llm triage-eval-claude ``` ``` METRIC triage-eval-llm triage-eval-claude Pass Rate 100.0% 75.0% Mean Score 0.920 0.850 Tokens 4200 3800 Mean Latency 2.1s 1.8s ``` The comparison API also returns per-sample deltas accessible via the REST API: ```bash curl "http://localhost:8080/v1/eval-runs/compare?names=triage-eval-llm,triage-eval-claude" ``` ### Step 6: Manual Review (Optional) For subjective tasks where automated scoring is not appropriate, use `manual` scoring: ```bash orlojctl eval run \ --dataset triage-golden \ --system support-triage-system \ --scoring manual ``` Once the tasks complete, the run transitions to **PendingReview**. Export the results: ```bash orlojctl eval export triage-golden-run-xyz --format csv > review.csv ``` The CSV contains columns: `sample_name`, `input`, `output`, `score`, `pass`, `reasoning`. Fill in the `score`, `pass`, and `reasoning` columns, then import: ```bash orlojctl eval import triage-golden-run-xyz -f review.csv ``` Finalize to compute aggregate metrics: ```bash orlojctl eval finalize triage-golden-run-xyz ``` ### Step 7: Inspect Results Get full details for any run: ```bash orlojctl eval get triage-eval-llm -o yaml ``` List all runs: ```bash orlojctl eval list ``` ``` NAME DATASET SYSTEM PHASE PASS RATE SAMPLES triage-eval-exact triage-golden support-triage-system Succeeded 75.0% 4 triage-eval-llm triage-golden support-triage-system Succeeded 100.0% 4 triage-eval-claude triage-golden support-triage-system Succeeded 75.0% 4 ``` ### Tips * **Start small:** begin with `exact_match` on 5–10 samples to validate the workflow before scaling up. * **Use concurrency wisely:** higher concurrency runs faster but consumes more model quota. Start with 2–4. * **Version your datasets:** use descriptive names like `triage-golden-v2` to track dataset evolution. * **Automate with TaskSchedule:** create a [TaskSchedule](../concepts/tasks/task-schedule.md) or webhook to run evaluations on every deployment. * **Export for dashboards:** pipe `orlojctl eval export` output to your observability stack for trend tracking. ### Related * [Agent Evaluation (concept)](../concepts/evaluation/) -- framework overview * [EvalDataset reference](../reference/resources/eval-dataset.md) -- full spec documentation * [EvalRun reference](../reference/resources/eval-run.md) -- full spec documentation * [CLI Reference: `orlojctl eval`](../reference/cli.md) -- all eval subcommands ## Set Up Multi-Agent Governance This guide is for platform engineers who need to enforce tool authorization and model constraints on their agent systems. You will create policies, roles, and tool permissions, deploy a governed agent system, and verify that unauthorized tool calls are denied. ### Prerequisites * Orloj server (`orlojd`) and at least one worker running * `orlojctl` available * Familiarity with the [Governance and Policies](../concepts/governance/) concepts ### What You Will Build A governed version of the report pipeline where: * An AgentPolicy restricts model usage and blocks dangerous tools * AgentRoles grant specific tool permissions to agents * ToolPermissions define authorization requirements for each tool * An agent that attempts an unauthorized tool call is denied ### Step 1: Define the Tools Start by creating the tools your agents will reference: ```yaml apiVersion: orloj.dev/v1 kind: Tool metadata: name: web_search spec: type: http endpoint: https://api.search.com auth: secretRef: search-api-key ``` ```yaml apiVersion: orloj.dev/v1 kind: Tool metadata: name: vector_db spec: type: http endpoint: https://api.vector-db.local/query ``` Apply both: ```bash orlojctl apply -f web_search_tool.yaml orlojctl apply -f vector_db_tool.yaml ``` ### Step 2: Create Tool Permissions Define what permissions are required to invoke each tool: ```yaml apiVersion: orloj.dev/v1 kind: ToolPermission metadata: name: web-search-invoke spec: tool_ref: web_search action: invoke match_mode: all apply_mode: global required_permissions: - tool:web_search:invoke - capability:web.read ``` ```yaml apiVersion: orloj.dev/v1 kind: ToolPermission metadata: name: vector-db-invoke spec: tool_ref: vector_db action: invoke match_mode: all apply_mode: global required_permissions: - tool:vector_db:invoke ``` Apply: ```bash orlojctl apply -f web_search_invoke_permission.yaml orlojctl apply -f vector_db_invoke_permission.yaml ``` ### Step 3: Create Agent Roles Roles bundle permissions that can be bound to agents: ```yaml apiVersion: orloj.dev/v1 kind: AgentRole metadata: name: analyst-role spec: description: Can call web search style tools. permissions: - tool:web_search:invoke - capability:web.read ``` ```yaml apiVersion: orloj.dev/v1 kind: AgentRole metadata: name: vector-reader-role spec: description: Can query vector knowledge tools. permissions: - tool:vector_db:invoke ``` Apply: ```bash orlojctl apply -f analyst_role.yaml orlojctl apply -f vector_reader_role.yaml ``` ### Step 4: Create an Agent Policy The policy scopes model and tool constraints to a specific agent system: ```yaml apiVersion: orloj.dev/v1 kind: AgentPolicy metadata: name: cost-policy spec: apply_mode: scoped target_systems: - report-system-governed max_tokens_per_run: 50000 allowed_models: - gpt-4o blocked_tools: - filesystem_delete ``` Apply: ```bash orlojctl apply -f cost_policy.yaml ``` ### Step 5: Deploy the Governed Agent Create an agent that binds the `analyst-role` but **not** `vector-reader-role`. This means it will be authorized for `web_search` but denied for `vector_db`: ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: research-agent-governed spec: model_ref: openai-default prompt: | You are a research assistant. Produce concise evidence-backed answers. roles: - analyst-role tools: - web_search - vector_db memory: ref: research-memory limits: max_steps: 6 timeout: 30s ``` Even though `vector_db` is listed in `tools`, the agent lacks the `tool:vector_db:invoke` permission, so any attempt to call it will be denied. ### Step 6: Wire the System and Submit a Task ```yaml apiVersion: orloj.dev/v1 kind: AgentSystem metadata: name: report-system-governed spec: agents: - planner-agent - research-agent-governed - writer-agent graph: planner-agent: next: research-agent-governed research-agent-governed: next: writer-agent ``` ```bash orlojctl apply -f report_system_governed.yaml orlojctl apply -f weekly_report_governed_task.yaml ``` ### Step 7: Verify Governance Enforcement Check the task trace for authorization events: ```bash orlojctl trace task weekly-report-governed ``` In the trace output, look for: * Successful `web_search` invocations (agent holds required permissions) * `tool_permission_denied` errors for any `vector_db` attempts (agent lacks `tool:vector_db:invoke`) ### Allowing Both Tools To grant the agent access to both tools, add `vector-reader-role` to its roles: ```yaml spec: roles: - analyst-role - vector-reader-role ``` The pre-built example for this is `examples/resources/agents/research_agent_governed_allow.yaml` with `examples/resources/agent-systems/report_system_governed_allow.yaml`. ### Next Steps * [Governance and Policies](../concepts/governance/) -- deeper dive into the authorization model * [Security and Isolation](../operations/security.md) -- operational security controls * [Configure Model Routing](./configure-model-routing.md) -- set up provider-specific model endpoints ## Starter Blueprints Blueprints are ready-to-run templates that combine agents, an agent system (the graph), and a task into a single directory. They are the fastest way to see Orloj in action and to understand each orchestration pattern. For copy-paste **use case** bundles (YAML + README per scenario) that map these patterns to simple and production-scale problems, see [examples/use-cases/](https://github.com/OrlojHQ/orloj/tree/main/examples/use-cases). ### Available Patterns #### Pipeline Predictable stage-by-stage execution: `planner -> research -> writer`. ```bash orlojctl apply -f examples/blueprints/pipeline/ --run ``` #### Hierarchical Manager-led delegation: `manager -> leads -> workers -> editor`. ```bash orlojctl apply -f examples/blueprints/hierarchical/ --run ``` #### Swarm and Loop Parallel exploration with iterative coordination: `coordinator <-> scouts -> synthesizer`. Safety-bounded by `Task.spec.max_turns`. ```bash orlojctl apply -f examples/blueprints/swarm-loop/ --run ``` ### Runtime Compatibility Blueprints work in both execution modes: * **Sequential** -- run with `--embedded-worker` for single-process development. Good for getting started. * **Message-driven** -- run with `--agent-message-bus-backend=memory` (or `nats-jetstream`) and `--agent-message-consume` for distributed execution. Required for parallel fan-out in the swarm-loop pattern. ### What is Inside a Blueprint Each blueprint directory contains: * `agents/*.yaml` -- individual Agent resources with prompts, model config, and tool bindings. * `agent-system.yaml` -- the AgentSystem resource defining the graph topology (nodes and edges). * `task.yaml` -- a Task resource that targets the agent system with sample input. Apply the entire directory with `orlojctl apply -f / --run` to include runnable `Task` resources. ## Core Concepts This page introduces Orloj's key building blocks and how they fit together. Read this before diving into the individual concept pages. ### Resource Map ``` TaskSchedule ──creates──▶ Task ◀──creates── TaskWebhook │ triggers ▼ AgentSystem ╱ ╲ composes composes ╱ ╲ Agent A ─────────── Agent B ╱ │ ╲ ╱ │ calls invokes reads calls invokes ╱ │ ╲ ╱ │ ModelEndpoint Tool Memory │ │ │ │ │ │ resolves resolves │ │ auth via auth via │ │ ╲ ╱ │ │ Secret │ │ │ │ ┄┄┄┄┄┄┄┄ Governance ┄┄┄┄┄┄┄┄┄┄┄┄┄┤┄┄┄┄┄┄┤ ┆ ┆ ┆ AgentPolicy ┄┄ constrains ┄┄▶ Agent A, Agent B AgentRole ┄┄ grants permissions to ┄▶ Agents ToolPermission ┄ controls access to ┄▶ Tools Worker ──claims and executes──▶ Task ``` ### Agents An [**Agent**](../concepts/agents/agent.md) is a declarative unit of work backed by a language model. You define its prompt, model, tools, and constraints in YAML. ```yaml kind: Agent metadata: name: research-agent spec: model_ref: openai-default prompt: "You are a research assistant." tools: [web_search] limits: max_steps: 6 ``` ### Agent Systems An [**AgentSystem**](../concepts/agents/agent-system.md) composes agents into a directed graph -- pipelines, hierarchies, or swarm loops. ```yaml kind: AgentSystem metadata: name: report-system spec: agents: [planner, researcher, writer] graph: planner: edges: [{to: researcher}] researcher: edges: [{to: writer}] ``` ### Tasks A [**Task**](../concepts/tasks/task.md) is a request to execute an AgentSystem. Tasks track lifecycle state (`Pending` -> `Running` -> `Succeeded`), support retry, and produce output. ```yaml kind: Task metadata: name: weekly-report spec: system: report-system input: topic: AI startups ``` ### Tools A [**Tool**](../concepts/tools/tool.md) is an external capability agents can invoke. Seven transport types (HTTP, external, gRPC, webhook-callback, MCP, CLI, WASM) and four isolation modes (none, sandboxed, container, WASM). ```yaml kind: Tool metadata: name: web_search spec: type: http endpoint: https://api.search.com auth: secretRef: search-api-key ``` ### Model Endpoints A [**ModelEndpoint**](../concepts/tools/model-endpoint.md) configures a connection to a model provider (OpenAI, Anthropic, Azure OpenAI, Ollama). Agents reference endpoints by name, decoupling agent definitions from provider details. ```yaml kind: ModelEndpoint metadata: name: openai-default spec: provider: openai default_model: gpt-4o-mini auth: secretRef: openai-api-key ``` ### Memory [**Memory**](../concepts/memory/) gives agents persistent storage across execution steps and task runs. Three layers: conversation history, task-scoped shared state, and persistent backends (in-memory, pgvector, HTTP). ### Governance The [**governance layer**](../concepts/governance/) controls what agents can do at runtime: * [**AgentPolicy**](../concepts/governance/agent-policy.md) -- constrain models, block tools, cap tokens * [**AgentRole**](../concepts/governance/agent-role.md) -- grant named permissions to agents * [**ToolPermission**](../concepts/governance/tool-permission.md) -- require permissions to invoke tools Governance is fail-closed: unauthorized tool calls are denied, not silently ignored. ### Automation * [**TaskSchedule**](../concepts/tasks/task-schedule.md) -- create tasks on a cron schedule * [**TaskWebhook**](../concepts/tasks/task-webhook.md) -- create tasks from external HTTP events ### Infrastructure * [**Worker**](../concepts/infrastructure/worker.md) -- execution unit that claims and runs tasks * [**Secret**](../concepts/tools/secret.md) -- stores API keys and credentials * [**McpServer**](../concepts/tools/mcp-server.md) -- connects to MCP servers and auto-discovers tools ### Next Steps * [Architecture Overview](../concepts/architecture.md) -- understand the three-layer architecture * [Deploy Your First Pipeline](../guides/deploy-pipeline.md) -- build and run a multi-agent pipeline * [Explore Concepts](../concepts/agents/agent.md) -- dive into individual resource pages ## Install Orloj This guide covers how to install Orloj for local evaluation and production-like use: from source (clone and run or build), from **release binaries** (GitHub Releases), or from **container images** (GitHub Container Registry). Use release artifacts when you want a tagged, published build instead of building from source. ### Before You Begin * **From source:** Go `1.24+`, optionally Bun `1.3+` for docs/frontend * **Containers:** Docker * **API checks:** `curl` and `jq` *** ### Homebrew (macOS / Linux) The fastest way to install the CLI: ```bash brew tap OrlojHQ/orloj brew install orlojctl ``` Formula versions follow [Orloj releases](https://github.com/OrlojHQ/orloj/releases). Upgrade to the latest release: ```bash brew update && brew upgrade orlojctl ``` > Homebrew installs the **`orlojctl`** CLI only. For the server (`orlojd`) and worker (`orlojworker`) binaries, use the install script, release binaries, or container images below. *** ### From source Clone the repo, then either run in place or build binaries. ```bash git clone https://github.com/OrlojHQ/orloj.git && cd orloj ``` #### Run from source (no build) Single process with embedded worker: ```bash go run ./cmd/orlojd \ --storage-backend=memory \ --task-execution-mode=sequential \ --embedded-worker ``` #### Build local binaries ```bash go build -o ./bin/orlojd ./cmd/orlojd go build -o ./bin/orlojworker ./cmd/orlojworker go build -o ./bin/orlojctl ./cmd/orlojctl ``` Run the server: ```bash ./bin/orlojd --storage-backend=memory --task-execution-mode=sequential --embedded-worker ``` *** ### From release binaries (GitHub Releases) The install script detects your OS and architecture, downloads the matching binaries, verifies checksums, and installs to `/usr/local/bin` (or `~/.local/bin` if no sudo): ```bash curl -sSfL https://raw.githubusercontent.com/OrlojHQ/orloj/main/scripts/install.sh | sh ``` Install a specific version or a subset of binaries: ```bash # Specific version curl -sSfL https://raw.githubusercontent.com/OrlojHQ/orloj/main/scripts/install.sh | ORLOJ_VERSION=v0.1.1 sh # CLI only (for remote management of a hosted deployment) curl -sSfL https://raw.githubusercontent.com/OrlojHQ/orloj/main/scripts/install.sh | ORLOJ_BINARIES="orlojctl" sh ``` Or download manually from [GitHub Releases](https://github.com/OrlojHQ/orloj/releases). Artifacts are named by binary, git tag, OS, and arch (e.g. `orlojd_v0.1.0_linux_amd64.tar.gz`, `orlojctl_v0.1.0_darwin_arm64.tar.gz`). Verify with `checksums.txt` on the same release. Run the server: ```bash orlojd --storage-backend=memory --task-execution-mode=sequential --embedded-worker ``` #### CLI only for hosted deployments If `orlojd` and workers run elsewhere—Docker Compose on a VPS, Kubernetes, GHCR images, or a managed host—you **do not** need the full repo on your laptop. Install just the CLI: ```bash brew tap OrlojHQ/orloj brew install orlojctl ``` Or with the install script: ```bash curl -sSfL https://raw.githubusercontent.com/OrlojHQ/orloj/main/scripts/install.sh | ORLOJ_BINARIES="orlojctl" sh ``` Or download only the **`orlojctl_*__`** archive for your platform from [GitHub Releases](https://github.com/OrlojHQ/orloj/releases), verify it with `checksums.txt`, extract the binary, and put it on your `PATH`. Point `orlojctl` at your API with `--server` and authenticate with a bearer token; see [Remote CLI and API access](../deploy/remote-cli-access.md). Prefer a **CLI version that matches your server’s release tag** when possible. *** ### From container images (GHCR) Published releases are pushed to GitHub Container Registry. Pull and run the server and worker without building from source: ```bash docker pull ghcr.io/orlojhq/orloj-orlojd:latest docker pull ghcr.io/orlojhq/orloj-orlojworker:latest ``` Use a version tag instead of `latest` for production (e.g. `ghcr.io/orlojhq/orloj-orlojd:v0.1.0`). You still need Postgres and optionally NATS for persistence and message-driven mode; see [Deployment](../deploy/) for full-stack options. Example, server only with in-memory storage: ```bash docker run --rm -p 8080:8080 ghcr.io/orlojhq/orloj-orlojd:latest \ --addr=:8080 \ --storage-backend=memory \ --task-execution-mode=sequential \ --embedded-worker ``` For a full stack (Postgres, NATS, server, workers), use the [VPS](../deploy/vps.md) or [Kubernetes](../deploy/kubernetes.md) deployment guides with `image: ghcr.io/orlojhq/orloj-orlojd:` (and the worker image) instead of building from the repo. *** ### Docker Compose (from source) To run the full stack from the repo (Postgres, NATS, `orlojd`, two workers) with a local build: ```bash git clone https://github.com/OrlojHQ/orloj.git && cd orloj docker compose up --build ``` This builds the server and worker images from the Dockerfile. To use release images instead, override the service images to `ghcr.io/orlojhq/orloj-orlojd:` and `ghcr.io/orlojhq/orloj-orlojworker:` (see [Deployment](../deploy/)). ### Verify Installation ```bash curl -s http://127.0.0.1:8080/healthz | jq . go run ./cmd/orlojctl get workers ``` Expected result: * `healthz` returns healthy status. * At least one worker is `Ready`. ### Next Steps * [Deployment Overview](../deploy/) * [Local Deployment](../deploy/local.md) * [VPS Deployment](../deploy/vps.md) * [Kubernetes Deployment](../deploy/kubernetes.md) * [Quickstart](./quickstart.md) * [Configuration](../operations/configuration.md) ## Quickstart Get a multi-agent pipeline running in under five minutes. This quickstart uses sequential execution mode -- the simplest way to run Orloj with a single process and no external dependencies. > **Using Homebrew or release binaries?** Use the guided [5-minute tutorial](../guides/five-minute-tutorial.md) instead: it covers `orlojctl init`, a real OpenAI secret (`value=…`), and `orlojctl run` against `demo-system`. This page is aimed at **from-source** development with `go run` and the checked-in `examples/` blueprints. ### Before You Begin * Go `1.24+` is installed. * You are in repository root. ### 1. Start the Server Start `orlojd` with an embedded worker in sequential mode: ```bash go run ./cmd/orlojd \ --storage-backend=memory \ --task-execution-mode=sequential \ --embedded-worker ``` This runs the server and a built-in worker in a single process. No separate worker needed. **Web console:** Open [http://127.0.0.1:8080/](http://127.0.0.1:8080/) in your browser to view agents, systems, tasks, and the task trace. You can use it to inspect the pipeline and task status as you run the steps below. ### 2. Apply a Starter Blueprint ```bash go run ./cmd/orlojctl apply -f examples/blueprints/pipeline/ --run ``` This creates agents, an agent system (the pipeline graph), and a task in one command. ### 3. Verify Execution ```bash go run ./cmd/orlojctl get task bp-pipeline-task ``` Expected result: task reaches `Succeeded`. ### Scaling to Production When you are ready to run multi-worker, distributed workloads, switch to **message-driven** mode. This unlocks parallel fan-out, durable message delivery, and horizontal scaling. Start the server: ```bash go run ./cmd/orlojd \ --storage-backend=postgres \ --task-execution-mode=message-driven \ --agent-message-bus-backend=nats-jetstream ``` Start one or more workers: ```bash go run ./cmd/orlojworker \ --storage-backend=postgres \ --task-execution-mode=message-driven \ --agent-message-bus-backend=nats-jetstream \ --agent-message-consume ``` See [Execution and Messaging](../concepts/execution-model.md) for details on the message lifecycle, ownership guarantees, and retry behavior. ### Try with a Real Model The quickstart above uses the mock gateway, which returns placeholder output. To use a real provider (OpenAI, Anthropic, Ollama, etc.), create a **Secret** resource for your API key, create a **ModelEndpoint** that references it via `auth.secretRef`, and point your agents at that endpoint with `model_ref`. See [Configure Model Routing](../guides/configure-model-routing.md) for the full steps. ### Next Steps * [Starter Blueprints](../guides/starter-blueprints.md) -- pipeline, hierarchical, and swarm-loop topologies * [Configuration](../operations/configuration.md) -- all flags and environment variables ## Deploy and Operate This section covers deploying Orloj to any environment and running it in production. ### Deployment Targets Choose a deployment path based on your environment: | Target | Best For | Persistence | Process Management | Scope | | ---------- | ----------------------------------------------- | ------------------------------- | -------------------------- | ----------------------------- | | Local | Development and rapid iteration | Optional (`memory` or Postgres) | terminal or Docker Compose | single operator machine | | VPS | Single-node production-style self-hosting | Postgres volume | systemd + Docker Compose | small internal workloads | | Kubernetes | Cluster-based operations and lifecycle controls | PVC-backed Postgres | Kubernetes deployments | platform-managed environments | **Deployment runbooks:** 1. [Local Deployment](./local.md) 2. [VPS Deployment (Compose + systemd)](./vps.md) 3. [Kubernetes Deployment (Helm + Manifest Fallback)](./kubernetes.md) 4. [Kubernetes CRD Operator](./kubernetes-operator.md) -- optional GitOps-ready operator that syncs Orloj resources as real K8s CRDs 5. [Remote CLI and API access](./remote-cli-access.md) -- tokens, `orlojctl` profiles, and `config.json` after you expose the control plane ### Hosted Stack, Local CLI When the control plane runs in Compose, Kubernetes, or GHCR images, install **`orlojctl` alone** on the machine you use to operate the cluster: download the `orlojctl_*` archive for your OS and arch from [GitHub Releases](https://github.com/OrlojHQ/orloj/releases) (see [Install: CLI only for hosted deployments](../getting-started/install.md#cli-only-for-hosted-deployments)). Then follow [Remote CLI and API access](./remote-cli-access.md) for `--server`, tokens, and optional profiles. ### Day-to-Day Operations * [Configuration](../operations/configuration.md) -- flags, environment variables, and runtime tuning * [Runbook](../operations/runbook.md) -- startup, verification, and incident response * [Security and Isolation](../operations/security.md) -- secrets, auth, and network hardening * [Upgrades and Rollbacks](../operations/upgrades.md) -- version migration procedures * [Troubleshooting](../operations/troubleshooting.md) -- common issues and debugging ### Observability * [Observability](../operations/observability.md) -- OpenTelemetry tracing, Prometheus metrics, structured logging * [Monitoring and Alerts](../operations/monitoring-alerts.md) -- alerting rules and dashboards * [Backup and Restore](../operations/backup-restore.md) -- data protection procedures ### Security Defaults * Rotate default secrets before non-local use. * Restrict network exposure to required interfaces. * Keep API auth strategy explicit for each target. * After deployment, configure [remote CLI access](./remote-cli-access.md) (API tokens, env vars, optional `orlojctl config` profiles). ## Kubernetes CRD Operator ### Purpose The CRD sync operator is an **optional** component that makes Orloj resources (Agents, AgentSystems, Tools, etc.) real Kubernetes Custom Resource Definitions. When deployed, you can manage Orloj configuration with `kubectl apply`, store manifests in Git, and let Argo CD or Flux reconcile them — the operator watches CRD objects and syncs them into the Orloj Postgres store automatically. Without the operator, Orloj works exactly as before: you create resources via `orlojctl apply`, the REST API, or the web console. The operator adds an alternative input path; it does not replace anything. ### When Do You Need It? | Scenario | Operator needed? | | ---------------------------------------------------- | ---------------- | | Getting started / local dev | No | | Small team, `orlojctl apply` in CI | No | | GitOps (Argo CD, Flux, Crossplane) | **Yes** | | Multi-team platform with RBAC on manifests | **Yes** | | Unified `kubectl` workflow for all cluster resources | **Yes** | | You only use the web console | No | ### How It Works ``` ┌──────────────┐ watch ┌───────────────────┐ upsert ┌──────────────┐ │ K8s CRDs │──────────► │ orloj-operator │────────────►│ Postgres │ │ (etcd) │◄────────── │ (controller- │ │ (orlojd │ │ │ status │ runtime) │ │ store) │ └──────────────┘ └───────────────────┘ └──────────────┘ ▲ │ │ kubectl apply │ serves │ ▼ Git repo / CI orlojd API + UI ``` 1. You `kubectl apply` an Orloj CRD (e.g. `Agent`). 2. The operator's reconciler converts the CRD spec to an Orloj resource and upserts it into Postgres. 3. The resource is now visible in the REST API, web console, and `orlojctl get`. 4. On delete, the operator's finalizer (`orloj.dev/sync`) removes the resource from the store. 5. A periodic status writer syncs `status.phase`, `observedGeneration`, and `lastSyncedAt` back to the CRD subresource. Every synced resource gets the annotation `orloj.dev/managed-by: crd-sync`, which `orlojd` uses for conflict detection (see [CRD conflict policy](#crd-conflict-policy)). ### Install #### Enable via Helm ```bash helm upgrade --install orloj oci://ghcr.io/orlojhq/charts/orloj \ --namespace orloj \ --reuse-values \ --set operator.enabled=true \ --set operator.installCRDs=true ``` This deploys: * CRD manifests for all supported resource kinds * The `orloj-operator` Deployment (connects to the same Postgres as `orlojd`) * A ServiceAccount with RBAC for CRD watch/list/get/update/patch and status updates * A PodDisruptionBudget (`minAvailable: 1`) * Optional ServiceMonitor for Prometheus scraping #### Verify ```bash # CRDs registered kubectl get crd agents.orloj.dev # Operator running kubectl -n orloj rollout status deploy/orloj-operator # List Orloj resources (empty initially) kubectl get agents,agentsystems,tools,mcpservers,modelendpoints,memories,agentpolicies,secrets.orloj.dev ``` ### Helm Values Reference All operator values live under the `operator.*` key: | Value | Default | Description | | ------------------------------------ | ------------------------ | -------------------------------------------------------------- | | `operator.enabled` | `false` | Deploy the CRD sync operator. | | `operator.installCRDs` | `true` | Install CRD manifests with the chart. | | `operator.replicaCount` | `1` | Operator replicas. Leader election ensures only one is active. | | `operator.image.repository` | `orlojhq/orloj-operator` | Operator container image. | | `operator.image.tag` | `""` (appVersion) | Image tag override. | | `operator.resources.requests.cpu` | `100m` | CPU request. | | `operator.resources.requests.memory` | `128Mi` | Memory request. | | `operator.resources.limits.cpu` | `500m` | CPU limit. | | `operator.resources.limits.memory` | `256Mi` | Memory limit. | | `operator.statusSyncInterval` | `5s` | Interval for syncing CRD status back to Kubernetes. | | `operator.leaderElection` | `true` | Enable leader election for HA. | | `operator.healthzPort` | `8081` | Health probe port. | | `operator.metricsPort` | `8080` | Prometheus metrics port. | | `operator.serviceMonitor.enabled` | `false` | Create a Prometheus ServiceMonitor. | | `operator.serviceMonitor.interval` | `30s` | Scrape interval. | | `operator.serviceMonitor.labels` | `{}` | Extra labels on the ServiceMonitor. | | `operator.pdb.enabled` | `true` | Create a PodDisruptionBudget. | | `operator.pdb.minAvailable` | `1` | Minimum available replicas. | The operator also requires access to the same Postgres database as `orlojd`. The chart wires `--postgres-dsn` and `--secret-encryption-key` from the shared release secret automatically. ### CRD Conflict Policy When the operator is running, resources it creates are annotated `orloj.dev/managed-by: crd-sync`. If someone then edits that same resource through the REST API, the change will be overwritten on the next operator reconcile. The `--crd-conflict-policy` flag on `orlojd` controls how the API server handles this: | Mode | Behavior | | ---------------- | --------------------------------------------------------------------------------------------------- | | `off` | No conflict detection. REST writes silently proceed. | | `warn` (default) | REST writes succeed but `orlojd` logs a warning and sets the `X-Orloj-CRD-Managed` response header. | | `reject` | REST writes to CRD-managed resources return `409 Conflict`. | Set the policy via Helm: ```yaml crdConflictPolicy: reject ``` Or via the environment variable `ORLOJ_CRD_CONFLICT_POLICY`. ### Namespace Mapping By default, the operator maps the Kubernetes namespace of a CRD object directly to the Orloj namespace in the store. A resource in K8s namespace `team-a` is stored in Orloj namespace `team-a`. For setups where K8s namespaces don't align with Orloj namespaces (e.g., a single `gitops` namespace holds all manifests but they target different Orloj namespaces), use the `orloj.dev/target-namespace` annotation: ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: summarizer namespace: gitops-manifests # K8s governance namespace annotations: orloj.dev/target-namespace: production # Orloj store namespace spec: model_ref: gpt-4o prompt: "You are a concise summarizer." ``` | Annotation set? | Orloj namespace used | | --------------- | ------------------------------------ | | No | `metadata.namespace` (K8s namespace) | | Yes | `orloj.dev/target-namespace` value | This is useful when: * Your K8s namespace structure is governed by platform policy (e.g., one namespace per ArgoCD Application) * You have many Orloj namespaces but don't want to create a matching K8s namespace for each * You want to decouple K8s RBAC boundaries from Orloj resource organization If you don't set the annotation, behavior is unchanged — K8s namespace = Orloj namespace. ### First Use Walkthrough #### 1. Enable the operator ```bash helm upgrade --install orloj oci://ghcr.io/orlojhq/charts/orloj \ --namespace orloj --reuse-values \ --set operator.enabled=true \ --set operator.installCRDs=true ``` #### 2. Apply an Agent CRD ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: summarizer namespace: default spec: prompt: You are a concise summarizer. model_ref: gpt-4o limits: max_steps: 5 ``` ```bash kubectl apply -f summarizer-agent.yaml ``` #### 3. Verify sync ```bash # CRD status shows "Synced" kubectl get agent summarizer -o jsonpath='{.status.phase}' # Resource visible via orlojctl orlojctl get agent summarizer ``` #### 4. See in the web console Open the Orloj web console — the agent appears in the Agents list with a "CRD Managed" badge. ### GitOps Setup #### Argo CD ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: orloj-resources namespace: argocd spec: project: default source: repoURL: https://github.com/your-org/orloj-config targetRevision: main path: manifests/orloj destination: server: https://kubernetes.default.svc namespace: default syncPolicy: automated: { prune: true, selfHeal: true } ``` Place Orloj CRD manifests (Agents, Tools, AgentSystems, etc.) under `manifests/orloj/` in your config repo. Argo CD applies them; the operator syncs them into the store. #### Flux ```yaml apiVersion: source.toolkit.fluxcd.io/v1 kind: GitRepository metadata: name: orloj-config namespace: flux-system spec: url: https://github.com/your-org/orloj-config ref: branch: main interval: 1m --- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: orloj-resources namespace: flux-system spec: sourceRef: kind: GitRepository name: orloj-config path: ./manifests/orloj prune: true interval: 5m ``` ### Migrating Existing Resources to CRDs If you already have resources created via `orlojctl apply` or the REST API, you can adopt them as CRDs: 1. Export the resource: ```bash orlojctl get agent my-agent -o yaml > my-agent.yaml ``` 2. Add the CRD `apiVersion` and strip runtime status fields: ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: my-agent namespace: default spec: # ... your existing spec ``` 3. Apply: ```bash kubectl apply -f my-agent.yaml ``` The operator upserts the resource. Because the name matches, the existing store entry is updated and adopted — it gains the `orloj.dev/managed-by: crd-sync` annotation. From this point, manage it via `kubectl` and Git. ### Secrets and GitOps For secrets that tools and MCP servers need at runtime, you have two options: **Option A: Orloj Secret CRD** — store secrets directly in the Orloj store via the `Secret` CRD. Simple but means plaintext values in your manifests (not safe to commit to Git without additional encryption). **Option B: K8s-native Secrets + `secretRef`** (recommended for GitOps) — use an external secrets operator (Bitnami Sealed Secrets, External Secrets Operator, HashiCorp Vault) to manage K8s-native Secrets, then reference them from Orloj resources: ```yaml apiVersion: orloj.dev/v1 kind: Tool metadata: name: my-api-tool spec: type: http endpoint: "https://api.example.com" auth: secretRef: my-k8s-secret # K8s-native Secret created by Sealed Secrets ``` The `KubernetesSecretResolver` reads K8s-native Secrets at runtime. This keeps secrets out of Git, leverages K8s RBAC and audit logging, and integrates with existing secret management tooling. ### Phase 1 Scope The operator currently syncs these 8 resource kinds: | CRD | API Group | Store Kind | | -------------------------- | -------------- | ------------- | | `agents.orloj.dev` | `orloj.dev/v1` | Agent | | `agentsystems.orloj.dev` | `orloj.dev/v1` | AgentSystem | | `tools.orloj.dev` | `orloj.dev/v1` | Tool | | `mcpservers.orloj.dev` | `orloj.dev/v1` | McpServer | | `modelendpoints.orloj.dev` | `orloj.dev/v1` | ModelEndpoint | | `memories.orloj.dev` | `orloj.dev/v1` | Memory | | `agentpolicies.orloj.dev` | `orloj.dev/v1` | AgentPolicy | | `secrets.orloj.dev` | `orloj.dev/v1` | Secret | Runtime resources (Tasks, TaskSchedules, TaskWebhooks, Workers) are not CRDs — they are created through the API at runtime or via `orlojctl`. ### Relationship to Agent K8s Execution The CRD operator and the Kubernetes agent/tool execution backends are **orthogonal features** that stack: | Feature | What it does | Helm values | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | | **CRD Operator** | Syncs resource definitions (Agents, Tools, ...) from K8s CRDs into the Orloj store. Manages the *configuration* plane. | `operator.enabled` | | **Tool K8s Isolation** | Runs individual tool invocations as ephemeral K8s Jobs. Manages the *execution* plane for tools. | `toolIsolation.kubernetes.enabled` | | **Agent K8s Execution** | Runs entire agent steps as ephemeral K8s Jobs. Manages the *execution* plane for agents. | `agentExecution.kubernetes.enabled` | You can use any combination: CRDs without K8s execution, K8s execution without CRDs, or all three together. ### Related Docs * [Kubernetes Deployment (Helm)](./kubernetes.md) * [kubectl vs orlojctl](../guides/kubectl-vs-orlojctl.md) * [Architecture](../concepts/architecture.md) * [Server Flags — `--crd-conflict-policy`](../reference/server-flags.md) * [Troubleshooting — Operator](../operations/troubleshooting.md#operator) ## Kubernetes Deployment (Helm + Manifest Fallback) ### Purpose Deploy Orloj on Kubernetes with a Helm chart (recommended) or with raw manifests (fallback). ### Prerequisites * Kubernetes cluster access (`kubectl` context configured) * Helm 3 (`helm`) * `curl`, `jq` for verification (and `go` if running `orlojctl` from source) The release workflow publishes `orlojd` and `orlojworker` container images plus the Helm chart to GHCR — you do not need to build anything yourself unless you're deploying from a local checkout. ### Install #### 1. Install with Helm (Recommended) The chart is published as an OCI artifact on every `v*` release: ```bash helm upgrade --install orloj oci://ghcr.io/orlojhq/charts/orloj \ --version 0.14.2 \ --namespace orloj \ --create-namespace \ --set postgresql.auth.password='' \ --set secretEncryptionKey="$(openssl rand -hex 32)" \ --set auth.mode=native \ --set auth.setupToken="$(openssl rand -hex 32)" ``` Notes: * The chart defaults `image.registry`, `image.server.repository`, and `image.worker.repository` at the published GHCR images, so you do not need to set them. * `secretEncryptionKey` is a 256-bit AES key used to encrypt provider API keys at rest in Postgres. Generate with `openssl rand -hex 32` and store it as you would any other root secret. * `auth.mode=native` requires `auth.setupToken` for first-user bootstrap. See [Operations > Security](../operations/security.md). * Model provider API keys (Anthropic, OpenAI, Bedrock, etc.) are **not** chart values — they are encrypted `Secret` resources you create via `orlojctl` after the control plane is up, and `ModelEndpoint` resources reference them by name. See [ModelEndpoint](../concepts/tools/model-endpoint.md). To inspect effective values: ```bash helm get values orloj --namespace orloj ``` ##### Install from a source checkout If you've cloned the repo and want to deploy a development build, you can install from the chart directory directly. Subchart deps must be resolved first: ```bash helm dependency update charts/orloj helm upgrade --install orloj ./charts/orloj \ --namespace orloj \ --create-namespace \ --set postgresql.auth.password='' \ --set secretEncryptionKey="$(openssl rand -hex 32)" \ --set auth.mode=native \ --set auth.setupToken="$(openssl rand -hex 32)" ``` To pin custom image tags (for example, a locally-built image pushed to your own registry): ```bash --set image.registry=ghcr.io/ \ --set image.server.repository=/orloj-orlojd \ --set image.server.tag= \ --set image.worker.repository=/orloj-orlojworker \ --set image.worker.tag= ``` ##### GitOps (ArgoCD, Flux) ArgoCD `Application` example pointing at the OCI chart: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: orloj namespace: argocd spec: project: default source: repoURL: ghcr.io/orlojhq/charts chart: orloj targetRevision: 0.14.2 helm: valueFiles: - values.yaml destination: server: https://kubernetes.default.svc namespace: orloj syncPolicy: automated: { prune: true, selfHeal: true } syncOptions: [ CreateNamespace=true ] ``` #### 2. Manifest Fallback (No Helm) If you cannot use Helm, apply the baseline manifest set: 1. Edit `docs/deploy/kubernetes/orloj-stack.yaml` image references and rotate the baseline secrets (Postgres password, secret encryption key, setup token). 2. Apply manifests: ```bash kubectl apply -f docs/deploy/kubernetes/orloj-stack.yaml ``` ### Verify Wait for rollouts. The Helm release names follow the `-` convention; with `helm install orloj ...` you get: ```bash kubectl -n orloj rollout status statefulset/orloj-postgresql kubectl -n orloj rollout status statefulset/orloj-nats kubectl -n orloj rollout status deploy/orloj-server kubectl -n orloj rollout status deploy/orloj-worker ``` If you used the manifest fallback, the names are unprefixed: ```bash kubectl -n orloj rollout status deploy/postgres kubectl -n orloj rollout status deploy/nats kubectl -n orloj rollout status deploy/orlojd kubectl -n orloj rollout status deploy/orlojworker ``` Port-forward the API service: ```bash # Helm install kubectl -n orloj port-forward svc/orloj-server 8080:8080 # Manifest fallback kubectl -n orloj port-forward svc/orlojd 8080:8080 ``` In another terminal: ```bash curl -s http://127.0.0.1:8080/healthz | jq . orlojctl --server http://127.0.0.1:8080 get workers orlojctl --server http://127.0.0.1:8080 apply -f examples/blueprints/pipeline/ --run orlojctl --server http://127.0.0.1:8080 get task bp-pipeline-task ``` Done means: * all rollouts are successful. * API service is reachable through port-forward. * at least one worker is `Ready`. * sample task reaches `Succeeded`. ### Operate Scale workers (Helm install): ```bash kubectl -n orloj scale deploy/orloj-worker --replicas=3 kubectl -n orloj rollout status deploy/orloj-worker ``` For long-term scaling, prefer the HPA values: ```yaml worker: autoscaling: enabled: true minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 70 ``` Restart control plane: ```bash kubectl -n orloj rollout restart deploy/orloj-server kubectl -n orloj rollout status deploy/orloj-server ``` View logs: ```bash kubectl -n orloj logs deploy/orloj-server --tail=200 kubectl -n orloj logs deploy/orloj-worker --tail=200 ``` Upgrade chart release: ```bash helm upgrade orloj oci://ghcr.io/orlojhq/charts/orloj \ --version --namespace orloj --reuse-values ``` Rollback: ```bash helm rollback orloj --namespace orloj ``` ### Troubleshoot * pods in `ImagePullBackOff`: verify image names/tags and registry access. * workers not processing: verify `ORLOJ_AGENT_MESSAGE_CONSUME=true` and message-bus env values. * tasks not created: verify the API endpoint is reachable from `orlojctl`. ### Tool Isolation: Kubernetes Backend When `toolIsolation.kubernetes.enabled=true`, Orloj runs tool invocations with `isolation_mode: kubernetes` as ephemeral Kubernetes Jobs in the cluster. This eliminates the need for a Docker socket on worker nodes. #### RBAC Requirements The Helm chart automatically creates a Role (and RoleBinding) for the worker ServiceAccount with the following permissions: | API Group | Resource | Verbs | | --------- | ---------- | ------------------------------------------ | | `batch` | `jobs` | `create`, `get`, `list`, `watch`, `delete` | | (core) | `pods` | `get`, `list` | | (core) | `pods/log` | `get` | | (core) | `secrets` | `get` | The Role is scoped to the namespace configured by `toolIsolation.kubernetes.namespace` (defaults to the release namespace). #### Helm Values Configure the Kubernetes tool isolation backend under `toolIsolation.kubernetes`: ```yaml toolIsolation: kubernetes: enabled: false # Set to true to enable namespace: "" # Namespace for tool Jobs (default: release namespace) serviceAccount: "" # Service account for tool Pods (default: worker SA) jobTTLSeconds: 300 # TTL seconds after Job finishes (automatic cleanup) defaultImage: "curlimages/curl:8.8.0" # Fallback image for HTTP tools ``` When enabled, the chart sets `ORLOJ_TOOL_K8S_ENABLED=true` plus related env vars on both the `orlojd` server and `orlojworker` deployments. #### Coexistence with Container Backend Both `container` and `kubernetes` isolation backends can be active simultaneously. Each tool's `spec.runtime.isolation_mode` selects which backend handles that tool: * `isolation_mode: container` — runs via `docker run` on the worker host * `isolation_mode: kubernetes` — runs as a Kubernetes Job in the cluster This allows gradual migration from Docker-based isolation to Kubernetes-native execution. ### Agent Execution: Kubernetes Backend When `agentExecution.kubernetes.enabled=true`, Orloj runs each agent in a multi-agent task as an ephemeral Kubernetes Job instead of executing it in-process on the worker. This isolates agent execution at the pod level and allows independent scaling of agent workloads. Agents whose tools require Docker (container isolation mode or stdio MCP servers with a container image) automatically fall back to in-process execution. #### RBAC Requirements The Helm chart creates a Role (and RoleBindings for both the worker and server ServiceAccounts) with the following permissions: | API Group | Resource | Verbs | | --------- | ---------- | ------------------------------------------ | | `batch` | `jobs` | `create`, `get`, `list`, `watch`, `delete` | | (core) | `pods` | `get`, `list` | | (core) | `pods/log` | `get` | The Role is scoped to the namespace configured by `agentExecution.kubernetes.namespace` (defaults to the release namespace). #### Helm Values ```yaml agentExecution: kubernetes: enabled: false # Set to true to enable namespace: "" # Namespace for agent Jobs (default: release namespace) serviceAccount: "" # Service account for agent Pods image: "" # Container image (default: worker image) jobTTLSeconds: 600 # TTL seconds after Job finishes defaultMemory: "512Mi" # Default memory limit for agent Pods defaultCPU: "500m" # Default CPU limit for agent Pods ``` #### How It Works 1. The orchestrator (worker or server) checks whether the agent can run as a K8s Job (`CanRunAsJob`). 2. If eligible, it writes agent input to the task's status in Postgres and creates a K8s Job running the worker image with `--single-agent` mode. 3. The agent pod reads its input from Postgres, executes the agent, and writes the result back. 4. The orchestrator watches the Job for completion and reads the result. 5. If the orchestrator crashes and restarts, it detects the existing Job by its deterministic name and resumes watching. #### Crash Recovery Agent Jobs use deterministic names based on the task, agent, and attempt number. If the orchestrator pod restarts mid-execution, it detects the existing Job and either reads its result (if complete) or resumes watching (if still running). ### A2A Protocol To configure public A2A Agent Card URLs in a Helm deployment, set the A2A public base URL. Individual AgentSystems are exposed with `spec.a2a.enabled: true`. ```bash helm upgrade orloj ./charts/orloj --namespace orloj --reuse-values \ --set a2a.publicBaseURL=https://orloj.example.com ``` See the [Chart README](../../../charts/orloj/README.md#a2a-protocol) for the full list of `a2a.*` values and their defaults. ### CRD Sync Operator (Optional) The Orloj CRD operator makes Orloj resources (Agents, Tools, AgentSystems, etc.) real Kubernetes Custom Resource Definitions. When enabled, you can manage configuration with `kubectl apply` and integrate with GitOps tools like Argo CD and Flux. ```bash helm upgrade --install orloj oci://ghcr.io/orlojhq/charts/orloj \ --namespace orloj --reuse-values \ --set operator.enabled=true \ --set operator.installCRDs=true ``` The operator is independent of tool isolation and agent execution backends — it manages the *configuration* plane (resource definitions), not the *execution* plane (how tools and agents run). You can use any combination. See [Kubernetes CRD Operator](./kubernetes-operator.md) for full documentation, values reference, GitOps examples, and migration guide. ### Security Defaults * This baseline is not HA — `server.replicaCount` defaults to 1. Multi-replica `orlojd` requires leader election (see roadmap). * Rotate secrets before non-test use: * `postgresql.auth.password` (or `postgresql.auth.existingSecret` for a pre-sealed value). * `secretEncryptionKey` — losing this makes every encrypted Orloj `Secret` unrecoverable. * `auth.setupToken` — single-use bootstrap; rotate after the first admin account is created. * `auth.apiToken` — set this only if you also need a static bearer for CLI/automation; otherwise rely on user-issued tokens minted through the native auth flow. * `ORLOJ_AUTH_MODE` defaults to `native` (the chart's `auth.mode` value). `auth.mode=off` disables authentication entirely and is intended only for local development. * Restrict namespace and service exposure based on cluster policy. The chart's `server.ingress` is opt-in and emits a `networking.k8s.io/v1 Ingress`; for Gateway API environments, leave `server.ingress.enabled=false` and ship an `HTTPRoute` alongside the release. ### Related Docs * [Deployment Assets (`docs/deploy/kubernetes`)](../../deploy/kubernetes/README.md) * [Configuration](../operations/configuration.md) * [Operations Runbook](../operations/runbook.md) * [ModelEndpoint](../concepts/tools/model-endpoint.md) * [Security](../operations/security.md) ## Local Deployment ### Purpose Run Orloj locally for development and deterministic feature validation. ### Prerequisites * Go `1.25+` * Docker * `curl` and `jq` * repository checked out locally ### Install #### Option A: Run from Source Terminal 1: ```bash go run ./cmd/orlojd \ --storage-backend=memory \ --task-execution-mode=message-driven \ --agent-message-bus-backend=memory ``` Terminal 2: ```bash go run ./cmd/orlojworker \ --storage-backend=memory \ --task-execution-mode=message-driven \ --agent-message-bus-backend=memory \ --agent-message-consume ``` #### Option B: Docker Compose Stack ```bash docker compose up --build -d ``` Uses [`docker-compose.yml`](../../../docker-compose.yml). ### Verify Health and worker readiness: ```bash curl -s http://127.0.0.1:8080/healthz | jq . go run ./cmd/orlojctl get workers ``` Sample task execution: ```bash go run ./cmd/orlojctl apply -f examples/blueprints/pipeline/ --run go run ./cmd/orlojctl get task bp-pipeline-task ``` Done means: * `/healthz` returns healthy status. * at least one worker is `Ready`. * `bp-pipeline-task` reaches `Succeeded`. ### Operate Source-mode stop: terminate both processes. Compose-mode stop: ```bash docker compose down ``` Compose logs: ```bash docker compose logs -f orlojd orlojworker-a orlojworker-b ``` ### Troubleshoot * If workers do not appear, check worker process logs for DSN/backend mismatch. * If tasks remain pending, verify execution mode and message-consumer flags match. * If `orlojctl` calls fail, confirm `--server` points to the active API address. ### Security Defaults For local-only use, API auth may remain disabled. Do not expose local ports publicly. ### Related Docs * [Quickstart](../getting-started/quickstart.md) * [Configuration](../operations/configuration.md) * [Troubleshooting](../operations/troubleshooting.md) ## Remote CLI and API access This guide is for **operators and users** who already have `orlojd` reachable on a network (self-hosted, VPS, Kubernetes, or internal URL) and need to call the API from **`orlojctl`**, scripts, or CI. It complements the [quickstart](../getting-started/quickstart.md), which focuses on a single-machine dev loop. For deeper security context (generation, rotation, threat model), see [Control plane API tokens](../operations/security.md#control-plane-api-tokens). ### Install `orlojctl` locally You need the CLI on **your** machine (or in CI), not inside the server container. The easiest path is Homebrew: ```bash brew tap OrlojHQ/orloj brew install orlojctl ``` Alternatively, download the standalone binary from [GitHub Releases](https://github.com/OrlojHQ/orloj/releases) (`orlojctl___`), verify with `checksums.txt`, extract, and add it to your `PATH`. Details and naming conventions are in [Install: CLI only for hosted deployments](../getting-started/install.md#cli-only-for-hosted-deployments). If you already cloned the repo with Go installed, `go run ./cmd/orlojctl` works the same way against a remote `--server`. ### API tokens (shared secret) Orloj **does not** issue API tokens from the web console. The **operator** generates a random string, configures **the same value** on the server and on every client that uses `Authorization: Bearer `. ```bash openssl rand -hex 32 ``` Store the value in your secrets manager or deployment environment—**not** in git. On the server, set **`orlojd --api-key=...`** or **`ORLOJ_API_TOKEN=...`** (or **`ORLOJ_API_TOKENS`** for multiple `name:token:role` entries; legacy `token:role` remains supported). See [Control plane API tokens](../operations/security.md#control-plane-api-tokens) for details. ### Server-side wiring Where you set `ORLOJ_API_TOKEN` depends on how you run `orlojd`: * **Docker Compose / systemd** — env var or secret in the service definition (e.g. [VPS deployment](./vps.md)). * **Kubernetes / Helm** — `runtimeSecret` or equivalent env injection (see [Kubernetes deployment](./kubernetes.md)). ### Client-side: environment and flags From any machine that should talk to the API: | Mechanism | Purpose | | ------------------------------ | ------------------------------------------------------- | | `ORLOJ_SERVER` | Default API base URL when `--server` is omitted | | `ORLOJCTL_SERVER` | Same default; **takes precedence** over `ORLOJ_SERVER` | | `ORLOJ_API_TOKEN` | Bearer token | | `ORLOJCTL_API_TOKEN` | Same token; checked before `ORLOJ_API_TOKEN` by the CLI | | `orlojctl --api-token ` | Overrides env for that process | | `orlojctl --server ` | Overrides per-command default server | ### Precedence **Token** (first match wins): 1. `orlojctl --api-token ...` 2. `ORLOJCTL_API_TOKEN` 3. `ORLOJ_API_TOKEN` 4. Active profile: `token` field, else value of the env var named by `token_env` **Default `--server`** when the flag is omitted (first match wins): 1. `ORLOJCTL_SERVER` 2. `ORLOJ_SERVER` 3. Active profile `server` 4. `http://127.0.0.1:8080` Explicit `--server` on a subcommand always overrides the default above. ### `orlojctl config` and `config.json` Named **profiles** are stored as JSON: * **Path:** `orlojctl config path` (typically `~/.config/orlojctl/config.json` on Unix). * **Permissions:** file is written with mode `0600` when created or updated. **The file does not exist until the first successful save** (for example `orlojctl config set-profile ...`). Until then, only environment variables and flags apply—if you open the path early, an empty or missing file is normal. Commands: ```bash orlojctl config path orlojctl config set-profile production --server https://orloj.example.com --token-env ORLOJ_PROD_TOKEN orlojctl config use production orlojctl config get ``` `set-profile` creates or updates a profile. The first profile you create also becomes **`current_profile`** if none was set. Prefer **`--token-env`** so the token is not stored in the JSON file. #### Example `config.json` Shape matches the CLI (field names are JSON): ```json { "current_profile": "production", "profiles": { "local": { "server": "http://127.0.0.1:8080" }, "production": { "server": "https://orloj.example.com", "token_env": "ORLOJ_PROD_TOKEN" } } } ``` You can hand-edit this file if you prefer; invalid JSON will cause `orlojctl` to error on load. ### `orlojctl auth login` (native mode) When the server runs **`--auth-mode=native`**, you can authenticate the CLI directly with your username and password instead of manually configuring tokens: ```bash orlojctl config set-profile production --profile-server https://orloj.example.com orlojctl config use production orlojctl auth login # Username: admin # Password: ******** # logged in as admin (role=admin) on https://orloj.example.com # token saved to profile "production" ``` This calls `POST /v1/auth/cli-token` on the server, which validates your credentials and mints a new bearer token. The token is automatically saved to the active profile in `config.json`. You can also pass credentials non-interactively (useful in CI): ```bash orlojctl auth login -u admin -p "$MY_PASSWORD" ``` After login, all subsequent commands use the saved bearer token. Run `orlojctl auth whoami` to verify. ### Local UI auth vs API tokens If you use **`--auth-mode=native`**, the web UI uses an **admin username/password** and **session cookies**. The CLI uses **bearer tokens**. You can obtain a CLI token in two ways: 1. **`orlojctl auth login`** — authenticates with your username/password and saves a token to the active profile (recommended). 2. **Manual token configuration** — an operator generates a token with `ORLOJ_API_TOKEN` / `--api-key` on the server, and you configure it with `--token-env` or `--token` in a profile. See [Control plane API tokens](../operations/security.md#control-plane-api-tokens) and [CLI reference: orlojctl](../reference/cli.md#orlojctl). ### Related docs * [CLI reference](../reference/cli.md) — full command list and flags * [Configuration](../operations/configuration.md) — `orlojd` / `orlojworker` environment variables * [VPS deployment](./vps.md) — single-node Compose + systemd * [Kubernetes deployment](./kubernetes.md) — Helm and manifests ## VPS Deployment (Compose + systemd) ### Purpose Run Orloj on a single VPS with Docker Compose managed by systemd for automatic restart and reboot recovery. ### Prerequisites * Linux VPS with systemd (for example Ubuntu 22.04+) * Docker Engine with Compose plugin * `git`, `curl`, and `jq` * sudo access ### Install #### 1. Place Repository on Host ```bash sudo mkdir -p /opt/orloj sudo chown "$USER":"$USER" /opt/orloj git clone https://github.com/OrlojHQ/orloj.git /opt/orloj cd /opt/orloj ``` #### 2. Configure Runtime Variables ```bash cp docs/deploy/vps/.env.vps.example docs/deploy/vps/.env.vps ``` Edit `docs/deploy/vps/.env.vps` and rotate at minimum: * `POSTGRES_PASSWORD` * `ORLOJ_POSTGRES_DSN` password component #### 3. Validate Compose Config ```bash docker compose --env-file docs/deploy/vps/.env.vps -f docs/deploy/vps/docker-compose.vps.yml config ``` #### 4. Install systemd Unit ```bash sudo cp docs/deploy/vps/orloj-compose.service /etc/systemd/system/orloj.service sudo systemctl daemon-reload sudo systemctl enable --now orloj ``` ### Verify Service status: ```bash sudo systemctl status orloj --no-pager ``` Stack and health checks: ```bash docker compose --env-file docs/deploy/vps/.env.vps -f docs/deploy/vps/docker-compose.vps.yml ps curl -s http://127.0.0.1:8080/healthz | jq . go run ./cmd/orlojctl get workers ``` Sample task execution: ```bash go run ./cmd/orlojctl apply -f examples/blueprints/pipeline/ --run go run ./cmd/orlojctl get task bp-pipeline-task ``` Done means: * `orloj` systemd unit is active. * stack survives restart (`sudo systemctl restart orloj`). * health and worker checks pass. * sample task reaches `Succeeded`. ### Operate Restart stack: ```bash sudo systemctl restart orloj ``` Tail service logs: ```bash sudo journalctl -u orloj -f ``` Tail compose logs: ```bash docker compose --env-file docs/deploy/vps/.env.vps -f docs/deploy/vps/docker-compose.vps.yml logs -f ``` Upgrade flow: 1. `git pull` in `/opt/orloj`. 2. `sudo systemctl reload orloj`. 3. rerun verification checks. ### Troubleshoot * `docker compose ... config` fails: fix missing/invalid `.env.vps` values. * systemd start fails: verify docker binary path and service logs (`journalctl -u orloj`). * workers absent: verify `ORLOJ_AGENT_MESSAGE_CONSUME=true` and message-bus settings. ### Security Defaults * This is a single-node baseline, not HA. * Bind or firewall `8080` to trusted networks only. * API auth defaults to `ORLOJ_AUTH_MODE=native`; complete `/setup` on first boot. * Generate and rotate an API token (`openssl rand -hex 32`), set `ORLOJ_API_TOKEN` on the server, and reuse the same value for CLI/automation—see [Control plane API tokens](../operations/security.md#control-plane-api-tokens). ### Related Docs * [Deployment Assets (`docs/deploy/vps`)](../../deploy/vps/README.md) * [Operations Runbook](../operations/runbook.md) ## A2A Interoperability Orloj implements the [Agent-to-Agent (A2A) protocol](https://github.com/a2aproject/A2A) to enable cross-platform agent communication. Any A2A-compliant client can discover Orloj agents, submit tasks, and stream results -- and Orloj agents can call external A2A agents as tools. ### Why A2A? Agent runtimes are converging on a shared interoperability layer. A2A provides a standard for agent discovery (Agent Cards), task lifecycle (JSON-RPC), and streaming updates (SSE). Supporting A2A means Orloj agents can participate in multi-vendor agent ecosystems without custom integration code. ### Architecture A2A support in Orloj has two directions: **Inbound** -- external A2A clients call Orloj agents: ``` A2A Client → GET /.well-known/agent-card.json (discovery) → POST /a2a or /v1/agent-systems/{name}/a2a (JSON-RPC) → Orloj creates a Task, runs the AgentSystem → Returns A2A status updates and artifacts ``` **Outbound** -- Orloj agents call external A2A agents as tools: ``` Orloj Agent step → GovernedToolRuntime (policy, auth, retry) → A2AToolRuntime → GET {remote}/.well-known/agent-card.json (discover capabilities) → POST {remote}/a2a (tasks/send or tasks/sendSubscribe) → Map remote A2A result → tool result ``` Both directions reuse existing Orloj infrastructure: Tasks, auth, governance, webhooks, and SSE streaming. ### Agent Cards Every Orloj AgentSystem exposed via A2A publishes an **Agent Card** -- a JSON document describing the system's name, capabilities, skills, authentication requirements, and endpoint URL. Cards are generated automatically from the agent's metadata, tools, and runtime configuration: * **Name** and **description** come from the AgentSystem `metadata.name` and `metadata.annotations["orloj.dev/description"]`. * **Skills** are derived from the agent's attached tools, including each tool's `description` and `input_schema`. * **Capabilities** reflect runtime support: streaming (from task trace SSE), push notifications (from TaskWebhook support), and state transition history. * **Authentication** reflects the server's configured auth mode. Cards are served at: * `GET /.well-known/agent-card.json` -- only when exactly one AgentSystem is A2A-enabled * `GET /v1/agent-systems/{name}/.well-known/agent-card.json` -- specific AgentSystem ### State Mapping A2A defines its own task lifecycle states. Orloj maps them bidirectionally: | A2A State | Orloj Phase | Direction | | ---------------- | --------------------------------------------------- | --------- | | `submitted` | `Pending` | inbound | | `working` | `Running` | both | | `input-required` | `WaitingApproval` (with `a2a-input-required` label) | inbound | | `completed` | `Succeeded` | both | | `failed` | `Failed` | both | | `canceled` | `Failed` (with `orloj.dev/a2a-cancelled` label) | inbound | | `rejected` | `Failed` (with rejection reason) | inbound | Orloj task output is converted to A2A artifacts, and trace/watch events are converted to A2A streaming status updates for `tasks/sendSubscribe` calls. ### Inbound Routing Two routing modes are supported for inbound A2A requests: 1. **Per-system endpoints** (recommended): `POST /v1/agent-systems/{name}/a2a` -- the AgentSystem name is in the URL path. Each system's card `url` field points to its per-system endpoint. 2. **Shared endpoint**: `POST /a2a` -- the target is resolved from request params. When a single AgentSystem is A2A-enabled, this endpoint defaults to it. ### Outbound: A2A Tools External A2A agents are consumed as `type: a2a` tools. The tool spec includes the remote agent URL, optional protocol version, and streaming preference. At invocation time, the A2A tool runtime fetches the remote card, sends a JSON-RPC request, and maps the response back to Orloj's tool result format. Existing `spec.auth` profiles (bearer, API key, basic, OAuth2) work for authenticating with remote A2A agents. ### Security Model A2A endpoints participate in Orloj's existing security model: * **Agent Card discovery** endpoints are public (no auth required) -- cards contain only metadata, not secrets. * **JSON-RPC endpoints** enforce per-system auth via `spec.a2a.auth` (`"bearer"` by default, or `"public"` for unauthenticated access). All four methods (`tasks/send`, `tasks/get`, `tasks/cancel`, `tasks/sendSubscribe`) require a valid bearer token on `auth: bearer` systems. Native-auth browser sessions are not accepted for A2A invocation. * **Scoped `a2a` tokens** can invoke only the AgentSystems listed in their `a2a_agent_systems` scope and cannot read or mutate control-plane resources. * **Outbound calls** to remote agents use tool-level auth (`spec.auth`) and are subject to governance (AgentRole, ToolPermission, AgentPolicy). * **Private endpoint protection**: by default, outbound A2A calls to private/internal endpoints are blocked (`a2a.allowPrivateEndpoints` defaults to `false`). ### Configuration A2A is enabled per AgentSystem with `spec.a2a.enabled: true`. Server configuration controls advertised URLs and outbound registry behavior: | Setting | Description | | --------------------- | ------------------------------------------------- | | `a2a.publicBaseURL` | Public base URL for Agent Card endpoint URLs | | `a2a.protocolVersion` | A2A protocol version to advertise | | `a2a.remoteAgents[]` | Pre-configured remote A2A agents for the registry | | `a2a.cardCacheTTL` | Cache TTL for fetched remote Agent Cards | ### Related * [Guide: Expose Agents via A2A](../guides/a2a-expose-agents.md) * [Guide: Use Remote A2A Agents](../guides/a2a-remote-agents.md) * [Reference: A2A JSON-RPC](../reference/a2a-jsonrpc.md) * [Reference: Agent Card](../reference/resources/agent-card.md) * [Tool](./tools/tool.md) -- tool types including `a2a` ## Architecture Overview Orloj is organized into three layers: a **server** that manages resources and scheduling, **workers** that execute agent workflows, and a **governance layer** that enforces policies and permissions at runtime. ``` ┌─────────────────────────────────────────────────────┐ │ Server (orlojd) │ │ │ │ ┌──────────────┐ ┌────────────────┐ │ │ │ API Server │──►│ Resource Store │ │ │ │ (REST) │ │ mem/postgres │ │ │ └──────┬───────┘ └────────────────┘ │ │ │ │ │ ▼ │ │ ┌──────────────┐ ┌────────────────┐ │ │ │ Services │──►│ Task Scheduler │ │ │ └──────────────┘ └───────┬────────┘ │ └─────────────────────────────┼───────────────────────┘ │ assign tasks ▼ ┌─────────────────────────────────────────────────────┐ │ Workers (orlojworker) │ │ │ │ ┌──────────────┐ ┌───────────────┐ │ │ │ Task Worker │──►│ Model Gateway │ │ │ │ │ └───────────────┘ │ │ │ │ ┌───────────────┐ │ │ │ │──►│ Tool Runtime │ │ │ │ │ └───────────────┘ │ │ │ │ ┌───────────────┐ │ │ │ ◄──────┼───│ Message Bus │ │ │ │ │──►│ mem/nats-js │ │ │ └──────────────┘ └───────────────┘ │ │ ▲ │ └─────────┼───────────────────────────────────────────┘ │ enforced at runtime ┌─────────┴───────────────────────────────────────────┐ │ Governance │ │ │ │ ┌─────────────┐ ┌───────────┐ ┌────────────────┐ │ │ │ AgentPolicy │ │ AgentRole │ │ ToolPermission │ │ │ └─────────────┘ └───────────┘ └────────────────┘ │ └─────────────────────────────────────────────────────┘ ``` ### Server The server runs as `orlojd` and provides: **API Server** -- HTTP REST API for creating, reading, updating, and deleting Orloj resources. Supports watch endpoints for real-time event streaming, optimistic concurrency via `resourceVersion` / `If-Match`, and namespace scoping. Also serves the built-in web console at the root path (`/`) by default, configurable via `--ui-path` / `ORLOJ_UI_PATH`. **Resource Store** -- Pluggable storage backend for all resources. Two implementations: * `memory` -- in-memory store for local development and testing. Fast, no dependencies, data is lost on restart. * `postgres` -- PostgreSQL-backed store for production. Uses `FOR UPDATE SKIP LOCKED` for safe concurrent task claiming. **Services** -- Background processes for each resource type: Agent, AgentSystem, ModelEndpoint, Tool, Memory, AgentPolicy, Task, TaskScheduler, TaskSchedule, and Worker. Services drive resources toward their desired state and update status fields. **Task Scheduler** -- Matches pending tasks to available workers based on requirements (region, GPU, model), respects TaskSchedule cron triggers, and manages the assignment lifecycle. ### Workers Workers run as `orlojworker` and execute the actual agent workflows: **Task Worker** -- Claims assigned tasks via the lease mechanism, executes the agent graph step by step, and reports results back through status updates. Supports concurrent task execution up to `max_concurrent_tasks`. **Model Gateway** -- Routes model requests to the appropriate provider based on the agent's `model_ref` configuration. Handles provider-specific request formatting, authentication, and response parsing for OpenAI, Anthropic, Azure OpenAI, Ollama, and mock backends. **Tool Runtime** -- Executes tool invocations with the configured isolation backend (none, sandboxed, container, or WASM). Enforces timeouts, manages retries with capped exponential backoff and jitter, and normalizes responses into the standard tool contract envelope. **Message Bus** -- Handles agent-to-agent communication within a task's graph. Two implementations: * `memory` -- in-memory bus for local development. * `nats-jetstream` -- NATS JetStream for production with durable delivery guarantees. ### Governance Layer The governance layer is not a separate process -- it is enforced inline during worker execution: **AgentPolicy** -- Evaluated before each agent turn. Checks `allowed_models`, `blocked_tools`, and `max_tokens_per_run`. Policies can be scoped to specific systems/tasks or applied globally. **AgentRole + ToolPermission** -- Evaluated before each tool invocation. The worker collects the agent's permissions from all bound roles and checks them against the tool's ToolPermission requirements. Unauthorized calls return `tool_permission_denied`. All governance decisions are deterministic and fail-closed. Denied actions produce structured errors that flow into task trace and history for auditability. ### CRD Operator (Optional) The **CRD sync operator** (`orloj-operator`) is an optional component that provides an alternative input path into the resource store. Instead of going through the REST API, resources can be defined as Kubernetes Custom Resource Definitions and synced into Postgres by the operator. ``` kubectl apply ──► K8s CRDs ──► orloj-operator ──► Postgres store orlojctl apply ──► REST API ──► orlojd ──────────► Postgres store ``` Both paths write to the same store and produce identical runtime behavior. The operator enables GitOps workflows (Argo CD, Flux) and `kubectl`-native management for teams that prefer Kubernetes-style resource definitions. Resources synced by the operator are annotated `orloj.dev/managed-by: crd-sync`; the `--crd-conflict-policy` flag on `orlojd` controls whether REST API writes to CRD-managed resources are warned or rejected. The operator is not required for any Orloj functionality — it is purely an integration convenience. See [Kubernetes CRD Operator](../deploy/kubernetes-operator.md) for deployment and configuration. ### A2A Integration Orloj supports the [Agent-to-Agent (A2A) protocol](./a2a-interoperability.md) as an integration point for cross-system agent communication. When enabled, the server publishes Agent Cards describing local agents and exposes JSON-RPC 2.0 endpoints for inbound task delegation. Outbound A2A calls are handled by the `a2a` tool type, allowing local agents to delegate work to remote A2A-compatible agents. The A2A layer sits alongside the existing Tool Runtime and uses the same SSRF protection, auth enforcement, and governed runtime pipeline. ### Execution Modes Orloj supports two execution modes. Start with sequential for development, then graduate to message-driven for production. | Mode | How it works | When to use | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | | `sequential` | The server drives execution directly in a single process. Simpler, lower latency, easy to debug. | Getting started, development, single-agent systems | | `message-driven` | Workers consume from the message bus. Agents hand off via durable queued messages. Enables parallel fan-out and horizontal scaling. | Production, multi-agent systems, distributed workloads | **Sequential** is the default and requires no external dependencies. Use `--embedded-worker` to run everything in one process. **Message-driven** requires `--task-execution-mode=message-driven` and a message bus backend (`memory` for local testing, `nats-jetstream` for production). This mode provides lease-based ownership, idempotent replay, and dead-letter handling. ### Reliability Characteristics Orloj's runtime provides several reliability guarantees: * **Lease-based task ownership** -- Workers hold time-bounded leases on tasks. If a worker crashes, the lease expires and another worker can safely take over. * **Owner-only message execution** -- Only the worker that holds the task lease can process messages for that task, preventing duplicate execution. * **Idempotency tracking** -- Message idempotency keys prevent duplicate processing during replay and crash recovery. * **Capped exponential retry with jitter** -- Both task-level and message-level retries use bounded backoff with configurable jitter to avoid thundering herds. * **Dead-letter transitions** -- Messages and tasks that exhaust all retries move to a terminal `DeadLetter` phase for manual investigation rather than being silently dropped. ### Related Docs * [Execution and Messaging](./execution-model.md) * [Agents](./agents/agent.md) * [Tasks](./tasks/task.md) * [Tools](./tools/tool.md) * [Governance](./governance/) * [Runbook](../operations/runbook.md) * [Configuration](../operations/configuration.md) ## Execution and Messaging This page documents task routing, message lifecycle, and ownership guarantees. ### Graph Routing `AgentSystem.spec.graph` supports two edge styles: * `next`: legacy single edge * `edges[]`: preferred route list with labels/policy metadata #### Conditional Edge Routing Edges in the `edges[]` list can carry an optional `condition` that is evaluated against the completing agent's output. When conditions are present, only edges whose condition matches will fire. Edges without conditions are unconditional and always fire. If no conditional edge matches, a `default: true` edge fires as a fallback. Conditions support both string-based operators (`output_contains`, `output_not_contains`, `output_matches`) and JSON path operators (`output_json_path` with `equals`, `not_equals`, `contains`, `greater_than`, `less_than`). JSON path conditions pair with `output_schema` on the Agent spec for guaranteed structured routing. See [AgentSystem -- Conditional Routing](./agents/agent-system.md#conditional-routing) for full details and patterns, and [AgentSystem -- Structured Output](./agents/agent-system.md#structured-output) for provider-native schema enforcement. Conditional routing requires message-driven execution mode. ### Fan-out and Fan-in * Fan-out: one node routes to multiple downstream edges. When conditional routing is active, only matched edges fan out. * Fan-in: downstream join gate with: * `wait_for_all` * `quorum` (`quorum_count` or `quorum_percent`) Join gates automatically adjust their expected branch count when conditional routing reduces the set of dispatched upstream agents. Join state persists in `Task.status.join_states`. ### Delegation Graph nodes can declare `delegates` alongside `edges`. This gives a node two-phase execution: 1. **Dispatch**: After the node's first execution, the output filters `delegates` (same condition logic as edges). Matched delegates receive messages with `delegate_of` metadata. 2. **Collect**: A delegation gate tracks returns. Join modes (`wait_for_all`, `quorum`) control when the gate fires. 3. **Review**: The node re-executes with delegation context (`inbox.delegation.*`). 4. **Forward**: After review, normal `edges` fire. Delegates that reach a terminal point (no outgoing edges match) automatically route back to the delegator. The `delegate_of` field propagates through sub-branches, enabling multi-hop delegation trees. Delegation state persists in `Task.status.delegation_states`. See [AgentSystem -- Delegation](./agents/agent-system.md#delegation) for full details, patterns, and examples. ### Message Lifecycle `Task.status.messages` includes: * lifecycle phase: `queued|running|retrypending|waitingapproval|succeeded|deadletter` * retry fields: `attempts`, `max_attempts`, `next_attempt_at` * worker ownership fields: `worker`, `processed_at`, `last_error` * routing/tracing fields: `branch_id`, `parent_branch_id`, `trace_id`, `parent_id` When a workflow hits a review checkpoint, the current message moves to `waitingapproval` until the linked `TaskApproval` is approved, denied, expired, or rerouted through `request_changes`. ### Tool Selection Model * `Agent.spec.tools[]` defines candidate tools. * Model responses select specific tool calls for each step. * Only selected and authorized tools are executed. * Unauthorized tool selections fail closed as `tool_permission_denied`. ### Ownership and Safety Guarantees * only `Task.status.claimedBy` worker may process messages * leases are renewed during active processing * lease expiry allows safe takeover by another worker * idempotency keys protect replay and crash recovery ### Choosing an Execution Mode Orloj supports two execution modes that share the same resource model and graph definitions. **Sequential mode** (`--task-execution-mode=sequential`) runs the entire graph in-process on the server or embedded worker. Best for getting started, development, and single-agent systems. No message bus required. **Message-driven mode** (`--task-execution-mode=message-driven`) distributes execution across workers via the message bus. Each agent step is a queued message with durable delivery, retry, and dead-letter guarantees. Best for production, parallel fan-out, and horizontal scaling. Both modes produce the same task trace, history, and output. You can develop in sequential mode and deploy to production in message-driven mode without changing your resource definitions. See [Configuration](../operations/configuration.md) for the full set of flags. ### Related Docs * [Architecture Overview](./architecture.md) * [Configuration](../operations/configuration.md) ## Define a CLI Tool This guide shows how to define a CLI tool that invokes a local binary (e.g., `kubectl`, `gh`, `aws`) under Orloj's governance and isolation model. ### Quick start ```yaml apiVersion: orloj.dev/v1 kind: Tool metadata: name: kubectl-get-pods spec: type: cli description: "List Kubernetes pods in a namespace" input_schema: type: object properties: namespace: type: string format: type: string enum: [json, yaml, wide] cli: command: kubectl args: - get - pods - -n - "{{ .namespace }}" - -o - "{{ .format }}" image: bitnami/kubectl:1.30 env_from: - name: KUBECONFIG secretRef: k8s-kubeconfig risk_level: medium operation_classes: [read] runtime: timeout: 15s ``` Apply: ```bash orlojctl apply -f kubectl-get-pods.yaml ``` ### How it works 1. The agent selects the tool and provides JSON input matching `input_schema`. 2. Orloj evaluates `cli.args` templates against the parsed JSON input to build an argv array. 3. Secrets referenced in `cli.env_from` are resolved from the secret store. 4. The command runs inside a container (the default) or directly on the worker host (`isolation_mode: none`). 5. Stdout is returned to the agent as the tool result. ### Argument templates Each entry in `cli.args` is evaluated as a Go `text/template` with the model's JSON input as the data context. Static entries (without `{{ }}`) pass through unchanged. ```yaml args: - get - pods - -n - "{{ .namespace }}" ``` If the model sends `{"namespace": "production", "format": "json"}`, the resulting argv is `["get", "pods", "-n", "production"]`. Each template produces exactly one argv entry -- there is no shell splitting. Templates that evaluate to an empty string are dropped from the final argv. ### Passing input via stdin For tools that read structured input from stdin (e.g., `jq`, custom CLIs), set `stdin_from_input: true`: ```yaml cli: command: jq args: [".items[].metadata.name"] image: ghcr.io/jqlang/jq:1.7 network: none stdin_from_input: true ``` Both templated args and stdin can be combined. ### Credentials CLI tools do not use `spec.auth` (it is rejected at validation time). Instead, map Orloj secrets to the environment variables your binary expects using `env_from`: ```yaml cli: command: gh args: ["pr", "list", "--repo", "{{ .repo }}"] image: ghcr.io/cli/cli:2.50 env_from: - name: GITHUB_TOKEN secretRef: gh-api-token ``` For multiple credentials (e.g., AWS): ```yaml env_from: - name: AWS_ACCESS_KEY_ID secretRef: aws-creds key: access_key - name: AWS_SECRET_ACCESS_KEY secretRef: aws-creds key: secret_key ``` Use `env` for non-secret literals: ```yaml env: AWS_DEFAULT_REGION: us-east-1 ``` ### Container isolation (default) CLI tools default to `container` isolation. The operator provides a container image containing the binary via `cli.image`. The container runs with: * `--read-only` filesystem * `--cap-drop=ALL` * `--security-opt no-new-privileges` * `--network none` by default (inherits `--tool-container-network`; configurable per tool via `cli.network`) * Resource limits from `cli.resources` (per-tool) or the global worker config (`--tool-container-memory`, `--tool-container-cpus`, `--tool-container-pids-limit`) Set `cli.network: bridge` when the binary needs outbound network access (e.g., `kubectl`, `gh`, `curl`): ```yaml cli: command: gh image: ghcr.io/cli/cli:2.50 network: bridge env_from: - name: GITHUB_TOKEN secretRef: gh-api-token ``` Tools that do not need network access (e.g., `jq`, `yq`) can leave `cli.network` unset or set it explicitly to `none`: ```yaml cli: command: jq image: ghcr.io/jqlang/jq:1.7 network: none ``` #### Per-tool container resources Tools that need more resources than the global defaults (e.g. Chromium-based tools) can declare per-tool overrides via `cli.resources`. When set, these take precedence over the global `--tool-container-*` flags: ```yaml cli: command: screenshot image: my-chromium:latest network: bridge resources: memory: 1g cpus: "1.0" pids_limit: 256 ``` | Field | Format | Description | | ------------ | ----------------------------------- | ----------------------- | | `memory` | Docker memory string (`128m`, `1g`) | Container memory limit. | | `cpus` | Decimal string (`0.50`, `1.0`) | Container CPU limit. | | `pids_limit` | Integer | Container PID limit. | Operators can set a ceiling with `--tool-container-max-memory`, `--tool-container-max-cpus`, and `--tool-container-max-pids-limit` on `orlojd`. Manifests exceeding the ceiling are rejected at apply time. ### Kubernetes isolation When `isolation_mode: kubernetes`, CLI tools run as ephemeral Kubernetes Jobs instead of local Docker containers. The same `cli.image`, `cli.command`, `cli.args`, and `cli.resources` fields are used, but execution happens in the cluster via the Kubernetes API rather than via `docker run`. ```yaml spec: type: cli cli: command: kubectl args: ["get", "pods", "-n", "{{ .namespace }}"] image: bitnami/kubectl:1.30 resources: memory: 256m cpus: "0.50" runtime: isolation_mode: kubernetes timeout: 30s ``` This mode requires `--tool-k8s-enabled=true` on the server and worker. The runtime creates a Job in the configured namespace, waits for completion (bounded by `runtime.timeout`), captures stdout/stderr from the Pod logs, and cleans up via `ttlSecondsAfterFinished`. Key differences from `container` isolation: * Execution happens in-cluster; no Docker socket is required on the worker host. * Resource limits from `cli.resources` map to Kubernetes resource requests/limits on the Pod spec. * `runtime.timeout` sets `activeDeadlineSeconds` on the Job. * Network isolation is managed via Kubernetes NetworkPolicies rather than Docker network modes (`cli.network` is ignored). * Credentials from `cli.env_from` are injected as environment variables on the Job's container spec. See [Kubernetes Deployment](../../deploy/kubernetes.md) for RBAC and Helm configuration. ### Direct execution (no container) For trusted tools on the worker host, set `isolation_mode: none`. The binary must exist on the worker's filesystem. `cli.image` is not required in this mode. ```yaml spec: type: cli cli: command: /usr/local/bin/my-tool args: ["--flag", "{{ .value }}"] runtime: isolation_mode: none ``` ### Output capture `cli.output` controls what is returned to the agent: * `stdout` (default) -- return stdout only * `stderr` -- return stderr only * `both` -- return `{"stdout": "...", "stderr": "..."}` as JSON Non-zero exit codes produce a tool error with the exit code and stderr tail in the error details. ### Worker flags | Flag | Env | Default | Description | | ----------------------------- | --------------------------------- | ------- | ---------------------------------------------------- | | `--cli-tool-allowed-commands` | `ORLOJ_CLI_TOOL_ALLOWED_COMMANDS` | (empty) | Comma-separated command allowlist. Empty allows all. | | `--cli-tool-max-argv-length` | `ORLOJ_CLI_TOOL_MAX_ARGV_LENGTH` | `4096` | Max total argv byte length. | ### See also * [Tool reference](../../reference/resources/tool.md) * [Security and Isolation](../../operations/security.md) ## McpServer An **McpServer** represents a connection to an external MCP (Model Context Protocol) server. The McpServer controller discovers tools via `tools/list` and auto-generates [Tool](./tool.md) resources (type=mcp) for each discovered tool. ### Defining an McpServer **stdio transport** (spawns a child process): ```yaml apiVersion: orloj.dev/v1 kind: McpServer metadata: name: everything-server spec: transport: stdio command: npx args: - -y - "@modelcontextprotocol/server-everything" env: - name: API_KEY secretRef: mcp-api-key tool_filter: include: - echo - add reconnect: max_attempts: 3 backoff: 2s ``` **stdio transport with container image** (runs inside Docker): ```yaml apiVersion: orloj.dev/v1 kind: McpServer metadata: name: gmail spec: transport: stdio image: mcp/gmail idle_timeout: "5m" env: - name: GMAIL_OAUTH_PATH secretRef: gmail-creds/oauth_keys mountPath: /secrets/gcp-oauth.keys.json - name: GMAIL_CREDENTIALS_PATH secretRef: gmail-creds/credentials mountPath: /secrets/credentials.json ``` When `image` is set, the MCP server runs inside a container (`docker run --rm -i`) with sandboxing (read-only filesystem, no capabilities, no privilege escalation). If `command` is also set, it overrides the image's entrypoint. If only `image` is set, the image's built-in entrypoint is used. When `mountPath` is set on an env entry, the resolved value is written to an ephemeral host file and bind-mounted read-only into the container at that path. The env var is set to the mount path so the MCP server can locate the file. This enables MCP servers that require file-based credentials (OAuth JSON keys, service account files, TLS certificates). **HTTP transport** (connects to a running server): ```yaml apiVersion: orloj.dev/v1 kind: McpServer metadata: name: remote-server spec: transport: http endpoint: https://mcp.example.com auth: secretRef: mcp-auth-token profile: bearer ``` #### Key Fields | Field | Description | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `transport` | Required. `stdio` or `http`. | | `command` | stdio: command to spawn the MCP server process. Required unless `image` is set. | | `args` | stdio: command arguments. | | `env` | stdio: environment variables. Each entry supports `value` (literal) or `secretRef` (resolve from Secret). | | `env[].mountPath` | Absolute path inside the container where the resolved value is written as a file. Only valid with `image`. | | `image` | stdio: container image. When set, the MCP server runs inside a Docker container. | | `idle_timeout` | Duration after which an idle session is shut down (e.g. `5m`). Default `0` means never evict. | | `endpoint` | http: the MCP server URL. | | `auth` | http: authentication configuration (`secretRef` + `profile`). | | `tool_filter.include` | Optional allowlist of MCP tool names. When set, only listed tools are generated. | | `reconnect` | Reconnection policy: `max_attempts` (default 3) and `backoff` (default 2s). | | `resources` | Container resource overrides: `memory`, `cpus`, `pids_limit`. Overrides global `--tool-container-*` flags. Only applies when `image` is set. | ### How It Works When an McpServer resource is applied: 1. The McpServer controller establishes a connection using the configured transport. 2. It calls `tools/list` to discover available tools. 3. For each discovered tool (filtered by `tool_filter.include` if set), it creates a `Tool` resource with `type: mcp`, `mcp_server_ref`, and `mcp_tool_name`. 4. The `description` and `input_schema` from the MCP server are propagated to the generated Tool, giving the LLM rich tool definitions. At invocation time, the `MCPToolRuntime` resolves the server reference, obtains a session from the `McpSessionManager`, and sends a `tools/call` JSON-RPC 2.0 request through the appropriate transport. ### Container Resources Container-backed MCP servers inherit the global `--tool-container-memory`, `--tool-container-cpus`, and `--tool-container-pids-limit` defaults. For servers that need more resources (e.g. Chromium-based tools like Playwright), override per-server: ```yaml apiVersion: orloj.dev/v1 kind: McpServer metadata: name: playwright-mcp spec: transport: stdio image: playwright-mcp:latest resources: memory: 1g cpus: "1.0" pids_limit: 512 ``` Operators can enforce an upper bound with `--tool-container-max-memory`, `--tool-container-max-cpus`, and `--tool-container-max-pids-limit` on `orlojd`. Manifests exceeding the ceiling are rejected at apply time. ### Idle Timeout and Session Lifecycle When `idle_timeout` is set, sessions are automatically shut down after the specified duration of inactivity: 1. **Apply** -- session spins up, `tools/list` discovers tools, Tool resources are written to the store. 2. **Idle period** -- after `idle_timeout` with no `tools/call`, the reaper closes the session (kills the process or container). 3. **Tool resources persist** -- agents can still see the discovered tools even while the session is down. 4. **Next task** -- when an agent calls a tool, `GetOrCreate` transparently recreates the session. 5. **Warm reuse** -- if multiple tool calls arrive within the timeout window, they reuse the same session. This is especially useful with container-backed MCP servers: the container is only running while tools are actively being called, then automatically shuts down between tasks. #### Image Choice and Cold Start When a session is recreated after being reaped, the cold-start cost depends on the image strategy: * **Pre-built image** (MCP server baked in): 1-3 seconds. Recommended for production. * **Generic base image + `npx -y`**: re-downloads the package on every cold start because `--rm` wipes the container. Not recommended with `idle_timeout`. * **Bare host process** (no image): npm cache persists between restarts on the host. ### Status The McpServer status tracks connection and tool sync state: | Field | Description | | ----------------- | ----------------------------------------------------------- | | `phase` | `Pending`, `Connecting`, `Ready`, or `Error`. | | `discoveredTools` | All tool names from the MCP server's `tools/list` response. | | `generatedTools` | Names of the `Tool` resources actually created. | | `lastSyncedAt` | Timestamp of last successful tool sync. | ### Related * [Tool](./tool.md) -- the auto-generated tool resources * [Secret](./secret.md) -- credentials for MCP server auth * [Resource Reference: McpServer](../../reference/resources/mcp-server.md) * [Guide: Connect an MCP Server](../../guides/connect-mcp-server.md) ## ModelEndpoint Orloj decouples agents from specific model providers through **ModelEndpoint** resources. A ModelEndpoint declares a provider, base URL, default model, and authentication -- and agents reference it by name. This lets you swap providers, manage credentials centrally, and route different agents to different models without modifying agent manifests. ### Defining a ModelEndpoint ```yaml apiVersion: orloj.dev/v1 kind: ModelEndpoint metadata: name: openai-default spec: provider: openai base_url: https://api.openai.com/v1 default_model: gpt-4o-mini auth: secretRef: openai-api-key ``` #### Supported Providers | Provider | `provider` value | Default `base_url` | | ----------------- | ------------------- | ------------------------------ | | OpenAI | `openai` | `https://api.openai.com/v1` | | Anthropic | `anthropic` | `https://api.anthropic.com/v1` | | AWS Bedrock | `bedrock` | (SDK-managed) | | Azure OpenAI | `azure-openai` | (must be set explicitly) | | Ollama (native) | `ollama` | `http://127.0.0.1:11434` | | OpenAI-compatible | `openai-compatible` | (must be set explicitly) | | Mock | `mock` | (no network calls) | #### Provider-Specific Options Some providers require additional configuration via the `options` field: **Anthropic:** ```yaml spec: provider: anthropic base_url: https://api.anthropic.com/v1 default_model: claude-3-5-sonnet-latest options: anthropic_version: "2023-06-01" max_tokens: "1024" auth: secretRef: anthropic-api-key ``` Anthropic credentials may be either a standard API key (`sk-ant-api...`, sent as `x-api-key`) or an OAuth access token (`sk-ant-oat...`, sent as `Authorization: Bearer`). Store either value in the same `secretRef`; Orloj selects the header from the token prefix. **Azure OpenAI:** ```yaml spec: provider: azure-openai base_url: https://YOUR_RESOURCE_NAME.openai.azure.com default_model: gpt-4o-deployment options: api_version: "2024-10-21" auth: secretRef: azure-openai-api-key ``` **AWS Bedrock** (uses the Converse API via the AWS SDK): ```yaml spec: provider: bedrock default_model: anthropic.claude-sonnet-4-20250514-v1:0 options: region: us-east-1 max_tokens: "4096" auth: secretRef: aws-credentials ``` Bedrock uses AWS IAM credentials instead of a simple API key. If `auth.secretRef` is set, the secret must contain a JSON blob with `access_key_id`, `secret_access_key`, and optionally `session_token`. If `auth.secretRef` is omitted, the AWS SDK resolves credentials from the environment (env vars, `~/.aws/credentials`, EC2/ECS IAM roles, etc.). | Option | Description | Default | | ------------ | -------------------------------------- | --------- | | `region` | AWS region (required) | -- | | `max_tokens` | Default max output tokens | `1024` | | `profile` | AWS named profile from `~/.aws/config` | (default) | Cross-region inference profiles (e.g. `us.anthropic.claude-sonnet-4-20250514-v1:0`) work transparently -- use the profile ID as `default_model`. **Ollama** (native `/api/chat` endpoint, no auth required): ```yaml spec: provider: ollama base_url: http://127.0.0.1:11434 default_model: llama3.1 ``` > **Ollama base URL tip:** For `provider: ollama`, use the server root (`http://host:11434`) and do **not** append `/v1`. #### OpenAI-Compatible Providers The `openai-compatible` provider uses the OpenAI Chat Completions protocol (`/chat/completions`) with a custom `base_url`. This lets you connect to any service that exposes an OpenAI-compatible API. `auth.secretRef` is optional for this provider. The following table lists tested providers and their configuration. Any service that implements the `/chat/completions` endpoint should work -- this list is not exhaustive. | Provider | `base_url` | Example `default_model` | Auth required | | ------------------------- | --------------------------------------------------------- | --------------------------------------------------- | ------------- | | Groq | `https://api.groq.com/openai/v1` | `llama-3.3-70b-versatile` | Yes | | Together AI | `https://api.together.xyz/v1` | `meta-llama/Llama-3.3-70B-Instruct-Turbo` | Yes | | Fireworks AI | `https://api.fireworks.ai/inference/v1` | `accounts/fireworks/models/llama-v3p3-70b-instruct` | Yes | | Mistral AI | `https://api.mistral.ai/v1` | `mistral-large-latest` | Yes | | DeepSeek | `https://api.deepseek.com/v1` | `deepseek-chat` | Yes | | xAI (Grok) | `https://api.x.ai/v1` | `grok-3` | Yes | | Google Gemini | `https://generativelanguage.googleapis.com/v1beta/openai` | `gemini-2.5-pro` | Yes | | Perplexity | `https://api.perplexity.ai` | `sonar-pro` | Yes | | OpenRouter | `https://openrouter.ai/api/v1` | `anthropic/claude-sonnet-4` | Yes | | Cerebras | `https://api.cerebras.ai/v1` | `llama-4-scout-17b-16e-instruct` | Yes | | SambaNova | `https://api.sambanova.ai/v1` | `Meta-Llama-3.3-70B-Instruct` | Yes | | vLLM | `http://localhost:8000/v1` | (your deployed model) | No | | text-generation-inference | `http://localhost:8080/v1` | (your deployed model) | No | | LM Studio | `http://localhost:1234/v1` | (your loaded model) | No | | LiteLLM proxy | `http://localhost:4000/v1` | (your configured model) | No | | Ollama (OpenAI mode) | `http://127.0.0.1:11434/v1` | `llama3.1` | No | **Example -- Groq:** ```yaml spec: provider: openai-compatible base_url: https://api.groq.com/openai/v1 default_model: llama-3.3-70b-versatile auth: secretRef: groq-api-key ``` **Example -- Google Gemini (via OpenAI-compatible endpoint):** ```yaml spec: provider: openai-compatible base_url: https://generativelanguage.googleapis.com/v1beta/openai default_model: gemini-2.5-pro auth: secretRef: gemini-api-key ``` **Example -- local vLLM server:** ```yaml spec: provider: openai-compatible base_url: http://localhost:8000/v1 default_model: meta-llama/Llama-3.1-8B-Instruct allowPrivate: true ``` > **Local endpoint note:** For `provider: openai-compatible`, set `allowPrivate: true` when `base_url` points at localhost or a private network. The native `ollama` provider defaults `allowPrivate` to `true`. > > **Ollama note:** Ollama exposes both a native API (`/api/chat`, used by the `ollama` provider) and an OpenAI-compatible API (`/v1/chat/completions`). Use whichever suits your setup -- the `openai-compatible` provider works with Ollama's `/v1` endpoint. > > **Not listed here?** Any service that implements OpenAI's `/chat/completions` endpoint should work. Set `provider: openai-compatible`, point `base_url` at the service's API root, and add `auth.secretRef` if the service requires an API key. ### Binding Agents to Models Agents configure model routing through `spec.model_ref`, which points to a ModelEndpoint: ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: writer-agent spec: model_ref: openai-default prompt: | You are a writing agent. ``` ### Fallback Routing When a model provider is down or rate-limited, you can configure `fallback_model_refs` on an agent to cascade through backup endpoints automatically: ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: writer-agent spec: model_ref: anthropic-claude fallback_model_refs: - openai-gpt4 - ollama-local prompt: | You are a writing agent. ``` The router tries endpoints in order -- primary first, then each fallback. The first successful response wins. Fallback is triggered only on **retryable errors**: * **429** (rate limit) * **5xx** (server errors: 500, 502, 503, etc.) * Connection failures, DNS errors, and timeouts **Non-retryable errors** (400, 401, 403, 404, etc.) fail immediately without trying fallbacks -- these indicate configuration problems that retrying with a different provider won't solve. If all endpoints are exhausted, the last error is returned to the agent worker. Fallback is handled entirely within the model router. The agent worker and execution engine are unaware of it -- they still call `Complete()` once per step. [AgentPolicy](../governance/agent-policy.md) governance applies independently to each endpoint in the fallback chain. ### How Routing Works When a worker executes an agent turn: 1. The runtime resolves the agent's referenced ModelEndpoint from `model_ref`. 2. The model gateway constructs a provider-specific API request using the endpoint's `base_url`, `default_model`, `options`, and auth credentials. 3. The request is sent to the provider and the response is returned to the agent execution loop. ModelEndpoint references are resolved by name within the same namespace, or by `namespace/name` for cross-namespace references. ### Authentication Model authentication is managed through [Secret](./secret.md) resources referenced by `auth.secretRef`. * `openai`, `anthropic`, and `azure-openai` require `auth.secretRef`. * `bedrock` accepts either with or without `auth.secretRef`. When set, the secret must be a JSON blob containing `access_key_id` and `secret_access_key`. When omitted, the AWS SDK default credential chain is used (env vars, instance profiles, SSO, etc.). * `openai-compatible` accepts either with or without `auth.secretRef`. * `ollama` usually runs without `auth.secretRef`. The simplest way to create a Secret is the imperative CLI command: ```bash orlojctl create secret openai-api-key --from-literal value=sk-your-api-key-here ``` Or with a YAML manifest via `orlojctl apply -f`: ```yaml apiVersion: orloj.dev/v1 kind: Secret metadata: name: openai-api-key spec: stringData: value: sk-your-api-key-here ``` In production, you can also skip `Secret` resources entirely and inject values via environment variables (`ORLOJ_SECRET_`). See [Secret Handling](../../operations/security.md#secret-handling) for details. ### Governance Integration AgentPolicy resources can restrict which models an agent is allowed to use via the `allowed_models` field: ```yaml apiVersion: orloj.dev/v1 kind: AgentPolicy metadata: name: cost-policy spec: allowed_models: - gpt-4o max_tokens_per_run: 50000 ``` If an agent's resolved endpoint `default_model` is not in the policy's `allowed_models` list, execution is denied. ### Related * [Secret](./secret.md) -- credential storage for model auth * [Agent](../agents/agent.md) -- agents that reference ModelEndpoints * [Resource Reference: ModelEndpoint](../../reference/resources/model-endpoint.md) * [Configuration](../../operations/configuration.md) * [Guide: Configure Model Routing](../../guides/configure-model-routing.md) ## Orloj Built-in Tools Orloj provides built-in tools under the `orloj.*` namespace that let agents interact with the Orloj control plane. These tools follow the same governance model as any other tool -- ToolPermission, AgentPolicy `blocked_tools`, and ToolApproval all apply. ### Available Tools #### `orloj.task.create` Creates a new task from a template. The task runs independently (fire-and-forget) and the tool returns immediately with the created task's name and initial phase. **Parameters:** | Field | Type | Required | Description | | ---------- | ------ | -------- | ------------------------------------------------------- | | `template` | string | Yes | Name of an existing task with `mode: template` | | `input` | object | No | Key-value input overrides merged with template defaults | | `labels` | object | No | Additional labels attached to the new task | **Response:** ```json { "status": "created", "name": "write-article-parent-task-a1b2c3d4", "phase": "Pending", "template": "write-article", "system": "writing-dept" } ``` #### `orloj.task.list` Lists tasks in the current namespace, optionally filtered by labels. **Parameters:** | Field | Type | Required | Description | | -------- | ------- | -------- | -------------------------------------- | | `labels` | object | No | Label key-value pairs to filter tasks | | `limit` | integer | No | Maximum results to return (default 20) | ### Enabling Orloj Tools Add the desired tool names to the agent's `spec.allowed_tools` list: ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: researcher spec: model_ref: gpt-4o prompt: "You are a research agent..." allowed_tools: - web-search - orloj.task.create - orloj.task.list ``` ### Task Templates `orloj.task.create` references tasks that have `mode: template`. Define a template task that serves as a blueprint: ```yaml apiVersion: orloj.dev/v1 kind: Task metadata: name: write-article spec: mode: template system: writing-dept input: topic: "" research_findings: "" ``` When an agent creates a task from this template, the child task gets `mode: run` and enters the normal execution pipeline. ### Lineage Tracking Child tasks are automatically labeled for traceability: * `orloj.dev/parent-task` -- name of the parent task * `orloj.dev/depth` -- nesting depth (0 for root tasks, incremented for each level) * `orloj.dev/created-by` -- always `orloj.task.create` Use `orloj.task.list` with a label filter to find tasks spawned by a specific parent: ```json {"labels": {"orloj.dev/parent-task": "my-research-task"}} ``` ### Safety Limits AgentPolicy supports two optional fields to prevent runaway task creation: | Field | Default | Description | | ----------------- | ------- | ------------------------------------------------- | | `max_child_depth` | 5 | Maximum nesting depth for chained task creation | | `max_child_tasks` | 20 | Maximum child tasks a single execution can create | ```yaml apiVersion: orloj.dev/v1 kind: AgentPolicy metadata: name: limit-task-creation spec: apply_mode: global max_child_depth: 3 max_child_tasks: 10 ``` ### Governance Orloj tools are governed like any other tool: * **Block with AgentPolicy:** `blocked_tools: [orloj.task.create]` * **Require approval with ToolPermission:** ```yaml apiVersion: orloj.dev/v1 kind: ToolPermission metadata: name: gate-task-creation spec: tool_ref: orloj.task.create operation_rules: - operation_class: write verdict: approval_required ``` If no governance resources are configured, agents with orloj tools in their `allowed_tools` can use them freely. ## Secret A **Secret** stores sensitive values (API keys, tokens, passwords) used by other resources. ModelEndpoints, Tools, McpServers, and TaskWebhooks reference Secrets for authentication. If you need to commit encrypted secret manifests to git, use [SealedSecret](../../reference/resources/sealed-secret.md). `SealedSecret` is decrypted by `orlojd` and reconciled into a normal `Secret`, while consumers continue to reference the generated `Secret`. ### Defining a Secret The simplest way to create a Secret is with the CLI: ```bash orlojctl create secret openai-api-key --from-literal value=sk-your-api-key-here ``` Or with a YAML manifest: ```yaml apiVersion: orloj.dev/v1 kind: Secret metadata: name: openai-api-key spec: stringData: value: sk-your-api-key-here ``` #### Key Fields | Field | Description | | ------------ | ------------------------------------------------------------------------------------------------------------------ | | `data` | Base64-encoded key-value pairs. This is what the runtime reads at execution time. | | `stringData` | Write-only plaintext convenience input. Entries are base64-encoded into `data` during normalization, then cleared. | ### How Secrets Work * `stringData` entries are merged into `data` as base64 during normalization. * Every `data` value must be non-empty valid base64. * `stringData` is cleared after normalization (write-only behavior) -- it is never stored or returned by the API. * Secret resolution is performed fresh per tool invocation. There is no caching of raw secret values, so rotated secrets take effect immediately. ### Environment Variable Override In production, you can skip `Secret` resources entirely and inject values via environment variables: ``` ORLOJ_SECRET_= ``` See [Secret Handling](../../operations/security.md#secret-handling) for details. ### Related * [ModelEndpoint](./model-endpoint.md) -- uses Secrets for model provider auth * [Tool](./tool.md) -- uses Secrets for tool auth * [McpServer](./mcp-server.md) -- uses Secrets for MCP server auth * [Resource Reference: Secret](../../reference/resources/secret.md) * [Resource Reference: SealedSecret](../../reference/resources/sealed-secret.md) ## Tool A **Tool** is an external capability that agents can invoke during execution. Orloj provides a standardized tool contract, multiple isolation backends, and runtime controls for timeout, retry, and risk classification. ### Defining a Tool Tools are declared as resources that describe the tool's endpoint, auth requirements, risk level, and runtime configuration. ```yaml apiVersion: orloj.dev/v1 kind: Tool metadata: name: web_search spec: type: http endpoint: https://api.search.com auth: secretRef: search-api-key ``` For tools that require isolation and runtime controls: ```yaml apiVersion: orloj.dev/v1 kind: Tool metadata: name: wasm-echo spec: type: wasm wasm: module: echo.wasm entrypoint: run max_memory_bytes: 16777216 fuel: 100000 enable_wasi: true capabilities: - wasm.echo.invoke risk_level: low runtime: isolation_mode: wasm timeout: 5s ``` #### Key Fields | Field | Description | | ------------------------ | ----------------------------------------------------------------------- | | `type` | Tool type. Determines the transport and execution model. See below. | | `endpoint` | The tool's network endpoint. | | `capabilities` | Declared operations this tool provides. Used for permission matching. | | `risk_level` | `low`, `medium`, `high`, or `critical`. Affects default isolation mode. | | `runtime.isolation_mode` | Execution isolation backend (see below). | | `runtime.timeout` | Maximum execution time. Defaults to `30s`. | | `runtime.retry` | Retry policy for failed invocations. | | `auth.secretRef` | Reference to a [Secret](./secret.md) resource for tool authentication. | ### Tool Types `Tool.spec.type` determines how the runtime communicates with the tool. Eight types are supported: | Type | Transport | Contract | Use case | | ------------------ | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `http` | HTTP POST to `endpoint` | Raw body or `ToolExecutionResponse` | Simple API integrations. Default when omitted. | | `external` | HTTP POST to `endpoint` | Strict `ToolExecutionRequest` / `ToolExecutionResponse` | Tools running as standalone microservices that need the full execution context. | | `grpc` | Unary gRPC call to `endpoint` | `ToolExecutionRequest` / `ToolExecutionResponse` as JSON over `orloj.tool.v1.ToolService/Execute` | Teams that prefer gRPC for tool communication. | | `webhook-callback` | HTTP POST to `endpoint`, then poll `{endpoint}/{request_id}` | `ToolExecutionRequest` / `ToolExecutionResponse` | Long-running tools, batch jobs, or tools that require human-in-the-loop steps. | | `mcp` | JSON-RPC 2.0 via stdio or HTTP | MCP `tools/call` / `tools/list` | Tools exposed by MCP servers. Auto-generated by the [McpServer](./mcp-server.md) controller. | | `cli` | execve (container or direct) | stdout / stderr captured as result | Local binaries (`kubectl`, `gh`, `aws`, etc.) run under Orloj's isolation model. See [CLI Tools](./cli-tool.md). | | `wasm` | Embedded wazero runtime | stdin/stdout JSON contract | Tools compiled to WebAssembly, executed in-process with host-enforced resource limits. | | `a2a` | A2A JSON-RPC 2.0 | A2A protocol task lifecycle | Delegates execution to remote [A2A](../a2a-interoperability.md)-compatible agents. | All types flow through the same governed runtime pipeline -- policy enforcement, retry, timeout, auth injection, and error taxonomy behave identically regardless of tool type. Unknown type values are rejected at apply time. #### MCP The `mcp` type represents tools provided by an MCP (Model Context Protocol) server. These tools are auto-generated by the [McpServer](./mcp-server.md) controller -- you do not create them manually. Each `type=mcp` tool carries `mcp_server_ref` (the McpServer that owns it) and `mcp_tool_name` (the tool name on the MCP server). At invocation time, the `MCPToolRuntime` resolves the server reference, obtains a session from the `McpSessionManager`, and sends a `tools/call` JSON-RPC 2.0 request through the appropriate transport (stdio or Streamable HTTP). MCP tools also carry `description` and `input_schema` from the MCP server's `tools/list` response. These are propagated to the model gateway so the LLM receives rich, structured tool definitions instead of generic parameter schemas. See the [Connect an MCP Server](../../guides/connect-mcp-server.md) guide for a complete walkthrough. #### CLI The `cli` type invokes a local binary (`kubectl`, `gh`, `aws`, `jq`, etc.) using an execve-style call — no shell involved. Each entry in `cli.args` is a Go `text/template` evaluated against the agent's JSON input; each template produces exactly one argv element. CLI tools default to `container` isolation regardless of `risk_level`, running the binary inside a Docker container specified by `cli.image`. Credentials are injected as environment variables via `cli.env_from` (referencing Secret resources); `spec.auth` is not supported for this type. See [Define a CLI Tool](./cli-tool.md) for a complete reference and examples. #### HTTP (default) The `http` type sends the agent's tool input as an HTTP POST body to `spec.endpoint`. The runtime accepts both raw text responses and structured `ToolExecutionResponse` JSON envelopes. Auth is injected as an `Authorization: Bearer` header when `auth.secretRef` is configured. #### External The `external` type sends the full `ToolExecutionRequest` contract envelope as JSON to `spec.endpoint` and expects a `ToolExecutionResponse` back. This gives the external service access to the full execution context (task ID, agent, namespace, trace IDs, attempt number). Use this when your tool needs to be aware of the Orloj execution context. #### gRPC The `grpc` type calls `orloj.tool.v1.ToolService/Execute` as a unary gRPC method on `spec.endpoint`, using a JSON codec. The request and response payloads are the same `ToolExecutionRequest` / `ToolExecutionResponse` envelopes as `external`. Use this when your tool infrastructure is gRPC-native. #### Webhook-Callback The `webhook-callback` type supports asynchronous tool execution: 1. The runtime POSTs a `ToolExecutionRequest` to `spec.endpoint`. 2. The tool returns `202 Accepted` (or `200 OK` with an immediate result). 3. If `202`: the runtime polls `{endpoint}/{request_id}` at regular intervals until a `ToolExecutionResponse` with a terminal status arrives, or the configured timeout expires. This is useful for tools that take minutes to complete (e.g., batch processing, code review, CI pipeline triggers) or that require external approval before returning a result. ### Isolation Modes Isolation modes control the execution boundary of a tool, independent of tool type. | Mode | Description | Default for | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | | `none` | Direct execution in the worker process. The `http` type makes real HTTP calls; other types use their respective transports. | `low` and `medium` risk tools | | `sandboxed` | Restricted container execution with secure defaults: read-only filesystem, no capabilities, no privilege escalation, no network, non-root user, memory/CPU/pids limits. | `high` and `critical` risk tools | | `container` | Each tool invocation runs in an isolated container. Full filesystem and network isolation. | Explicitly configured | | `kubernetes` | Executes the tool as an ephemeral Kubernetes Job. Requires `--tool-k8s-enabled=true`. The tool's `spec.cli.image`, `command`, `args`, and `resources` are mapped to the Job's Pod spec. | Explicitly configured | | `wasm` | Tool runs as a WebAssembly module with a host-guest stdin/stdout contract. Memory-safe and deterministic. | Explicitly configured | The isolation mode defaults are based on `risk_level`: * `low` or `medium` risk: defaults to `none` * `high` or `critical` risk: defaults to `sandboxed` You can always override the default by setting `runtime.isolation_mode` explicitly. #### Sandboxed Defaults When `isolation_mode` is `sandboxed`, the container backend enforces these secure defaults: * `--read-only` filesystem * `--cap-drop=ALL` (no Linux capabilities) * `--security-opt no-new-privileges` * `--network none` (no network access) * `--user 65532:65532` (non-root) * `--memory 128m` * `--cpus 0.50` * `--pids-limit 64` These defaults can be overridden with `--tool-container-*` flags on `orlojd` and `orlojworker`, but the default posture is restrictive. ### Tool Contract v1 Every tool interaction follows a standardized request/response envelope. This contract ensures tools are portable, testable, and observable regardless of the isolation backend. **Request envelope** (sent to the tool): ```json { "request_id": "req-abc-123", "tool": "web_search", "action": "invoke", "parameters": { "query": "enterprise AI adoption trends" }, "auth": { "type": "bearer", "token": "sk-..." }, "context": { "task": "weekly-report", "agent": "research-agent", "attempt": 1 } } ``` **Response envelope** (returned from the tool): ```json { "request_id": "req-abc-123", "status": "success", "result": { "data": "..." } } ``` **Error response:** ```json { "request_id": "req-abc-123", "status": "error", "error": { "tool_code": "rate_limited", "tool_reason": "API rate limit exceeded", "retryable": true } } ``` The tool contract defines a canonical error taxonomy with `tool_code`, `tool_reason`, and `retryable` fields, enabling the runtime to make intelligent retry decisions. ### WASM Tools WASM tools run as WebAssembly modules inside an embedded [wazero](https://wazero.io) runtime. No external binary is required -- the runtime is pure Go and always available regardless of `--tool-isolation-backend`. WASM tools communicate over stdin/stdout using a JSON contract (v1). The host writes the request to the module's stdin and reads a JSON response from stdout. This provides memory-safe, sandboxed execution with no filesystem or network access unless WASI is explicitly enabled. Local `module` paths must be **relative** and resolve under `--tool-wasm-cache-dir` (default `~/.orloj/wasm-cache`). Absolute paths, `..` segments, and plain `http://` URLs are rejected; use `https://` or `oci://` for remote modules. #### Per-Tool Configuration Each WASM tool declares its module and resource limits in `spec.wasm`. The `module` field accepts a relative path under `--tool-wasm-cache-dir`, an HTTPS URL, or an OCI artifact reference: ```yaml apiVersion: orloj.dev/v1 kind: Tool metadata: name: wasm-echo spec: type: wasm wasm: module: echo.wasm # Relative path under --tool-wasm-cache-dir, or HTTPS / oci:// URL entrypoint: run # Exported function (default: run) max_memory_bytes: 67108864 # 64 MB (default) fuel: 1000000 # Execution fuel limit (default: 1M) enable_wasi: true # Enable stdin/stdout/stderr via WASI image_pull_secret: ghcr-creds # Optional: Secret for private OCI registries capabilities: - wasm.echo.invoke risk_level: low runtime: isolation_mode: wasm timeout: 5s ``` #### Writing a WASM Tool Any language that compiles to WebAssembly can produce a WASM tool. The simplest path is Go with `GOOS=wasip1 GOARCH=wasm`: ```go package main import ( "encoding/json" "io" "os" ) func main() { data, _ := io.ReadAll(os.Stdin) var req struct { Input string `json:"input"` } _ = json.Unmarshal(data, &req) resp := map[string]string{ "contract_version": "v1", "status": "ok", "output": req.Input, } _ = json.NewEncoder(os.Stdout).Encode(resp) } ``` Build with: `GOOS=wasip1 GOARCH=wasm go build -o tool.wasm tool.go` #### Resource Limits Memory and fuel limits are enforced by the host runtime, not by the guest module: * **Memory**: `max_memory_bytes` limits the WASM linear memory. Defaults to 64 MB. * **Fuel**: `fuel` limits execution steps. Prevents runaway modules from consuming unbounded CPU. * **WASI**: When `enable_wasi` is false, the module has no access to stdin/stdout/stderr. Most tools require WASI enabled. #### Coexistence with Containers WASM tools have their own dedicated runtime slot and work independently of `--tool-isolation-backend`. You can run WASM tools and container-isolated tools in the same agent system without conflict. #### CLI Development Tools `orlojctl` provides scaffold and test commands for WASM tool development: * **`orlojctl tool scaffold --lang go|rust`** generates a ready-to-build project with guest code, Makefile, tool manifest, and test fixtures. * **`orlojctl tool test --fixtures `** runs the module against JSON test fixtures and validates contract compliance, expected output, and resource budgets. For a complete walkthrough including the full stdin/stdout contract specification, error handling, multi-language examples, and local testing, see the [Build a WASM Tool](../../guides/build-wasm-tool.md) guide. ### Error Taxonomy Tool failures use a canonical error taxonomy with three fields: | Field | Purpose | | ------------- | ------------------------------------------------------------------------------------------------- | | `tool_code` | Machine-readable error code (e.g. `rate_limited`, `unsupported_tool`, `secret_resolution_failed`) | | `tool_reason` | Human-readable explanation | | `retryable` | Whether the runtime should retry the invocation | HTTP status codes are mapped automatically: `429` and `5xx` are retryable, `4xx` are not. HTTP `401` maps to `auth_invalid` and `403` maps to `auth_forbidden` -- both non-retryable. All tool types share the same taxonomy, so policy and observability behave consistently. ### Auth Profiles Tools support four authentication profiles via `spec.auth.profile`: | Profile | Secret format | Injection | | --------------------------- | ----------------------------------------------------- | ------------------------------------------------------------- | | `bearer` (default) | Single token value | `Authorization: Bearer ` | | `api_key_header` | Single key value | Custom header via `spec.auth.headerName` | | `basic` | `username:password` | `Authorization: Basic ` | | `oauth2_client_credentials` | Multi-key secret with `client_id` and `client_secret` | Token exchange at `spec.auth.tokenURL`, then bearer injection | When `spec.auth.secretRef` is set without an explicit profile, the default is `bearer` for backward compatibility. #### Secret Rotation Secret resolution is performed fresh per tool invocation -- there is no caching of raw secret values. If a secret is rotated between invocations, the new value takes effect on the next call without requiring a restart. For `oauth2_client_credentials`, access tokens are cached with a TTL derived from the token endpoint's `expires_in` response. Tokens are evicted on expiry or when the tool endpoint returns HTTP 401, triggering a fresh token exchange. ### Retry and Timeout Each tool can configure its own retry policy independently of the task-level retry: ```yaml runtime: timeout: 5s retry: max_attempts: 3 backoff: 1s max_backoff: 30s jitter: full ``` Retry uses capped exponential backoff. The `jitter` field controls randomization: `none` (deterministic), `full` (random between 0 and backoff), or `equal` (half deterministic, half random). ### Operation Classes Tools can declare operation classes via `spec.operation_classes` (e.g. `read`, `write`, `delete`, `admin`). When omitted, the default is `["read"]` for low/medium risk tools and `["write"]` for high/critical risk tools. Operation classes are used by `ToolPermission.spec.operation_rules` to define per-class policy verdicts: * **allow**: proceed with the tool call (default). * **deny**: block the call with a `permission_denied` error. * **approval\_required**: pause the task and create a [ToolApproval](../governance/tool-approval.md) resource. An external actor (human or system) must approve or deny the request before the task can continue. When multiple rules match, the most restrictive verdict wins: `deny` > `approval_required` > `allow`. ### Governance Integration Tool invocations are gated by the [governance layer](../governance/). An agent must have the required permissions (via AgentRole) to invoke a tool, and the tool must not be blocked by any applicable AgentPolicy. Unauthorized calls fail closed with a `tool_permission_denied` error. ### Related * [McpServer](./mcp-server.md) -- auto-discover tools from MCP servers * [Secret](./secret.md) -- store credentials for tool auth * [ModelEndpoint](./model-endpoint.md) -- model provider configuration * [Resource Reference: Tool](../../reference/resources/tool.md) * [Guide: Build a Custom Tool](../../guides/build-custom-tool.md) * [Guide: Build a WASM Tool](../../guides/build-wasm-tool.md) * [Guide: Connect an MCP Server](../../guides/connect-mcp-server.md) ## TaskSchedule A **TaskSchedule** creates [Tasks](./task.md) on a cron-based schedule from a template task. Use this to automate recurring work like daily reports, periodic data processing, or scheduled monitoring runs. ### Defining a TaskSchedule ```yaml apiVersion: orloj.dev/v1 kind: TaskSchedule metadata: name: weekly-report spec: task_ref: weekly-report-template schedule: "0 9 * * 1" time_zone: America/Chicago suspend: false starting_deadline_seconds: 300 concurrency_policy: forbid successful_history_limit: 10 failed_history_limit: 3 ``` #### Key Fields | Field | Description | | --------------------------- | ---------------------------------------------------------------- | | `task_ref` | Reference to a Task with `mode: template`. | | `schedule` | Standard 5-field cron expression. | | `time_zone` | IANA timezone (defaults to `UTC`). | | `concurrency_policy` | `forbid` prevents overlapping runs. | | `starting_deadline_seconds` | Maximum lateness before a missed trigger is skipped. | | `suspend` | Set to `true` to pause scheduling without deleting the resource. | ### How It Works When the cron fires, the scheduler: 1. Checks `concurrency_policy` -- if `forbid` and a previous run is still active, the trigger is skipped. 2. Checks `starting_deadline_seconds` -- if the trigger is later than the deadline, it is skipped. 3. Creates a new Task from the template, inheriting all `spec` fields from the referenced task. 4. The new task enters `Pending` and follows the normal [Task lifecycle](./task.md#task-lifecycle). ### Related * [Task](./task.md) -- the tasks that schedules create * [TaskWebhook](./task-webhook.md) -- trigger tasks from external events * [Resource Reference: TaskSchedule](../../reference/resources/task-schedule.md) ## TaskWebhook A **TaskWebhook** creates [Tasks](./task.md) in response to external HTTP events, with built-in signature verification and idempotency. ### Defining a TaskWebhook A TaskWebhook can reference a separate template Task via `task_ref`, or define the task spec inline via `task_template`. Exactly one must be set. #### Using a template reference ```yaml apiVersion: orloj.dev/v1 kind: TaskWebhook metadata: name: report-github-push spec: task_ref: weekly-report-template auth: profile: github secret_ref: webhook-shared-secret idempotency: event_id_header: X-GitHub-Delivery dedupe_window_seconds: 86400 payload: mode: raw input_key: webhook_payload ``` #### Using an inline template When only one webhook uses a template, you can embed the task spec directly in the webhook to avoid creating a separate Task resource: ```yaml apiVersion: orloj.dev/v1 kind: TaskWebhook metadata: name: ingest-events spec: task_template: system: event-pipeline priority: normal input: webhook_payload: "" auth: profile: generic secret_ref: ingest-secret idempotency: event_id_header: X-Event-Id payload: input_key: webhook_payload ``` ### Auth Profiles TaskWebhooks verify incoming requests using signature verification. Four profiles are supported: | Profile | Signature Method | Headers | | -------------- | ----------------------------------------------------------------------- | ------------------------------------------ | | `generic` | HMAC-SHA256 over `timestamp + "." + rawBody` | `X-Signature`, `X-Timestamp`, `X-Event-Id` | | `github` | HMAC-SHA256 over raw body | `X-Hub-Signature-256`, `X-GitHub-Delivery` | | `hmac` | Configurable HMAC (algorithm, payload format, encoding, header parsing) | User-defined | | `shared_token` | Constant-time comparison of a static token header | User-defined | The shared secret is stored in a [Secret](../tools/secret.md) resource referenced by `auth.secret_ref`. #### `hmac` Profile The `hmac` profile provides full control over HMAC verification for services that don't match the `generic` or `github` presets. | Field | Description | Default | | -------------------- | ----------------------------------------------------------------------------------------- | -------- | | `algorithm` | Hash algorithm: `sha256`, `sha1`, `sha512` | `sha256` | | `payload_format` | How to construct the signing input: `body`, `timestamp_dot_body`, `prefix_timestamp_body` | `body` | | `payload_prefix` | Literal prefix for `prefix_timestamp_body` (e.g., `v0`) | | | `payload_separator` | Separator between payload parts | `.` | | `signature_encoding` | Signature encoding: `hex` or `base64` | `hex` | | `header_format` | How to parse the signature header: `plain` or `kv_pairs` | `plain` | | `signature_key` | Key holding the signature in a `kv_pairs` header | | | `timestamp_key` | Key holding the timestamp in a `kv_pairs` header | | Example -- Stripe: ```yaml auth: profile: hmac secret_ref: stripe-secret algorithm: sha256 payload_format: timestamp_dot_body signature_header: Stripe-Signature header_format: kv_pairs signature_key: v1 timestamp_key: t signature_encoding: hex ``` Example -- Slack: ```yaml auth: profile: hmac secret_ref: slack-signing-secret algorithm: sha256 payload_format: prefix_timestamp_body payload_prefix: "v0" payload_separator: ":" signature_header: X-Slack-Signature signature_prefix: "v0=" timestamp_header: X-Slack-Request-Timestamp signature_encoding: hex ``` #### `shared_token` Profile For services that send a static secret token in a header (no HMAC). Comparison is constant-time. Example -- Telegram: ```yaml auth: profile: shared_token secret_ref: telegram-bot-secret signature_header: X-Telegram-Bot-Api-Secret-Token ``` ### Idempotency TaskWebhooks deduplicate deliveries using the event ID header. If a delivery with the same event ID arrives within the `dedupe_window_seconds`, it is rejected as a duplicate. ### How It Works When an HTTP request hits the webhook endpoint: 1. The runtime verifies the request against the shared secret using the configured auth profile (HMAC signature or shared token comparison). 2. The event ID is checked against the deduplication window. 3. If valid and not a duplicate, a new Task is created from the template. 4. The webhook payload is injected into the task input under `input_key`. ### Related * [Task](./task.md) -- the tasks that webhooks create * [TaskSchedule](./task-schedule.md) -- cron-based task automation * [Resource Reference: TaskWebhook](../../reference/resources/task-webhook.md) ## Task A **Task** is a request to execute an [AgentSystem](../agents/agent-system.md). Tasks are the unit of work in Orloj -- they carry input, track execution state, and produce output. ### Defining a Task ```yaml apiVersion: orloj.dev/v1 kind: Task metadata: name: weekly-report spec: system: report-system input: topic: AI startups priority: high retry: max_attempts: 3 backoff: 5s message_retry: max_attempts: 2 backoff: 250ms max_backoff: 2s jitter: full requirements: region: default model: gpt-4o ``` ### Task Lifecycle Every task moves through a well-defined set of phases: ``` Pending ──► Running ──► Succeeded └──► Failed └──► DeadLetter ``` | Phase | Meaning | | ------------ | --------------------------------------------------------------------------------- | | `Pending` | Task is created and waiting for a worker to claim it. | | `Running` | A worker has claimed the task and is executing the agent graph. | | `Succeeded` | All agents in the graph completed successfully. | | `Failed` | Execution failed and retries are not exhausted. May transition back to `Pending`. | | `DeadLetter` | All retry attempts exhausted. Terminal state requiring manual investigation. | ### Worker Assignment and Leases The scheduler assigns tasks to workers based on `requirements` (region, GPU, model). Workers claim tasks through a lease mechanism: 1. Scheduler matches task requirements to worker capabilities. 2. Worker claims the task and acquires a time-bounded lease. 3. Worker renews the lease via heartbeats during execution. 4. If the lease expires (worker crash, network partition), another worker may safely take over. This guarantees exactly-once processing semantics even under failure. ### Retry Configuration Tasks support two levels of retry: **Task-level retry** (`spec.retry`) -- retries the entire task from the beginning if it fails. ```yaml retry: max_attempts: 3 backoff: 5s ``` **Message-level retry** (`spec.message_retry`) -- retries individual agent-to-agent messages within the graph without restarting the full task. ```yaml message_retry: max_attempts: 2 backoff: 250ms max_backoff: 2s jitter: full ``` Retry uses capped exponential backoff with configurable jitter (`none`, `full`, `equal`). Messages that exhaust retries transition to `deadletter` phase. ### Cyclic Graphs For AgentSystems with cycles (loops), `spec.max_turns` bounds the number of iterations to prevent infinite execution: ```yaml spec: system: manager-research-loop-system input: topic: AI coding assistants max_turns: 6 ``` ### Task Templates Tasks with `mode: template` serve as templates for [TaskSchedules](./task-schedule.md) and [TaskWebhooks](./task-webhook.md). They are not executed directly. ```yaml spec: mode: template system: report-system input: topic: AI startups ``` ### Related * [TaskSchedule](./task-schedule.md) -- automate task creation with cron * [TaskWebhook](./task-webhook.md) -- trigger tasks from external events * [Worker](../infrastructure/worker.md) -- the execution units that run tasks * [Resource Reference: Task](../../reference/resources/task.md) * [Execution and Messaging](../execution-model.md) ## Memory Memory gives agents the ability to store, retrieve, and search information across execution steps and across tasks. Orloj implements memory as a layered system: conversation history provides short-term context within a single task turn, a task-scoped shared store lets agents in the same task exchange state, and persistent backends retain knowledge across task runs. ### How Memory Works When an agent has `spec.memory.ref` set to a Memory resource, the runtime attaches that memory backend to the agent. Built-in memory operations are granted explicitly through `spec.memory.allow`, and the runtime exposes only those allowed operations as callable built-in tools. They behave like tools during execution, but are handled internally by the runtime without network calls. ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: research-agent spec: model_ref: openai-default prompt: | You are a research assistant. Use memory tools to store and retrieve findings. tools: - web_search memory: ref: research-memory allow: - read - write - search limits: max_steps: 10 ``` The `memory.ref` field points to a Memory resource that configures the backing store: ```yaml apiVersion: orloj.dev/v1 kind: Memory metadata: name: research-memory spec: type: vector provider: in-memory ``` For vector-similarity search with PostgreSQL and pgvector: ```yaml apiVersion: orloj.dev/v1 kind: Memory metadata: name: production-memory spec: type: vector provider: pgvector endpoint: postgres://orloj@pgvector-host:5432/memories embedding_model: openai-embeddings # references a ModelEndpoint auth: secretRef: pg-password ``` For a custom vector database via the HTTP adapter: ```yaml apiVersion: orloj.dev/v1 kind: Memory metadata: name: custom-vectordb spec: type: vector provider: http endpoint: https://my-vector-adapter.example.com auth: secretRef: vector-db-api-key ``` See [Memory Providers](./providers.md) for the full list of supported providers. ### Memory Layers #### Conversation History Every agent accumulates a message history during multi-turn execution within a single task turn. The system prompt, user context, model responses, and tool results are all appended to the conversation and sent to the model on each step. This gives the model continuity across reasoning steps without explicit memory tool calls. Conversation history is ephemeral -- it exists only for the duration of the agent's current activation and is not shared between agents. #### Task-Scoped Shared Memory When no persistent backend is configured (or as a fallback), memory tools operate on an in-process key-value store scoped to the current task. All agents within the same task share this store, enabling coordination: * Agent A writes `memory.write({"key": "findings", "value": "..."})` * Agent B reads `memory.read({"key": "findings"})` This store is ephemeral and cleared when the task completes. #### Persistent Backends When a Memory resource specifies a persistent provider, memory tools delegate to the configured backend. Data written by one task is available to future tasks that reference the same Memory resource. There are two ways to connect a vector database: **Built-in providers** -- Orloj ships Go implementations that connect directly to popular databases. Users configure `spec.endpoint` and `spec.auth.secretRef` on the Memory CRD and Orloj handles the rest. No extra infrastructure needed. **HTTP adapter** -- For databases without a built-in provider, users deploy a lightweight adapter service that speaks a simple JSON contract and set `provider: http`. The adapter can be written in any language. See [Memory Providers](./providers.md) for full details on each provider, configuration examples, and how to build custom providers. ### Built-in Memory Tools When `spec.memory.ref` is set and `spec.memory.allow` grants the corresponding operations, the runtime exposes the following built-in tools. They do not need to be listed in `spec.tools`. #### `memory.read` Retrieve a value by key. ```json {"key": "research-findings"} ``` Returns `{"found": true, "key": "research-findings", "value": "..."}` or `{"found": false, "key": "research-findings"}`. #### `memory.write` Store a value under a key. Overwrites any existing value. ```json {"key": "research-findings", "value": "The study shows..."} ``` Returns `{"status": "ok", "key": "research-findings"}`. #### `memory.search` Search stored entries by keyword (or vector similarity when a persistent backend with embeddings is configured). ```json {"query": "climate data", "top_k": 5} ``` Returns `{"results": [{"key": "...", "value": "...", "score": 1.0}], "count": 3}`. #### `memory.list` List stored entries, optionally filtered by key prefix. ```json {"prefix": "research/"} ``` Returns `{"entries": [{"key": "...", "value": "..."}], "count": 5}`. #### `memory.ingest` Chunk a document and store the pieces for later search. Useful for loading text files, reports, or other documents into memory. ```json { "source": "quarterly-report", "content": "Full text of the document...", "chunk_size": 1000, "overlap": 200 } ``` The tool splits the content into overlapping windows and stores each chunk under `{source}/chunk-{NNNN}`. Returns `{"status": "ok", "source": "quarterly-report", "chunks_stored": 12}`. `chunk_size` and `overlap` are optional and default to 1000 and 200 characters respectively. ### Memory in Agent Systems In a multi-agent system, memory enables coordination between agents without requiring direct message passing for every piece of state: * A **research agent** writes findings to memory. * A **writer agent** reads those findings and produces a report. * A **coordinator agent** lists memory entries to track overall progress. All agents that reference the same Memory resource (via `spec.memory.ref`) and execute within the same task share the same backing store. ### Memory Resource Configuration The Memory resource is a declarative configuration. It tells the runtime which backend to use and how to configure it. #### `spec` Fields | Field | Description | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | Categorization of the memory use case (e.g. `vector`, `kv`). Informational; does not affect runtime behavior in v1. | | `provider` | Backend implementation. `in-memory` (default), `pgvector`, `http` (external adapter), or a registered built-in provider name. See [Memory Providers](./providers.md). | | `embedding_model` | Reference to a ModelEndpoint resource that provides an embeddings API. Required for vector providers like `pgvector`. The endpoint's `base_url`, `auth`, and `default_model` are used to generate embeddings. | | `endpoint` | URL or connection string for the database or adapter service. Required for `pgvector`, `http`, and cloud-hosted built-in providers. Not needed for `in-memory`. | | `auth.secretRef` | Reference to a Secret resource containing credentials (API key, password, bearer token). | #### Status The controller reconciles Memory resources and reports backend health: * **Ready** -- backend is configured and reachable. * **Error** -- provider is unsupported or connectivity check failed. See `status.lastError` for details. ### Frontend The Memory detail page in the UI includes an **Entries** tab that displays stored memory entries. You can search entries by keyword and browse keys and values. This is useful for debugging agent behavior and inspecting what data has been stored. ### Related Resources * [Memory Providers](./providers.md) * [Resource Reference: Memory](../../reference/resources/memory.md) * [Agents](../agents/agent.md) * [Architecture](../architecture.md) * [API Reference](../../reference/api.md) ## Memory Providers Memory providers are the backends that store and retrieve data for Orloj's built-in memory tools. There are two paths to connect a vector database, both coexisting: **Built-in providers** -- Orloj ships Go implementations that connect directly to popular databases. Users configure `spec.endpoint` and `spec.auth.secretRef` on the Memory CRD and Orloj handles the rest. No extra infrastructure needed. **HTTP adapter** -- For databases without a built-in provider, users deploy a lightweight adapter service that speaks a simple JSON contract and set `provider: http`. The adapter can be written in any language. Both paths use the same CRD fields: `spec.endpoint` for the database URL and `spec.auth.secretRef` for credentials. ### Built-in Providers | Provider | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `in-memory` | In-process map. No endpoint needed. Useful for testing and single-instance deployments. Data is lost on restart. | | `pgvector` | PostgreSQL with the pgvector extension. Full vector-similarity search via embeddings. Requires `endpoint` (Postgres DSN), `embedding_model` (ModelEndpoint reference), and optionally `auth.secretRef` (Postgres password). See [pgvector](#pgvector). | | `http` | Delegates to an external HTTP service at `spec.endpoint`. See [HTTP Adapter](#http-adapter). | ### pgvector The `pgvector` provider stores memory entries in PostgreSQL using the [pgvector](https://github.com/pgvector/pgvector) extension. Every write generates a vector embedding, enabling true cosine-similarity search via `memory.search`. #### Requirements * A PostgreSQL instance with the `vector` extension installed (pgvector). * A ModelEndpoint that serves an OpenAI-compatible `/embeddings` API (OpenAI, Azure OpenAI, Ollama, or any compatible provider). #### Configuration ```yaml apiVersion: orloj.dev/v1 kind: Memory metadata: name: team-knowledge namespace: production spec: type: vector provider: pgvector endpoint: postgres://orloj@pgvector-host:5432/memories embedding_model: openai-embeddings auth: secretRef: pg-password ``` The `embedding_model` field references a ModelEndpoint by name: ```yaml apiVersion: orloj.dev/v1 kind: ModelEndpoint metadata: name: openai-embeddings namespace: production spec: provider: openai default_model: text-embedding-3-small auth: secretRef: openai-api-key ``` | Field | Description | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `endpoint` | Full Postgres connection string (DSN). Example: `postgres://user@host:5432/dbname`. Mutually exclusive with `endpoint_secret_ref`. | | `endpoint_secret_ref` | Reference to a Secret whose first data value contains the full Postgres DSN (including password). Use this instead of `endpoint` + `auth.secretRef` to keep all sensitive connection details in a single Secret. Mutually exclusive with `endpoint`. | | `embedding_model` | Name of a ModelEndpoint in the same namespace (or `namespace/name` for cross-namespace). The endpoint's `base_url` and `auth` are used to call the embeddings API, and `default_model` selects the model. | | `auth.secretRef` | Optional. Reference to a Secret containing the Postgres password. Injected into the DSN if the connection string doesn't already include one. Not needed when using `endpoint_secret_ref` with a full DSN that includes the password. | When the connection string is sensitive, use `endpoint_secret_ref` to store the entire DSN (including password) in a single Secret: ```yaml apiVersion: orloj.dev/v1 kind: Memory metadata: name: team-knowledge namespace: production spec: type: vector provider: pgvector endpoint_secret_ref: pg-connection-string embedding_model: openai-embeddings ``` #### How It Works On creation, the memory controller: 1. Resolves the `embedding_model` ModelEndpoint and builds an embedding provider. 2. Connects to PostgreSQL using the DSN from `endpoint`. 3. Generates a test embedding to auto-detect the vector dimension. 4. Creates the `vector` extension, table, and HNSW index if they don't exist. 5. Runs `Ping` to verify connectivity. The table schema (default table name `orloj_memory`, overridable via `spec.options.table`): ```sql CREATE TABLE orloj_memory ( key TEXT PRIMARY KEY, value TEXT NOT NULL, embedding vector(), created_at TIMESTAMPTZ DEFAULT now(), updated_at TIMESTAMPTZ DEFAULT now() ); CREATE INDEX orloj_memory_embedding_idx ON orloj_memory USING hnsw (embedding vector_cosine_ops); ``` * **`memory.write`** embeds the value and upserts the row. * **`memory.search`** embeds the query and performs cosine-similarity search. * **`memory.read`** and **`memory.list`** operate on key/prefix without embeddings. * **`memory.ingest`** chunks the document and stores each chunk with its embedding. ### HTTP Adapter When `provider: http` is set, Orloj delegates all memory operations to an external service at `spec.endpoint`. This is the escape hatch for vector databases that don't have a built-in provider yet. The adapter can be written in any language and deployed anywhere Orloj can reach over HTTP. #### Contract The service must implement five endpoints. All POST endpoints accept and return `application/json`. **`POST /put`** -- Store a key-value pair. ```json // Request {"key": "findings/chunk-0001", "value": "The quarterly report shows..."} // Response {"status": "ok"} ``` **`POST /get`** -- Retrieve a value by key. ```json // Request {"key": "findings/chunk-0001"} // Response {"found": true, "key": "findings/chunk-0001", "value": "The quarterly report shows..."} ``` **`POST /search`** -- Search entries by keyword or vector similarity. ```json // Request {"query": "quarterly revenue", "top_k": 5} // Response {"results": [{"key": "...", "value": "...", "score": 0.92}]} ``` **`POST /list`** -- List entries by key prefix. ```json // Request {"prefix": "findings/"} // Response {"entries": [{"key": "...", "value": "..."}]} ``` **`GET /ping`** -- Health check. ```json // Response {"status": "ok"} ``` Errors are signaled via HTTP status codes (4xx/5xx) with an optional `{"error": "message"}` body. #### Authentication When `spec.auth.secretRef` is set, Orloj sends an `Authorization: Bearer ` header on every request. The token is resolved from the referenced Secret resource. #### Example ```yaml apiVersion: orloj.dev/v1 kind: Memory metadata: name: custom-vectordb spec: provider: http endpoint: https://my-adapter.example.com auth: secretRef: adapter-api-key ``` ### Custom Providers For contributors adding first-party vector database support, or users building custom Orloj binaries, providers can be registered directly in Go. Implement the `PersistentMemoryBackend` interface and register a factory at startup: ```go import agentruntime "github.com/OrlojHQ/orloj/runtime" func init() { agentruntime.DefaultMemoryProviderRegistry().Register("qdrant", func(cfg agentruntime.MemoryProviderConfig) (agentruntime.PersistentMemoryBackend, error) { // cfg.Endpoint, cfg.AuthToken, cfg.Embedder are available return NewQdrantBackend(cfg) }) } ``` The `MemoryProviderConfig` passed to the factory contains: | Field | Description | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Type` | The `spec.type` from the Memory CRD (e.g. `vector`, `kv`). | | `Provider` | The `spec.provider` value that matched the registration. | | `EmbeddingModel` | The raw `spec.embedding_model` string from the Memory CRD. | | `Endpoint` | The `spec.endpoint` URL or connection string. | | `AuthToken` | Resolved bearer token from `spec.auth.secretRef`. | | `Options` | Provider-specific configuration (currently unused). | | `Embedder` | An `EmbeddingProvider` interface (with `Embed` and `Dimensions` methods). Non-nil when `spec.embedding_model` references a valid ModelEndpoint. Vector providers should use this for generating embeddings. | The Memory controller calls the factory, runs `Ping` to verify connectivity, and moves the resource to `Ready` if successful. ## Worker A **Worker** is an execution unit that claims and runs [Tasks](../tasks/task.md). Workers register their capabilities (region, GPU, supported models) and the scheduler uses these for task matching. ### Defining a Worker ```yaml apiVersion: orloj.dev/v1 kind: Worker metadata: name: worker-a spec: region: default max_concurrent_tasks: 1 capabilities: gpu: false supported_models: - gpt-4o ``` #### Key Fields | Field | Description | | ------------------------------- | ------------------------------------------------------------------------------- | | `region` | Region label used for task requirement matching. | | `max_concurrent_tasks` | Maximum number of tasks this worker will claim simultaneously. Defaults to `1`. | | `capabilities.gpu` | Whether this worker has GPU access. | | `capabilities.supported_models` | Model identifiers this worker can serve. | ### How Workers Operate Workers connect to the Orloj server and participate in task assignment: 1. The worker registers itself with its capabilities. 2. The scheduler matches tasks to workers based on `Task.spec.requirements` (region, GPU, model). 3. When matched, the worker claims the task and acquires a time-bounded lease. 4. The worker renews the lease via heartbeats during execution. 5. If the lease expires (worker crash, network partition), another worker may safely take over. Workers can run as separate processes (`orlojworker`) or embedded in the server process (`orlojd --embedded-worker`) for single-process development. ### Status | Field | Description | | --------------- | --------------------------------------- | | `phase` | Worker lifecycle phase. | | `lastHeartbeat` | Timestamp of last heartbeat. | | `currentTasks` | Tasks currently claimed by this worker. | ### Related * [Task](../tasks/task.md) -- the work units that workers execute * [Resource Reference: Worker](../../reference/resources/worker.md) * [Deployment: Local](../../deploy/local.md) ## AgentPolicy An **AgentPolicy** sets execution constraints on agent systems or tasks. Policies can restrict model usage, block specific tools, and cap token consumption. ### Defining an AgentPolicy ```yaml apiVersion: orloj.dev/v1 kind: AgentPolicy metadata: name: cost-policy spec: apply_mode: scoped target_systems: - report-system max_tokens_per_run: 50000 allowed_models: - gpt-4o blocked_tools: - filesystem_delete ``` #### Key Fields | Field | Description | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `apply_mode` | `scoped` (default) applies only to listed targets. `global` applies to all systems/tasks. | | `target_systems` | AgentSystem names this policy applies to (when `scoped`). | | `target_tasks` | Task names this policy applies to (when `scoped`). | | `target_agents` | Agent names this policy applies to. When set, only listed agents are constrained; other agents in the same system are unaffected. When empty, the policy applies to all agents. | | `allowed_models` | Whitelist of permitted model identifiers. Agents configured with unlisted models are denied. | | `blocked_tools` | Tools that may not be invoked under this policy, regardless of agent permissions. | | `max_tokens_per_run` | Maximum token budget for a single task execution. When `target_agents` is set, acts as a per-agent budget for the listed agents rather than a system-wide total. | #### Per-Agent Budgets Use `target_agents` to set different token limits for different agents in the same system: ```yaml apiVersion: orloj.dev/v1 kind: AgentPolicy metadata: name: verdict-budget spec: apply_mode: scoped target_systems: - fraud-system target_agents: - verdict-agent max_tokens_per_run: 4000 --- apiVersion: orloj.dev/v1 kind: AgentPolicy metadata: name: analyst-budget spec: apply_mode: scoped target_systems: - fraud-system target_agents: - velocity-analyst - geo-risk-analyst - pattern-analyst max_tokens_per_run: 1500 ``` The verdict agent gets 4000 tokens per run while each analyst gets 1500. Agents not listed in any `target_agents` are only constrained by policies without `target_agents` (system-wide policies). ### How It Works AgentPolicy is the first check in the [authorization flow](./). When an agent selects a tool call, the runtime checks all applicable policies before evaluating role-based permissions: * If the tool is in `blocked_tools`, the call is immediately denied. * If the agent's model is not in `allowed_models`, execution is denied. * If `max_tokens_per_run` is exceeded, execution is stopped. A `global` policy applies to every execution. A `scoped` policy applies only to the listed `target_systems` and `target_tasks`. ### Related * [Governance Overview](./) -- how the three governance resources work together * [AgentRole](./agent-role.md) * [ToolPermission](./tool-permission.md) * [Resource Reference: AgentPolicy](../../reference/resources/agent-policy.md) ## AgentRole An **AgentRole** is a named set of permission strings. Agents bind to roles through their `spec.roles` field, which grants them the associated permissions. ### Defining an AgentRole ```yaml apiVersion: orloj.dev/v1 kind: AgentRole metadata: name: analyst-role spec: description: Can call web search style tools. permissions: - tool:web_search:invoke - capability:web.read ``` #### Permission String Conventions | Pattern | Meaning | | ------------------------------ | ---------------------------------------- | | `tool::invoke` | Permission to invoke a specific tool. | | `capability:` | Permission to use a declared capability. | ### How It Works An agent that binds multiple roles accumulates the union of all granted permissions: ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: research-agent-governed spec: model_ref: openai-default roles: - analyst-role - vector-reader-role tools: - web_search - vector_db ``` During the [authorization flow](./), the runtime collects permissions from all bound roles and checks them against the [ToolPermission](./tool-permission.md) requirements for the requested tool. Permissions are trimmed and deduplicated (case-insensitive) during normalization. ### Related * [Governance Overview](./) -- how the three governance resources work together * [AgentPolicy](./agent-policy.md) * [ToolPermission](./tool-permission.md) * [Resource Reference: AgentRole](../../reference/resources/agent-role.md) ## Governance and Policies Orloj provides a built-in governance layer that controls what agents can do at runtime. Three resource types work together to enforce authorization: **[AgentPolicy](./agent-policy.md)** constrains execution parameters, **[AgentRole](./agent-role.md)** grants named permissions to agents, and **[ToolPermission](./tool-permission.md)** defines what permissions are required to invoke a tool. Governance is fail-closed: if an agent uses roles and lacks the required permissions for a tool call, the call is denied with a `tool_permission_denied` error. ### Simple Path: `allowed_tools` For most agents, you can skip roles and ToolPermission entirely by listing tools in the agent's `spec.allowed_tools` field. Tools in this list are pre-authorized and bypass RBAC checks: ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: research-agent spec: model_ref: openai-default tools: - web_search - vector_db allowed_tools: - web_search - vector_db prompt: | You are a research assistant. ``` This agent can invoke both `web_search` and `vector_db` without any AgentRole or ToolPermission resources. `spec.tools` declares which tools the agent can select during execution; `spec.allowed_tools` declares which of those tools are pre-authorized. AgentPolicy constraints (like `blocked_tools` and `max_tokens_per_run`) still apply. `allowed_tools` only bypasses the role-based permission check. ### Advanced Path: Roles and ToolPermission For fine-grained access control, use [AgentRole](./agent-role.md) and [ToolPermission](./tool-permission.md) resources. This is recommended when you need per-tool permission auditing, scoped tool access across teams, or separation of duties between agent authors and platform operators. ### How Authorization Works When an agent selects a tool call during execution, the runtime evaluates authorization in this order: 1. **AgentPolicy check** -- Is the tool in the policy's `blocked_tools` list? If yes, deny. 2. **ToolPermission lookup** -- Find the ToolPermission for this tool and action. 3. **Permission matching** -- Collect the agent's permissions from all bound AgentRoles. Check them against `required_permissions` using the configured `match_mode`. 4. **Decision** -- If all checks pass, the tool is invoked. If any check fails, the call returns `tool_permission_denied`. ``` Agent selects tool call │ ▼ AgentPolicy check (blocked_tools?) │ ┌───┴───┐ │blocked │──► Denied └───┬───┘ │ allowed ▼ ToolPermission lookup │ ▼ Permission matching (agent roles vs required) │ ┌───┴───┐ │ fail │──► Denied (tool_permission_denied) └───┬───┘ │ pass ▼ Tool invoked ``` ### Approval Workflows Orloj now supports two approval layers: * [ToolApproval](./tool-approval.md): "may this tool call happen?" * [TaskApproval](./task-approval.md): "is this agent output or final task output acceptable to continue?" When a tool call is flagged as `approval_required` by a [ToolPermission](./tool-permission.md) operation rule, the task pauses and a [ToolApproval](./tool-approval.md) resource is created. When an `AgentSystem` review checkpoint is reached, the task pauses and a [TaskApproval](./task-approval.md) resource is created. In both cases the task moves to `WaitingApproval`. ### End-to-End Example To set up a governed agent that can search the web but not access the filesystem: **1. Define the role:** ```yaml apiVersion: orloj.dev/v1 kind: AgentRole metadata: name: analyst-role spec: description: Can call web search style tools. permissions: - tool:web_search:invoke - capability:web.read ``` **2. Define the tool permission:** ```yaml apiVersion: orloj.dev/v1 kind: ToolPermission metadata: name: web-search-invoke spec: tool_ref: web_search action: invoke match_mode: all required_permissions: - tool:web_search:invoke - capability:web.read ``` **3. Define the policy:** ```yaml apiVersion: orloj.dev/v1 kind: AgentPolicy metadata: name: cost-policy spec: apply_mode: scoped target_systems: - report-system-governed allowed_models: - gpt-4o blocked_tools: - filesystem_delete ``` **4. Bind the role to the agent:** ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: research-agent-governed spec: model_ref: openai-default roles: - analyst-role tools: - web_search - vector_db ``` In this configuration, `research-agent-governed` can invoke `web_search` (it holds the required permissions) but cannot invoke `vector_db` (it lacks `tool:vector_db:invoke`). Any attempt to call `filesystem_delete` is blocked by the policy regardless of permissions. ### Related * [AgentPolicy](./agent-policy.md) * [AgentRole](./agent-role.md) * [ToolPermission](./tool-permission.md) * [ToolApproval](./tool-approval.md) * [TaskApproval](./task-approval.md) * [Security and Isolation](../../operations/security.md) * [Guide: Set Up Multi-Agent Governance](../../guides/setup-governance.md) ## TaskApproval `TaskApproval` extends Orloj approvals from "may this tool call happen?" to "is this output safe and acceptable to continue?" Use it when you want a human to review: * a sensitive agent handoff before downstream agents continue * a regulated draft before publication or external delivery * a final task result before the task is marked `Succeeded` ### How It Works 1. An `AgentSystem` checkpoint is configured on a node (`spec.graph..review`) or on final completion (`spec.completion_review`). 2. The agent runs normally. 3. Orloj pauses the task in `WaitingApproval`, stores the exact blocker in `Task.status.blocked_on`, and creates a `TaskApproval`. 4. A reviewer chooses: * `approve`: resume the workflow * `deny`: fail the task * `request_changes`: rerun the same producing agent with reviewer feedback injected as `review.*` runtime input 5. If a rerun hits the same checkpoint again, Orloj creates a new `TaskApproval` with an incremented `review_cycle` and a `supersedes` link to the prior review. `request_changes` is optional per checkpoint. If `allow_request_changes` is set to `false`, reviewers can only approve or deny. When `max_review_cycles` is reached, Orloj rejects further `request_changes` decisions with `409 Conflict`. ### Checkpoint Configuration ```yaml apiVersion: orloj.dev/v1 kind: AgentSystem metadata: name: regulated-writer spec: agents: - writer-agent - compliance-agent graph: writer-agent: review: checkpoint_id: writer-review display_name: Writer Review reason: Human reviewer must approve the draft before compliance continues. ttl: 30m allow_request_changes: true max_review_cycles: 3 next: compliance-agent completion_review: checkpoint_id: publish-review reason: Final human signoff is required before the task is marked succeeded. ``` ### Review Context For `request_changes`, Orloj injects: * `review.feedback` * `review.previous_output` * `review.checkpoint_id` * `review.cycle` * `review.requested_by` That lets the same agent revise its output with concrete human guidance instead of starting from scratch. Reviewers send that feedback through `POST /v1/task-approvals/{name}/request-changes` or `orlojctl request-changes task-approval ...`. The request must include `comment` or the legacy `reason` field. ### When To Use It * Healthcare: review a triage note or patient-facing summary before delivery. * Finance: review a risk classification or client communication before release. * Insurance: review a claim denial rationale or settlement summary before sending. See also: * [ToolApproval](./tool-approval.md) * [Human Review Checkpoints guide](../../guides/human-review-checkpoints.md) * [TaskApproval resource reference](../../reference/resources/task-approval.md) ## ToolApproval A **ToolApproval** captures a pending human/system approval request for a tool invocation that was flagged by a [ToolPermission](./tool-permission.md) `operation_rules` verdict of `approval_required`. `ToolApproval` is about authorizing a tool action. If you need human review of agent output or final task output, use [TaskApproval](./task-approval.md). ### How the Approval Workflow Works When a tool call is flagged as `approval_required`: 1. The `GovernedToolRuntime` returns an `ErrToolApprovalRequired` sentinel error. 2. The task controller transitions the task to the `WaitingApproval` phase. 3. A `ToolApproval` resource is created with details about the pending call. 4. An external actor approves or denies the request via the API. 5. The task controller reconciles the approval status: * **Approved**: task resumes to `Running`. * **Denied**: task transitions to `Failed` with `approval_denied`. * **Expired** (TTL elapsed): task transitions to `Failed` with `approval_timeout`. Approval-related outcomes are non-retryable and do not consume retry budget. ### ToolApproval Fields ```yaml apiVersion: orloj.dev/v1 kind: ToolApproval metadata: name: db-write-approval-001 spec: task_ref: weekly-report tool: database_tool operation_class: write agent: research-agent input: '{"query": "INSERT INTO ..."}' reason: "Write operation requires human approval" ttl: 10m ``` | Field | Description | | ----------------- | --------------------------------------------------- | | `task_ref` | Name of the Task waiting for approval. | | `tool` | Tool name that triggered the approval request. | | `operation_class` | The operation class that requires approval. | | `agent` | Agent that attempted the tool call. | | `input` | Tool input payload (for audit context). | | `reason` | Human-readable reason for the approval request. | | `ttl` | Time-to-live before auto-expiry. Defaults to `10m`. | ### Status | Field | Description | | ------------ | ------------------------------------------- | | `phase` | `Pending`, `Approved`, `Denied`, `Expired`. | | `decision` | `approved` or `denied`. | | `decided_by` | Identity of the approver/denier. | | `decided_at` | Timestamp of the decision. | | `comment` | Optional reviewer comment. | | `expires_at` | Timestamp when the approval expires. | ### API Endpoints * `POST /v1/tool-approvals` -- create an approval request. * `GET /v1/tool-approvals` -- list approval requests. * `GET /v1/tool-approvals/{name}` -- get a specific approval. * `DELETE /v1/tool-approvals/{name}` -- delete an approval. * `POST /v1/tool-approvals/{name}/approve` -- approve a pending request. Body: `{"decided_by": "...", "comment": "..."}` (`comment` optional; `reason` remains a compatibility alias). * `POST /v1/tool-approvals/{name}/deny` -- deny a pending request. Body: `{"decided_by": "...", "comment": "..."}` (`comment` optional; `reason` remains a compatibility alias). ### Related * [ToolPermission](./tool-permission.md) -- defines which operations require approval * [TaskApproval](./task-approval.md) -- review task or agent output instead of tool execution * [Governance Overview](./) -- how the governance resources work together * [Resource Reference: ToolApproval](../../reference/resources/tool-approval.md) ## ToolPermission A **ToolPermission** defines what permissions are required to invoke a specific [Tool](../tools/tool.md). When an agent attempts to call a tool, the runtime checks the agent's accumulated permissions against the tool's ToolPermission. ### Defining a ToolPermission ```yaml apiVersion: orloj.dev/v1 kind: ToolPermission metadata: name: web-search-invoke spec: tool_ref: web_search action: invoke match_mode: all apply_mode: global required_permissions: - tool:web_search:invoke - capability:web.read ``` #### Key Fields | Field | Description | | ---------------------- | --------------------------------------------------------------------------- | | `tool_ref` | The tool this permission gate applies to. Defaults to `metadata.name`. | | `action` | The action being gated. Defaults to `invoke`. | | `match_mode` | `all` requires every listed permission. `any` requires at least one. | | `apply_mode` | `global` applies to all agents. `scoped` applies only to `target_agents`. | | `required_permissions` | Permission strings the agent must hold (via [AgentRoles](./agent-role.md)). | ### Operation Rules ToolPermissions can define per-operation-class verdicts using `operation_rules`: ```yaml spec: tool_ref: database_tool operation_rules: - operation_class: read verdict: allow - operation_class: write verdict: approval_required - operation_class: delete verdict: deny ``` | Verdict | Behavior | | ------------------- | ------------------------------------------------------------------------ | | `allow` | Proceed with the tool call (default). | | `deny` | Block the call with a `permission_denied` error. | | `approval_required` | Pause the task and create a [ToolApproval](./tool-approval.md) resource. | Operation classes are declared on the [Tool](../tools/tool.md) resource via `spec.operation_classes`. When multiple rules match, the most restrictive verdict wins: `deny` > `approval_required` > `allow`. ### Related * [Governance Overview](./) -- how the three governance resources work together * [AgentPolicy](./agent-policy.md) * [AgentRole](./agent-role.md) * [ToolApproval](./tool-approval.md) -- what happens when `approval_required` is triggered * [Resource Reference: ToolPermission](../../reference/resources/tool-permission.md) ## Agent Evaluation Orloj includes a built-in evaluation framework for measuring and comparing agent system quality. Define golden datasets as declarative YAML, run them against your agent systems, score the results with multiple strategies (programmatic, LLM-as-judge, or human review), and compare runs side-by-side. ### Why Evaluate? Model changes, prompt edits, tool updates, and graph topology changes can all silently degrade agent behavior. An evaluation framework lets you: * **Detect regressions** before they reach production by running golden datasets against every change. * **Compare configurations** (models, prompts, topologies) with objective metrics. * **Involve humans** in scoring subjective output quality via an export/review/import workflow. * **Track quality over time** with pass rates, mean scores, and latency trends. ### How It Works The evaluation framework introduces two resource kinds: | Resource | Purpose | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | | **[EvalDataset](../../reference/resources/eval-dataset.md)** | A list of (input, expected output) pairs with optional per-sample scoring rubrics. | | **[EvalRun](../../reference/resources/eval-run.md)** | A single evaluation execution: run a dataset against an agent system and collect scores. | The workflow is: ``` 1. Define a dataset ──► orlojctl apply -f dataset.yaml 2. Start an eval run ──► orlojctl eval run --dataset golden --system my-system 3. Controller creates ──► one Task per sample, respecting concurrency limits 4. Workers execute tasks ──► normal agent system execution 5. Scoring pipeline runs ──► exact_match, llm_judge, manual, or custom 6. Results aggregated ──► pass rate, mean score, latency, tokens 7. Compare runs ──► orlojctl eval compare run-a run-b ``` ### Scoring Strategies Each sample can be scored with one of four strategies: #### exact\_match Compares agent output against expected fields using the same matching logic as graph edge conditions: `output_contains`, `output_matches` (regex), `output_json_path` with comparison operators. Binary 0/1 score. ```yaml expected: output_contains: "billing" output_json_path: "$.category" equals: "billing" ``` #### llm\_judge Sends the input, agent output, and a rubric to a judge model. The judge returns a score (0.0–1.0), pass/fail verdict, and reasoning. This is a single model call per sample, not a full agent execution. ```yaml scoring: strategy: llm_judge model_ref: gpt-4o-judge rubric: "The response should correctly identify the customer's intent." ``` #### manual Tasks execute and outputs are collected, but no automated scoring occurs. The run transitions to **PendingReview** and results can be exported for human review (CSV or JSON), annotated via CLI or API, and then finalized. ```yaml scoring: strategy: manual ``` #### custom Invokes an external [Tool](../tools/tool.md) with the sample input, expected output, and actual output. The tool returns a JSON score object. This lets you plug in any custom scoring logic. ```yaml scoring: strategy: custom tool_ref: my-custom-scorer ``` ### EvalRun Lifecycle ``` Pending ──► Running ──► Scoring ──► Succeeded └──► PendingReview ──► Succeeded (after finalize) └──► Failed └──► Cancelled ``` | Phase | Meaning | | --------------- | ------------------------------------------------------------------ | | `Pending` | Run is created, waiting for the controller to start task creation. | | `Running` | Tasks are being created and executed. | | `Scoring` | All tasks completed; the scoring pipeline is running. | | `PendingReview` | Manual scoring: waiting for human annotations. | | `Succeeded` | All scoring complete; summary metrics computed. | | `Failed` | The eval run encountered a fatal error (e.g., missing dataset). | | `Cancelled` | Cancelled by user; in-flight tasks are also cancelled. | ### Manual Review Workflow When using `manual` scoring (or a mix where some samples use it): 1. **Tasks execute** normally and outputs are collected. 2. The run enters **PendingReview** once all tasks complete. 3. **Export** results for review: ```bash orlojctl eval export my-run --format csv > results.csv ``` 4. **Annotate** individual samples: ```bash orlojctl eval annotate my-run --sample billing-q --score 0.8 --pass --comment "Good" ``` 5. **Bulk import** from a reviewed CSV: ```bash orlojctl eval import my-run -f reviewed.csv ``` 6. **Finalize** to compute aggregates and transition to Succeeded: ```bash orlojctl eval finalize my-run ``` ### Comparing Runs The comparison API and CLI show side-by-side metrics across multiple runs: ```bash orlojctl eval compare run-gpt4o run-claude run-gemini ``` ``` METRIC run-gpt4o run-claude run-gemini Pass Rate 88.0% 85.0% 82.0% Mean Score 0.910 0.890 0.850 Tokens 13100 11800 12200 ``` This is the primary tool for A/B testing model changes, prompt revisions, or topology experiments. Use `agent_overrides` in the EvalRun spec to swap prompts or models without modifying the base agent resources. ### Related * [EvalDataset reference](../../reference/resources/eval-dataset.md) -- full spec documentation * [EvalRun reference](../../reference/resources/eval-run.md) -- full spec documentation * [Guide: Run Your First Agent Evaluation](../../guides/run-agent-evaluation.md) -- step-by-step tutorial * [CLI: `orlojctl eval`](../../reference/cli.md) -- command reference ## AgentSystem An **AgentSystem** composes multiple [Agents](./agent.md) into a directed graph that Orloj executes as a coordinated workflow. The graph defines how messages flow between agents during task execution. Agent systems can also declare human review checkpoints: * `spec.graph..review`: review a node's output before downstream routing continues * `spec.completion_review`: review the final task output before the task is marked `Succeeded` ### Defining an AgentSystem ```yaml apiVersion: orloj.dev/v1 kind: AgentSystem metadata: name: report-system labels: orloj.dev/domain: reporting orloj.dev/usecase: weekly-report spec: agents: - planner-agent - research-agent - writer-agent graph: planner-agent: next: research-agent research-agent: next: writer-agent ``` ### Graph Topologies The `graph` field supports three fundamental patterns: **Pipeline** -- sequential stage-by-stage execution where each agent hands off to the next. ```yaml graph: planner-agent: edges: - to: research-agent research-agent: edges: - to: writer-agent ``` **Hierarchical** -- a manager delegates to leads, who delegate to workers, with a join gate that waits for all branches before proceeding. ```yaml graph: manager-agent: edges: - to: research-lead-agent - to: social-lead-agent research-lead-agent: edges: - to: research-worker-agent social-lead-agent: edges: - to: social-worker-agent research-worker-agent: edges: - to: editor-agent social-worker-agent: edges: - to: editor-agent editor-agent: join: mode: wait_for_all ``` **Swarm with loop** -- parallel scouts report back to a coordinator in iterative cycles, bounded by `Task.spec.max_turns`. ```yaml graph: coordinator-agent: edges: - to: scout-alpha-agent - to: scout-beta-agent - to: synthesizer-agent scout-alpha-agent: edges: - to: coordinator-agent scout-beta-agent: edges: - to: coordinator-agent ``` ### Fan-out and Fan-in When a graph node has multiple outbound edges, messages fan out to all targets in parallel. Fan-in is handled through join gates: | Join Mode | Behavior | | -------------- | --------------------------------------------------------------------------------- | | `wait_for_all` | Waits for every upstream branch to complete before activating the join node. | | `quorum` | Activates after `quorum_count` or `quorum_percent` of upstream branches complete. | If an upstream branch fails, the `on_failure` policy determines behavior: `deadletter` (default), `skip`, or `continue_partial`. ### Human Review Checkpoints Attach a review checkpoint to a graph node when a human must inspect that output before the workflow continues. ```yaml graph: writer-agent: review: checkpoint_id: writer-review reason: Editor must approve the draft before compliance runs. ttl: 30m allow_request_changes: true max_review_cycles: 3 next: compliance-agent completion_review: checkpoint_id: final-review reason: Final human signoff before success. ``` When a checkpoint is reached, Orloj creates a `TaskApproval`, pauses the task in `WaitingApproval`, and records the exact blocker in `Task.status.blocked_on`. If `allow_request_changes` is `false`, reviewers can only approve or deny that checkpoint. If a reviewer keeps sending work back, `max_review_cycles` caps the number of rerun loops before Orloj rejects additional `request_changes` decisions. ### Conditional Routing Edges can carry a `condition` that is evaluated against the completing agent's output. Only edges whose condition matches will fire. This enables data-dependent graph routing where agents decide which downstream agents should run. ```yaml graph: classifier-agent: edges: - to: billing-agent condition: output_contains: "BILLING" - to: tech-agent condition: output_contains: "TECH" - to: general-agent condition: default: true ``` #### Condition Operators **String-based operators** -- evaluate against the raw output text: | Operator | Behavior | | --------------------- | ----------------------------------------------------------------------------- | | `output_contains` | Matches if the agent's output contains the string (case-insensitive). | | `output_not_contains` | Matches if the agent's output does NOT contain the string (case-insensitive). | | `output_matches` | Matches if the agent's output matches the regex pattern. | | `default` | Marks the edge as a fallback — fires only when no conditional edge matches. | **JSON path operators** -- extract a value from JSON output and compare: | Operator | Behavior | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | `output_json_path` | Dot-notation path (e.g. `$.route`, `$.result.category`) to extract from JSON output. Required when using comparison operators. | | `equals` | Matches when the extracted value equals this string. | | `not_equals` | Matches when the extracted value does NOT equal this string. | | `contains` | For arrays: matches when any element equals this value. For strings: matches on substring (case-insensitive). | | `greater_than` | Matches when the extracted numeric value is greater than this threshold. | | `less_than` | Matches when the extracted numeric value is less than this threshold. | JSON path operators require the agent's output to be valid JSON. When `output_json_path` is set, at least one comparison operator (`equals`, `not_equals`, `contains`, `greater_than`, `less_than`) must also be set. If the output is not valid JSON or the path does not exist, the condition evaluates to false. #### Evaluation Rules * Edges **without** a condition are unconditional and always fire (backward-compatible). * When any edges have conditions: conditional edges are evaluated first. If one or more match, only matched edges (plus any unconditional edges) fire. * If no conditional edge matches, `default: true` edges fire (plus unconditional edges). * If no conditional edge matches and no default exists, the task completes at this node. * At most one `default` edge is allowed per source node. * A `default` edge must not combine with other condition fields. * Multiple condition fields on the same edge are combined with AND logic. #### Patterns **Triage / routing** -- a classifier agent routes to the right specialist: ```yaml graph: intake-agent: edges: - to: refund-agent condition: output_contains: "REFUND" - to: support-agent condition: output_contains: "SUPPORT" - to: general-agent condition: default: true ``` **Quality gates** -- skip expensive downstream stages when not needed: ```yaml graph: screener-agent: edges: - to: deep-research-agent condition: output_contains: "VIABLE" - to: rejection-agent condition: output_contains: "REJECT" ``` **Intelligent hierarchical delegation** -- a manager activates only the leads that are needed, saving cost and time: ```yaml graph: manager-agent: edges: - to: research-lead condition: output_contains: "NEEDS_RESEARCH" - to: engineering-lead condition: output_contains: "NEEDS_ENGINEERING" - to: legal-lead condition: output_contains: "NEEDS_LEGAL" research-lead: edges: - to: editor-agent engineering-lead: edges: - to: editor-agent legal-lead: edges: - to: editor-agent editor-agent: join: mode: wait_for_all ``` When the manager only activates research-lead and legal-lead, the editor's `wait_for_all` join gate automatically adjusts to wait for 2 branches instead of 3. **Structured JSON routing** -- combine `output_schema` with JSON path conditions for reliable, typed routing: ```yaml graph: classifier-agent: edges: - to: research-lead condition: output_json_path: "$.route" equals: "research" - to: legal-lead condition: output_json_path: "$.domains" contains: "legal" - to: high-priority-agent condition: output_json_path: "$.confidence" greater_than: "0.9" - to: general-agent condition: default: true ``` This pairs naturally with `output_schema` on the classifier agent to guarantee valid JSON output. See [Structured Output](#structured-output) below. **Iterative refinement with exit conditions** -- a review loop that terminates on quality, not just a turn counter: ```yaml graph: writer-agent: edges: - to: critic-agent critic-agent: edges: - to: writer-agent condition: output_contains: "REVISION_NEEDED" - to: publisher-agent condition: output_contains: "APPROVED" ``` Conditional routing requires **message-driven** execution mode (`--task-execution-mode=message-driven`). ### Structured Output Agents can declare an `output_schema` in their execution spec to constrain model responses to valid JSON matching a specific schema. This uses provider-native structured output features (constrained decoding) for guaranteed schema compliance. ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: classifier-agent spec: model_ref: openai-gpt4 prompt: Classify the incoming request. execution: output_schema: type: object properties: route: type: string enum: [research, engineering, legal] confidence: type: number domains: type: array items: type: string required: [route, confidence, domains] additionalProperties: false ``` #### Provider Support | Provider | Structured Output | Mechanism | | ----------------- | ----------------------- | --------------------------------------------------------------------- | | OpenAI | Full schema enforcement | `response_format.json_schema` with constrained decoding | | OpenAI-compatible | Full schema enforcement | Same as OpenAI | | Azure OpenAI | Full schema enforcement | Same as OpenAI | | Anthropic | Full schema enforcement | `output_config.format` with constrained decoding | | Ollama | Best-effort JSON | `format` field with schema; enforcement depends on model capabilities | #### Combining with JSON Path Conditions Structured output and JSON path conditions are designed to work together. The `output_schema` guarantees the agent produces valid, typed JSON, and `output_json_path` conditions route on the structured fields: ```yaml # Agent definition apiVersion: orloj.dev/v1 kind: Agent metadata: name: classifier-agent spec: model_ref: openai-gpt4 prompt: Classify the request and assess confidence. execution: output_schema: type: object properties: route: type: string enum: [research, engineering, legal] confidence: type: number required: [route, confidence] additionalProperties: false --- # AgentSystem graph routing on structured output graph: classifier-agent: edges: - to: research-lead condition: output_json_path: "$.route" equals: "research" - to: engineering-lead condition: output_json_path: "$.route" equals: "engineering" - to: legal-lead condition: output_json_path: "$.route" equals: "legal" ``` ### Delegation The `delegates` field on a graph node gives an agent two-phase execution: **dispatch** to downstream agents, **collect** their reports, **review** with all results, and **forward** onward. ```yaml graph: vp-engineering: delegates: - to: backend-lead condition: output_json_path: "$.teams" contains: "backend" - to: security-lead condition: output_json_path: "$.risk_level" equals: "high" delegate_join: mode: wait_for_all edges: - to: synthesizer ``` #### How Delegation Works 1. **Dispatch phase**: The node executes. Its output filters the `delegates` list (same condition logic as edges). Matched delegates receive messages with `delegate_of` metadata pointing back to the delegator. 2. **Collect phase**: Delegates execute and report back. A delegation gate collects returns using the same join modes as regular fan-in (`wait_for_all`, `quorum`). 3. **Review phase**: Once enough delegates return, the node **re-executes** with all delegate outputs in context via `inbox.delegation.*` keys. 4. **Forward phase**: After the review execution, normal `edges` fire for onward routing. #### Automatic Return Routing When a delegate-activated agent reaches a terminal point (no outgoing edges match), the runtime automatically routes back to the delegator. The `delegate_of` field propagates through sub-edges, so multi-hop delegate branches still return to the correct delegator. * **Simple delegate** (leaf node): executes once, reports back * **Delegate with edges**: follows its own edges; the terminal node reports back * **Delegate with its own delegates** (nested): inner delegation completes first, then the delegate reviews and reports back #### Agent Context The agent sees delegation context through `inbox.*` keys: * **Dispatch phase**: Normal `inbox.from`, `inbox.content` — no delegation keys * **Review phase**: `inbox.delegation.enabled: true`, `inbox.delegation.mode`, `inbox.delegation.sources`, `inbox.delegation.payloads` #### Delegation Join Modes `delegate_join` supports the same modes as regular join gates: | Mode | Behavior | | -------------- | ------------------------------------------------------------------------ | | `wait_for_all` | Re-execute after every dispatched delegate returns. | | `quorum` | Re-execute after `quorum_count` or `quorum_percent` of delegates return. | #### Patterns **Hierarchical company** -- CEO delegates to VPs, VPs delegate to leads, each manager gets dispatch + review: ```yaml graph: ceo-agent: delegates: - to: vp-engineering condition: output_json_path: "$.departments" contains: "engineering" - to: vp-product condition: output_json_path: "$.departments" contains: "product" delegate_join: mode: wait_for_all edges: - to: board-report-agent vp-engineering: delegates: - to: backend-lead - to: security-lead delegate_join: mode: wait_for_all ``` **Fast-response quorum** -- review fires after the first 2 of 3 analysts complete: ```yaml graph: research-manager: delegates: - to: analyst-a - to: analyst-b - to: analyst-c delegate_join: mode: quorum quorum_count: 2 ``` **Node with both join and delegates** -- composes correctly. The join gate fires first (collecting upstream), then the dispatch phase, then the delegation gate, then the review phase, then edges. ### Labels Labels on AgentSystem metadata follow Kubernetes conventions and are useful for filtering, governance scoping, and operational grouping: ```yaml metadata: labels: orloj.dev/domain: reporting orloj.dev/usecase: weekly-report orloj.dev/env: dev ``` ### Related * [Agent](./agent.md) -- the individual agents that compose a system * [Task](../tasks/task.md) -- how to execute an AgentSystem * [Resource Reference: AgentSystem](../../reference/resources/agent-system.md) * [Execution and Messaging](../execution-model.md) * [Starter Blueprints](../../guides/starter-blueprints.md) ## Agent An **Agent** is a declarative unit of work backed by a language model. It defines what the agent does (its prompt), what model powers it, what tools it can call, and what constraints bound its execution. ### Defining an Agent ```yaml apiVersion: orloj.dev/v1 kind: Agent metadata: name: research-agent spec: model_ref: openai-default prompt: | You are a research assistant. Produce concise evidence-backed answers. tools: - web_search - vector_db memory: ref: research-memory roles: - analyst-role limits: max_steps: 6 timeout: 30s ``` #### Key Fields | Field | Description | | ------------------ | ----------------------------------------------------------------------------------------------------------- | | `model_ref` | Required reference to a [ModelEndpoint](../tools/model-endpoint.md) resource for provider-aware routing. | | `prompt` | The system instruction that defines the agent's behavior. | | `tools` | List of [Tool](../tools/tool.md) names this agent may call. Tool calls are subject to governance checks. | | `roles` | Bound [AgentRole](../governance/agent-role.md) names. Roles carry permissions that authorize tool usage. | | `memory.ref` | Reference to a [Memory](../memory/) resource. This attaches the memory backend to the agent. | | `memory.allow` | Explicit list of built-in memory operations the agent may use: `read`, `write`, `search`, `list`, `ingest`. | | `limits.max_steps` | Maximum execution steps per task turn. Defaults to `10`. | | `limits.timeout` | Maximum wall-clock time per task turn. | ### How an Agent Executes When the runtime activates an agent during a task, it: 1. Initializes the agent's conversation history with the system prompt and current task context. 2. If `memory.ref` is set, wires the backing memory store into the runtime. If `memory.allow` is also set, the runtime exposes only those built-in memory operations as available tools. 3. Routes the request to the configured model via the model gateway, sending the full conversation history. 4. If the model selects tool calls, the runtime checks governance (AgentPolicy, AgentRole, ToolPermission) and executes authorized tools. Memory tool calls are handled internally without network calls. Tool results are sent back using the provider's native structured tool protocol (`role: "tool"` with `tool_call_id` for OpenAI, `tool_result` content blocks for Anthropic). 5. Results are appended to the conversation history and sent back to the model for the next step. The agent completes when the model produces text output without requesting further tools, or when `max_steps` / `timeout` is reached. Already-called tools are removed from the available list to prevent duplicate calls. Conversation history is maintained for the full duration of the agent's activation, giving the model continuity across reasoning and tool-use steps. See [Memory](../memory/) for details on memory layers and built-in tools. ### Related * [AgentSystem](./agent-system.md) -- compose agents into directed graphs * [Resource Reference: Agent](../../reference/resources/agent.md) * [Memory](../memory/) * [Guide: Deploy Your First Pipeline](../../guides/deploy-pipeline.md)