Yuno lifecycle under the agent

This document describes what the Yuneta agent (yuno_agent) actually does when a yuno is created, started, paused, killed, updated or deleted on a host. It is the on-disk + in-memory + on-the-wire picture, not the marketing one.

Yuno = binary + configuration. Those two halves are stored independently, versioned independently, and linked by the yuno record. Understanding that is the prerequisite to understanding everything else here.

The agent’s authoritative source is src/c_agent.c (~11k lines) and its treedb schema src/treedb_schema_yuneta_agent.c. Every claim below cites the file:line it comes from.

1. Mental model¶

                         ┌───────────────────────┐
                         │       yuno_agent      │
                         │   (one per host)      │
                         └───────────┬───────────┘
                                     │ owns
                                     ▼
                         ┌───────────────────────┐
                         │  treedb (timeranger2) │
                         └───┬───────┬───────┬───┘
                             │       │       │
              ┌──────────────┘       │       └───────────────┐
              ▼                      ▼                       ▼
       ┌─────────────┐       ┌─────────────────┐      ┌─────────────┐
       │  binaries   │       │ configurations  │      │    yunos    │
       │  (per role  │       │ (per role+name  │      │ (registered │
       │  + version) │       │   + version,    │      │  instances) │
       │             │       │  blob zcontent) │      │             │
       └──────┬──────┘       └────────┬────────┘      └──────┬──────┘
              │                       │                      │
   resolves   │            materialised at start             │  references:
   yuno_role  │            into JSON files inside            │   role+role_version
   + role_    │            <realm>/<yuno>/bin/                │   role+name+name_version
   version    │                                              │
              ▼                                              ▼
   /yuneta/repos/<tags>/                          /yuneta/realms/<owner>/
   <role>/<version>/<role>                        <realm>/<yuno>/{bin,logs}

The agent is the only thing that should touch those directories. Manual edits create drift with treedb that won’t show up until the next start fails.

2. Data model¶

2.1 The binaries topic¶

Defined in treedb_schema_yuneta_agent.c:499-604. Composite key:

Multiple versions per role coexist. The yuno record decides which version is used (see §2.3).

On-disk path, built by yuneta_repos_yuno_dir() (c_agent.c:7088) and yuneta_repos_yuno_file() (c_agent.c:7109):

<yuneta_root_dir>/repos/<tags>/<role>/<version>/<role>

The file inside is the executable; the filename is the role again.

2.2 The configurations topic¶

Defined in treedb_schema_yuneta_agent.c:607-674. Composite key:

The blob lives in treedb. At start time the agent materialises it to disk under the yuno’s bin/ directory and hands the path to the binary via --config-file='[<paths>]'. The materialisation site is c_agent.c:7722-7735 and 7857-7868; the launcher line is c_agent.c:8025.

2.3 The yunos topic¶

Defined in treedb_schema_yuneta_agent.c:275-495. Per-yuno record. Important columns:

A yuno record without a matching binaries row or configurations row will fail at create time, not at start time (c_agent.c:4466, 4487-4492).

2.4 Realm and per-yuno layout¶

Built by build_yuno_private_domain() (c_agent.c:7135):

<yuneta_root_dir>/realms/<realm_owner>/<realm_name>.<realm_role>.<realm_env>/<yuno_role>_<yuno_name>/
  ├── bin/        ← N-<role>_<name>.json (materialised configs)
  └── logs/       ← N.log files

The bin/ directory is not the binary. It is the working dir the yuno gets config files from. The actual binary lives in /yuneta/repos/ (see §2.1).

3. Command inventory¶

Registered in the agent’s command table at c_agent.c:806-900. Yuno + binary + config commands only (admin, realm, certs and console commands omitted):

Binaries¶

Configurations¶

Yunos¶

Permission gating is per-command via pm_<name> schemas (c_agent.c:849-898).

4. Lifecycle, step by step¶

4.1 State machine of a single yuno (as the agent sees it)¶

                                  delete-yuno
                              ┌─────────────────────┐
                              │  (only if stopped)  │
                              ▼                     │
                          ┌────────┐               ┌┴────────────┐
   create-yuno  ─────────►│ STOPPED│◄──────────────┤  DELETED    │
                          └────┬───┘  EV_ON_CLOSE  └─────────────┘
                               │                        ▲
                               │                        │
                  run-yuno     │                        │ kill-yuno
                  (fork+exec   │                        │ (SIGQUIT)
                   + handshake)│                        │
                               ▼                        │
                          ┌────────┐                    │
                          │STARTING│                    │
                          │(pid    │                    │
                          │ alive, │                    │
                          │ no chan│                    │
                          │ yet)   │                    │
                          └────┬───┘                    │
                               │  EV_ON_OPEN            │
                               │  (yuno reports pid +   │
                               │   watcher_pid)         │
                               ▼                        │
                          ┌────────┐    EV_PAUSE_YUNO   │
                          │RUNNING ├────────────────┐   │
                          │playing │                │   │
                          │=true   │◄──────┐        │   │
                          └───┬────┘       │ ACK    ▼   │
                              │            │   ┌────────┴┐
                              └────────────┘   │ RUNNING │
                                               │ paused  │
                       EV_PLAY_YUNO_ACK        │ =true   │
                                               └─────────┘

Note: the agent itself stays in ST_IDLE always (c_agent.c:11015). It is the event types (c_agent.c:11022-11054) that drive transitions on the per-yuno record fields (yuno_running, yuno_playing, yuno_pid).

4.2 Registration: create-yuno¶

c_agent.c:4427. Requires realm + binary (yuno_role, role_version) + config (yuno_role.yuno_name, name_version) to already exist. Defaults missing versions to latest. Writes the row with yuno_running=false, yuno_disabled=false, yuno_pid=0 (c_agent.c:4813-4814).

4.3 Start: run-yuno¶

c_agent.c:4805 → run_yuno() at c_agent.c:7943.

1. Select yunos: disabled=false ∧ running=false (c_agent.c:8362-8363).

2. Resolve binary via get_yuno_binary() (c_agent.c:7275): prefer active snapshot (gobj_list_snaps(), c_agent.c:7279-7284), fallback to direct (role, role_version) lookup (c_agent.c:7291-7297).

3. Materialise each configurations.zcontent blob to a JSON file under the yuno’s bin/ directory (c_agent.c:7722-7735, 7857-7868).

4. Build a launcher shell script that runs the binary with:

   <binary> --config-file='["bin/1-role_name.json", "bin/2-role_name.json", …]' "$@"

   (c_agent.c:8025).

5. run_process2(bfbinary, argv) (c_agent.c:7995) — fork/exec via a wrapper.

6. The yuno opens a channel back to the agent and emits EV_ON_OPEN carrying its pid and watcher_pid (handler ac_on_open(), c_agent.c:10233). Only after this handshake is yuno_running set to true (c_agent.c:10408) and yuno_pid / watcher_pid stored (c_agent.c:10410-10411).

7. If must_play=true, the agent fires play-yuno automatically right after the open (ac_on_open()), unless the command was issued with play=0 (see below).

Implication: a yuno that forks fine but never opens the channel never becomes running from the agent’s point of view, even if ps shows the process alive. See §5 Stale pid.

Command response: one answer per command, and the play=0 knob¶

run-yuno returns a single command answer once all launched yunos have connected back — like kill-yuno/pause-yuno/play-yuno. It aggregates the per-yuno EV_ON_OPEN ACKs into one C_COUNTER (max_count=total) created after the launch loop, exactly mirroring the other three commands.

The implicit auto-play of step 7, however, is inherently per-yuno and async (each yuno connects at its own time), so in the default play=1 mode the caller also sees one extra play-yuno answer per must_play yuno. Scripts that need exactly one answer per command should split the two phases:

ycommand -c 'run-yuno play=0'   # launch only → 1 answer ("N yunos found to run")
ycommand -c 'play-yuno'         # play already-running yunos → 1 aggregated answer

play=0 (default 1, backward-compatible) suppresses the auto-play for that launch only. The agent records the launch_id in an in-memory set (priv->no_play_launches); ac_on_open() consumes it by matching the connecting yuno’s identity_cardlaunch_idand deletes it on first connect. It is not a treedb column and does not touchmust_play, so a watcher crash relaunch (which reuses the same launch_id, now absent) still reconciles must_play` per §4.6.

4.4 Pause / Play¶

These are not process signals. They are gobj events delivered through the yuno’s open channel:

• play-yuno → EV_PLAY_YUNO → yuno does whatever “playing” means for it → EV_PLAY_YUNO_ACK → agent sets yuno_playing=true (c_agent.c:8156).

• pause-yuno → EV_PAUSE_YUNO → … → EV_PAUSE_YUNO_ACK → agent sets yuno_playing=false (c_agent.c:8192).

Most yunos use the play/paused gate to enable/disable I/O processing without exiting. The process never stops; only its inputs are gated.

4.5 Stop: kill-yuno¶

c_agent.c:4998 → kill_yuno() at c_agent.c:8040. Orderly shutdown, not a SIGKILL.

1. Read the signal2kill attribute (default SIGQUIT, c_agent.c:8045-8048).

2. kill(yuno_pid, signal2kill) (c_agent.c:8084).

3. If the chosen signal is SIGKILL, the watcher is killed too (c_agent.c:8104-8127).

4. No timer-based escalation in code. The agent trusts the yuno’s signal handler to actually shut down. If it doesn’t, the yuno stays “running” from the agent’s record forever (see §5).

5. When the channel closes, ac_on_close() (c_agent.c:10501) flips yuno_running=false, yuno_playing=false, yuno_pid=0 (c_agent.c:10563-10565).

4.6 Crash detection and reconciliation¶

From the agent’s point of view, a crash is indistinguishable from a clean kill: EV_ON_CLOSE fires, ac_on_close() runs, the treedb row flips yuno_running=false. The agent has no SIGCHLD handler and does not poll pids. It does not know “exited normally” from “segfaulted”.

The restart, however, does not depend on the agent. Every yuno started with --start runs under a per-yuno watcher process (the first-fork survivor in ydaemon.c). When the yuno child dies abnormally (any signal other than SIGKILL, or any non-zero exit code), the watcher sleeps 2s and re-execs the same binary — and it does so regardless of whether the agent is running. A yuno is an autonomous machine; the watcher is what makes that concrete. See ENTRY_POINT.md §4 for the full decision matrix.

What the agent contributes on top:

• It logs the closed channel and clears yuno_pid/yuno_playing. A fresh run-yuno is not needed; the watcher has already spawned a new child with a new pid that will reconnect to the agent on its own (it goes through the normal EV_ON_OPEN handshake).

• At agent boot the timer ac_timeout() (c_agent.c:10902) runs:

  1. run_util_yunos() (c_agent.c:8389) — yunos tagged util, ignoring disabled.

  2. run_enabled_yunos() (c_agent.c:8331) — every row with disabled=false ∧ running=false.

  This reconciliation matters only for yunos that don’t have a live watcher (e.g. the agent itself was killed and brought back; any yunos whose own watchers happened to die too).

Forensics: a crashed yuno also dumps a core at /var/crash/core.<role> (sysctl + PAM limits configured by the .deb, see ENTRY_POINT.md §8). The watcher emits a Daemon relaunched log line on every relaunch (MSGSET_SYSTEM) — grep for it to spot silent crash loops.

4.7 Deletion: delete-yuno¶

c_agent.c:4686. Refuses if yuno_running=true (c_agent.c:4721-4735). Optional refusal on tagged yunos unless force=1 (c_agent.c:4738-4752). Removes the treedb row; the on-disk bin/ directory is cleaned by treedb cascade.

5. Sharp edges (read these before touching production)¶

5.1 update-binary fails while the yuno is running (text-file-busy)¶

update-binary (c_agent.c:3234) base64-decodes content64 to /yuneta/realms/agent/agent/temp/<role>, then copies it over /yuneta/repos/<tags>/<role>/<version>/<role>. The agent execs running yunos directly from that repos path, so if a process is running that exact file the copy is refused by the kernel (ETXTBSY) and the command returns:

ERROR -1: Cannot copy '/yuneta/realms/agent/agent/temp/<role>'
          to '/yuneta/repos/.../<version>/<role>'

It does not corrupt the live process — Linux simply won’t let you overwrite a busy executable. The command’s description still says “WARNING: Don’t use in production!” (c_agent.c:850).

So the same-version hot-patch order is mandatory: kill-yuno <role> FIRST, then update-binary, then run-yuno. ($$(<role>) in ycommand reads the freshly-built binary from $YUNETAS_YUNOS = outputs/yunos/<role>, so a plain make install is enough to stage the new build.) For a real release, prefer install-binary with a bumped version.

Note — list-binaries shows the binary IN USE (the primary node per role). A runtime update-binary (same version) mutates that primary node in place, so the new size/date appear immediately. An install-binary of a NEW version does not change what list-binaries shows until deactivate-snap promotes and reloads it — which is correct: the new binary is not in use until then. To see every installed (role, version) from the moment it lands, use list-binaries-instances (gobj_list_instances, the pkey2 iterator refreshed at runtime by dbf532ec9).

5.2 Stale yuno_running=true after a hard crash¶

If the yuno dies in a way that leaves the channel open (rare, but SIGKILL-from-outside is one of them) ac_on_close() is never invoked. The record keeps yuno_running=true and yuno_pid=<old-pid>.

On the next run-yuno, ac_on_open() checks getpgid(_pid) >= 0 (c_agent.c:10304). If the old pid happens to have been reused, the agent kills the new occupant (c_agent.c:10318). This is the worst flavour of flapping. If you suspect a stale pid, manually clear yuno_running and yuno_pid in treedb before retrying.

5.3 No SIGKILL escalation, and the watcher gotcha¶

kill-yuno sends one signal and waits. There’s no “after 30 seconds, try SIGKILL”. A yuno that swallows SIGQUIT (or whose signalfd handler is wedged) stays alive and the agent’s record stays stuck in running.

Worse, a naive kill -9 <yuno_pid> from the shell does not kill the yuno permanently: the watcher classifies SIGKILL on the child as “abnormal” and relaunches after 2s. You need to kill both the child and its watcher, or — better — use kill-yuno force=1 / set-quick-kill, which sends SIGKILL to both pids and is the only way the agent can actually take down an uncooperative yuno (c_agent.c:8104-8127, and ENTRY_POINT.md §7).

5.4 pause ≠ SIGSTOP, play ≠ SIGCONT¶

Pause/Play are channel events. The process is never frozen at the kernel level. If you actually want the process suspended (e.g. for gdb attach), the agent gives you nothing — use the shell.

5.5 update-config does not hot-reload¶

c_agent.c:3744 updates the treedb blob. The launcher script is rebuilt only at run-yuno. So changes take effect on next start. There is no re-materialisation of the JSON files on disk for a running yuno.

5.6 disable-yuno does not stop a running yuno¶

c_agent.c:5601 only flips the flag. A disabled=true yuno that was already running keeps running until you kill-yuno. The flag only prevents the next run-yuno from picking it up.

5.7 install-binary vs update-binary¶

Pick install-binary for new versions. Pick update-binary only when you truly want to overwrite — and then only if no yuno is running that version.

6. Operational recipes¶

All examples assume ycommand is talking to the local agent.

Bulk reconciliation. To compare every binary the agent has installed against the freshly built ones in outputs/yunos and push the differences in one pass, use tools/agent/sync_binaries.py (drives from the agent’s installed set; proposes install-binary for version bumps and update-binary for same-version rebuilds; -n for a dry run). The recipes below are the manual, per-yuno equivalents.

6.1 Onboard a brand-new yuno¶

# 1. install the binary (new role or new version)
ycommand -c 'install-binary content64=$$(my_role)'

# 2. install/upsert its configuration
ycommand -c 'update-config id=my_role.my_name version=1 zcontent=$$(my_role_my_name.json)'

# 3. create the yuno record (links binary + config to a realm)
ycommand -c 'create-yuno realm_id=<realm> yuno_role=my_role yuno_name=my_name'

# 4. enable, launch, then play (two steps = one response each)
ycommand -c 'enable-yuno id=<yuno_id>'
ycommand -c 'run-yuno play=0 id=<yuno_id>'
ycommand -c 'play-yuno id=<yuno_id>'

# 5. verify
ycommand -c 'list-yunos'

6.2 Hot-patch the binary at the same version (update-binary)¶

Use this when the version number in main.c (APP_VERSION) is unchanged — e.g. a RelWithDebInfo rebuild for a quick fix during a debugging session. update-binary overwrites the existing {role}/{version}/ slot in /yuneta/repos/... and the matching treedb row. There is no rollback path; the previous bytes are gone.

Always orderly shutdown first; never update-binary over a live mmap. The command itself does NOT refuse if a yuno using the binary is running (see §5.1).

# 1. build (APP_VERSION unchanged in main.c)
cd /yuneta/development/yunetas/yunos/c/<yuno>/build && make clean && make install

# 2. orderly shutdown via the agent (NOT a manual kill)
ycommand -c 'kill-yuno yuno_role=<role>'

# 3. wait until it really left
ycommand -c 'list-yunos'    # expect yuno_running=false

# 4. overwrite the same-version slot in the agent repo
ycommand -c 'update-binary id=<role> content64=$$(<role>)'
ycommand -c 'list-binaries' # verify size/date

# 5. launch back, then play (two steps = one response each)
ycommand -c 'run-yuno play=0'
ycommand -c 'play-yuno'

For a real version bump (1.3.1.0 → 1.3.1.1, 7.3.4 → 7.4.0, …) use the upgrade flow in §6.5 instead — update-binary is the wrong tool, and the WARNING: Don't use in production! description in command-yuno help applies precisely to that misuse.

6.3 Change a yuno’s configuration¶

update-config does not hot-reload — the yuno must be restarted.

ycommand -c 'update-config id=<role>.<name> version=<v> zcontent=$$(<file>.json)'
ycommand -c 'kill-yuno id=<yuno_id>'
ycommand -c 'run-yuno  play=0 id=<yuno_id>'
ycommand -c 'play-yuno id=<yuno_id>'

# verify the EFFECTIVE merged config (not the stored one)
ycommand -c 'command-yuno id=<yuno_id> service=__yuno__ command=view-config'

6.4 Retire a yuno¶

ycommand -c 'kill-yuno    id=<yuno_id>'    # orderly shutdown
ycommand -c 'list-yunos'                   # confirm running=false
ycommand -c 'disable-yuno id=<yuno_id>'    # belt and braces
ycommand -c 'delete-yuno  id=<yuno_id>'    # refuses if running
# binary stays in /yuneta/repos until you delete-binary it

6.5 Upgrade a yuno to a new release (version bump)¶

When APP_VERSION in main.c changes (1.3.1.0 → 1.3.1.1, or a yunetas-side bump like 7.3.4 → 7.4.0), the canonical flow is not kill-yuno + run-yuno. Both the old and the new yuno-instance rows exist after registration, and the agent’s in-memory primary index for the yunos topic still points at the old pkey2 (yuno_release). Plain run-yuno will re-launch the older release.

# 1. Build the new version
cd /yuneta/development/yunetas/yunos/c/<yuno>/build && make clean && make install

# 2. Push the new binary to the agent
#    install-binary creates a new <role>/<version>/ slot; refuses if
#    (role, version) already exists — that is the safety vs update-binary.
ycommand -c 'install-binary id=<role> content64=$$(<role>)'

# 3. Register a yuno-instance row at the new role_version
#    create=1 actually persists; without it, find-new-yunos just lists
#    the commands it would run. Both pkey2 rows now coexist:
#       <id> <realm> <role>  <yuno_release=OLD>
#       <id> <realm> <role>  <yuno_release=NEW>
#    Visible via `list-yunos-instances`; `list-yunos` still shows OLD
#    as primary.
ycommand -c 'find-new-yunos create=1'

# 4. Force the agent to promote the newest release to primary and restart
#    deactivate-snap (no args, no active snap) is the only command that
#    triggers `restart_nodes()`: SIGKILL every running yuno, then —
#    BEFORE the reload — `promote_highest_release_yunos()` re-appends the
#    highest non-disabled `yuno_release` per id so it becomes the highest
#    rowid. (The treedb primary is the highest-ROWID record, not the
#    highest version; lifecycle/snap writes can leave an older release on
#    top, which is why a plain reload alone is not enough — the old
#    "force volatil" TODO.) gobj_stop/start then rebuilds the primary
#    index from disk with the promoted release on top, and every
#    must_play yuno runs on it. Equivalent to `yshutdown` +
#    `restart-yuneta` but without restarting the agent process itself.
ycommand -c 'deactivate-snap'

# 5. Verify
ycommand -c 'list-yunos yuno_role=<role> yuno_running=true'
# release column should now read the new version. The OLD pkey2 row
# stays in treedb for rollback (see §6.6 below).

Caveats¶

• Node-wide bounce. restart_nodes() SIGKILLs every running yuno on the node, not just the one being upgraded. Tolerable for kernel-yuno rotations (auth_bff, emailsender, logcenter); worth flagging before doing it during a busy window on a realm with many citizen yunos. (The version-promotion half of the old “force volatil” TODO is now handled by promote_highest_release_yunos(); what remains is making the bounce per-role instead of node-wide.)

• No orderly shutdown. The SIGKILL means yunos do NOT run their mt_stop / FSM stop callbacks. For protocols that flush state on exit (mqtt close packets, treedb final saves), prefer manually kill-yuno-ing each role before deactivate-snap.

• find-new-yunos create=1 without an actual binary will not break anything, but the row it creates will have no corresponding binary file and run-yuno will then fail with “primary binary not found” once deactivate-snap promotes it. Always do step 2 first.

6.6 Rollback after an upgrade¶

The OLD pkey2 row stays in yunos, and the OLD binary stays in /yuneta/repos/<role>/<old_version>/. To revert:

# Take a snap of the current state first (optional but recommended)
ycommand -c 'shoot-snap name=<rollback-tag> description="pre-upgrade"'

# … later, if the new release misbehaves, activate the snap
ycommand -c 'activate-snap name=<rollback-tag>'
# This calls the same restart_nodes() cycle, but with the snap
# active get_yuno_binary refuses to fall back to list_instances
# and primary-only lookup is enforced — the OLD row wins again.

# To remove the pin once you've decided:
ycommand -c 'deactivate-snap'

Snaps pin the binaries they reference — delete-binary respects that. shoot-snap stamps the snap’s id on every topic’s current-primary record (its md2 user_flag, surfaced as __md_treedb__.tag), binaries included. A binary the snap tagged must survive for activate-snap to roll back to it — otherwise the treedb pointer is restored but the file is gone and run-yuno fails with “primary binary not found”. So delete-binary refuses to remove a snap-tagged binary (the kernel’s treedb_delete_node enforces it; the agent also reports it clearly and never reaches the rmrdir). Pass force=1 to delete anyway — that breaks the rollback the snap was protecting.

6.7 Inspecting a snap (snaps / snap-content)¶

A snap is a point-in-time tag (a numeric user_flag, 1..65534) applied to the treedb records that were current when it was shot. snaps lists them; snap-content shows what a given snap captured — useful to see what a rollback snap (§6.6) would restore before you activate-snap it.

# List the snaps (id, name, date, active, description)
ycommand -c 'snaps'

# Overview: WHERE the snap points — every topic it tags and how many
# records each. Select the snap by name, id, or the legacy snap_id.
ycommand -c 'snap-content name=<tag>'
#   → realms:3  yunos:16  binaries:15  configurations:16  public_services:2
#     "snap 2 spans 5 topic(s); add topic_name=<topic> to see the records"

# Drill into one topic's foto (the records as captured in that snap)
ycommand -c 'snap-content name=<tag> topic_name=yunos'

snap-content selects the snap by any of:

• name=<tag> — the friendly name, resolved against __snaps__,

• id=<n> / snap_id=<n> — the numeric id shown by snaps (snap_id is the legacy form, kept for backward compatibility).

topic_name is optional: omit it for the per-topic overview (a cheap count-only walk that does not load records), or pass topic_name=<topic> for the full record foto of that topic. With neither a valid snap nor a name it answers What snap? give snap_id/id (1..65534) or name.

7. Code pointers (one-pager)¶