Release trains

A release train ships several repositories together. When the train is cut, fremforge pins the commit of every repository in it (its components) in an immutable, hashed manifest. It then takes that release through its stages, for example tst and then prd, by starting each repository’s own deploy workflows with the pinned commits.

• One release across repositories. A platform, its infrastructure and an ingestion service ship as one train with one manifest, not as three deploys that happen to run close together.
• Cut on a rule. fremforge cuts the train when the oldest merge that is not yet in a train is old enough, for example 60 minutes. You can also cut by hand.
• No merges slip in. At the cut, a merge hold stops merges into the components until the stage you name has passed. The merge button, the API and the merge queue all wait.
• The deploy gates apply unchanged. A train only starts workflow runs. Each run meets its environment’s gates (approvers, allowed branches, time windows, required checks, concurrency, the pause flag) exactly as a run you start yourself.

Release trains are off until an organisation owner switches them on. When they are switched on, they start in shadow only: they record what they would deploy and deploy nothing until an owner turns that off.

Define a train

A train is one file, trains/<name>.yaml, in a repository of your organisation that you choose (a small release repository works well). The file name is the train’s name: 1 to 40 lower-case letters, digits and hyphens. A change to a train is a pull request like any other, so it is reviewed, and the repository’s history is the train’s history.

version: 1
owner: hk                       # the person the train acts for
description: Platform, infrastructure and ingestion
components:
  platform: { repo: platform }
  infra:    { repo: infrastructure-azure, branch: main }
  ingest:   { repo: ingestion }
cut:
  after_oldest_unreleased_minutes: 60
hold:
  release_after: tst            # hold merges until stage tst has passed
stages:
  - name: tst
    timeout_minutes: 90
    steps:
      - component: platform
        workflow: code-deploy.yaml
        inputs: { app: migrate, confirm: migrate, sha: "{{ sha }}" }
        environment: tst
      - component: platform
        workflow: code-deploy.yaml
        inputs: { app: web, confirm: web, sha: "{{ sha }}" }
        environment: tst
  - name: prd
    steps:
      - component: platform
        workflow: code-deploy-prd.yaml
        inputs: { app: migrate, confirm: migrate, sha: "{{ sha }}", infra_pin: "{{ sha.infra }}" }
        environment: prd

Each step starts one workflow run:

Input values can use these placeholders:

A workflow that receives the commit as an input should build or deploy exactly that commit, not the branch head. The branch can move on after the train’s hold is released.

Check a definition before it merges

curl -X POST https://frem.sh/_app/api/v1/orgs/<org>/trains/validate \
  -H "Authorization: Bearer $FREMFORGE_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg s "$(cat trains/platform.yaml)" '{path: "trains/platform.yaml", source: $s}')"

The answer has valid, errors and warnings. Besides the file itself, fremforge checks it against your organisation: the owner is a member, every repository and branch exists, every workflow exists on its branch, accepts workflow_dispatch and the inputs the step sends (and is sent every input it requires), and every environment is configured. A warning tells you when a workflow never names the step’s environment, or when the hold covers a branch where fremforge/hold is not a required check. Run it from the pull request that changes the train. The token needs deployments:read.

Switch trains on

curl -X PUT https://frem.sh/_app/api/v1/orgs/<org>/trains/settings \
  -H "Authorization: Bearer $FREMFORGE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"enabled": true, "definitions_repo": "release", "definitions_branch": "main"}'

The token needs policy:write. While trains are on, fremforge reads trains/*.yaml every two minutes, all at one commit of the branch.

The answer, and GET .../trains/settings, also has shadow_only. It is true until an owner turns it off in the console; see Shadow only.

For the hold to stop the merge button, turn on merge-hold enforcement for the components’ branches. Without it the hold is recorded and read by tooling, but the merge button does not wait.

What a cut does

1. fremforge checks the definition against the organisation, as validate does. A train that cannot run is not cut.
2. It places the merge hold: one per branch the components use, owned by the train’s owner, with the reason Release train <name> #<n>: merges held until stage <stage> has passed.
3. It reads every component’s head commit, and compares it with the commit the previous train pinned.
4. It writes the manifest and the train, together.

If anything after the hold fails, fremforge releases the hold again. A failed cut never leaves a branch held.

A train definition has one train in progress at a time. The next cut waits until the current train has completed, failed or been cancelled.

The rule. For each component, the unreleased commits are those since the previous train’s commit. When the oldest of them, on any component, is after_oldest_unreleased_minutes old, the train is cut. Age is the commit’s committer date, which is the merge time for a squash merge. The first train of a definition counts from each branch’s head commit.

By hand. An organisation owner, or a token acting for a person, can cut now:

curl -X POST https://frem.sh/_app/api/v1/orgs/<org>/trains/<name>/cut \
  -H "Authorization: Bearer $FREMFORGE_TOKEN"

The token needs deployments:write, and must act for a person: a cut by hand is recorded as theirs.

The manifest

The manifest is what the train shipped, and it never changes: the database refuses any update to it. It holds:

• every component’s repository, branch and pinned commit, the commit the previous train pinned, and how many commits are new;
• the plan files changed since the previous train, each with the sha256 of its content at the pinned commit;
• the train definition as it was at the cut, and the commit of the definitions repository it was read from, so a later edit of the file never changes a train that is already running.

manifest_sha256 is the sha256 of the manifest written as canonical JSON: object keys sorted at every level, no whitespace. You can recompute it from the stored manifest:

curl -s https://frem.sh/_app/api/v1/orgs/<org>/trains/runs/<id> \
  -H "Authorization: Bearer $FREMFORGE_TOKEN" \
  | jq -cS '.manifest' | tr -d '\n' | sha256sum

Stages

A cut train starts its first stage at the next run of the job, within two minutes. Each stage runs its steps one after the other:

1. fremforge starts the step’s workflow with workflow_dispatch on the component’s branch, with the inputs filled in.
2. The run meets its environment’s gates. While it waits for an approval, the train says so (waiting for approval in environment tst). Approvers approve the deployment as usual, at Build & deploy → Deployments.
3. When the run succeeds, and (if the step names an environment) its deployment to that environment passed the gates, the step has passed and the next one starts.

When every step of a stage has passed, the train moves to the next stage. The hold is released after the stage named in hold.release_after, and at the latest when the train ends.

fremforge starts the runs with the platform’s Forgejo account, as a schedule would. So the train’s owner can approve the train’s own deployments: an approval by the person who started a run is refused.

A train stops (failed, with the reason) when:

• a run fails or is cancelled, or an environment’s gates refuse it (the reason names the gate);
• a run succeeds without going through the step’s environment;
• a stage runs past its timeout_minutes;
• a dispatch fails or its outcome is unknown. fremforge never starts a step twice: if it cannot tell whether a run started, it stops the train and asks you to look at the repository’s Actions.

When a train stops, its holds are released. Commits already merged stay merged; the next cut carries the fix.

Cancel a train that has not finished:

curl -X POST https://frem.sh/_app/api/v1/orgs/<org>/trains/runs/<id>/cancel \
  -H "Authorization: Bearer $FREMFORGE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"reason": "Re-cut after the hotfix"}'

The holds are released. Workflow runs the train already started keep running.

In the console, org owners see the same at Build & deploy → Release trains (https://frem.sh/<org>/_admin/release-trains): every train, the one in progress with what it waits for, recent trains with their pinned commits, and buttons to cut now or cancel.

Shadow mode

Set dispatch: none to try a train before it deploys anything, for example next to the release process you use today:

version: 1
owner: hk
dispatch: none
# components, cut, hold and stages as usual

A train in shadow mode:

• is cut on its rule, places its merge hold and writes its manifest, exactly as a train that deploys;
• records at the cut, for every step, what it would dispatch: the repository, branch, workflow and inputs with the pinned commits (would_dispatch on each step, with status shadow);
• then walks its stages without starting a single workflow run, releases its hold after the stage named in hold.release_after, and completes;
• is marked shadow: true in the API, with a Shadow badge in the console, and in every event and audit record.

Compare what it would have dispatched with what you shipped. When they match, remove dispatch: none. The next cut deploys once the organisation is no longer shadow only.

Shadow only

dispatch: none is a line in a file, and anyone who can merge into the definitions repository can remove it. Shadow only is the organisation’s own setting, and fremforge enforces it whatever a definition says. While it is on:

• every train is a shadow train: each step records what it would dispatch, and no workflow run is started;
• no merge hold is placed, even when a definition asks for one;
• a cut whose definition asked to deploy or to hold says so in the manifest (definition.org_shadow_only) and in the audit log (release_train.shadow_only_enforced);
• a train cut while shadow only was off stops, before its next dispatch, if shadow only is turned back on. Its holds are released.

Shadow only is on for every organisation until an owner turns it off. Turn it off in the console, at Build & deploy → Release trains: open Let trains deploy, type the organisation’s name exactly and confirm.

• It takes an organisation owner, signed in to the console.
• If your organisation has its own sign-in step-up configured, you are asked to sign in again first, unless you did in the last 10 minutes.
• Every owner of the organisation gets an email saying who turned it off, and a train.shadow_only_cleared event goes on the event feed.
• The API can turn shadow only on ("shadow_only": true in PUT .../trains/settings) but never off. "shadow_only": false is refused for every token.

Turning shadow only back on is one button on the same page. The audit log records release_train.shadow_only_cleared and release_train.shadow_only_set, with who changed it.

Read trains

All need deployments:read. A train’s status is cut, running, completed, failed or cancelled, and shadow is true for a train in shadow mode.

Events and audit

The event feed has one event per component repository for each of these, with sha the commit the train pinned there:

Events of a train in shadow mode carry data.shadow: true.

The audit log records release_train.settings_updated, release_train.shadow_only_set, release_train.shadow_only_cleared, release_train.shadow_only_enforced, release_train.cut, release_train.step_dispatched, release_train.stage_promoted, release_train.completed, release_train.failed and release_train.cancelled, together with the hold’s own merge_hold.created and merge_hold.released. A cut by the rule and every step are recorded as system:release-trains, acting for the train’s owner; a cut or cancel by hand as the person who made it.

What a train does not do

• It does not approve anything. Approvals are given by approvers on the deployments, as without a train.
• It does not change the gates. An environment with a gate bypass, a pause flag or required reviewers behaves exactly as it does for any run.
• It does not roll back. A failed stage stops the train. Merged commits stay merged; ship the fix in the next train.
• It does not cancel runs. Cancelling a train leaves the runs it started alone.