CI runners

fremforge provides hosted runners included in every seat plan, and supports BYO runners for teams that need on-prem, custom hardware, or specific network placement. Both runner types execute Forgejo Actions workflows using the same GitHub Actions YAML syntax.

Hosted runners

Hosted runners are per-commit virtual machines on T Cloud Public in the eu-de region (Biere/Magdeburg). A VM is provisioned from a custom debian image with Docker pre-installed, runs jobs from one commit of one repository, and is destroyed once that commit’s jobs are done. Nothing persists between commits.

Available labels

Reference a label in your workflow:

jobs:
  build:
    runs-on: fremforge

Container builds work two ways out of the box — see Runner image contents § Container builds below for the full split:

• Native Docker (default) — docker build, docker compose, docker pull just work. The Docker daemon runs natively on the VM (no Docker-in-Docker sidecar), so docker behaves the same as on a GitHub-hosted runner. Suits 90% of workflows.
• kaniko — daemonless, faster cold-start, better remote-registry layer cache. Use when you don’t need BuildKit-specific features.

jobs:
  build-image:
    runs-on: fremforge
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t registry.example.com/$GITHUB_REPOSITORY:$GITHUB_SHA .
      - run: docker push registry.example.com/$GITHUB_REPOSITORY:$GITHUB_SHA 2>&1 | stdbuf -oL tr '\r' '\n' | grep -vE '^(Waiting|)$'

ARM64 hosted runners and additional OS variants (Ubuntu, Fedora, …) are on the roadmap (see changelog) — fremforge today targets x86_64 Linux only. BYO runners (see below) are the path for ARM64 or non-Linux workflows in the meantime.

Isolation model

Jobs run on dedicated virtual machines with their own kernel. The platform watches the Forgejo Actions queue and starts a fresh VM for a pending job. A cold start takes about 40 seconds before the job’s first step; there is no pool of pre-started VMs.

One commit, one trust domain. A VM only ever runs jobs from the commit and repository it was started for. When a job finishes, the VM waits up to 60 seconds for another job from the same commit, in any workflow, and runs it without a new boot (about 5 seconds to start). After that it powers off and is deleted. No VM runs jobs from two commits, two repositories or two organisations, so treat all jobs of one commit as one trust domain. Credentials the platform knows of (Docker and npm logins, the usual credential files) are reset between jobs, but tool caches and pulled images stay on the VM. See Hosted runner model for the full design.

Sizing jobs for hosted runners

Every fresh VM costs about 40 seconds of boot before your first step, and often the 60-second wait after your last one. A job that does 15 seconds of work therefore holds a VM for about two minutes. So:

• Give a job its own VM only when its work takes about 2 minutes or more. Shorter checks (lint, a small test suite, a guard script) are cheaper and faster as extra steps in an existing job.
• Parallelise long, independent work (test shards, image builds, end-to-end suites) across jobs. That is what concurrency is for.
• Parallelise short work inside one job instead of across jobs, for example several tofu plan runs in background processes of one step. They mostly wait on APIs, not CPU.
• Chains reuse VMs automatically. A needs: job that becomes ready while its predecessor’s VM is still waiting (within 60 seconds) runs on that VM with no boot. A wide fan-out of jobs that all start at once cannot reuse anything: each needs its own VM.
• Keep a run inside your concurrency. A matrix wider than your organisation’s guaranteed share runs in waves, and the last wave waits for a whole job to finish. Size shards so a run fits in one wave.

Memory

A hosted runner VM has 4 GiB of memory and no swap (2 vCPU). There is no larger hosted runner size today.

A job that runs out of memory fails within about a minute. When available memory drops below about 4% (roughly 130 MiB), the runner stops the job’s largest process, preferring build tools such as node, tsc, java and vitest workers. It sends SIGTERM first and SIGKILL if memory keeps falling. The step fails with exit code 143 or 137, and the step log says why:

earlyoom: killed node (pid 4242) because the runner VM ran out of memory. ...

The runner itself is never the process that is stopped, so the job’s result and logs are still reported. Before this change, a job that outgrew the VM could stall silently until its timeout-minutes.

If you see this line, reduce the job’s peak memory:

• Split the work across jobs. Running a large build next to its integration tests, or a docker build next to test containers, adds the two peaks together on one VM.
• Bound the tool’s memory. For example NODE_OPTIONS=--max-old-space-size=2048 for Node, --checkers 1 and GOMEMLIMIT for TypeScript 7 (tsc written in Go), -Xmx for Java, fewer test workers (--maxWorkers / --pool-options).
• Stream instead of loading. For large image scans, see Scanning large images with Trivy.

If a job cannot fit in 4 GiB, contact support@frem.sh, or run it on a BYO runner sized for it.

Included minutes and concurrency

Jobs above your organisation’s limit wait in the queue rather than fail. Contact support@frem.sh if you need a higher guaranteed share.

Overage billing (opt-in): by default, new jobs queue when the pool is exhausted. Org owners can enable metered overage at Org admin → Billing → Runner-minute usage. When enabled, minutes above the pool are charged at €0,01/min on the next invoice. Overage can be disabled at any time from the same page.

Usage dashboard

Track runner minute consumption at Org admin → Billing → Runner-minute usage.

The usage dashboard shows:

Minutes are counted from job start to job end, rounded up to the nearest second. Queue time (waiting for a runner) is not counted.

The monthly period resets on the 1st of each month UTC. Unused minutes do not roll over.

BYO runner minutes are tracked separately in the same dashboard and do not count against the hosted pool.

Customer-tunable usage-alert thresholds aren’t surfaced yet — fremforge sends a default warning email when the org reaches 80% of the monthly pool. Contact support@frem.sh to adjust the threshold.

Runner image contents

The hosted runner image is rebuilt weekly (Mon 06:00 UTC) from upstream tool versions tracked by Renovate. Every build-time download is sha256-pinned (build fails on drift). A Trivy CVE gate runs at build time — any image with HIGH or CRITICAL findings is blocked from shipping until patched.

Before a rebuilt image goes live on the runner fleet, it is checked again against the known-exploited vulnerability lists (ENISA EUVD and CISA KEV). An image with a known-exploited CVE that has a fix available is not rolled out, at any severity, unless a reviewed exception of at most 30 days covers that CVE. Runners keep using the previous image until a patched one is built.

Two-layer model, both EU-resident:

• System layer (~300 MB) — debian base + single-version system tools (Node 24 LTS, build essentials, scanners, IaC + cloud CLIs, browsers, db clients). Part of the VM boot disk; no per-job download.
• hostedtoolcache layer (~7 GB) — multi-version language toolchains (Node, Go, Python, .NET, Java, Ruby) + Android SDK + Miniconda3 + R (since 2026-06-06). Baked into the same VM boot disk at /opt/hostedtoolcache. setup-* actions resolve from here with no per-job download.

How we deliver full coverage in ~7 GB total when GitHub’s ubuntu-latest is ~50 GB: see the size story below.

Pre-installed in the system layer:

Pre-installed in the hostedtoolcache (resolved by setup-* actions, no per-job download):

The canonical version list with exact resolved versions + image digest + Trivy scan summary + CycloneDX SBOM is published with each weekly rebuild — see the runner image reference for the current toolstack.

Container builds: native Docker or kaniko

Two paths, pick by workload:

Native Docker (default) — the VM ships with the Docker daemon installed and running on the host. Your docker CLI talks to it over the standard local socket — no DOCKER_HOST indirection, no sidecar. Works for docker build, docker compose, docker pull, multi-stage builds, BuildKit features. Drop-in parity with GitHub-hosted runner docker.

- run: docker build -t myapp:${{ github.sha }} .
- run: docker push myapp:${{ github.sha }} 2>&1 | stdbuf -oL tr '\r' '\n' | grep -vE '^(Waiting|)$'

Scanning large images with Trivy

trivy image <ref> (daemon-mediated) holds layer metadata for the full image in memory while it walks the filesystem — multi-GB images can spike well past the runner VM’s memory limit and trigger an OOM-kill mid-scan. The reliable pattern for any image size is to stream the image via docker save:

- run: docker build -t myapp:${{ github.sha }} .
- run: |
    docker save myapp:${{ github.sha }} | trivy image \
      --severity HIGH,CRITICAL --ignore-unfixed \
      --scanners vuln --pkg-types os \
      --offline-scan --no-progress --quiet \
      --input -

Why it works: docker save reads layers from disk and streams the tarball into trivy’s stdin one layer at a time. Trivy parses each layer for the OS package metadata (~tens of KB per layer) and discards it before the next. Combined peak RSS stays around 500 MB regardless of total image size, so the scan fits comfortably within the runner VM’s memory budget. GOGC=10 in front of trivy (e.g. GOGC=10 trivy ...) tightens the heap further if you ever push close to the limit.

The tr '\r' '\n' filter on docker push is recommended: Docker writes layer-upload progress with carriage-return overwrites that the Forgejo Actions step-log treats as a single ever-growing line and silently truncates. Without the filter, a push that stalls or errors mid-upload shows only <layer>: Preparing in the log with no failure detail — by the time the step times out, the actual error has been overwritten. Piping through tr turns each progress tick into its own line that gets captured intact, and grep -vE '^(Waiting|)$' drops empty/idle padding. Layer-push throughput is unaffected.

Customer registries (ghcr.io, dockerhub, quay.io, your own) are reachable via the runner egress; the proxy carries a 30 min per-request timeout so multi-GB layer PUTs complete cleanly. Image-size budget: 50 GB per build (the VM scratch-disk cap) — file a ticket if you need more.

kaniko — daemonless, rootless OCI builder. Faster cold-start than the full Docker daemon, better layer-cache behaviour against remote registries. Use when the build doesn’t need BuildKit-specific features.

- run: |
    /usr/local/bin/kaniko \
      --context=$GITHUB_WORKSPACE \
      --dockerfile=Dockerfile \
      --destination=registry.example.com/$GITHUB_REPOSITORY:$GITHUB_SHA

For kind / testcontainers workloads that need KVM-nested guests, add the dind-privileged label to runs-on. fremforge routes these to a VM flavor with nested virtualisation enabled.

Size story vs GitHub-hosted

GitHub’s ubuntu-latest is ~50 GB because it bakes:

• Several versions per language — 3 recent series for Node, Python, Java and Ruby; 2 for Go and .NET (~3 GB total). The table above is the authoritative set.
• Android SDK + NDK (~10 GB on GitHub-hosted) — we bake the core SDK (platform 34 + build-tools + platform-tools, ~3 GB); extra API levels + NDK + emulators install per-job via setup-android@v3 (no re-download of what’s baked).
• Pre-pulled Docker service-container images (postgres, mysql, mongo, redis, mssql ~5 GB) — not baked; a docker pull fetches them from Docker Hub, or through your organisation’s dependency proxy (see package caches).
• Microsoft Edge browser — not baked (Chromium is the same engine for headless CI testing).
• Mono / Julia / Swift / Haskell / Lua / Crystal / Nim / Elixir — not baked (niche, install per-job via setup-* or apt).
• Multiple GCC versions, old/legacy toolchains kept for compatibility — we ship one current set.

We don’t miss tools the typical workflow uses — every common language, every major cloud CLI, every common scanner, both major browsers, Docker (native, root via dind-privileged label), Android (core SDK), and data-science (Miniconda3 + R). iOS / Xcode is Linux-incompatible and tracked separately; niche languages (Julia/Swift/Haskell/Mono) install per-job. Let us know if your team needs something we don’t bake and we’ll evaluate for the next weekly image rebuild.

For full per-tool details see docs.frem.sh/build/actions/runner-image/.

BYO runners

Register your own runners against any fremforge org (on-prem servers, cloud VMs, Mac minis, Raspberry Pis). BYO runner minutes do not count against the hosted pool.

Registration

1. Go to Org admin → Settings → Runners → New runner.
2. Copy the registration token.
3. Download the Forgejo runner binary. The current stable release is published at code.forgejo.org/forgejo/runner/releases, pin to a recent tag when you install:

# Linux x86_64 — replace <version> with the tag from the releases page (e.g. v6.4.0)
curl -sSL https://code.forgejo.org/forgejo/runner/releases/download/<version>/forgejo-runner-<version>-linux-amd64 \
  -o forgejo-runner && chmod +x forgejo-runner

# macOS arm64
curl -sSL https://code.forgejo.org/forgejo/runner/releases/download/<version>/forgejo-runner-<version>-darwin-arm64 \
  -o forgejo-runner && chmod +x forgejo-runner

fremforge tracks current Forgejo releases; runners on older binaries continue to work but should be upgraded within a release cycle to pick up CVE patches.

4. Register the runner:

./forgejo-runner register \
  --instance https://frem.sh \
  --token <registration-token> \
  --name my-runner \
  --labels linux,self-hosted,my-custom-label

5. Start the runner:

./forgejo-runner daemon

Supported platforms

Custom labels

Labels assigned at registration are what you reference in runs-on::

jobs:
  deploy:
    runs-on: [self-hosted, my-custom-label]

Multiple labels in a list mean “find a runner that has ALL of these labels”. The self-hosted label is conventional. Include it to avoid accidentally routing a BYO job to a hosted runner.

Don’t use reserved labels. The labels fremforge and linux/amd64 route to our hosted-runner pool (the legacy fremforge-cci and cci aliases are deprecated and route the same way). Pick distinct labels for your BYO runners (e.g. self-hosted, on-prem, my-team-runner) so workflows can address them unambiguously.

Network requirements for the runner host

The runner binary opens long-poll HTTPS connections to https://frem.sh and clones repositories over HTTPS. From the host’s perspective the connectivity requirement is:

No inbound connectivity is needed — the runner initiates all connections outbound. A runner behind NAT or a firewall works fine as long as it can reach frem.sh:443.

Security and isolation responsibility

For hosted runners, fremforge provides kernel-level isolation between commits and between customers (each commit runs in a fresh ephemeral ECS VM; jobs of the same commit may share it). For BYO runners, isolation is your responsibility. A self-hosted runner that handles workflows from multiple repositories shares state across jobs unless you scope each runner to a single repository or org.

If you run workflows from multiple teams or trust-domains, treat your BYO runner the same way GitHub Actions self-hosted runners are treated:

• One runner per repository (or per trust-boundary) when secrets are sensitive.
• Don’t register a BYO runner against a public repository — anyone with merge access could land a workflow change that executes arbitrary code on your runner host.
• Treat the runner host as production: harden the OS, keep the runner binary patched, and route its egress through your own SSRF / egress controls.

The fremforge SSRF outbound proxy does NOT apply to BYO runners — only to hosted runners. Your runner’s network policy is whatever you wire on the host.

Egress and network

Org-level egress allowlist: to allow outbound access to additional CIDRs or ports from runner jobs, contact support@frem.sh. Egress rules are configured at the platform level.

The SSRF proxy is a platform floor. It cannot be disabled. It prevents runner jobs from reaching T Cloud Public instance metadata or internal infrastructure.

Secrets in jobs

Secrets are injected as environment variables and masked in job logs. Any log line containing a secret value is replaced with ***.

Reference syntax:

steps:
  - name: Deploy
    env:
      API_KEY: ${{ secrets.MY_API_KEY }}
    run: ./deploy.sh

Secret scopes

Environment secrets require the job to declare an environment:

jobs:
  deploy:
    runs-on: fremforge
    environment: production
    steps:
      - run: echo ${{ secrets.PROD_API_KEY }}

For cloud provider credentials, prefer OIDC token federation over long-lived secrets. No secret to store, no secret to rotate.

Troubleshooting

A step fails with earlyoom: killed <process> ... ran out of memory.

The job used more than the runner VM’s 4 GiB of memory, so the runner stopped its largest process instead of letting the VM stall. See Memory for how to reduce the job’s peak.

Job is stuck in queue.

The most common cause is hitting the per-tenant concurrency cap (default 30, controlled by RUNNER_PER_TENANT_CONCURRENCY_CAP). Check Org admin → Billing → Runner-minute usage for current active job count. If you are at the cap, either wait for a running job to finish or contact support@frem.sh for an Enterprise-on-Demand uplift.

docker command works but a privileged operation fails (e.g. kind cluster create with KVM nesting).

The default runs-on: fremforge VM runs Docker natively with rootless user defaults — docker build / docker compose / docker pull work out of the box, but operations that need extra capabilities (KVM nesting, raw /dev access for a nested guest, certain kind cluster network modes) will not. Add the dind-privileged label to runs-on (runs-on: [fremforge, dind-privileged]) — fremforge routes onto a VM flavor with nested-virt enabled. Same per-commit VM isolation; same image; just elevated capabilities.

For pure image copy/retag between registries (no build), the pre-installed skopeo works without needing the full Docker daemon at all.

Secret is masked in logs but the step behaves as if it is empty.

The secret exists in the runtime (masking confirms this), but the value at the correct scope is not set. Check:

1. Is the secret set at org level, repo level, or environment level?
2. If it is an environment secret, does the job declare environment: <name> matching the secret’s environment?
3. Is the secret name spelled exactly right (case-sensitive)?

The Secrets UI at the relevant scope shows the secret name and last-updated timestamp. Use that to confirm the secret exists before looking at the workflow.

Cross-references

• OIDC token federation, keyless cloud auth from CI jobs
• Secrets, managing org, repo, and environment secrets
• Marketplace compatibility, which GitHub Marketplace actions work on fremforge runners
• AI posture, what AI tooling is and is not present in the runner environment