Hosted runner security model

fremforge hosted runners execute customer Forgejo Actions workflows in T Cloud CCE Turbo — a dedicated Kubernetes cluster running in the EU-DE region. Every queued job spawns a fresh, single-task pod; the pod registers ephemerally with Forgejo, picks up exactly one task, runs it, and exits. There is no persistent runner pool that a previous customer’s code could leave residue in.

This page documents the isolation guarantees fremforge makes, the per-tenant limits that apply, and the threat model: what customer code can do, what it can’t, and the design choices a workflow author should make to keep their secrets safe.

Where workflows run

Isolation guarantees

Network isolation (per pod)

Every runner pod gets its own VPC ENI (Yangtse Cloud Native Network 2.0) bound to a per-pod Security Group fremforge-prd-runner-pod. The SG denies, by default, all traffic except:

• DNS (53/UDP, 53/TCP) to the cluster’s CoreDNS service
• Egress through outbound-proxy-runners (port 80) — the only outbound HTTP path

Customer code cannot reach:

• The fremforge api or its private database
• Forgejo’s internal ClusterIP (runner pods talk to Forgejo through the public Bunny-fronted apex with a runner-specific scope; the internal service is unreachable)
• T Cloud platform services (RDS, DCS, OBS, SFS-Turbo, IAM, KMS)
• Other tenants’ runner pods
• The cluster’s metadata IP (169.254.169.254) — explicitly blocked at the SSRF guard
• RFC1918 private ranges or loopback

A Kubernetes NetworkPolicy enforces the same rules at the CNI layer (default-deny on fremforge-prd-runners namespace), so even an SG misconfiguration cannot open a path.

Outbound egress: outbound-proxy-runners

Customer workflow steps that need the public internet (cloning a public repo, downloading a tool, calling a SaaS API) route through outbound-proxy-runners via HTTPS_PROXY injected into the pod. This proxy:

• Accepts any public unicast IP including Carrier-Grade NAT (matches GitHub-hosted runner behaviour — needed for T Cloud’s own services like SWR which sit in CGNAT)
• Blocks RFC1918 private, link-local, loopback, multicast, reserved, cluster Service-IP CIDR, and explicit cloud-metadata IPs (169.254.169.254, T Cloud’s 100.100.100.200)
• Logs every CONNECT tunnel attempt with hostname + resolved IP for audit retention

The proxy is deliberately permissive in the GitHub-hosted-runner sense: customer workflows can reach any public host they need. The boundary is at SSRF + cluster-internal, not at vendor allowlists.

Pod-level isolation

Cross-tenant + intra-tenant kernel isolation

This is the load-bearing claim of the hosted-runner model. Every queued job gets its own CCE Turbo VM node — a guest Linux kernel running on T Cloud’s hypervisor. The pod runs on that node, the job runs in the pod, the job completes, the controller deletes the pod, and the controller deletes the node. The next job (same tenant or another) gets a fresh node with a fresh kernel.

Per-tenant limits

When a tenant exceeds the runner-minute pool and overage isn’t enabled (or is frozen), the spawner refuses dispatch: the workflow’s queued job transitions to Cancelled in Forgejo with reason runner_minute_cap. Tenants with runner_overage_enabled=true keep running and are invoiced for the excess.

When a tenant hits the concurrent cap, additional queued jobs wait — the spawner polls every 2s and dispatches as soon as a slot frees.

What customer code can NOT do

• Reach the fremforge api, Forgejo’s internal ClusterIP, or any other platform service (the api is only available through the public Bunny apex with proper authentication, just like for any external caller)
• Read another tenant’s runner pod (separate VPC ENI; cross-pod traffic explicitly denied)
• Read another tenant’s secrets (workflow secrets are mounted only into the pod that’s running the workflow that owns them; pod exit destroys the mount)
• Persist state across job invocations (single-task pod, no shared volume)
• Reach the Kubernetes API server (no SA token mounted)
• Reach cloud metadata endpoints (169.254.169.254, 100.100.100.200 — both explicitly denied at the SSRF guard)
• Exhaust cluster capacity (ResourceQuota + global cap)
• Cause queue starvation across tenants (per-tenant concurrent cap)
• Run privileged or root containers (PSA baseline enforced on the namespace)
• Use raw network sockets, capabilities, host networking, or hostPID/hostIPC

Recommended workflow shapes

Split validate and apply jobs

Run untrusted code (PR builds, dependency scans, contributor-supplied test fixtures) in a validate job with no production credentials, and gate the apply job behind an environment approval. A compromised validate job cannot reach apply-job secrets.

Pin uses: references by SHA when supply-chain matters

uses: actions/checkout@v4 resolves to frem.sh/actions/checkout (fremforge’s Forgejo native pull-mirror of github.com/actions/checkout, 8-hour sync). The mirror tracks upstream tags including force-pushes, so pinning by SHA is the only way to guarantee bytewise stability:

# Bytewise-stable across maintainer force-pushes
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4.2.0

# Picks up upstream's current HEAD at last mirror sync — including
# any maintainer-credential-compromise force-push
- uses: actions/checkout@v4

Use OIDC for cloud auth, not long-lived secrets

Every runner pod gets a fresh OIDC token with aud=fremforge. Configure your cloud provider’s trust policy to accept this token and exchange it for short-lived credentials. Avoid storing long-lived AWS access keys / GCP service-account JSONs in workflow secrets where they end up in pod env vars.

Cap workflow step deadlines

Set timeout-minutes: on individual steps to bound how long a hung step can run. The pod-level activeDeadlineSeconds=3600 is the platform floor; step-level timeouts give faster feedback.

Threat-model summary

Auditability

Every dispatch is logged in three places:

1. runner_jobs table — tenant_id, repo, forgejo_task_id, started_at, finished_at, duration_seconds, pod_name. Source for billing aggregation and the operator console’s Infrastructure tab
2. metering_events table — code=runner_minute rows derived from runner_jobs.duration_seconds. Drives invoice line items and the tenant’s “Runner minutes used” panel
3. LTS stream fremforge-prd-runner-spawner — structured pod-spawn logs for the operator. Keyword alarms on runner_dispatch_refused, runner_pod_stuck_pending, runner_job_overran_deadline, runner_quota_exhausted, runner_pool_at_max

Tenants see their own usage at Org admin → CI runners → Usage, including a per-month minute count and a per-job durations table.