What your CI is actually talking to.
A regional healthcare provider asked us the question every security review starts with: do we have holes? Their zero-trust setup was solid. The hole was a self-hosted CI runner someone had allowlisted through it years ago, and nobody on the current team knew it was there.
What we walked into.
The posture was honestly good. Identity-aware access, segmented networks, and egress policy on the workloads that mattered. Around forty repos building and deploying through GitHub Actions, mostly on hosted runners. The deploy jobs that had to reach inside the network ran on one self-hosted runner, and that runner had an egress exception straight through the zero-trust controls.
That runner was the finding before we ran a single tool. It got registered three years earlier for a project that had since wrapped up. Still online, still picking up jobs from any repo in the org, still allowlisted through egress because that project had needed a vendor. Nobody on the current team had built it and it wasn't in any inventory. A compromised action on that box could reach anywhere the exception allowed, and the rest of the stack had been told to trust it.
The same cancelled project had also left behind a six-node AKS cluster running one dashboard and a Renovate bot. Utilization sat around 4 percent, and it had been on the bill for fourteen months. That cluster is what made the fix cheap.
What made it awkward.
- One runner outside the controls with a standing exception nobody could explain. Turning it off cold would've broken the deploys that still depended on it.
- And no visibility anywhere else. Hosted runners don't give you egress logs you can read.
- Forty repos to cover. Anything that meant editing every workflow would take a quarter and be out of date the week after.
What we did about it.
cargowall is an eBPF firewall for CI runners. It sits on the runner's network interface, resolves DNS through its own proxy, enforces a deny-by-default allowlist per connection, and logs every verdict along with the process that made it. In audit mode it just logs, in enforce mode it blocks. It's Apache-2.0 and it's young, so we pinned the version and verified the checksum, same as we would for anything else that runs with CAP_NET_ADMIN.
The trick to getting it into every job without touching every job is owning the runner. We moved the org onto actions-runner-controller on that idle cluster, with a runner image that starts cargowall before the runner agent comes up. Every step of every job runs behind that firewall. No workflow has to add a step, and no workflow can skip one.
# The runner image. One change from upstream: cargowall is baked in, and the
# entrypoint starts it before the runner so every step of every job runs
# behind it. Nobody has to add a step to their workflow, and nobody can
# forget to.
FROM ghcr.io/actions/actions-runner:2.321.0
ARG CARGOWALL_VERSION=0.6.3
ARG CARGOWALL_SHA256=<pinned> # young software moves. pin it, verify it.
USER root
RUN curl -fsSL -o /usr/local/bin/cargowall \
"https://github.com/code-cargo/cargowall/releases/download/v${CARGOWALL_VERSION}/cargowall-linux-amd64" \
&& echo "${CARGOWALL_SHA256} /usr/local/bin/cargowall" | sha256sum -c - \
&& chmod 0755 /usr/local/bin/cargowall \
&& mkdir -p /etc/cargowall /var/log/cargowall
# The base policy is part of the image. Per-repo additions come in as a
# mounted ConfigMap, never from the workflow, so a workflow can't loosen its
# own firewall. See 02-cargowall-policy.json.
COPY cargowall-policy.json /etc/cargowall/config.json
COPY entrypoint.sh /entrypoint.sh
RUN chmod 0755 /entrypoint.sh
USER runner
ENTRYPOINT ["/entrypoint.sh"]
# --- entrypoint.sh -----------------------------------------------------------
# #!/usr/bin/env bash
# set -euo pipefail
#
# MODE=${CARGOWALL_MODE:-enforce} # "audit" for the first two weeks
# EXTRA=${CARGOWALL_EXTRA_CONFIG:-} # per-repo allowlist, mounted read-only
#
# sudo cargowall start \
# --config /etc/cargowall/config.json ${EXTRA:+--config "$EXTRA"} \
# --dns-upstream 168.63.129.16:53 \
# --audit-log /var/log/cargowall/audit.ndjson \
# $([ "$MODE" = audit ] && echo --audit-mode) &
#
# exec /home/runner/run.sh "$@"
The base policy lives in the image and it's short. GitHub, the two package registries, the container registry, Entra login, the Azure resolver. Everything else is denied. If a repo needs somewhere else, it gets added to an overlay ConfigMap that's reviewed in the platform repo, not in the repo asking for it. A workflow can't loosen its own firewall.
{
"_comment": "Base policy baked into the runner image. Deny by default. Everything a job needs to talk to GitHub and pull packages is here; anything else is a finding until a repo owner adds it to their own overlay.",
"defaultAction": "deny",
"rules": [
{ "type": "hostname", "value": "github.com", "ports": [{ "port": 443, "protocol": "tcp" }], "action": "allow" },
{ "type": "hostname", "value": "api.github.com", "ports": [{ "port": 443, "protocol": "tcp" }], "action": "allow" },
{ "type": "hostname", "value": "**.actions.githubusercontent.com", "ports": [{ "port": 443, "protocol": "tcp" }], "action": "allow" },
{ "type": "hostname", "value": "objects.githubusercontent.com", "ports": [{ "port": 443, "protocol": "tcp" }], "action": "allow" },
{ "type": "hostname", "value": "ghcr.io", "ports": [{ "port": 443, "protocol": "tcp" }], "action": "allow" },
{ "type": "hostname", "value": "**.pkg.github.com", "ports": [{ "port": 443, "protocol": "tcp" }], "action": "allow" },
{ "type": "hostname", "value": "registry.npmjs.org", "ports": [{ "port": 443, "protocol": "tcp" }], "action": "allow" },
{ "type": "hostname", "value": "pypi.org", "ports": [{ "port": 443, "protocol": "tcp" }], "action": "allow" },
{ "type": "hostname", "value": "files.pythonhosted.org", "ports": [{ "port": 443, "protocol": "tcp" }], "action": "allow" },
{ "type": "hostname", "value": "acrappprod.azurecr.io", "ports": [{ "port": 443, "protocol": "tcp" }], "action": "allow" },
{ "type": "hostname", "value": "login.microsoftonline.com", "ports": [{ "port": 443, "protocol": "tcp" }], "action": "allow" },
{ "type": "hostname", "value": "management.azure.com", "ports": [{ "port": 443, "protocol": "tcp" }], "action": "allow" },
{ "type": "cidr", "value": "168.63.129.16/32", "ports": [{ "port": 53, "protocol": "udp" }, { "port": 53, "protocol": "tcp" }], "action": "allow" }
],
"searchDomains": []
}
The scale set itself is a Helm chart and a values file. A GitHub App instead of a PAT, BPF and NET_ADMIN on the runner container and nothing else, a node selector pinning runners to the one pool we kept, and a sidecar shipping the audit log to Log Analytics.
# helm values for gha-runner-scale-set (actions-runner-controller).
#
# helm install arc-secure oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
# --namespace arc-runners --create-namespace -f 03-runner-scale-set.values.yaml
#
# Workflows opt in with `runs-on: arc-secure`. That's the whole migration
# from hosted runners, one line per workflow.
githubConfigUrl: https://github.com/example-org
githubConfigSecret: arc-github-app # a GitHub App, not a PAT. The App's key lives in this one secret.
runnerScaleSetName: arc-secure
minRunners: 1 # one warm runner so the first job of the morning doesn't wait for a pod
maxRunners: 12 # about what the node takes before it's the bottleneck
template:
spec:
nodeSelector:
workload: automation # the one node pool we kept. see the post.
tolerations:
- key: workload
operator: Equal
value: automation
effect: NoSchedule
containers:
- name: runner
image: acrappprod.azurecr.io/ci/runner-cargowall:2.321.0-0.6.3
command: ["/entrypoint.sh"]
env:
- name: CARGOWALL_MODE
value: enforce # was "audit" for the first two weeks
- name: CARGOWALL_EXTRA_CONFIG
value: /etc/cargowall/overlay/config.json
securityContext:
capabilities:
add: ["BPF", "NET_ADMIN"] # what the eBPF firewall needs. not privileged.
resources:
requests: { cpu: "1", memory: 2Gi }
limits: { cpu: "2", memory: 4Gi }
volumeMounts:
- name: overlay
mountPath: /etc/cargowall/overlay
readOnly: true
- name: audit
mountPath: /var/log/cargowall
# Ships the NDJSON audit log to Log Analytics. The firewall's verdicts
# are the deliverable; a verdict nobody can read is not one.
- name: audit-shipper
image: acrappprod.azurecr.io/ci/audit-shipper:1.2.0
volumeMounts:
- name: audit
mountPath: /var/log/cargowall
readOnly: true
volumes:
- name: overlay
configMap:
name: cargowall-overlay # per-org additions, reviewed in the platform repo
optional: true
- name: audit
emptyDir: {}
Then we sat in audit mode for two weeks, on the new runners and on the old allowlisted VM with cargowall just watching it.
What two weeks of audit mode turned up.
- On the allowlisted runner: outbound attempts on every deploy to an IP range that used to belong to the old vendor, two years after that contract ended, from an action still pinned in three workflows. Nothing was answering anymore. Something had been trying through that exception the entire time.
- Same box: a weekly workflow from a repo with no owner, reading a file share the runner could reach. Nobody could tell us what it was for.
- A postinstall script buried in a transitive npm dependency, POSTing the CI environment variables to an analytics endpoint. That package gets 200k downloads a week.
- Two workflows shipping build metadata to a domain the company had let expire. Someone else owns it now.
None of that was an active breach. All of it was the kind of thing that turns into one, and two of the four were on the one machine the controls had been told not to look at.
Enforce mode went on one repo at a time over four days. Twenty-two workflows broke, and every single one broke because it reached somewhere legitimate that wasn't in the base policy yet. Each one got an overlay entry after someone looked at it. The three deploy workflows that had leaned on the old runner moved to the new ones with only the internal destinations they actually needed. Then we deregistered the old VM, deleted its exception, and shut it off. Not one person noticed.
# .github/workflows/build.yml
#
# The only line that changed in any workflow across the org. Everything
# else about the job is the same; it just runs on a runner that won't let
# a step talk to anything the policy doesn't name.
name: build
on:
push:
branches: [main]
pull_request:
jobs:
build:
runs-on: arc-secure # was: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm ci
- run: npm test
# No cargowall step. It started before this job did.
Where it landed.
"Do we have holes" has a better answer now. The one exception to the policy is gone, and what replaced it sits inside the policy. Every CI job runs behind the same deny-by-default egress control, and every outbound connection from the last ninety days is one Log Analytics query away, with the process that made it and the rule that let it through.
The cluster dropped from six nodes to two. One pool runs the whole automation platform now: ARC, the runners, Renovate, the scheduled jobs, the dashboard. The hosted-runner minutes line on the bill went to zero. CI costs less than it did before it had a firewall in it, and the migration was one line per repo, runs-on: arc-secure. Renovate opened the PRs for us.
Go check your own runners.
- Which machines have a standing exception through your zero-trust policy, and can someone explain each one? The one nobody can explain is your finding.
- Can a single workflow step send your job's secrets wherever it wants? On hosted runners, yes it can.
- Do you actually own the runner? That's what makes everything else here possible.
- Is your allowlist deny-by-default, and can a workflow loosen its own rules?
- Did audit mode run long enough to catch the weekly jobs and not just the daily ones?
Have a system that rhymes with one of these?
[ TALK TO US ]