In modern cloud-native software development pipelines, GitHub Actions Runner Controller (ARC) has emerged as an industry standard for orchestrating self-hosted runner scale sets on Kubernetes clusters. However, engineering teams operating within highly restricted environments—such as air-gapped networks, heavily fortified corporate clusters, or enterprises bound by strict internet-egress compliance policies—frequently encounter rigid operational hurdles. A prominent architectural friction point exists within version 0.15.0 of the gha-runner-scale-set Helm chart, specifically regarding the containerMode type configured as dind (Docker-in-Docker). Under this configuration, the chart hardcodes the underlying daemon container image as docker:dind, which the local kubelet aggressively attempts to pull directly from Docker Hub. Because current stable releases lack a direct values parameter to modify this image registry source, platform engineers have been forced to implement advanced manifest-patching strategies utilizing tools like Kustomize and Kapitan to redirect traffic through secure pull-through mirrors or internal enterprise registries.
Background Context and the Core Architectural Challenge
The genesis of this operational bottleneck traces back to how ARC manages container execution modes for continuous integration and continuous deployment (CI/CD) workloads. When developers execute builds that require building container images or spinning up local services within their runner pods, enabling Docker-in-Docker is a fundamental requirement. Historically, the actions-runner-controller project utilized traditional runner deployments. The evolution toward scale sets introduced more efficient pod lifecycle management via AutoscalingRunnerSet custom resources, backed by the gha-runner-scale-set Helm chart.
Within the 0.15.0 release of the chart, the dind container image is explicitly declared as a literal string within the template helper functions—specifically defined as image: docker:dind in _helpers.tpl. For public cloud environments or developer sandboxes with open internet access, this default behavior is seamless. However, for organizations operating behind corporate firewalls or entirely disconnected air-gapped infrastructure, calls to Docker Hub routinely fail due to missing internet routes or strict rate-limiting policies enforced on unauthenticated external traffic.
Engineering teams attempting to bypass this limitation have encountered persistent roadblocks within the open-source community. The underlying issue was formally documented in GitHub issue actions/actions-runner-controller#3997, opened in March 2025, which quickly accumulated substantial community engagement. The issue specifically highlighted the operational risks of reliance on Docker Hub pull limits and requested a native configuration field to override the default registry path. Despite initial contributions—such as pull request #3851, which sought to make the image parameter fully configurable—the path to native integration has faced resistance from project maintainers, leaving infrastructure teams to engineer custom downstream patch pipelines.
Community Discourse and Maintainer Stance
The friction between enterprise compliance requirements and upstream maintenance philosophies came to a head when maintainers formally addressed community pull requests. When related pull requests, including #4219, were closed without merging, ARC maintainer nikola-jokic clarified the project’s architectural direction. The maintainer noted that the core development team intentionally avoided implementing simple point-patches that only modify the dind image string, pointing instead toward advanced Helm implementation patterns.
According to the official documentation and maintainer guidance, teams wishing to customize the pod spec entirely are advised to leave the containerMode parameter empty and author the exhaustive pod specification manually within the template block. This manual approach requires platform operators to explicitly define multiple init containers, the primary runner container, and complex volume mounts. Furthermore, it places the burden of synchronization squarely on the infrastructure team: every time upstream releases a new chart version containing updates to runner internals, operators must manually reconcile their bespoke pod templates against the official chart definitions.
Although an experimental variant of the chart—published alongside version 0.15.0 under the designation gha-runner-scale-set-experimental—introduced a tentative value (runner.dind.container.image), production readiness standards strictly prohibit its usage. Official release notes explicitly warn that experimental charts are unsupported for critical production workloads and are subject to immediate modification or complete removal without prior notice. Consequently, enterprise platform engineering groups require reliable, repeatable, and automated manifest transformation methodologies.
Technical Chronology and Environment Setup
To establish a production-grade mitigation strategy, rigorous testing protocols were executed across standardized local development environments. Each patch mechanism was rendered from the official gha-runner-scale-set 0.15.0 chart using Helm 4.3.1, Kustomize 5.8.3, and Kapitan 0.36.3 equipped with the mutate 0.1.0 generator. All experiments were validated on a local kind (Kubernetes in Docker) cluster running Kubernetes version 1.37.0, alongside cross-verification tests on legacy Kubernetes 1.28.0 environments.
A critical architectural consideration during chart rendering involves Kubernetes version awareness. Starting with Kubernetes 1.29, sidecar containers graduated to general availability and were enabled by default. Consequently, the gha-runner-scale-set chart dynamically alters its template structure based on the targeted cluster version. Under Kubernetes 1.29 and above, autoscalingrunnerset.yaml evaluates capability checks and positions the dind daemon as the second init container, complete with a dedicated restartPolicy: Always and a startup probe executing docker info to ensure daemon readiness before the runner container initializes. Conversely, for clusters running version 1.28 or earlier, the chart places dind directly within the standard containers array.
Because helm template operates offline without querying a live cluster control plane, it defaults to built-in fallback values—specifically targeting Kubernetes 1.37.0 under Helm 4.3.1. Operators deploying to older cluster topologies must explicitly supply the --kube-version flag to ensure templates render with the correct container structures.
Implementing Solutions: Kustomize Over Helm Template
For organizations utilizing Kustomize as part of their GitOps pipelines (such as Argo CD or Flux), leveraging Kustomize over raw helm template output provides an elegant transformation path. The Helm values file must be properly initialized to enable the dind mode and anchor the primary runner image by both explicit tag and immutable content digest:
githubConfigUrl: https://github.com/example-org
githubConfigSecret: arc-github
controllerServiceAccount:
namespace: arc-systems
name: arc-gha-rs-controller
containerMode:
type: dind
template:
spec:
containers:
- name: runner
image: ghcr.io/actions/actions-runner:2.338.0@sha256:4ffadc0002b2581327e06101fc8c06cd189232baf79fe561fac9caeb76f5e807
command: ["/home/runner/run.sh"]Executing the initial template generation and subsequent Kustomize build requires a structured directory arrangement:
helm template arc-runners oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
--version 0.15.0 --namespace arc-runners -f values.yaml --kube-version 1.37.0 > helm.yaml
kustomize build .To achieve the image override, a corresponding kustomization.yaml file is placed adjacent to the rendered output:
resources:
- helm.yaml
images:
- name: docker
newName: mirror.gcr.io/library/docker
newTag: 29.9.1-dind
digest: sha256:3e5756e2b6dad6a5f1f5deacf0a19f9eaa2581b76ae092b2d4a118162e1d4d4eKustomize’s native images transformer is uniquely suited for this task because it recursively scans containers and initContainers arrays across arbitrary nesting depths within any Kubernetes manifest. As a result, it successfully identifies and transforms the dind reference inside the custom resource definition (AutoscalingRunnerSet) regardless of whether the target cluster utilizes init-container or standard-container sidecar layouts. Crucially, strategic merge patches are inadequate here due to the absence of an integrated OpenAPI schema for custom resources like AutoscalingRunnerSet, which would cause Kustomize to inadvertently wipe out peer configuration elements like init-dind-externals or privileged security contexts.
Alternative Implementation: Kapitan’s Mutate Generator
For enterprises standardizing on Kapitan for complex configuration management, the template rendering and mutation process can be codified entirely within parameters and classes. A standard dependency and compilation block manages the Helm chart retrieval and local output generation:
parameters:
kube_version: 1.37.0
kapitan:
dependencies:
- type: helm
source: oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set@sha256:bdfdad17cf8e45ae064c73ed6764f0c75d4bf71a6e2912e6363a43d6dfc787de
chart_name: gha-runner-scale-set
version: 0.15.0
output_path: charts/gha-runner-scale-set
compile:
- input_type: helm
input_paths: [charts/gha-runner-scale-set]
output_path: gha-runner-scale-set
helm_values_files: [values.yaml]
helm_params:
name: arc-runners
namespace: arc-runners
kube_version: $kube_versionBecause Kapitan’s dedicated images action primarily targets native workloads such as Pods and standard workload controllers rather than arbitrary custom resources, the required modification is handled via JSON Patch (JSON6902) execution:
parameters:
mutations:
dind-image:
match: apiVersion: actions.github.com/v1alpha1, kind: AutoscalingRunnerSet, name: arc-runners
json6902:
- op: test, path: /spec/template/spec/initContainers/1/name, value: dind
- op: replace
path: /spec/template/spec/initContainers/1/image
value: mirror.gcr.io/library/docker:29.9.1-dind@sha256:3e5756e2b6dad6a5f1f5deacf0a19f9eaa2581b76ae092b2d4a118162e1d4d4eThe incorporation of the test operation acts as a reliable defensive programming safeguard: compilation halts immediately if the second init container does not match the expected dind identifier, preventing silent drift caused by upstream chart restructuring. For legacy clusters operating on Kubernetes versions prior to 1.29, operators simply shift the mutation target path from initContainers to standard containers.
Air-Gapped Registry Replication via ORAS
While public pull-through mirrors like mirror.gcr.io provide convenient caching mechanisms that mirror exact upstream image indexes and digests, fully isolated air-gapped environments require local artifact mirroring. To achieve this without altering content hashes, operators can utilize the OCI Registry As Storage (ORAS) CLI tool to replicate the exact multi-architecture image index directly to an internal corporate registry:
oras cp -r mirror.gcr.io/library/docker:29.9.1-dind@sha256:3e5756e2b6dad6a5f1f5deacf0a19f9eaa2581b76ae092b2d4a118162e1d4d4e
registry.example.internal/library/docker:29.9.1-dindBy leveraging oras cp, administrators ensure that the multi-platform index—supporting architectures including linux/amd64, linux/arm64v8, linux/armv6, and linux/armv7—retains its cryptographic content addressability (sha256:3e5756e2b...). Consequently, downstream manifest patches require only a simple URI substitution pointing to the internal registry domain while preserving absolute execution integrity.
Verification and Operational Implications
Post-implementation validation within test clusters confirms that API servers successfully accept mutated manifests across both Kustomize and Kapitan pipelines. Inspection of stored custom resource definitions verifies that sidecar container attributes—including restart policies and execution arguments—remain intact while pointing successfully to private mirrors. Furthermore, verification pods executed against temporary runners demonstrate nominal initialization speeds, clean container handoffs from the dind daemon startup probe to the primary runner execution context, and accurate reporting of daemon version strings.
Ultimately, while the lack of a native Helm value in Actions Runner Controller v0.15.0 introduces initial friction for secure and isolated deployments, robust post-render tooling bridges the gap effectively. Enterprise platform teams can successfully maintain compliance, bypass external rate limits, and secure their software supply chains without sacrificing the scalability benefits of modern GitHub Actions scale sets.
