Mutual TLS authentication

This topic describes mutual TLS (mTLS) for Scalr self-hosted agent pools.

📘

This page covers mTLS between a Scalr Agent and Scalr. For mTLS on the webhook that Scalr calls for a serverless agent pool, see Webhook mutual TLS.

Overview

Mutual TLS, or mTLS, is an industry-standard protocol for mutual authentication between a client and a server. Both parties present a certificate issued by a trusted certificate authority (CA), and each side verifies the other before the connection is established. Unlike standard TLS, which authenticates only the server, mTLS authenticates both sides.

Scalr supports mTLS for both Run agents, which execute Terraform/OpenTofu workloads, and VCS agents, which connect Scalr to self-hosted VCS providers.

Benefits

  • Stolen token protection - a JWT intercepted in transit or leaked from a secrets store is useless without the agent's private key. Credential stuffing and token replay attacks cannot succeed against pools with mTLS enabled.
  • On-path attack prevention - an on-path attacker cannot authenticate to Scalr without the agent's private key, making interception and modification of agent traffic ineffective.
  • Spoofing prevention - an attacker cannot imitate a legitimate agent. Both the JWT and the certificate must be valid and must match the same pool, so neither side can be spoofed independently.
  • Compliance - supports strong-authentication requirements common in regulated environments, including those governed by frameworks such as PCI-DSS and HIPAA.

How it works

Scalr maintains a private Certificate Authority (CA) that signs agent client certificates. The
agent's operator generates a keypair locally and sends only the public key to Scalr as a Certificate
Signing Request (CSR)
. Scalr signs it and returns a client certificate scoped to a specific agent pool and account. The private key never leaves the operator's environment.

Certificate identity is set server-side: Scalr discards the CSR subject and stamps the issued
certificate with the agent pool ID and account ID. The same identity fields are checked on every
connection, so a certificate signed for one pool cannot authenticate against another.

Before the agent starts, the operator places the signed certificate and the private key on disk
and points the agent to them with SCALR_AGENT_TLS_CERT_FILE
and SCALR_AGENT_TLS_KEY_FILE.
The agent uses those files for every outbound HTTPS connection to Scalr, including API requests,
relay connections, and blob service traffic. All agents in a pool share the same certificate and
key - the same pair is distributed to every host.

On every connection, the agent presents both the JWT and the certificate. Scalr verifies both -
if either check fails, the request is rejected. The two layers serve different purposes:

LayerMechanismPurpose
AuthenticationJWT Bearer tokenIdentifies the agent pool and authorizes operations
Transport securitymTLS client certificateProves the JWT is being presented by a legitimate agent host

Session tokens that Scalr issues to the agent are bound to the specific certificate presented at
issuance. The binding stays in effect for the token's lifetime, so rotating the certificate
invalidates any cached tokens until the agent reconnects and obtains new ones.

Requirements

  • Enterprise plan
  • Scalr Agent version 1.1.0 or later
  • The agent-pools:update permission on the target agent pool

Setup

Complete these steps before launching the agent for the first time against an agent pool. The
setup applies to both Run and VCS agents - certificates are scoped per agent pool, regardless of
which workload the pool runs.

1. Create an agent pool with mTLS enforcement

mTLS is enforced per agent pool. The pool owner must create a new agent pool with mTLS
enforcement enabled - this is the setting that tells Scalr to validate client certificates on
incoming agent connections. Scalr checks mTLS status only on pools where enforcement is enabled;
pools without it continue to authenticate agents using JWT only, even if the agent presents a
certificate.

Plan the rollout per pool. Existing pools without mTLS enforcement keep working unchanged, and
agents that already hold a JWT for those pools do not need to be reprovisioned. To migrate an
existing workload onto mTLS, create a new pool with enforcement enabled, issue certificates, and
move agents to the new pool.

2. Generate a keypair and CSR

Scalr recommends EC P-256 (prime256v1) keys. EC signatures are smaller and faster to verify than
RSA, which keeps per-connection mTLS handshake overhead low.

These instructions place the certificate and key under ~/.scalr-agent/ssl/. Use this path or
substitute your own, and update the references in later steps accordingly.

Run the following commands on the host where the agent will run:

mkdir -p ~/.scalr-agent/ssl
openssl ecparam -genkey -name prime256v1 -noout \
  -out ~/.scalr-agent/ssl/scalr-agent.key
openssl req -new -key ~/.scalr-agent/ssl/scalr-agent.key \
  -out ~/.scalr-agent/ssl/scalr-agent.csr \
  -subj "/CN=agent-pool"

This produces scalr-agent.key (private key) and scalr-agent.csr (the CSR). The CSR subject
is discarded by Scalr - identity is set server-side at signing time using the agent pool ID and
account ID.

🚧

Store scalr-agent.key in a secure location. It cannot be recovered or re-downloaded - if lost,
you must generate a new keypair and re-register with Scalr.

3. Upload the CSR

Open the agent pool page in the Scalr UI and submit scalr-agent.csr from the Certificates
tab. Scalr signs the CSR and returns the signed certificate. Save the returned certificate as
scalr-agent.crt in the same directory as scalr-agent.key.

4. Configure and start the agent

The agent reads the certificate and key paths from
SCALR_AGENT_TLS_CERT_FILE
and SCALR_AGENT_TLS_KEY_FILE,
and authenticates to the pool with SCALR_AGENT_TOKEN.
Both TLS file variables must be set together - setting only one is rejected at startup. If neither
is set, the agent runs in JWT-only mode and is rejected by any pool that requires mTLS.

Choose the deployment that matches your environment. Once running, the agent validates the
certificate and key files at startup and uses them for every outbound HTTPS connection to Scalr -
API requests, relay traffic, and blob service requests.

Mount ~/.scalr-agent/ssl/ from the host and point the env vars at the in-container path:

docker run -d \
  --name scalr-agent \
  -e SCALR_AGENT_TOKEN="<agent pool token>" \
  -e SCALR_AGENT_TLS_CERT_FILE=/etc/scalr-agent/ssl/scalr-agent.crt \
  -e SCALR_AGENT_TLS_KEY_FILE=/etc/scalr-agent/ssl/scalr-agent.key \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/lib/scalr-agent:/var/lib/scalr-agent \
  -v ~/.scalr-agent/ssl:/etc/scalr-agent/ssl:ro \
  --pull=always \
  scalr/agent:latest

5. Verify

After the agent starts, confirm mTLS is working by checking the agent log for two INFO
messages:

INFO tacoagent.cmd: mTLS client certificate loaded tls_cert_file=scalr-agent.crt tls_key_file=scalr-agent.key san=urn:scalr:agent-pool:apool-xxx, urn:scalr:account:acc-xxx not_valid_after=2027-06-22T11:35:14+00:00
INFO tacoagent.cmd: Connected to agent pool agent_name=<your-agent> pool_name=<your-pool>

The first line confirms the agent successfully loaded the certificate and key files. The second
confirms the mTLS handshake succeeded and the agent established a session with the pool. Together
they show mTLS is operational.

📘

Scalr does not emit an audit-log event for each successful mTLS handshake - only rejections are recorded. The agent-side log is the source of truth for healthy connections.

Rotation and revocation

Pool certificates are issued with a default lifetime of 90 days. The minimum lifetime is 1 day and
the maximum is 730 days (2 years). Routine periodic rotation is not required - rotate when the
private key is compromised, when you reprovision the pool, or when the certificate is approaching
expiry.

A pool can have multiple active certificates at once, which makes zero-downtime rotation
straightforward. Each issued certificate is tracked individually and can be revoked without
affecting the others.

The agent logs a WARNING at startup when the loaded certificate is approaching its expiry:

WARNING tacoagent.cmd: mTLS client certificate is expiring soon. Once expired the agent will fail to establish TLS connections to the Scalr platform causing all runs to fail, renew the certificate before it expires not_valid_after=2026-05-07T11:35:14+00:00 seconds_until_expiry=604800

Treat this warning as a rotation signal - reissue and redeploy the certificate before
not_valid_after passes.

View certificates

The Certificates tab on the agent pool page lists all certificates issued for the pool, along
with their status (active, expired, or revoked), fingerprint, and expiry timestamp.

Rotate a certificate

To replace a certificate without disrupting running agents:

  1. Generate a new keypair and CSR on the host (see Generate a keypair and CSR).
  2. Submit the new CSR from the Certificates tab. The new certificate is added to the pool's
    active set alongside the existing one - both are accepted during the migration window.
  3. Distribute the new certificate and key to the agent hosts and restart the agents so they pick
    up the new credentials.
  4. Once all agents are running with the new certificate, revoke the old one.

For emergency rotation following a suspected key compromise, Scalr supports issuing a new
certificate that atomically replaces all previously active fingerprints for the pool. This
invalidates every prior certificate in a single step, so all agents in the pool must be
reprovisioned with the new credentials before they can reconnect.

Revoke a certificate

Revoke a specific certificate from the Certificates tab. Revocation removes the certificate
fingerprint from the pool's active set, so any agent presenting it is rejected on its next
connection. Revocation cannot be undone - issue a new certificate to bring the agent back online.

Limitations

  • Automatic certificate rotation is not yet available. Certificates must be reissued and redistributed manually before they expire. Track expiry against the lifetimes shown on the Certificates tab.
  • mTLS protects only agent-to-Scalr connections. Traffic from workloads the agent runs - for example, Terraform/OpenTofu and its plugins fetching modules or providers from a private Scalr registry - is sent by the IaC process itself, not by the agent. Those connections do not present the agent's client certificate, so mTLS does not extend to them.

Troubleshooting

When an agent fails to connect to a pool with mTLS enforcement enabled, Scalr returns a
401 Unauthorized response with a descriptive message. Two recommendations cover most cases:

  • Reissue the mTLS certificate. If the certificate is expired, revoked, or no longer accepted
    by the pool, issue a new one from the Certificates tab on the agent pool page and redeploy
    it to the agent host. Restart the agent so it presents the new credentials and obtains a fresh
    session token bound to the current certificate.
  • Check the audit log for extended details. Open Audit Logs and look for
    the matching rejection event - it identifies the specific check that failed and provides
    fingerprint and identity context for correlating the rejection with pool state.

Did this page help you?