OpenTofu/Terraform architecture is built around the use of plugins. Each initialization triggers the download of all providers defined in the configuration. Provider sizes vary, but on average can reach hundreds of megabytes. Large configurations with many providers may result in gigabytes of data being downloaded and processed during each init, potentially taking several minutes just to start, and this process is repeated for every run in remote cloud environments.

To address this issue, the Scalr Agent can cache OpenTofu/Terraform providers across all Scalr runs on a single Scalr Agent service node, ensuring that each provider is downloaded and stored on disk only once — significantly improving OpenTofu/Terraform initialization times.

Architecture

The Provider Cache uses an opt-in architecture and is enabled by default. See SCALR_AGENT_PROVIDER_CACHE_ENABLED for details.

The Provider Cache is implemented by maintaining a filesystem_mirror in the providers directory within the agent data directory.

🚧
The examples below use the same configuration with 20 providers and an estimated disk usage of 1.4 GB on an M1 macOS machine.

When running a Scalr Plan for the first time, you will see the following log output in the Run console during stage initialization:

Initializing plugins and modules...
Initialized 20 plugins and 0 modules in 79.39s (20 plugins downloaded)
Caching 20 locked plugins after changes to the HCL lockfile...
Cached 20 plugins in 7.26s
Validating configuration files...

During this step, OpenTofu/Terraform downloads and installs all required plugins into a per-run directory. This directory is configured for the Terraform process using the TF_PLUGIN_CACHE_DIR environment variable.

After each run completes, the Scalr Agent re-verifies provider checksums using the Provider Network Protocol and caches valid providers in $SCALR_AGENT_DATA_DIR/providers. You can find them at

/var/lib/scalr-agent
├── providers/
│   ├── registry.opentofu.org
│   │   ├── datadog
│   │   │   └── datadog
│   │   │       └── 3.69.0
│   │   │           └── linux_arm64
│   │   │               ├── CHANGELOG.md
│   │   │               ├── LICENSE
│   │   │               ├── LICENSE-3rdparty.csv
│   │   │               ├── README.md
│   │   │               └── terraform-provider-datadog_v3.69.0
│   │   └── hashicorp
│   │       ├── archive
│   │       │   └── 2.7.1
│   │       │       └── linux_arm64
│   │       │           ├── CHANGELOG.md
│   │       │           ├── LICENSE
│   │       │           ├── README.md
│   │       │           └── terraform-provider-archive
│   │       └── aws
│   │           └── 5.89.0
│   │               └── linux_arm64
│   │                   ├── CHANGELOG.md
│   │                   ├── LICENSE
│   │                   ├── README.md
│   │                   └── terraform-provider-aws
│   └── …
├── providers-metadata/
│   └── …
└── …

The Provider Cache also tracks metadata for installed providers. The metadata directory is located alongside the providers at $SCALR_AGENT_DATA_DIR/providers-metadata and contains information for each cached provider, including registry details, hashes, installation paths, and usage data (which is incremented after each run). This metadata is later used for garbage collection and monitoring Provider Cache usage via OpenTelemetry Metrics

/var/lib/scalr-agent
├── providers/
│   └── …
├── providers-metadata/
│   ├── registry.opentofu.org-datadog-datadog-3.69.0-linux_arm64.json
│   ├── registry.opentofu.org-hashicorp-archive-2.7.1-linux_arm64.json
│   ├── registry.opentofu.org-hashicorp-aws-5.89.0-linux_arm64.json
│   └── …
└── …

An example of metadata record:

{
  "source": "registry.opentofu.org/hashicorp/aws",
  "version": "5.89.0",
  "h1_hash": "h1:z/jQTjIfsqd7hiFrn+5l07jWs0aWJ5pHpUGkXcJUL1s=",
  "zh_hash": "zh:2ee7bc119ef769eef4c02e3a590b23228ff660d15f1e2b3352cae0cafc0f1fc5",
  "usage": 27,
  "last_used_at": 1753961593.216591,
  "size_bytes": 632306088,
  "provider_path": "/var/lib/scalr-agent/providers/registry.opentofu.org/hashicorp/aws/5.89.0/linux_arm64"
}

All subsequent runs will mount all registries in $SCALR_AGENT_DATA_DIR/providers as a filesystem_mirror in read-only mode, allowing OpenTofu/Terraform to use existing unpacked providers.

A following run would show a log like this:

Initializing plugins and modules...
Initialized 20 plugins and 0 modules in 6.09s (20 plugins used from cache)
Validating configuration files...

As a result, provider initialization time is reduced from ~87 seconds to ~6 seconds for all subsequent runs using this configuration.

Resolving Checksum Mismatch Errors

When running OpenTofu/Terraform configurations in a remote environment using Scalr Agent, you may encounter an error like this:

Initializing plugins and modules...
╷
│ Error: Failed to install provider
│
│ Error while installing hashicorp/archive v2.7.0: the local package for
│ registry.opentofu.org/hashicorp/archive 2.7.0 doesn't match any of the
│ checksums previously recorded in the dependency lock file (this might be
│ because the available checksums are for packages targeting different
│ platforms); for more information:
│ https://opentofu.org/docs/language/files/dependency-lock/#checksum-verification
╵

This error occurs when the checksums in the Terraform Dependency Lock File not match the provider's checksums for the target platform the agent is running on.

The lockfile can include two types of checksums:

zh: checksums: Represent zip archive checksums (SHA256) for provider packages downloaded over the network.
h1: checksums: Represent directory checksums (SHA256 of all files inside directory) of unpacked providers on the local filesystem.

A typical provider lock entry within the file looks like this:

provider "registry.opentofu.org/hashicorp/aws" {
  version     = "5.89.0"
  constraints = ">= 5.60.0, <= 5.89.0"
  hashes = [
    "h1:8l5APULZzN4d0bLnMzZ2X+3a++zO77uo8JfvhrKa+NY=",
    "zh:17d7f69ef81f8c6f8e330d92b269b54e85a19e8956475f25b2902900fcbea873",
    "zh:2ee7bc119ef769eef4c02e3a590b23228ff660d15f1e2b3352cae0cafc0f1fc5",
    "zh:32078c00deb0a99b92d7e013434ffff6925f43e01f5734d64b7a2bb469d372d1",
    "zh:53c6e72358ffb5bf9dbf7df23f8b0291dd03176b8eec5957a2b4beb110d4c6bf",
    "zh:62804018e90989434b5d561382f7ed28e7a524376baa3a79cb764924eecfb2b0",
    "zh:a0206e4676e0b9ce84d16e16660cc1aa5e86f1ddba5980386b26edd1567ef17f",
    "zh:ac489622ff0e5e75a404c8dd385c91aa3a4a292e12dea4c5d0b968865841cc95",
    "zh:b2ba22dfab6427b4d2c01a7e560a93a6078d249fd5a588480f936b97d757f4b3",
    "zh:bd4c7a1f199acabc82dd764cd2845eeb72ab3b0ed0d9a4e4cf714bba0d1ff2e8",
    "zh:df392bf5dd4bb2f0fdfc3c68829bd943c713eb656632d51b30ac5a9545548fb0",
  ]
}

This entry includes zh: checksums all platforms available for a plugin – such as linux_amd64, linux_arm64, windows_amd64, darwin_arm64 and so on. However, the h1: checksum is included only for the platform on which the lockfile was originally generated - unless explicitly set using the tofu provider lock --platform option.

The zh: checksum is used to verify fresh installations against network archives, while the h1: checksum is used for subsequent installations with cached providers from the local filesystem_mirror (if the provider cache is enabled).

In case the lockfile was generated on a different platform, the h1: checksums for the target platform may be missing. A common scenario is when the lockfile is generated on a desktop operating system such as macOS or Windows, while the agent is running on a server platform like linux_arm64 or linux_amd64.

To resolve this, the Scalr Agent updates the lockfile with the correct h1: checksums for the current platform from the provider cache directory before running the tofu/terraform init command.

The agent will log a message like this:

2025-07-30 12:09:29.322 | INFO      | tacoagent.app.terraform.service.lock: Injected h1 hashes of cached providers for the linux_arm64 platform into the dependency lockfile. account_id=acc-svrcncgh453bi8g agent_id=agent-v0oti6vuf0l8a524u container_id=523289daf9feea61bfa3baa6a324c4fb421b5ebc92b86ea6c3d1033181fb2e9a diff={'registry.terraform.io/datadog/datadog': 'h1:jmBLy5PKzWIYqEnzyIJ+V0/rNLsNHLBYeOGWuaF7FCI=', 'registry.terraform.io/hashicorp/google': 'h1:KHIoAQs+PP62BY6N+BV9DKqcb5EnvuT/vvsiX+2Xrmw=', 'registry.terraform.io/hashicorp/null': 'h1:obXguGZUWtNAO09f1f9Cb7hsPCOGXuGdN8bn/ohKRBQ='} id=atask-v0oti8pasd1tapuat name=terraform.plan plan_id=plan-v0oti8pamcohafsuc run_id=run-v0oti8palmddf611k workspace_id=ws-v0ot0loj7k1r4qr4e

The Scalr Agent will inject a new h1: hash into an existing provider only if the hash from the filesystem_mirror cache also matches one of the existing zh: hashes. Providers installed in the Scalr Agent–managed filesystem_mirror track record both h1: and zh: hashes.

📘
In summary, to avoid provider installation errors caused by checksum mismatches, the lockfile must include either the h1: or zh: checksum for the platform on which the agent is running. Otherwise, the provider cannot be installed securely.

If it is not possible to include the appropriate zh: checksums (for example, when installing providers from a mirror that does not provide official signed checksums), you should generate the lockfile with the agent’s platforms explicitly specified:

tofu providers lock \
  -platform=linux_arm64 \
  -platform=linux_amd64

Provider Cache Warm-Up

By default, OpenTofu/Terraform downloads providers sequentially during the tofu/terraform init process.

The Scalr Agent offers an optimization feature, enabled by default and configurable via the SCALR_AGENT_PROVIDER_CACHE_WARM_UP_FROM_LOCKFILE environment variable.

This feature pre-warms the provider cache by concurrently downloading providers listed in the dependency lockfile before OpenTofu/Terraform initialization begins.

Installing 20 plugins from the dependency lock file...
Installed 20 plugins from the dependency lock file in 42.47s (20 downloaded)
Initializing plugins and modules...
Initialized 20 plugins and 0 modules in 7.25s (20 plugins used from cache)
Validating configuration files...

During this process, the Scalr Agent concurrently downloads, unpacks, and installs providers into the cache using its own implementation of the Provider Registry Protocol. This optimization can reduce cold startup times by up to 50%.

🚧
The warm-up feature does not yet support the network mirror setting, which can be injected using a custom .terraformrc file.
Set SCALR_AGENT_PROVIDER_CACHE_WARM_UP_FROM_LOCKFILE=0 when using network mirrors.

Garbage Collection

Over time, the plugin cache can grow significantly on disk. The Scalr Agent performs periodic cleanups of stale or rarely used providers based on the last_used_at, usage, and size_bytes attributes from the metadata file.

Garbage collection runs each time the agent starts and daily at 00:00 (midnight).

The garbage collection task deletes providers that have not been used for more than 10 days, as well as providers that exceed the SCALR_AGENT_PROVIDER_CACHE_SIZE_LIMIT_MB limit, unless they were installed in the cache within the last 12 hours.

Providers used less frequently are prioritized for deletion over those used more often.

Limitations

Terraform < 0.14 doesn't support the Provider Cache.
Terragrunt doesn't support the Provider Cache yet.