Guides
Log InSign UpBlog
Start typing to search…

Providers Cache

OpenTofu/Terraform architecture is built around the use of plugins. Each initialization triggers the download of all providers defined in the configuration. Provider sizes vary, but on average can reach hundreds of megabytes. Large configurations with many providers may result in gigabytes of data being downloaded and processed during each init, potentially taking several minutes just to start, and this process is repeated for every run in remote cloud environments.

To address this issue, the Scalr Agent can cache OpenTofu/Terraform providers across all Scalr runs on a single Scalr Agent service node, ensuring that each provider is downloaded and stored on disk only once — significantly improving OpenTofu/Terraform initialization times.

Architecture

The Provider Cache uses an opt-in architecture and is enabled by default. See SCALR_AGENT_PROVIDER_CACHE_ENABLED for details.

The Provider Cache is implemented by maintaining a filesystem_mirror in the providers directory within the agent data directory.

🚧

The examples below use the same configuration with 20 providers and an estimated disk usage of 1.4 GB on an M1 macOS machine.

When running a Scalr Plan for the first time, you will see the following log output in the Run console during stage initialization:

Initializing plugins and modules...
Initialized 21 plugins and 0 modules in 79.39s (21 plugins downloaded)
Caching 21 locked plugins after changes to the HCL lockfile...
Cached 21 plugins in 7.26s
Validating configuration files...

During this step, OpenTofu/Terraform downloads and installs all required plugins into a per-run directory. This directory is configured for the Terraform process using the TF_PLUGIN_CACHE_DIR environment variable.

After each run completes, the Scalr Agent re-verifies provider checksums using the Provider Network Protocol and caches valid providers in $SCALR_AGENT_DATA_DIR/providers. You can find them at

  * /var/lib/scalr-agent
    * **providers/**
      * registry.opentofu.org
        * datadog
          * datadog
            * 3.69.0
              * linux\_arm64
                * CHANGELOG.md
                * LICENSE
                * LICENSE-3rdparty.csv
                * README.md
                * terraform-provider-datadog\_v3.69.0
        * hashicorp
          * archive
            * 2.7.1
              * linux\_arm64
                * CHANGELOG.md
                * LICENSE
                * README.md
                * terraform-provider-archive
          * aws
            * 5.89.0
              * linux\_arm64
                * CHANGELOG.md
                * LICENSE
                * README.md
                * terraform-provider-aws
          * …
    * providers-metadata/
      * …
    * …

The Provider Cache also tracks metadata for installed providers. The metadata directory is located alongside the providers at $SCALR_AGENT_DATA_DIR/providers-metadata and contains information for each cached provider, including registry details, hashes, installation paths, and usage data (which is incremented after each run). This metadata is later used for garbage collection and monitoring Provider Cache usage via OpenTelemetry Metrics

  * /var/lib/scalr-agent
    * providers/
      * …
    * **providers-metadata/**
      * registry.opentofu.org-datadog-datadog-3.69.0-linux\_arm64.json
      * registry.opentofu.org-hashicorp-archive-2.7.1-linux\_arm64.json
      * registry.opentofu.org-hashicorp-aws-5.89.0-linux\_arm64.json
      * …
    * …

An example of metadata record:

{
  "source": "registry.opentofu.org/hashicorp/aws",
  "version": "5.89.0",
  "h1_hash": "h1:z/jQTjIfsqd7hiFrn+5l07jWs0aWJ5pHpUGkXcJUL1s=",
  "zh_hash": "zh:2ee7bc119ef769eef4c02e3a590b23228ff660d15f1e2b3352cae0cafc0f1fc5",
  "usage": 27,
  "last_used_at": 1753961593.216591,
  "size_bytes": 632306088,
  "provider_path": "/var/lib/scalr-agent/providers/registry.opentofu.org/hashicorp/aws/5.89.0/linux_arm64"
}

All subsequent runs will mount all registries in $SCALR_AGENT_DATA_DIR/providers as a filesystem_mirror in read-only mode, allowing OpenTofu/Terraform to use existing unpacked providers.

A following run would show a log like this:

Initializing plugins and modules...
Initialized 20 plugins and 0 modules in 6.09s (20 plugins used from cache)
Validating configuration files...

As a result, provider initialization time is reduced from ~87 seconds to ~6 seconds for all subsequent runs using this configuration.

Handling Lockfile h1 Checksums

OpenTofu and Terraform lockfiles typically include two types of checksums:

  • zh: checksums: Represent zip archive checksums (SHA256) for provider packages downloaded over the network.
  • h1: checksums: Represent directory checksums (SHA256 of all files inside directory) of unpacked providers on the local filesystem.

A typical provider lock entry might look like this:

provider "registry.opentofu.org/hashicorp/aws" {
  version     = "5.89.0"
  constraints = ">= 5.60.0, <= 5.89.0"
  hashes = [
    "h1:8l5APULZzN4d0bLnMzZ2X+3a++zO77uo8JfvhrKa+NY=",
    "zh:17d7f69ef81f8c6f8e330d92b269b54e85a19e8956475f25b2902900fcbea873",
    "zh:2ee7bc119ef769eef4c02e3a590b23228ff660d15f1e2b3352cae0cafc0f1fc5",
    "zh:32078c00deb0a99b92d7e013434ffff6925f43e01f5734d64b7a2bb469d372d1",
    "zh:53c6e72358ffb5bf9dbf7df23f8b0291dd03176b8eec5957a2b4beb110d4c6bf",
    "zh:62804018e90989434b5d561382f7ed28e7a524376baa3a79cb764924eecfb2b0",
    "zh:a0206e4676e0b9ce84d16e16660cc1aa5e86f1ddba5980386b26edd1567ef17f",
    "zh:ac489622ff0e5e75a404c8dd385c91aa3a4a292e12dea4c5d0b968865841cc95",
    "zh:b2ba22dfab6427b4d2c01a7e560a93a6078d249fd5a588480f936b97d757f4b3",
    "zh:bd4c7a1f199acabc82dd764cd2845eeb72ab3b0ed0d9a4e4cf714bba0d1ff2e8",
    "zh:df392bf5dd4bb2f0fdfc3c68829bd943c713eb656632d51b30ac5a9545548fb0",
  ]
}

This example includes zh: checksums all available platforms - such as linux_amd64, linux_arm64, windows_amd64, darwin_arm64 and so on. However, the h1: checksum is included only for the platform on which the lockfile was originally generated - unless explicitly set using the tofu provider lock --platform option.

In a configuration using filesystem_mirror, this setup works fine for fresh installs using network archives. However, when using cached providers from the local filesystem_mirror, a checksum mismatch can occur:

Initializing plugins and modules...
╷
│ Error: Failed to install provider
│
│ Error while installing hashicorp/archive v2.7.0: the local package for
│ registry.opentofu.org/hashicorp/archive 2.7.0 doesn't match any of the
│ checksums previously recorded in the dependency lock file (this might be
│ because the available checksums are for packages targeting different
│ platforms); for more information:
│ https://opentofu.org/docs/language/files/dependency-lock/#checksum-verification
╵

This may happen because when OpenTofu/Terraform core selects a provider from the filesystem_mirror, it matches checksums using h1 values, not zh. If the lockfile was generated on desktop operating systems like Windows or macOS, and the agent is running on a different platform (e.g., linux_arm64 or linux_amd64).

To fix this, the Scalr Agent updates the Terraform Dependency Lock File (.terraform.lock.hcl) with the correct h1 checksums for the current platform from the provider cache directory before running the tofu/terraform init command.

The agent will log a message like this:

2025-07-30 12:09:29.322 | INFO      | tacoagent.app.terraform.service.lock: Injected h1 hashes of cached providers for the linux_arm64 platform into the dependency lockfile. account_id=acc-svrcncgh453bi8g agent_id=agent-v0oti6vuf0l8a524u container_id=523289daf9feea61bfa3baa6a324c4fb421b5ebc92b86ea6c3d1033181fb2e9a diff={'registry.terraform.io/datadog/datadog': 'h1:jmBLy5PKzWIYqEnzyIJ+V0/rNLsNHLBYeOGWuaF7FCI=', 'registry.terraform.io/hashicorp/google': 'h1:KHIoAQs+PP62BY6N+BV9DKqcb5EnvuT/vvsiX+2Xrmw=', 'registry.terraform.io/hashicorp/null': 'h1:obXguGZUWtNAO09f1f9Cb7hsPCOGXuGdN8bn/ohKRBQ='} id=atask-v0oti8pasd1tapuat name=terraform.plan plan_id=plan-v0oti8pamcohafsuc run_id=run-v0oti8palmddf611k workspace_id=ws-v0ot0loj7k1r4qr4e

The checksum will be injected only after verifying the corresponding zh hash, which is included in the metadata record of each provider installed in the cache.

Provider Cache Warm-Up

By default, OpenTofu/Terraform downloads providers sequentially during the tofu/terraform init process.

The Scalr Agent offers an optimization feature, enabled by default and configurable via the SCALR_AGENT_PROVIDER_CACHE_WARM_UP_FROM_LOCKFILE environment variable.

This feature pre-warms the provider cache by concurrently downloading providers listed in the dependency lockfile before OpenTofu/Terraform initialization begins.

Installing 20 plugins from the dependency lock file...
Installed 20 plugins from the dependency lock file in 42.47s (20 downloaded)
Initializing plugins and modules...
Initialized 20 plugins and 0 modules in 7.25s (20 plugins used from cache)
Validating configuration files...

During this process, the Scalr Agent concurrently downloads, unpacks, and installs providers into the cache using its own implementation of the Provider Registry Protocol. This optimization can reduce cold startup times by up to 50%.

🚧

The warm-up feature does not yet support the network mirror setting, which can be injected using a custom .terraformrc file.

Set SCALR_AGENT_PROVIDER_CACHE_WARM_UP_FROM_LOCKFILE=0 when using network mirrors.

Garbage Collection

Over time, the plugin cache can grow significantly on disk. The Scalr Agent performs periodic cleanups of stale or rarely used providers based on the last_used_at, usage, and size_bytes attributes from the metadata file.

Garbage collection runs each time the agent starts and daily at 00:00 (midnight).

The garbage collection task deletes providers that have not been used for more than 10 days, as well as providers that exceed the SCALR_AGENT_PROVIDER_CACHE_SIZE_LIMIT_MB limit, unless they were installed in the cache within the last 12 hours.

Providers used less frequently are prioritized for deletion over those used more often.

Limitations

  • Terraform < 0.14 doesn't support the Provider Cache.
  • Terragrunt doesn't support the Provider Cache yet.

Updated 1 day ago