Private Module Registry

The module registry can be used to publish modules directly from VCS providers without having to provide or share VCS credentials with other users that need to consume the module. The Scalr module registry is not designed in an opinionated way, all repository structures are supported (module per repo, mono-repo, sub-modules, etc). When adding modules to Scalr, they can be used in two different ways:

• Library module - Boilerplate code with the internal source reference that can be used to call the module from any Terraform or OpenTofu configuration in a Scalr environment.
• Deployable module - The same as a library module, but can be deployed through the module registry UI to create a no-code workspace.

The module registry also provides version control for modules. Modules can only be registered if Git releases or tagged versions have been created in the repository. Module calls via the registry must use versions, thus new versions can be released without impacting existing deployments.

Here is an example module call from the registry:

module "instance" {
  source  = "my-account.scalr.io/account-name/instance/aws"
  version = "1.0.1"
  instance_type = var.instance_type
  instance_count = var.instance_count
  subnet = var.subnet
  sg = var.security_group
  key = var.ssh_key
  vpc_id = var.vpc_id
  ami = var.ami
}

📘

Note

A git release is a tagged version of the code. Once created a release never changes as it is pinned to a specific commit. Tags must use semantic version numbering (m.m.p, e.g v1.0.4, 0.7.3) and can be created via the CLI (see git tag) or in the git VCS console via the “releases” page.

Modules in the registry are automatically pulled into workspaces where they are called and the registration process automatically creates internal references to the module to be used in the Terraform configuration.

Alternatively, it is possible to use external registries, such as those stored in Git, and use SSH keys to pull the modules into the run as needed.

Publishing Modules

A module can be registered at the Scalr environment or account scope. Modules registered at the account scope can be used in any environment or workspace. Scalr supports a module per repo or sub-modules (aka mono repo). In both cases, the modules use semantic versioning from the VCS repository. In a module per repo, it pulls the versions for the entire repo, in the sub-module use case, Scalr will reference tag prefixes so each individual module can have its own versioning. To publish a module, navigate to “Modules”, and click on “New Module”:

Module-Per-Repo

For modules in their own unique repository, the repository itself must be named using the format terraform-<provider_name>-<module_name>, e.g. terraform-aws-vcs_module. Here is an example from GitHub:

A release tag is required for every module for versioning:

To create the module in Scalr, just define the repository and publish:

Sub-Modules (Mono Repo)

For modules in a mono repo, the repository itself can have any name, but the sub-folders for each module must use the format terraform-<provider_name>-<module_name>, e.g. terraform-aws-instance. Here is an example from Github:

For the tagging, Scalr will use a tag prefix so each individual module can have its own versioning. In this example, there are two modules, one for sg and one for instance, each one of these has a tag prefix like so:

Let’s say you wanted to add the sg module to Scalr, this would be done with the following settings:

Scalr will now only show the versions for sg/ in that repo:

No-Code Deployments

A no-code workspace can be deployed through the Scalr UI by creating a workspace from the module registry. By default, when a module is registered in the Scalr module registry it will be added to the list of modules that can be used to create a workspace. In some cases, modules should only be library modules and users should not see a module in the dropdown of deployable modules in the UI. To prevent a module from appearing in the list of deployable modules, you must add a scalr-module.hcl file to the repo that contains the module. The scalr-module.hcl uses rules based on gitignore specifications to restrict which modules can be used for no-code workspaces.

Modules that can be used to create a workspace are also controlled through Git releases. If you update the scalr-module.hcl to include or exclude a module, then a new release must be published and only that module release will be included or excluded from the list of deployable modules. If you want former module versions to be excluded from the deployable module dropdown list, you must re-tag those versions from your VCS provider.

Here are some scalr-module.hcl examples:

Exclude all modules in the repository:

version="v2"
root-modules = ["!**"]

All modules in the modules/ directory are included, except for the system module:

version="v2"
root-modules = ["modules/**", "!modules/system"]

Any module at the root directory and only modules/foo and modules/bar in the sub directory are included:

version="v2"
root-modules = [".", "modules/foo", "modules/bar"]

See more syntax examples in the gitignore documentation.

Module Source Types

When a module is called from a Terraform configuration the source parameter specifies where Terraform should pull the module from. Terraform configuration files executed in Scalr can use 3 source types:

Modules from Scalr

This is an example of a Terraform configuration file executed via the CLI calling a module via the Scalr module registry. You have the option of using the Scalr account/environment name or ID when calling the module source. Note the source and version parameters.

terraform {
  backend "remote" {
  hostname = "my-account.scalr.io"
    organization = "org-sfgari365m7sck0"
    workspaces {
      name = "module-test"
    }
  }
}

provider "aws" {
  region = var.region
}

module "instance" {
  source  = "my-account.scalr.io/environment-name/instance/aws"
  version = "1.0.1"
  instance_type = var.instance_type
  instance_count = var.instance_count
  subnet = var.subnet
  sg = var.security_group
  key = var.ssh_key
  vpc_id = var.vpc_id
  ami = var.ami
}

Modules from Git

📘

Note

This only works for VCS integration. It does not work for CLI runs

Terraform configuration files running in a Scalr VCS integrated workspaces can call modules directly via git HTTPS URLs. This option will work if the VCS Provider used by the workspace to pull the configuration will also allow access to the modules repository. When a workspace is integrated with VCS the access token is propagated to the workspace, thus making it possible to pull the module as well.

provider "aws" {
  region = var.region
}

module "instance" {
  source  = "github.com/scalr-eap/instance"
  instance_type = var.instance_type
  instance_count = var.instance_count
  subnet = var.subnet
  sg = var.security_group
  key = var.ssh_key
  vpc_id = var.vpc_id
  ami = var.ami
}

Using SSH Keys

If you cannot use the VCS integration, you also have the option of using SSH keys to authenticate to Git. For, example, if the following module needed to be called, then a SSH key can be added to Scalr to provide the authentication:

module "null_resource" {
       source  = "[email protected]:<your-repo>/null_resource_module.git"
}

See more on how to apply SSH keys in Scalr here.

Modules from Paths or HTTPS

If a module exists as a sub-directory in the Terraform configuration directory it can also be sourced using a relative or absolute path. This works for VCS integration and the CLI as the directory and all it’s sub-directories are always loaded into the Scalr workspace.

Alternatively, if it is not in a subdirectory, it can be pulled using the HTTPS protocol as seen in the examples below

terraform {
  backend "remote" {
  hostname = "my-account.scalr.io"
    organization = "org-sfgari365m7sck0"
    workspaces {
      name = "module-test"
    }
  }
}

provider "aws" {
  region = var.region
}

module "instance" {
  source  = "./instance"
  instance_type = var.instance_type
  instance_count = var.instance_count
  subnet = var.subnet
  sg = var.security_group
  key = var.ssh_key
  vpc_id = var.vpc_id
  ami = var.ami
}

##Using HTTPS Protocol

module "kafka" {
  source = "git::https://github.com/some-org/terraform-kubernetes-kafka?ref=v0.0.2-9"
  # the rest of parameters below. 
}

Scaling Module Usage

Creating and using modules is just the first step in managing modules. Just because the module registry has been created, doesn't mean that users will necessarily use it correctly. Based on your model, you might want to put more controls in place to ensure that the correct modules, the version of the modules, and the source for the modules are used. Scalr has features like OPA enforcement and module reporting, which will help with module management at scale.

Using OPA with Modules

Users can pull modules from many different locations, but if you are standardizing on specific code, then you may want to ensure they are using the correct module code as well as source. With Open Policy Agent, you can create policies to check a number of module details. Here are a few examples:

Source of the Module:

The following policy will enforce module usage as it will forbid an AWS DB or S3 bucket from being created unless it is using the sources listed below.

#This policy will forbid resources from getting created unless done so through a module and a specific source.
package terraform

import input.tfplan as tfplan


# Map of resource types which must be created only using module
# with corresponding module source
resource_modules = {
    "aws_db_instance": "terraform-aws-modules/rds/aws",
    "aws_s3_bucket": "<account-name>.scalr.io/<account-id>/s3-bucket/aws"
}

contains(arr, elem) {
  arr[_] = elem
}

deny[reason] {
    resource := tfplan.resource_changes[_]
    action := resource.change.actions[count(resource.change.actions) - 1]
    contains(["create", "update"], action)
    module_source = resource_modules[resource.type]
    not resource.module_address
    reason := sprintf(
        "%s cannot be created directly. Module '%s' must be used instead",
        [resource.address, module_source]
    )
}

deny[reason] {
    resource := tfplan.resource_changes[_]
    action := resource.change.actions[count(resource.change.actions) - 1]
    contains(["create", "update"], action)
    module_source = resource_modules[resource.type]
    parts = split(resource.module_address, ".")
    module_name := parts[1]
    actual_source := tfplan.configuration.root_module.module_calls[module_name].source
    not actual_source == module_source
    reason := sprintf(
        "%s must be created with '%s' module, but '%s' is used",
        [resource.address, module_source, actual_source]
    )
}

For example, if a user tries to use the public S3 module, Scalr will prevent the run from going through when it gets to the OPA checks:

Pin a Module Version

Another example of using OPA to manage module usage is by pinning the module version to be used in an OPA policy:

# Enforce specific versions for modules

package terraform

import input.tfplan as tfplan


pins = {
    "terraform-aws-modules/rds/aws": "2.5.0",
    "<account-name>.scalr.io/<account-id>/s3-bucket/aws": "5.0.0"
}

version_str(module) = v1 {
    v1 := module.version_constraint
} else = v2 {
    v2 := "undefined"
}

deny[reason] {
    walk(tfplan.configuration.root_module.module_calls, [path, value])
    pinned_version = pins[module_source]
    module_source == value.source
    not pinned_version == value.version_constraint
    
    reason := sprintf("Module %q (%s) is of an unsupported version %q. Allowed version is %s",
                      [path[count(path)-1], module_source, version_str(value), pinned_version])
}

See all OPA examples in our Github repo here.

Module Reporting

Once you have your controls in place you'll likely want to do an audit on what is being used, which is where the Scalr reporting feature comes into play.

The module reports will tell you:

• What modules are being used and by what workspaces
• What versions of modules are being used
• The source of the modules

This will help you ensure usage is compliant as well as which modules you can potentially deprecate or focus more time on based on usage.

Renaming Modules

The ability to rename a module is not possible in Scalr as it would break many dependencies on that module. If there is an event that occurs where a rename is needed, the following process can be followed:

• Publish a new module with the new name.
• Review which environments/workspaces use the module in the modules usage report.
• Update the workspaces with the new module.
• Review the module usage report again to ensure the module is not used.
• Delete the old module.

Demo: Define, Enforce, and Report on Modules