Terraform Tips & Tricks – Part 1 – Building A Constant Reference

One of the most common problems I see in large organizations when working with terraform is consistency. When we have a large amount of resources being managed and multiple writing terraform code it often gets hard to maintain consistency in naming or abbreviations used for naming.

Problem

Consider the following organization with a 2 tier hierarchy:

  1. Research and Development
    • AI Research
    • Robotics Research
    • Software Development
  2. Business Development
    • New Product Development
    • Partner Development
    • Strategic Planning
  3. Marketing and Sales
    • Product Marketing
    • Demand Generation
    • Sales
  4. Operations
    • Manufacturing
    • Supply Chain Management
    • Customer Service

If we were to have a standard set of naming conventions that delineate which category the resource belongs to you can see we may run into some issues, for example for:

Research and Development -> Robotics Research

You may get some inconsistencies such as.

Person 1: rnd-robo

Person 2: rd-robotics

Person 3: rr-rnd

Since there are various ways to abbreviate and name a resource it’s often hard to maintain consistency and given the character limit in certain resources such as a GCP project of 30 characters we may run into issues down the line.

Solution

Creating a central location for common references that can be used across all terraform code can be a very powerful way to unify and provide a “single source of truth” for constant values used across an organization.

Defining a Standard Input

The first thing is we need to establish a standardization of input for terraform when referencing any of the above units.

The first category will be called a Business Unit

The second unit will be called a Tech Unit

All input references will be as it is shown above with the following modifications

  • Convert to lowercase
  • Spaces will be replaced with _

This gives us consistency when working with terraform, for example we want to give the inputs for:

Business Development -> New Product Development

Our terraform code will look like this.

business_unit = business_development
tech_unit     = new_product_development

Building a Constants Module

The second part is to build a module that we can reference that holds constant values, or values that should stay constant and generally should not change. Let’s build a list of abbreviations for the above fake organization.

locals.tf

locals {
  billing_account = "1234-ABCD-5678-EFGH"

  org_id          = "1234567"

  abbreviations   = {
    # Research and Development Abbreviations
    research_and_development = "rnd"
    ai_research              = "ai"
    robotics_research        = "robo"
    software_development     = "swd"

    # Business Development Abbreviations
    business_development     = "bd"
    new_product_development  = "npd"
    partner_development      = "partners"
    strategic_planning       = "strgy"

    # Marketing and Sales Abbreviations
    marketing_and_sales      = "mns"
    product_marketing        = "prdmkt"
    demand_generation        = "demand"
    sales                    = "sales"

    # Operations Abbreviations
    operations               = "ops"
    manufacturing            = "mfg"
    supply_chain_management  = "scm"
    customer_service         = "cs"
  }

  default_env = ["dev", "stage", "prod"] 
}

You probably have noticed I have defined a couple of additional values such as the billing account, the org id and a list of default environments. The idea behind this constants module is to put values that might be hard to remember or need to be standardized in an area available for reference, so feel free to add any values you see fit.

Next we need to create terraform outputs for each of these local variables so that they can be referenced in other modules. Let’s keep the same map keys for the output names.

outputs.tf

output "billing_account" {
  value = local.billing_account
}

output "abbreviations" {
  value = local.abbreviations
}

output "default_env" {
  value = local.default_env
}

Let’s put our module in the following folder structure:

.
└──modules
    └── constants
        ├── locals.tf
        └── outputs.tf

Now we are ready to use our module!

Creating a Project Module Using Our Constants

Now that we have created our constants module we can now use it, let’s create a project module so we can create our projects easier.

Lets standardize our project names as follows:

[business_unit]-[tech_unit]-[environment]

Let’s create our module in the following directory above.

.
└── modules
    ├── constants
    │   ├── locals.tf
    │   └── outputs.tf
    └── projects
        ├── projects.tf
        ├── outputs.tf
        └── variables.tf

modules/projects/project.tf

module "constants" {
 source = "../constants"
}

locals {
  bu = module.constants.abbreviations[var.business_unit]
  tu = module.constants.abbreviations[var.tech_unit]
}

resource "google_project" "project" {
  name                = "${local.bu}-${local.tu}-${var.environment}"
  project_id          = "${local.bu}-${local.tu}-${var.environment}"
  org_id              = module.constants.org_id
  billing_account     = module.constants.billing_account
  auto_create_network = var.auto_create_network
}

I first make my reference to the constants module.

Then I create some locals to simplify inputs and make them less lengthy for the project resource.

Then I compose my project resource substituting the name and project_id with our designated naming convention.

Note that for org_id and billing_account I am using a reference from the constants module. Finally I create a configurable input for auto_create_network, you may do this as well for any input the resource allows.

modules/projects/variables.tf

variable "business_unit" {
  type        = string
  description = "The business unit in all lowercase, replacing spaces with _"
}

variable "tech_unit" {
  type        = string
  description = "The tech unit in all lowercase, replacing spaces with _"
}

variable "environment" {
  type        = string
  description = "The environment for the project"
}

variable "auto_create_network" {
  type        = bool
  description = "Automatically create networks"
  default     = true
}

I define some variables so that we just configure them when we use the projects module, I set the auto_create_network variables default to true to make the variable optional.

modules/projects/outputs.tf

output "project" {
  value = resource.google_project.project
}

Finally I create an output for this module so that other modules can easily reference any output values, in this specific case we get the id and number. As defined in the terraform gcp project resource.

Using the Project Module

Now that we have created our constants module and a project module that uses our constants module we are ready to create a project.

Let’s make some terraform code for creating the Business Development -> New Product Development projects

First let’s setup the directory we will write this code in.

.
├── modules
|   ├── constants
|   │   ├── locals.tf
|   │   └── outputs.tf
|   └── projects
|       ├── projects.tf
|       ├── outputs.tf
|       └── variables.tf
└── terraform
    └── projects
        └── main.tf     

Now that the folder structure has been established we can start writing our code.

terraform/projects/main.tf

module "constants" {
  source = "../../modules/constants"
}

module "project" {
  source        = "../../modules/projects"
  for_each      = toset(module.constants.default_env)
  business_unit = "business_development"
  tech_unit     = "new_product_development"
  environment   = each.key
}

I took advantage of terraform looping as well as the constants module to create 3 projects in 1 terraform block.

I did this by specifying a for_each which loops through a set of values, in my case the default_env value we defined in the constants module. But first I need to change the type from list to set with the toset function in terraform.

I then defined all the inputs required by our project module and set the environment input to each.key which is a special variable that is present when using a for_each loop. This variable will give you the key for each iteration of the loop.

When ran this module will create the following projects:

  • bd-npd-dev
  • bd-npd-stage
  • bd-npd-prod

Final Notes

In our demonstration we created a constants module that provides referenceable values to other modules and terraform code so that we can maintain consistency as well as a single location for change that propagates to all modules that reference it.

I hope that you have enjoyed this article. I use this concept on a daily basis with my terraform code and it allows me to reuse code and prevent redundant declarations that can be a nightmare to change when working with hundreds or thousands of resources.

I have included a very basic example of the concept and this can be expanded to many other use cases such as:

  • A list of apis to enable by default
  • A list of emails for project owners
  • Firewall rules that should be applied to all networks
  • A map of tags that should be added based on business_unit and tech_unit
  • And many others…

Note: the entire modules folder should be stored in a git repository of its own and only be editable by infrastructure managers so it can remain as constant as possible and allow changes to be made by people who can make decisions regarding its contents

Propagation Example

Let’s say that we have decided that we want to add a new environment to all of our projects called qa.

We can modify our constants module to include this addition by adding it to the default_envs local variable:

modules/constants/local.tf

locals {
# Unchanged Code Above
# Changed Code Below
  default_envs = ["dev", "stage", "qa", "prod"]
}

Now that we added that change, anything that references default_envs like our project creation example above will now create an additional qa project as soon as the terraform code is rerun.

Related Posts

How to Use Packer and Subiquity on WSL2

Background My main dev environment is windows convenient WSL2, however the WSL2 instance is isolated from the host machines IP, leading to some issues when it comes…

How to deploy Ubuntu 20.04 packer templates for terraform on VSphere without duplicate IP’s

Background For a while now I have been struggling with Packer templates built with Ubuntu 20.04 server deployed on VSphere 7.0. When I build the packer image…

Leave a Reply