One of the most common problems I see in large organizations when working with terraform is consistency. When we have a large amount of resources being managed and multiple writing terraform code it often gets hard to maintain consistency in naming or abbreviations used for naming.
Problem
Consider the following organization with a 2 tier hierarchy:
- Research and Development
- AI Research
- Robotics Research
- Software Development
- Business Development
- New Product Development
- Partner Development
- Strategic Planning
- Marketing and Sales
- Product Marketing
- Demand Generation
- Sales
- Operations
- Manufacturing
- Supply Chain Management
- Customer Service
If we were to have a standard set of naming conventions that delineate which category the resource belongs to you can see we may run into some issues, for example for:
Research and Development -> Robotics Research
You may get some inconsistencies such as.
Person 1: rnd-robo
Person 2: rd-robotics
Person 3: rr-rnd
Since there are various ways to abbreviate and name a resource it’s often hard to maintain consistency and given the character limit in certain resources such as a GCP project of 30 characters we may run into issues down the line.
Solution
Creating a central location for common references that can be used across all terraform code can be a very powerful way to unify and provide a “single source of truth” for constant values used across an organization.
Defining a Standard Input
The first thing is we need to establish a standardization of input for terraform when referencing any of the above units.
The first category will be called a Business Unit
The second unit will be called a Tech Unit
All input references will be as it is shown above with the following modifications
- Convert to lowercase
- Spaces will be replaced with _
This gives us consistency when working with terraform, for example we want to give the inputs for:
Business Development -> New Product Development
Our terraform code will look like this.
business_unit = business_development
tech_unit = new_product_development
Building a Constants Module
The second part is to build a module that we can reference that holds constant values, or values that should stay constant and generally should not change. Let’s build a list of abbreviations for the above fake organization.
locals.tf
locals {
billing_account = "1234-ABCD-5678-EFGH"
org_id = "1234567"
abbreviations = {
# Research and Development Abbreviations
research_and_development = "rnd"
ai_research = "ai"
robotics_research = "robo"
software_development = "swd"
# Business Development Abbreviations
business_development = "bd"
new_product_development = "npd"
partner_development = "partners"
strategic_planning = "strgy"
# Marketing and Sales Abbreviations
marketing_and_sales = "mns"
product_marketing = "prdmkt"
demand_generation = "demand"
sales = "sales"
# Operations Abbreviations
operations = "ops"
manufacturing = "mfg"
supply_chain_management = "scm"
customer_service = "cs"
}
default_env = ["dev", "stage", "prod"]
}
You probably have noticed I have defined a couple of additional values such as the billing account
, the org id
and a list of default environments
. The idea behind this constants
module is to put values that might be hard to remember or need to be standardized in an area available for reference, so feel free to add any values you see fit.
Next we need to create terraform outputs for each of these local variables so that they can be referenced in other modules. Let’s keep the same map keys for the output names.
outputs.tf
output "billing_account" {
value = local.billing_account
}
output "abbreviations" {
value = local.abbreviations
}
output "default_env" {
value = local.default_env
}
Let’s put our module in the following folder structure:
.
└──modules
└── constants
├── locals.tf
└── outputs.tf
Now we are ready to use our module!
Creating a Project Module Using Our Constants
Now that we have created our constants
module we can now use it, let’s create a project module so we can create our projects easier.
Lets standardize our project names as follows:
[business_unit]-[tech_unit]-[environment]
Let’s create our module in the following directory above.
.
└── modules
├── constants
│ ├── locals.tf
│ └── outputs.tf
└── projects
├── projects.tf
├── outputs.tf
└── variables.tf
modules/projects/project.tf
module "constants" {
source = "../constants"
}
locals {
bu = module.constants.abbreviations[var.business_unit]
tu = module.constants.abbreviations[var.tech_unit]
}
resource "google_project" "project" {
name = "${local.bu}-${local.tu}-${var.environment}"
project_id = "${local.bu}-${local.tu}-${var.environment}"
org_id = module.constants.org_id
billing_account = module.constants.billing_account
auto_create_network = var.auto_create_network
}
I first make my reference to the constants
module.
Then I create some locals to simplify inputs and make them less lengthy for the project resource.
Then I compose my project resource substituting the name
and project_id
with our designated naming convention.
Note that for org_id
and billing_account
I am using a reference from the constants
module.
Finally I create a configurable input for auto_create_network
, you may do this as well for any input the resource allows.
modules/projects/variables.tf
variable "business_unit" {
type = string
description = "The business unit in all lowercase, replacing spaces with _"
}
variable "tech_unit" {
type = string
description = "The tech unit in all lowercase, replacing spaces with _"
}
variable "environment" {
type = string
description = "The environment for the project"
}
variable "auto_create_network" {
type = bool
description = "Automatically create networks"
default = true
}
I define some variables so that we just configure them when we use the projects
module, I set the auto_create_network
variables default to true
to make the variable optional.
modules/projects/outputs.tf
output "project" {
value = resource.google_project.project
}
Finally I create an output for this module so that other modules can easily reference any output values, in this specific case we get the id
and number
. As defined in the terraform gcp project resource.
Using the Project Module
Now that we have created our constants
module and a project
module that uses our constants
module we are ready to create a project.
Let’s make some terraform code for creating the Business Development -> New Product Development projects
First let’s setup the directory we will write this code in.
.
├── modules
| ├── constants
| │ ├── locals.tf
| │ └── outputs.tf
| └── projects
| ├── projects.tf
| ├── outputs.tf
| └── variables.tf
└── terraform
└── projects
└── main.tf
Now that the folder structure has been established we can start writing our code.
terraform/projects/main.tf
module "constants" {
source = "../../modules/constants"
}
module "project" {
source = "../../modules/projects"
for_each = toset(module.constants.default_env)
business_unit = "business_development"
tech_unit = "new_product_development"
environment = each.key
}
I took advantage of terraform looping as well as the constants
module to create 3 projects in 1 terraform block.
I did this by specifying a for_each
which loops through a set of values, in my case the default_env
value we defined in the constants
module. But first I need to change the type from list
to set
with the toset
function in terraform.
I then defined all the inputs required by our project
module and set the environment
input to each.key
which is a special variable that is present when using a for_each
loop. This variable will give you the key for each iteration of the loop.
When ran this module will create the following projects:
- bd-npd-dev
- bd-npd-stage
- bd-npd-prod
Final Notes
In our demonstration we created a constants
module that provides referenceable values to other modules and terraform code so that we can maintain consistency as well as a single location for change that propagates to all modules that reference it.
I hope that you have enjoyed this article. I use this concept on a daily basis with my terraform code and it allows me to reuse code and prevent redundant declarations that can be a nightmare to change when working with hundreds or thousands of resources.
I have included a very basic example of the concept and this can be expanded to many other use cases such as:
- A list of apis to enable by default
- A list of emails for project owners
- Firewall rules that should be applied to all networks
- A map of tags that should be added based on
business_unit
andtech_unit
- And many others…
Note: the entire modules
folder should be stored in a git repository of its own and only be editable by infrastructure managers so it can remain as constant as possible and allow changes to be made by people who can make decisions regarding its contents
Propagation Example
Let’s say that we have decided that we want to add a new environment to all of our projects called qa
.
We can modify our constants
module to include this addition by adding it to the default_envs
local variable:
modules/constants/local.tf
locals {
# Unchanged Code Above
# Changed Code Below
default_envs = ["dev", "stage", "qa", "prod"]
}
Now that we added that change, anything that references default_envs
like our project creation example above will now create an additional qa
project as soon as the terraform code is rerun.