Infrastructure as code templating overview

As of 2023, approximately 90% of organizations implementing DevOps practices have adopted infrastructure as code (IaC) to some extent (1). This adoption spans various industries, including technology, finance, healthcare, and retail (2). Companies adopt IaC to deploy infrastructure faster, with fewer errors, for a lower cost than they do without it.

However, there are challenges when adopting IaC:

• Skill Gaps: 60% of organizations cite a lack of skilled professionals as a significant barrier to fully implementing IaC practices (3)
• Security Concerns: Ensuring security and compliance through IaC remains a top concern for 50% of organizations, emphasizing the importance of integrating security best practices into IaC (4).

While the benefits are obvious, surmounting these challenges isn’t so straightforward. Many companies try to build their own infrastructure as code templating. In this overview we’ll run through some of the options available today, and introduce Resourcely: a tool built for infrastructure as code templating.

Infrastructure as code templating

Given infrastructure as code is….code, it is relatively easy to parameterize that code and allow users to make inputs into it without having to write the full code themselves. This has emerged as the most effective way to help developers deploy infrastructure in the new world of DevOps (where they are expected to do just that).

Generally, Platform, DevOps, and SRE teams create infrastructure as code modules or templates by defining reusable components that encapsulate specific infrastructure patterns. These often feature variables or parameters that can take input from end-users in order to customize the resulting IaC.

In practice, templating gets particularly sticky around:

• Learning Curve: While most IaC is declarative and designed to be human-readable, mastering the syntax can be challenging for newcomers.
• Limited Procedural Logic: Declarative languages advanced procedural programming constructs, potentially limiting complex automation scenarios.
• Cloud Infrastructure Complexity: IaC itself doesn’t solve the inherent problem of cloud infrastructure complexity. There are thousands of possible cloud resources, each with 10 - 100 different configuration possibilities. This heterogeneity is one of the primary causes of the frustration with tools like Terraform and CloudFormation.

Terraform

Terraform is the most widely adopted of IaC languages, with a modular architecture with support for nearly every major cloud provider that has made it incredibly popular.

Example: Simple Terraform Module that provisions an S3 bucket

# modules/s3_bucket/main.tf
resource "aws_s3_bucket" "this" {
  bucket = var.bucket_name
  acl    = var.acl

  tags = {
    Environment = var.environment
    Name        = var.bucket_name
  }
}

# modules/s3_bucket/variables.tf
variable "bucket_name" {
  description = "The name of the S3 bucket"
  type        = string
}

variable "acl" {
  description = "The ACL of the S3 bucket"
  type        = string
  default     = "private"
}

variable "environment" {
  description = "The environment for the bucket"
  type        = string
}

‍Usage:

module "example_bucket" {
  source      = "./modules/s3_bucket"
  bucket_name = "my-terraform-bucket"
  environment = "production"
}

Illustrating Terraform Module Complexities

Let's examine a more complex Terraform module that can be difficult for end-user developers to understand and use effectively.

Example: AWS VPC with Multiple Resources

# modules/advanced_vpc/main.tf

resource "aws_vpc" "this" {
  cidr_block           = var.cidr_block
  enable_dns_support   = var.enable_dns_support
  enable_dns_hostnames = var.enable_dns_hostnames

  tags = merge(var.tags, {
    Name = var.vpc_name
  })
}

resource "aws_subnet" "public" {
  count                   = length(var.public_subnets)
  vpc_id                  = aws_vpc.this.id
  cidr_block              = element(var.public_subnets, count.index)
  map_public_ip_on_launch = true
  availability_zone       = element(var.availability_zones, count.index)

  tags = merge(var.tags, {
    Name = "${var.vpc_name}-public-${count.index + 1}"
  })
}

resource "aws_subnet" "private" {
  count             = length(var.private_subnets)
  vpc_id            = aws_vpc.this.id
  cidr_block        = element(var.private_subnets, count.index)
  availability_zone = element(var.availability_zones, count.index)

  tags = merge(var.tags, {
    Name = "${var.vpc_name}-private-${count.index + 1}"
  })
}

resource "aws_internet_gateway" "this" {
  vpc_id = aws_vpc.this.id

  tags = merge(var.tags, {
    Name = "${var.vpc_name}-igw"
  })
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.this.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.this.id
  }

  tags = merge(var.tags, {
    Name = "${var.vpc_name}-public-rt"
  })
}

resource "aws_route_table_association" "public" {
  count          = length(aws_subnet.public)
  subnet_id      = aws_subnet.public[count.index].id
  route_table_id = aws_route_table.public.id
}

# modules/advanced_vpc/variables.tf

variable "vpc_name" {
  description = "Name of the VPC"
  type        = string
}

variable "cidr_block" {
  description = "CIDR block for the VPC"
  type        = string
}

variable "public_subnets" {
  description = "List of CIDR blocks for public subnets"
  type        = list(string)
}

variable "private_subnets" {
  description = "List of CIDR blocks for private subnets"
  type        = list(string)
}

variable "availability_zones" {
  description = "List of availability zones"
  type        = list(string)
}

variable "enable_dns_support" {
  description = "Enable DNS support"
  type        = bool
  default     = true
}

variable "enable_dns_hostnames" {
  description = "Enable DNS hostnames"
  type        = bool
  default     = true
}

variable "tags" {
  description = "Tags to apply to all resources"
  type        = map(string)
  default     = {}
}

Usage:‍

module "advanced_vpc" {
  source               = "./modules/advanced_vpc"
  vpc_name             = "production-vpc"
  cidr_block           = "10.0.0.0/16"
  public_subnets       = ["10.0.1.0/24", "10.0.2.0/24"]
  private_subnets      = ["10.0.11.0/24", "10.0.12.0/24"]
  availability_zones   = ["us-west-1a", "us-west-1b"]
  enable_dns_support   = true
  enable_dns_hostnames = true
  tags = {
    Environment = "production"
    Owner       = "DevOps Team"
  }
}

Using this module would be difficult, even for the most experienced of developers. It…

• requires the user to input multiple variables, including lists for subnets and availability zones. Managing and understanding these variables can be overwhelming for developers unfamiliar with networking concepts.
• assumes a specific structure and number of public and private subnets corresponding to the availability zones. Mismatched lengths between public_subnets, private_subnets, and availability_zones can lead to errors that are not immediately clear.
• doesn’t have comprehensive or embedded documentation, meaning end-users may struggle to understand the purpose of each variable and how to configure it properly.
• does not expose all possible configurations, limiting the end-user’s ability to tailor the infrastructure to their specific needs. This can force users to modify the module directly, which defeats the purpose of reusability and can introduce maintenance challenges.
• has a complex code structure, making it harder for end-users to quickly grasp its functionality. Looking at this example, your eyes will almost surely become bleary.

Imagine an end-user developer attempts to use our advanced_vpc module but provides mismatched lists for public_subnets, private_subnets, and availability_zones:

module "advanced_vpc" {
  source               = "./modules/advanced_vpc"
  vpc_name             = "staging-vpc"
  cidr_block           = "10.1.0.0/16"
  public_subnets       = ["10.1.1.0/24"]  # Only one subnet provided
  private_subnets      = ["10.1.11.0/24", "10.1.12.0/24"]
  availability_zones   = ["us-west-2a", "us-west-2b", "us-west-2c"]  # Three AZs provided
  enable_dns_support   = true
  enable_dns_hostnames = true
  tags = {
    Environment = "staging"
    Owner       = "QA Team"
  }
}

You can imagine this scenario happening, because it is a very possible one! If it were to happen, the terraform plan would fail with an error message like:

Error: Invalid function argument

Invalid index

  on modules/advanced_vpc/main.tf line 10, in resource "aws_subnet" "public":
  10: availability_zone = element(var.availability_zones, count.index)

Terraform modules don’t suffer from the problem of poor design. They are doing what they are designed for: helping cloud infrastructure experts move faster. Let’s cover some other tooling that is templated for configuring infrastructure:

AWS CloudFormation

Another popular IaC framework specifically for AWS is CloudFormation. CloudFormation templates use declarative JSON or YAML files (similar to Terraform).

Here’s an example of a CloudFormation template written in YAML. It takes parameters from a developer and creates an EC2 instance. You can see some of the customization available in the template parameters (allowed values, description, type, default, etc.):

AWSTemplateFormatVersion: '2010-09-09'
Description: >
  AWS CloudFormation Template to deploy an EC2 instance with customizable parameters.

Parameters:
  InstanceType:
    Description: Type of EC2 instance
    Type: String
    Default: t2.micro
    AllowedValues:
      - t2.micro
      - t2.small
      - t2.medium
    ConstraintDescription: Must be a valid EC2 instance type.

  KeyName:
    Description: Name of an existing EC2 KeyPair to enable SSH access
    Type: AWS::EC2::KeyPair::KeyName
    ConstraintDescription: Must be the name of an existing EC2 KeyPair.

  Environment:
    Description: Environment tag for the EC2 instance
    Type: String
    Default: Production
    AllowedValues:
      - Development
      - Staging
      - Production
    ConstraintDescription: Must be a valid environment.

  AMIId:
    Description: AMI ID for the EC2 instance
    Type: AWS::EC2::Image::Id
    Default: ami-0abcdef1234567890
    ConstraintDescription: Must be a valid AMI ID.

Resources:
  EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      KeyName: !Ref KeyName
      ImageId: !Ref AMIId
      Tags:
        - Key: Name
          Value: !Sub "${Environment}-EC2Instance"
        - Key: Environment
          Value: !Ref Environment

Outputs:
  InstanceId:
    Description: The Instance ID of the newly created EC2 instance
    Value: !Ref EC2Instance

  PublicIP:
    Description: Public IP address of the EC2 instance
    Value: !GetAtt EC2Instance.PublicIp

Interacting with this template would require using the AWS CLI or CDK. Here’s an example of how a developer would use it using the CLI:

aws cloudformation create-stack \
  --stack-name MyEC2Stack \
  --template-body file://ec2_instance.yaml \
  --parameters ParameterKey=InstanceType,ParameterValue=t2.small \
               ParameterKey=KeyName,ParameterValue=my-key-pair \
               ParameterKey=Environment,ParameterValue=Staging \
               ParameterKey=AMIId,ParameterValue=ami-0abcdef1234567890 \
  --capabilities CAPABILITY_IAM

Similar to Terraform modules, CloudFormation templates are not inherently modular, and can become very verbose. When linking together templates, deployment order becomes important and dependency hell can cause even the most experienced AWS developers to give up in frustration.

Other infrastructure as code templating

Ansible

Ansible is an agentless automation tool that uses YAML-based playbooks to define and manage configurations, deployments, and orchestration tasks. Given it is YAML-based, templating support for Ansible is with Jinja2.

Pulumi

Pulumi allows developers to define infrastructure using general-purpose programming languages like Python, TypeScript, and Go. While Pulumi provides templates on their website, those are general-purpose patterns. Given the variable language flexibility, developers must be familiar with that language in order to use it.

Chef

Chef uses a Ruby-based DSL (Domain-Specific Language) to define infrastructure configurations. Templating for Ruby-based frameworks is accomplished using Embedded Ruby (ERB), which can be prohibitive if your developer base isn’t fluent in Ruby.

SaltStack

SaltStack is an event-driven automation tool that uses YAML-based state files to define desired infrastructure states. Similar to Ansible, it is templated using Jinja2 without any specific quality-of-life features.

Infrastructure as code templating drawbacks

The primary issue with the infrastructure as code templating options available, is that they don’t go far enough. At their most basic (Jinja2), they allow for collection and insertion of inputs. At their most advanced (Terraform modules), they allow for defaults, descriptions, enums, types, etc.

All of these infrastructure as code templating options do not:

• embed any automated typing
• automatically detect linking between resources
• provide a UI
• automatically provide defaults for enums

Resourcely

Resourcely is a cloud configuration platform built to solve the cloud complexity problem. It both allows for IaC templating and maintains a knowledge graph of cloud resource metadata that makes creating and using IaC templates faster, simpler, and easier.

Resourcely Templating

Resourcely supports templating of Infrastructure as Code with a Liquid-style syntax.

Features that you would expect from templates (defaults, descriptions, etc) are included, while advanced quality-of-life concepts that utilize a proprietary knowledge graph help overcome cloud complexity. This includes automatic linking of resources, enum detection, parameterization, context-dependent options, and more are supported out of the box.

You can read more about building templates with Resourcely here.

Foundry (IaC Templating IDE)

Resourcely templates are created using Foundry: an integrated IDE for building, previewing, and testing templates. Platform teams report that with Resourcely, they can reduce their template creation time by 90% while helping developers ship infrastructure 80% faster.

Template UI

Every template (Blueprint, in Resourcely parlance) becomes an automatically hosted UI that developers can interact with to generate Infrastructure as Code. This UI includes a service catalog-style shopping cart where developers can build their own custom infrastructure stack from linkable, modular components.

Infrastructure Policy-as-Code

Resourcely also embeds infrastructure policy-as-code, allowing platform teams to control the infrastructure configuration that developers create with Resourcely templates. These policies are exposed to developers within the Resourcely UI, so that they don’t find out about policy violations after CI runs.

Conclusion

When evaluating an IaC templating tool, it is important to think about the developer experience of interacting with that template or module. Existing templating tools require a significant amount of effort by platform teams to build user-friendly modules.

Resourcely offers a next-generation infrastructure templating framework, with cloud resource parameter information embedded along with an automatically hosted UI. This is a unique alternative to Terraform modules or CloudFormation templates, which rely on platform teams to codify cloud complexity on their own.

References

• (1). DevOps Survey 2023. https://puppet.com/resources/report/devops-survey-2023/
• (2). State of the Cloud Report 2023. https://info.flexera.com/CM-REPORT-State-of-the-Cloud
• (3). State of DevOps Report 2023. https://cloud.google.com/devops/state-of-devops
• (4). State of the Cloud Report 2024. https://info.flexera.com/CM-REPORT-State-of-the-Cloud

‍