How do I structure my Terraform projects?

16 August 2018 · 3 minute read

Basics

No matter what I’m building with Terraform, I follow the same basic conventions. Every module I create has, at minimum, two files: main.tf and interface.tf.

In my top-level main.tf, I define my backend, providers, and any remote state required for the module. I like to keep the remote state declarations in a single place for each module to make it easier to keep track of module dependencies. Then, if there are any sub-modules, I’ll pass in the required remote state via variables. This can decrease coupling between modules, making reuse simpler. However, it can result in a large number of required variables for sub-modules (though this looks like the new ability in 0.12 to pass resources and modules as values may simplify this).

I declare all my inputs and outputs in the interface.tf. I’m honestly not sure where I picked up this convention, but I’ve also seen people separate out inputs and outputs into separate files. Either way, it’s useful to keep a modules interface in a standard place.

I try to keep most of my modules small enough that everything can fit into main.tf. When that’s not possible, I’ll break the resource declarations up into logical components. For example, I have a module that sets up HTTP Basic Auth for S3 objects using Lambda@Edge. In that module, the resources are declared in two files cloudfront.tf and http_basic_auth.tf. For another example, when I’m building an API Gateway, I like to define all the models in a models.tf and each resource and associated endpoints in a separate file (e.g., sessions.tf, publications.tf, etc.).

State Boundaries

As I’ve written previously, I recommend using multiple state files. I typically draw remote state boundaries around logical services. For example, in a recent project, I had the following top-level modules (each with their own state):

This makes it easier to manage dependencies between modules while keeping resources grouped coherently.

Module Dependencies

When I’m first laying out a Terraform project, I like sketch out my top-level modules and their dependencies in Tinderbox. This is, first of all, useful documentation for any developer trying to understand how everything fits together. It also helps me make certain I’m not introducing any cyclic dependencies between modules which could complicate rebuilding the infrastructure from scratch (in a new region or account).

Environments

I prefer to specify individual environments (i.e., dev, stage, prod) via Terraform modules (rather than using Terraform workspaces). Because I typically use separate AWS accounts for each environment, each environment gets its own state file, and each environment can vary in particulars via module variables. YMMV for other cloud providers.

Typically, stage and prod will be declared in a single module named env—this ensures they will remain as close to identical in everything except, perhaps, scale. My dev or sandbox environments often differ enough from stage and prod that I need to break them out into their own modules (and extract any shared configuration from env into additional sub-modules).

Coming Soon

I realize that this can all feel very abstract without concrete examples. To that end, I am working on an in-depth tutorial on how to build a system like the one I’ve diagrammed above from scratch.

When it’s ready, subscribers to my mailing list will hear it first.


If you find my work interesting, sign up for the Statics & Dynamics mailing list so you don’t miss a thing.