When I was first starting out with Terraform, I couldn’t find many resources on bootstrapping remote state or access control for multiple AWS accounts. Granted, it’s not difficult to figure out if you dig through the documentation, but it would have been nice to have a step-by-step walkthrough so that I could have jumped straight into more valuable work on the product I was creating. I suspect this is one of those things that seems too obvious to write about, but if you’re a developer trying to set things up using best practices, it can take a lot of reading to put it all together.

The bootstrap process is not hard, but it is tedious. Luckily, you shouldn’t have to run through it all that often. Naturally, we’ll be using fenna throughout.

We’re going to build four accounts: tools, dev, stage, and prod.

dev, stage, and prod will house our environments. tools will house our remote state and other cross-account tooling (such as CI/CD pipelines). We will use the root account for managing access control to the other accounts via IAM.

You can use this account structure to host multiple services or replicate it for each individual service. Separating your environments into their own accounts is an essential first step to minimizing blast radius on AWS. It may feel overly complex at first if you’re used to working within a single AWS account, but I assure you it’s worth it.

AWS Setup

We’re going to start in your current AWS account. This will be your root account. Ideally, this would be a fresh account with nothing else in it, but for the purposes of this tutorial it doesn’t really matter.

Account Limits

First off, you will need to request an organizations account limit increase.

Log into the AWS Console and click on the “Support” menu in the upper right and then on “Support Center”. Click on “Create Case” then “Service limit increase”. Select “Organizations” for “Limit type”, then “Number of Accounts” for “Limit”. Set “New limit value” to 5.

Write up a short explanation for what you’re planning on doing with the accounts, and then submit. Now, wait.

My last limit request was addressed by the next day, but the timing will vary. (I told you this was tedious.)

Access Control

Next, we’ll create the base access control from which we’ll bootstrap everything else. Head over to IAM in the AWS Console and create a group named ”Ops”. Attach the AdministratorAccess policy to the group. This group will provide access to any users assigned to it to all of the accounts.

Now, create a user named “Ops” (or whatever you like—the user name doesn’t really matter) with programmatic access enabled and make them a member of the “Ops” group. Be sure to record the access credentials somewhere secure (I prefer 1Password).

If you haven’t already, install the AWS CLI. I do this via Homebrew—brew install awscli.

We’re going to rely on fenna’s profile naming conventions, so we’ll need to configure our credentials accordingly. Select a name for your root profile. I have a bunch of these, so I typically stick to my client’s name, but for clients I recommend matching the profile name with the group or role it represents. So, for the purposes of this tutorial, I’m going to assume your root profile is named ops.

Configure your root profile with the credentials of the user you created above:

aws configure --profile ops

Repository Setup

Create a backends repository. This is where you will store the shared backend configuration your team will use with fenna.

Next create a base-infrastructure repository using my base infrastructure repo as a template and clone. You’ll want to keep your fork private since it will be specific to your organization and may contain sensitive information eventually.

Then, clone your backends repository into remote-state/backends within the base-infrastructure repo. This is where our remote-state Terraform modules will write the backend configuration, saving your team a lot of copy-paste in the future.

Bootstrapping

Once AWS has increased your account limit, you can finally start applying Terraform.

remote-state/root

First, we’re going to create the remote state infrastructure for the root account:

cd remote-state/root
terraform init
terraform plan -out plan
terraform apply plan && rm plan

When prompted during plan, provide a key_base. It doesn’t matter what you use. I typically use my client’s company name or an abbreviation of it, but whatever you enter should be URL-safe. When prompted for the profile, enter ops.

Commit the newly created terraform.tfvars file which will persist the key_base.

Then, cd ../backends, commit the root.tfvars file to the backends repo, and push to origin.

We now have our first backend and can bootstrap fenna.

If you haven’t already, install fenna.

Then, from remote-state/backends:

cd ../root
fenna bootstrap

Input the following when prompted:

What is this service named?
> remote-state

What's your backend config repository's URL?
> [PASTE YOUR BACKEND REPOSITORY URL HERE]

Will this service be developed in a sandbox? [yes no]
> no

Which backend would you like to use? [root]
> root

Where is your backend stored? [root]
> root

What AWS profile would you like to use as your root profile?
> ops

Replace backend "local" {} with backend "s3" {} in the terraform block in main.tf so that we can use our newly created backend. Then remove the (now) extra variable "profile" {} declaration from variables.tf.

Finally, run fenna init and answer yes when prompted to copy the Terraform state to S3:

Terraform detected that the backend type changed from "local" to "s3".
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Revert the changes in .gitignore (no reason to duplicate lines), and then commit all remaining changes to the repo.

This will make it easy for other developers to dive straight in with fenna onboard if changes need to be made to this service in the future. You can also now safely remove any tfstate files in the working directory if you like.

With that out of the way, we can move on to creating the other AWS accounts.

organizations

From remote-state/root:

cd ../../organizations
fenna bootstrap

When prompted, input the following:

What is this service named?
> organizations

What's your backend config repository's URL?
> [PASTE YOUR BACKEND REPOSITORY URL HERE]

Will this service be developed in a sandbox? [yes no]
> no

Which backend would you like to use? [root]
> root

Where is your backend stored? [root]
> root

What AWS profile would you like to use as your root profile?
> ops

Commit all of the changes to the repo.

Now, we can create the other accounts:

fenna init
fenna plan
fenna apply

You will be prompted during plan for base_email. Use one that you have access to since this will be used to create the root user for each account. Each account requires a unique email address for its root user, so the Terraform configuration will append +tools, +dev, etc. to the user name of your base_email (e.g., jd+tools@theconsultingcto.com). If this doesn’t work for you, make changes to the configuration accordingly.

Commit terraform.tfvars to the repo.

Then run ./bin/configure_aws_profiles.sh to automatically configure the remainder of your profiles for use with the AWS CLI. The script will make a backup of your ~/.aws/config and then cat the newly generated aws_config.ini to ~/.aws/config. (You’re welcome to do this manually if it makes you feel more comfortable.)

access-control

Now that we have all of our accounts created, we need to bring access control under management with Terraform. Creating each account resulted in the creation of an “Ops” role within each account, and we also need a way to manage the initial “Ops” group and user we created at the beginning of this process.

From organizations:

cd ../access-control
fenna bootstrap

When prompted, input the following:

What is this service named?
> access-control

What's your backend config repository's URL?
> [PASTE YOUR BACKEND REPOSITORY URL HERE]

Will this service be developed in a sandbox? [yes no]
> no

Which backend would you like to use? [root]
> root

Where is your backend stored? [root]
> root

What AWS profile would you like to use as your root profile?
> ops

Commit the changes to the repo.

Next:

fenna init
./bin/import.sh

import.sh will handle importing the existing resources into the Terraform state.

Note that the “Ops” roles within each account have an inline AdministratorAccess policy attached. If you want to change the permissions for those roles, you’ll need to manually remove those inline policies before making changes in Terraform.

Finally, to enable members of the “Ops” group in the root account to assume the “Ops” roles in the other accounts:

fenna plan
fenna apply

remote-state/tools

With our accounts and access control sorted, we can now bootstrap the remote state backends that we will be using more regularly.

From access-control:

cd ../remote-state/tools
terraform init
terraform plan -out plan

When prompted by plan, enter the following:

var.key_base
  Enter a value: remote-state

var.profile
  Enter a value: ops-tools

Then run terraform apply plan && rm plan.

Once the apply finishes, commit terraform.tfvars to the repository to retain the key_base.

Then, cd ../backends, commit the tools.tfvars and dev.tfvars files to the backends repo, and push to origin.

tools and dev are the backends you and your team will primarily use for development. Backend infrastructure for stage and prod is also created, but I recommend using those backends only from within your CI/CD pipelines (see my deployment-pipeline for an example of how that works).

Now that the infrastructure exists, we can switch to using fenna.

From remote-state/backends:

cd ../tools
fenna bootstrap

Enter the following when prompted:

What is this service named?
> remote-state

What's your backend config repository's URL?
> [PASTE YOUR BACKEND REPOSITORY URL HERE]

Will this service be developed in a sandbox? [yes no]
> no

Which backend would you like to use? [dev root tools]
> tools

Where is your backend stored? [dev root tools]
> tools

What AWS profile would you like to use as your root profile?
> ops

Again, replace backend "local" {} with backend "s3" {} in the terraform block in main.tf, remove the extra variable "profile" {} declaration from variables.tf, discard the changes on .gitignore, and commit all remaining changes to the repo.

Finally, move the Terraform state to the tools backend with:

fenna init

Answer yes when prompted.

Once init finishes, you can safely remove the tfstate files from the working directory.

Done

I know this process is a pain in the ass. Over time, I’ve automated more and more of it, but I haven’t found myself doing it often enough to take the time to automate it all end-to-end. (Besides, some steps can’t really be automated.) But by establishing conventions around your accounts, access control, and remote state early on, you will save yourself a lot of rework and free up your team to focus on the product.