Technology 15/03/2021

Securing Terraform for High Integrity Workloads

“Securing Terraform for High Integrity Workloads” or as I prefer to think of it “Keeping Khan Away from the Genesis Torpedo” (Ricardo Montalb√°n not Benedict Cumberbatch).

Here at G-Research we have a firm commitment to infrastructure as code, and make heavy use of HashiCorp’s Terraform Enterprise and Vault products.

In early 2020 we began a piece of work to secure our TFE environment, after a number of worrying security holes had been observed in it’s default behaviour. These were at odds to our requirements for High Integrity Infrastructure, where automated processes can be securely used for sensitive systems. Securing automated processes is not an unusual goal, but we also needed to ensure that no single human operator was able to load the genesis torpedo onto the transporter pad and teleport it out into the nebula (If you’ve not watched Wrath of Khan and are wondering what on Earth I am going on about, I would suggest that you do ūü§ď).

To stretch my tenuous metaphor to breaking point, you don’t want a single person responsible for the creation of a planet which subsequently explodes.

To translate that from Star Trek, to Terraform Enterprise – it should not be possible for a single engineer to be able to retrieve highly sensitive secrets without third party oversight.

State Secrets

You might at this stage be confused (especially if you’re not a fan of Star Trek). Secrets are held in Vault, Terraform Enterprise can be provided with an app role or app roles to access vault, and Terraform Enterprise only has access to those secrets its app role(s) are permissioned to?

Absolutely true, except that when Terraform makes use of a secret from Vault it writes that secret into the state file.

In plain text!

This means that Terraform state files have the potential to contain all manner of highly sensitive information and therefore need to be appropriately protected.

See No Evil – Apply Granular Permissions to State Files

Prior to release v202006-1, Terraform Enterprise only allowed Write or Read permissions to be set on a workspace, both of which still allowed the user to Download and or view the state file.

In TFE release v202006-1 HashiCorp added the capability to apply granular workspace permissions in place of Write or Read.

We were able to use this for certain select workspaces, to remove user access entirely to the state file.

  resource "tfe_team_access" "granular_tfe_team_access" {
    team_id      = "${data.tfe_team.granular_tfe_team.id}"
    workspace_id = "${module.granular_tfe_workspace.workspace_friendly_id}"
    permissions  = {
      state_versions    = "none"
      sentinel_mocks    = "none"
    }
  }

We also removed the ability to access the sentinel mock data. Sentinel mock data includes the state file and access to this needed to be similarly restricted.

Further reading around these permissions settings can be found here, along with this more general discussion of workspace permissions.

Whilst locking down human access to our sensitive workspaces, in parallel we were developing secure High Integrity Jenkins Pipelines that would be able to access specific workspace tokens within Terraform to carry out certain tasks relating to the state file, such as resource tainting, which could then be driven entirely from within Jenkins without a human operator having direct access to execute the commands themselves.

Speak No Evil – Creating a Forked Version of Terraform

Having locked the state file, we realised that it was also possible for secrets to be exposed when terraform plans were executed.

In certain innocuous circumstances it would be possible for a secret to appear in the output of a terraform plan. We also realised that a more nefarious individual could exploit this so as to deliberately view secrets within the terraform plan in the TFE GUI.

Enter the G-Research Opensource team, who were able to create forked versions of terraform 0.11, 0.12 and 0.13, adjusted so that when a particular environment variable was set on the workspace, the output produced by our terraform plans would only display the keys about to be changed and not their values:

So in this case Kahn could be stopped through the Power of the Fork ūü§ď.

If you are interested in our forked version of terraform it’s available on¬†GitHub, as Opensource software under a Mozilla Public License 2.0.

HashiCorp also have plans to improve data sensitivity from Terraform 0.14 onwards. This article provides some details around this new functionality.

Unfortunately there are no plans to back port this to previous versions of Terraform, but it provides further weight to the argument of upgrading to newer versions of Terraform.

Hear No Evil – TFE Plan Bot

Last on our list was the fact that it was possible to exploit the connectivity between GitHub and Terraform so that when raising Pull Requests into GitHub it would be possible to alter configurations around secrets and once again trick Terraform Enterprise into outputting sensitive values.

Once again the Opensource team were able to provide their expertise and assisted us with replacing the Webhook connections between TFE and GitHub with a custom GitHub App which would only allow a Pull Requests speculative plan to execute, once a third party or team mate had provided approval. This simple premise evolved into tfe-plan-bot, Opensource software published under an Apache 2.0 License.

We were also extremely lucky that at the same time HashiCorp made it possible to disable the native speculative plan functions of a tfe_workspace:

resource "tfe_workspace" "test" {
    name                = "my-workspace-name"
    organization        = "my-org-name"
    speculative_enabled = false
  }

In Closing

Our efforts on securing Terraform Enterprise within our environment are a source of great pride to G-Research, especially that we are able to share the Opensource software we have created.

Hopefully this insight is useful to other teams of engineers facing similar concerns around security.

Written by Carl Heinst, Engineer at G-Research

Stay up to-date with G-Research

Subscribe to our newsletter to receive news & updates

You can click here to read our privacy policy. You can unsubscribe at anytime.