Using Terraform continuously — Common traits in modules

This is the first blog post in the series about using Terraform continuously, which I am going to publish over the next few weeks.

UPD (15.07.2017): I wrote another blog post in the series — Terrapin: making good Terraform modules

I am Anton Babenko (github.com/antonbabenko) one of the active maintainers and contributors of several terraform-community-modules. Since I started using Terraform back in mid-2015, I have talked to many people (online and during conferences) who are using Terraform to build their infrastructures, so it feels natural to write some blog posts about what I am experiencing.

I want to hear feedback, so please leave a comment below if you have something to say (or agree, or disagree). You can also send me an email and I will reply to you.

Let’s start with important things first — Terraform modules.

What is Terraform module?

Modules in Terraform are self-contained packages of Terraform configurations that are managed as a group. Modules are used to create reusable components, improve organization, and to treat pieces of infrastructure as a black box.

(via https://www.terraform.io/intro/getting-started/modules.html)

Overall there are around 15 actively used Terraform community modules and lots of custom built modules (often niche one’s). Those modules were created to solve common scenarios — create network components (VPC, subnets, NAT gateways, etc) and various types of resources (RDS, ECS, Elasticache, etc).

Common traits in Terraform modules

Not clean code

Quote by Nick Dellamaggiore, Infrastructure Lead at Coursera, taken from the book «Terraform: Up & Running» by Yevgeniy Brikman:

If I look at a single file and it’s written by 10 different engineers, it should be almost indistinguishable which part was written by which person. To me, that is clean code.

In collaborative environment following coding style, naming conventions and agreed best-practices is important, because it lets others contribute faster.

Clean code is also aggregative term and is related to other traits listed below.

There is no single set of Terraform configurations best-practices yet, which most of Terraform users know and follow regularly. List of traits below may initiate discussions, which may turn into statements, which may turn into “Terraform modules best-practices”. Who knows?

Module features

Modules should be extensible. Often developers are taking single-purpose design principle too literally, which leads to not-extensible set of features which makes it impossible to extend without forking it (for example, there are 128 forks of very popular terraform-community-modules/tf_aws_vpc module). Good modules are as flexible as Terraform allows us to do without complicated hacks. For example, if there is a security group in the module, then it should be created as two resources — aws_security_group and aws_security_group_rule, so that user can append some rules, if necessary. One of the limitation which is still hard to do nicely is related to resource tags.

Tests

It is good practice in software development to include tests for your code and run them often. In my opinion it is still not well-covered area when it comes to Terraform configurations even though there are several tools which can help doing this. At the very least you should have a CI system run “terraform plan” with sane defaults. Apply code to create resources and then destroy them is more tricky, and it is fine to do it less often, but still automatically.

Sometimes there are syntax errors in completed code and in pull requests, which basic “terraform plan” can catch.

Defaults

Modules should have sane defaults. By sane I mean that it should be possible to use the module providing just essentially important arguments and use module’s defaults for the rest.

Examples

It is helpful for the users of the module to be able to see complete examples of the module and be able to run plan and apply on it effortlessly.

Documentation

…and deficit of it. Documentation should outline the purpose of a module, which resources it creates, how to use the module, required version of Terraform, how to report issues, etc.

Lifecycle

Module resources lifecycle should implement create, read, update and destroy functions. Resources creation is easy to handle in code, and this is often the main reason why the module was created. Read and destroy functions are also easy, because once created you should be able to read and destroy resources (bear with me “easy” is subjective). Update function is the most complicated, because of variety of resources and ways how they can be updated (replace in place, replace gradually, replace all at once, blue-green, etc). Modules are almost never implemented with a clear path through updates.

Versioning

Infrastructure as code should always be versioned. Versioning strategy and available releases should be documented. It is handy to be able to use specific release instead of using “master” branch for your critical infrastructure.

Security

You should always attempt to use the principle of least privilege when working with infrastructure. It is especially important when multiple operators handle infrastructure in self-service way. Module should specify set of least privileges it uses.

For now this is it, and in the next blog post I want to propose some ideas to the points listed above.

Thanks to Tim Hartmann, Steve Huff and Alex Matsuk for initial review and proof-reading.