grift: a clean approach to app configuration

Carolyn Ranti
Kensho Blog
Published in
4 min readMay 8, 2017

--

Our first open source Python package aims to make app configuration easier. Kensho Technologies has been using an internal version in production for a few months now, and we’re excited to be releasing it as an open source project!

What is app configuration?

In the code for a website, there are constants that should have different values depending on where the website is running. For example, the production site is (hopefully!) using a different database than the staging version, which is different from the database a developer uses when they’re running things locally.

So the question is — how do you write code that works for multiple environments? Rather than putting values for these settings in the code, you can read in values from another source (like a file, or environment variables). If you’re using a settings file, just create a separate file for each version of the website (production, staging, local, etc), and voila!

That doesn’t seem too bad. Why is it hard?

Settings can be sensitive. To name a few — database passwords, API tokens, AWS credentials, etc. These are secrets that typically are not stored in a repository that all developers can access.

Settings can be stored in multiple places. Less sensitive settings might be checked into a public repository, while environment variables might be set as part of the deploy process.

Misconfigured settings can cause failure at unpredictable times. What happens if a setting is missing or isn’t the right data type? It may not be clear until deploy time, and the failure may not happen until the setting is actually used in the code. This is no bueno.

Multiple solutions can complicate debugging. We found that without a unified way to configure apps, teams were approaching these challenges with bespoke solutions. That could make it more complicated for an on-call engineer to debug configuration issues — or even figure out where a setting value was being set.

So what does grift do?

It provides a clean way to define a schema for an app’s configuration, specify sources where settings are stored, and validate everything.

Configuration Schema

To get started, create a configuration class, defining each setting as an attribute of the class. Defining settings in a class makes it easy to figure out which settings are expected to exist — rather than having them sprinkled throughout the code.

Settings can be defined with a property type. When a setting value is loaded, it is converted to the correct data type.

A setting can also be defined with a default value. This is a nice way to provide a reasonable value, while still allowing an easy way to override.

Hierarchy of Sources

When the configuration class is initialized, it takes a list of loaders, each of which reads settings from a single source — a json file, environment variables, or a dict. The order of loaders is a hierarchy: each setting’s value is assigned from the first source in which it exists.

This strict hierarchy simplifies the process of figuring out where a setting’s value has been read from, when you have multiple sources.

Startup Guarantees

By default, every setting is required — a value must exist in one of the loaders. If any of the required settings are missing, an exception will be raised when the config class is initialized.

Initialization also fails if a value cannot be converted to the setting’s property type (though only if a property type was defined).

These validation steps provide a single failure point for misconfiguration — deploys are blocked if a required value is missing, or if the wrong type is provided.

Checking Service Configuration

One pain point for our developers was trying to figure out how a deployed service was configured. We started adding “varz” endpoints to our microservices — convenience endpoints that returned the non-sensitive part of the configuration in json format. The varz property of BaseConfig classes helps with exactly this — it is a dict with all of the non-sensitive settings. Settings can be marked as sensitive by defining them with exclude_from_varz=True.

How do I get started?

pip install grift

Where can I find more documentation?

Check out Kensho’s github for technical documentation and examples. You can get in touch with the maintainers by emailing grift-maintainer@kensho.com

Can I contribute?

Yes! Open issues or submit a pull request on our github.

What’s up with the name?

It’s a configuration manager — and like Paul Newman in The Sting, we’d like to think it’s pretty smooth.

(yeah, we’ll put a quarter in the pun jar…)

Interested in working on problems like this?

We’re hiring! Check out our careers page for details.

--

--