At Natural Intelligence, we have plenty of dockerized services that run on kubernetes and swarm cluster on AWS. Each service may require a different set of credentials. We use AWS System Manager (SSM) to store these credentials and to make them accessible in our services.
As part of our DevSecOps effort we want to make security easy for everyone.
The infrastructure teams set their sights on solving the challenge entailed by managing credentials. In a large scale, diverse environments, projects can be written in any of many programming languages, developed in several different environments and deployed through a number of continuous integration toolchains. This makes managing credentials even a complex task.
In this post, I will share the approach we have taken to bring secure and reliable applications to production.
We wrote an SDK that wrapped the access to SSM and allowed us access to credentials needed by services in a standard fashion, thus allowing for easier development and maintenance of credentials. The essence of using AWS SDK is to delegate standardization and consolidation in order to simplify development.
Instead of relying on an individual engineer to implement boto3 in order to retrieve the credentials from the Systems Manager, we decided to abstract this functionality in a shared Python module to support a unified interface.
Even if languages do not actively share ideas, parallels tend to emerge in tools, simply because most languages need to solve similar problems. We called this a polyglot package ni-secrets.
Authentication and Authorisation Flow
The basic requirement of the system is that each component will have its own set of permissions to AWS that apply to passwords, database strings, and license codes as parameter values. etc.
Naming Convention for Our Secrets
In order to provide a unified interface we also need to define a clear naming convention. AWS provides detailed instructions on how to organize your SSM Parameter Store to define and manage parameters easily.
This is our defined naming formula:
<Env>/<Component>/<AWS-Service-Name>
The flow looks like this:
It means that this service <AWS-Service-Name> requests credentials to this <Component>in this environment <Env>
Description:
<AWS-Service-Name> — the service
<Component> — a DB, Event bus, Queue, or any other service or infrastructure that requires credentials for access.
<Env> — production, staging etc — we have multiple “logical” environments in the same AWS account.
Using a well-defined hierarchy helps you to manage and retrieve parameters more efficiently, ensuring that all related parameters are bundled and easily found.
All of the above standardisation enables the following:
- Consistent naming convention
- Each service doesn’t need to know about the SSM
- One time only implementation: no need for repeated coding
- Easy to replace the secret storage
- Easy to manage and update passwords
Here is How we Support it in Python
We created a class wrapping all of the above and combined it with a couple of unit tests. Here are more details about the technical decisions we made.
SecretManager Class
The main class is the SecretManger. Its purpose is to abstract AWS SSM in order to provide the functionality of retrieving the credentials.
It enables the naming convention and storing the credentials in a Python dictionary. It also allows us a fallback for retrieval of the credentials from a local json file when AWS connection is unavailable.
The class provides a generic interface that allows for replacing the secret storage backend should we need to.
How Secret Manager establishes SSM Connection
The SecretManager constructor returns an object of the SSM resource. Now you can make requests and process responses from the service. Here you can see how we define the connection by the initSSM function:
https://gist.github.com/naturalett/bb9aa10bde4857b89077926285407795
The initSSM function uses boto3 the AWS SDK for python.
Retrieving the Credentials from SSM
We created an initialize function to retrieve from the SSM and store the credentials in a dictionary, following this process:
- Map the credential names in accordance with the naming convention
- Get the credential secrets using the SSM connection
- Store the credentials in a Python dictionary
Now that the credentials are in the dictionary, you can use them to access the necessary service — no need to access SSM the next time the service requires the credentials.
Unit-test
We believe in thoroughly testing our code. We chose to use unittest test cases and the unittest test runner. Unit tests promise that each method in our code works as intended, providing a baseline of quality and stability for the system as a whole.
https://gist.github.com/naturalett/136b6d4b0c677672cdd66d3c72c7d107
Argument Definition of the SecretManagerTest class
- name — the service name
- env — staging or production
- file_path — path to local json file storing cached credentials
Summarising the Test Logic
- The SecretManagerTest class defines attributes based on the arguments above: service name, environment and a file path.
- The SecretManager class creates a connection based on the attributes above.
- The initialize function gets credential names and will retrieve the credential values.
Packaging
The Python module includes one class and the logic structure was compiled as a Python egg, enabling an easy import for other Python projects.
To recap
This DevSecOps solution has several benefits:
- Using a naming convention will save you time by keeping your work organised and cohesive
- Reducing the abundance of boilerplate will keep your focus on coding
- Making it simple to manage and update credentials
- Abstracting away the secret storage so it can be easily replaced if needed