python - Recommended way to manage credentials with multiple AWS accounts?

Question

What is the best way to manage multiple Amazon Web Services (AWS) accounts through boto?

I am familiar with BotoConfig files, which I'm using. But each file describes only a single account...and I am working with more than just the one organization. For all the usual legal, financial, and security reasons, those accounts cannot be commingled.

Currently I am using one boto config file per account. E.g.:

• ~/.boto default account
• ~/.boto_clowncollege for "clowncollege" account
• ~/.boto_razorassoc for "razorassoc" account
• ~/.boto_xyz for "xyz" account

Then something like:

def boto_config_path(account=None):
    """
    Given an account name, return the path to the corresponding boto
    configuration file. If no account given, return the default config file.
    """
    path = '~/.boto' + ('_' + account if account else '')
    clean_path = os.path.abspath(os.path.expanduser(path))
    if os.path.isfile(clean_path):
        return clean_path
    else:
        errmsg = "cannot find boto config file {} for {}".format(clean_path, account)
        raise ValueError(errmsg)

def aws_credentials(account=None):
    """
    Return a tuple of AWS credentials (access key id and secret access key) for
    the given account.
    """
    try:
        cfg = INIConfig(open(boto_config_path(account)))
        return ( cfg.Credentials.aws_access_key_id, cfg.Credentials.aws_secret_access_key )
    except Exception:
        raise

conn = EC2Connection(*aws_credentials('razorassoc'))

Good, bad, or indifferent? Suggested improvements?

Answer 1

updated 2015-02-06, corrected 2015-03-19 by following top section

New standardized sharing of boto and AWSCLI credentials (boto>==2.29.0)

Since boto 2.29 there is new easy way for sharing BOTO and AWS CLI credentials as described by Mike Garnaat in A New and Standardized Way to Manage Credentials in the AWS SDKs

The aim is to:

1. allow sharing credentials by boto, AWSCLI and possibly other SDKs
2. keep all configuration in single file, located in user profile directory
3. allow using named profiles
4. keep it as simple as possible (e.g. prevent conflicts with other methods)

Create credentials file

Create the file ~/.aws/credentials (Mac/Linux) or %USERPROFILE%.awscredentials (Windwos) as follows:

[default]
aws_access_key_id = AxxxA
aws_secret_access_key = Zxxxr
region = eu-west-1

[jekyl]
aws_access_key_id = AxxxA
aws_secret_access_key = Zxxxr
region = eu-west-1

[hyde]
aws_access_key_id = AxxxZ
aws_secret_access_key = CxxxZ
region = eu-west-1

From now on, you may use a code like this:

Use default profile

import boto
con = boto.connect_s3()

Use explicit profile set by AWS_PROFILE env. var

(this is my favourite option keeping profile name out of code and still giving the deployer of my application a chance to pick specific profile)

$ export AWS_PROFILE=jekyl

and keep your code as simple as before:

import boto
con = boto.connect_s3()

Specify explicit profile in your code

import boto
con = boto.connect_s3(profile_name="jekyl")

This is all you typically need to do

The logic for picking proper credentials is described in boto issue #2292 as follows:

The loading order from highest to lowest precedence:

1.Directly passed from code

2. Environment variables for key/secret

3. Environment variables for profile

4. Shared credential file explicit profile

5. Shared credential file default profile

6. Config file explicit profile

7. Config file Credentials section

A profile passed from code overrides any set in an environment variable.

To keep things clean and simple, it is good to get rid of older methods, so remove any old style files (like ~/.aws/config or ~/.boto), unset environmental varialbe BOTO_CONFIG if set and possibly also the file, to which such variable points to.

And that is really all for boto >=2.29.0

Note: Do not attempt to control location of config file by env.variable (like AWS_CONFIG_FILE), it does not work as expected.

Use boto config profile (boto>=2.24.0)

Following description is kept here only for those, who cannot upgrade to boto 2.29.0 or higher

Since boto 2.24.0 there is a feature called profile_name

In your ~/.boto file you already have your [Credentials] section, this will serve as fallback option, and then [profile ] sections serving for different profiles:

[Credentials]
aws_access_key_id = AxxxA
aws_secret_access_key = Zxxxr

[profile jekyl]
aws_access_key_id = AxxxA
aws_secret_access_key = Zxxxr

[profile hyde]
aws_access_key_id = AxxxZ
aws_secret_access_key = CxxxZ

Then, when creating connection, you use this way:

import boto
con = boto.connect_s3(profile_name="jekyl")

Note, that this feature is available since boto 2.24.0.

Tutorial is here http://docs.pythonboto.org/en/latest/boto_config_tut.html?highlight=profile

There are even some notes about using keyrings, but I will first get used to this profile stuff, which I was dreaming of few years.

Sharing config file with AWSCLI

AWSCLI became really great tool. As format of config file is almost the same, I use it in following way:

1. keep ~/.aws/config file as created by AWSCLI (this is default location)
2. copy section [default] and rename it to [Credentials] (leaving the same values inside).
3. add whatever profiles I use
4. set BOTO_CONFIG variable to point to this ~/.aws/config file.

The ~/.boto would then become `~/.aws/config with following content:

[default]
aws_access_key_id = AxxxA
aws_secret_access_key = Zxxxr

[Credentials]
aws_access_key_id = AxxxA
aws_secret_access_key = Zxxxr

[profile jekyl]
aws_access_key_id = AxxxA
aws_secret_access_key = Zxxxr

[profile hyde]
aws_access_key_id = AxxxZ
aws_secret_access_key = CxxxZ

This way, it gets shared for both AWSCLI and boto including profiles.