Setting Up AWS Developer Environment with Python & Boto3

Setting Up Your AWS Developer Environment with Python Automation

Whether you are building your first serverless application or migrating enterprise workloads to AWS, you will quickly face a critical challenge: how to interact with AWS services efficiently and securely. The AWS Management Console is fine for exploring, but it becomes impractical when you need to manage hundreds of resources, automate deployments, or integrate AWS services into your applications.

The solution is a properly configured developer environment that combines the AWS Command Line Interface (CLI) for quick operations with Python and Boto3 for sophisticated automation.

However, setting up this environment correctly involves more than just installing a few tools. You need to consider security best practices, credential management, multi-account strategies, and automation workflows. A misconfigured environment can lead to security breaches, accidental resource deletion, or unexpected costs.

Solution Overview

This guide presents a systematic approach to setting up a professional AWS developer environment. The solution consists of several key components:

AWS CLI installation and configuration — providing command-line access to all AWS services
Python environment with Boto3 — enabling programmatic control and automation
Secure credential management — implementing best practices for access key handling
Multi-profile setup — supporting multiple AWS accounts and environments
Automated verification scripts — ensuring your environment is correctly configured

Prerequisites

Before we begin, ensure you have:

An AWS account (if you don't have one, create it at aws.amazon.com)
Python 3.8 or higher installed on your system
Administrative access to your local machine for software installation
Basic familiarity with command-line interfaces

Understanding AWS CLI Versions

AWS provides two major versions of the CLI. Understanding their differences matters for making the right choice.

AWS CLI v1

The original CLI, built with Python 2/3 compatibility. While still supported, it is considered legacy for new installations. You might encounter v1 in older documentation or existing systems.

Characteristics of v1:

Installed via pip (Python package manager)
Requires Python runtime
Limited features compared to v2
Slower performance
May have compatibility issues with newer AWS services

AWS CLI v2 (Recommended)

The current generation CLI, completely rewritten for better performance and features. This is what AWS recommends for all new installations.

Advantages of v2:

Standalone installer (no Python dependency)
Significantly faster command execution
Built-in auto-prompt for command completion
Improved error messages and debugging
Support for AWS IAM Identity Center (SSO)
Better handling of large responses
Automatic pagination for list commands
Wizards for complex configurations

Unless you have a specific requirement for v1 (such as legacy script compatibility), always choose v2 for new installations.

Step 1: Installing AWS CLI v2

The installation process varies by operating system, but AWS has streamlined it significantly compared to the older pip-based approach.

For macOS

You have two options: using Homebrew or the official installer.

# Option 1: Using Homebrew (recommended if you use Homebrew)
brew install awscli

# Option 2: Using the official installer
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

# Verify installation
aws --version
# Should show: aws-cli/2.x.x Python/3.x.x Darwin/x.x.x

For Linux

# Download the installer
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

# Extract and install
unzip awscliv2.zip
sudo ./aws/install

# Verify installation
aws --version

For Windows

Download the MSI installer from https://awscli.amazonaws.com/AWSCLIV2.msi
Run the downloaded MSI file
Follow the installation wizard
Open a new Command Prompt or PowerShell window
Verify with aws --version

Step 2: Understanding AWS Access Credentials

Before you can use the AWS CLI or any SDK, you need to understand how AWS authenticates requests. Unlike username/password authentication, AWS uses Access Keys for programmatic access.

What Are Access Keys?

Access Keys consist of two parts:

Access Key ID: A 20-character identifier (e.g., AKIAIOSFODNN7EXAMPLE)
Secret Access Key: A 40-character secret (e.g., wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY)

Think of them as a username and password specifically for API access. The Access Key ID identifies who is making the request, while the Secret Access Key proves you are who you claim to be.

Creating Your First IAM User

When you create an AWS account, you are logged in as the root user. This has unlimited power, which is dangerous for daily work. AWS best practice is to create an IAM (Identity and Access Management) user for development.

Log into the AWS Management Console using your root account credentials
Navigate to IAM: In the search bar, type "IAM" and select it
Create a new user:
- Click "Users" in the left sidebar
- Click "Create user"
- Username: Enter something like yourname-cli
- Click "Next"
Set permissions:
- Select "Attach policies directly"
- Search for and select AdministratorAccess
- Note: AdministratorAccess provides full AWS access. For production environments, use more restrictive policies
- Click "Next" and then "Create user"

A note on permissions: While PowerUserAccess might seem sufficient, it lacks permissions for creating IAM roles, which are required for Lambda functions, EC2 instances with roles, and many other AWS services. For learning and development, AdministratorAccess avoids permission roadblocks. In production, follow the principle of least privilege.

Create access keys:
- Click on the username you just created
- Go to the "Security credentials" tab
- Scroll to "Access keys" and click "Create access key"
- Select "Command Line Interface (CLI)"
- Check the confirmation box and click "Next"
- Add a description like "My development machine"
- Click "Create access key"
Save your credentials:
- You will see your Access Key ID and Secret Access Key
- IMPORTANT: This is the only time you will see the secret key
- Download the .csv file or copy both values to a secure location
- Click "Done"

Step 3: Configuring AWS CLI

Now that you have your access keys, configure the AWS CLI. It stores your credentials locally so you do not have to provide them with every command.

aws configure

You will be prompted for four pieces of information:

AWS Access Key ID: Paste the Access Key ID you just created
AWS Secret Access Key: Paste the Secret Access Key
Default region name: Enter your preferred AWS region (e.g., us-east-1, eu-west-1)
Default output format: Enter json (other options: text, table)

What Just Happened

The aws configure command created two files in your home directory:

~/.aws/credentials — Stores your access keys
~/.aws/config — Stores your preferences (region, output format)

These files are plain text, which is why protecting them is crucial. Never share these files or commit them to version control.

Testing Your Configuration

Verify everything is working by asking AWS "who am I?":

aws sts get-caller-identity

You should see output like this:

{
    "UserId": "AIDAI23HXD3KEXAMPLE",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/yourname-cli"
}

Step 4: Setting Up Python with Boto3

While the AWS CLI is great for one-off commands, Python with Boto3 (the AWS SDK) enables you to build sophisticated automation.

Why Boto3?

Automation: Write scripts that manage hundreds of resources
Integration: Build AWS functionality into your applications
Complex operations: Orchestrate multi-step processes
Error handling: Implement retry logic and graceful failure
Data processing: Work with AWS responses as Python objects

Creating a Python Virtual Environment

Virtual environments isolate your project dependencies, preventing conflicts between different projects.

# Create a virtual environment
python3 -m venv aws-dev-env

# Activate it
# On macOS/Linux:
source aws-dev-env/bin/activate

# On Windows:
aws-dev-env\Scripts\activate

# Your command prompt should now show (aws-dev-env)

Installing Boto3

pip install boto3 python-dotenv

This installs:

boto3: The AWS SDK for Python
python-dotenv: For managing environment variables

Your First Boto3 Script

Create a file called verify_aws_connection.py:

#!/usr/bin/env python3
"""
AWS Connection Verification Script

Verifies that your AWS credentials are properly configured
and displays information about your AWS environment.

Usage:
    python verify_aws_connection.py
"""

import boto3
from botocore.exceptions import ClientError, NoCredentialsError


def verify_aws_setup():
    """Verify that AWS credentials are properly configured."""
    try:
        sts = boto3.client('sts')
        identity = sts.get_caller_identity()

        print("Successfully connected to AWS")
        print(f"Account ID: {identity['Account']}")
        print(f"User ARN: {identity['Arn']}")
        print(f"User ID: {identity['UserId']}")

        username = identity['Arn'].split('/')[-1]
        print(f"Username: {username}")
        return True

    except NoCredentialsError:
        print("ERROR: No AWS credentials found")
        print("Run 'aws configure' to set up your credentials")
        print("Or set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables")
        return False

    except ClientError as e:
        print(f"ERROR: Could not connect to AWS: {e}")
        return False


if __name__ == "__main__":
    print("AWS Connection Verification")
    print("=" * 40)
    verify_aws_setup()

Run it:

source aws-dev-env/bin/activate
python verify_aws_connection.py

Expected output:

AWS Connection Verification
========================================
Successfully connected to AWS
Account ID: 123456789012
User ARN: arn:aws:iam::123456789012:user/yourname-cli
User ID: AIDAI23HXD3KEXAMPLE
Username: yourname-cli

Step 5: Managing Multiple AWS Accounts with Profiles

As you progress, you will likely work with multiple accounts — a personal account, a development account, and production access. AWS profiles let you switch between these easily.

Creating Additional Profiles

aws configure --profile work

This runs the same configuration wizard but saves credentials under the "work" profile name.

Using Profiles

With the CLI:

# Use default profile
aws s3 ls

# Use work profile
aws s3 ls --profile work

# Set profile for entire session
export AWS_PROFILE=work
aws s3 ls  # Now uses work profile

With Python:

import boto3

# Use default profile
s3 = boto3.client('s3')

# Use specific profile
session = boto3.Session(profile_name='work')
s3 = session.client('s3')

Profile Configuration Files

Your profiles are stored in:

~/.aws/credentials — Contains access keys for each profile
~/.aws/config — Contains configuration (region, output) for each profile

Step 6: Security Best Practices

Working with AWS credentials requires careful attention to security. A leaked access key can result in significant financial damage or data breaches.

Essential Security Rules

Never commit credentials to version control
- Add .aws to your .gitignore
- Never hardcode credentials in your scripts
- Use environment variables or AWS profiles
Enable MFA on your AWS account
- Go to IAM → Users → Your username → Security credentials
- Assign an MFA device (Google Authenticator, Authy, etc.)
Rotate access keys regularly
- Create a new key before deleting the old one
- Update your local configuration
- Test, then delete the old key
Use IAM roles when possible
- On EC2 instances, use IAM roles instead of access keys
- This provides temporary, automatically rotated credentials
Principle of least privilege
- Only grant the permissions actually needed
- Start with minimal permissions and add as required

Environment Variables for CI/CD

For production scripts or CI/CD pipelines, use environment variables instead of credential files:

export AWS_ACCESS_KEY_ID=your-access-key
export AWS_SECRET_ACCESS_KEY=your-secret-key
export AWS_DEFAULT_REGION=us-east-1

python your_script.py

Using .env Files for Local Development

WARNING: Never commit .env files to version control. Add .env to your .gitignore immediately.

Create a .env file in your project directory:

AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
AWS_DEFAULT_REGION=us-east-1

In your Python script:

from dotenv import load_dotenv
import os
import boto3

# Load environment variables from .env file
load_dotenv()

# Boto3 will automatically use the environment variables
s3 = boto3.client('s3')

Comprehensive Environment Validation

After setting up your environment, run this validation script to check that all components are properly configured. Create a file called validate_environment.py:

#!/usr/bin/env python3
"""
Comprehensive AWS Environment Validation

Checks CLI installation, Python setup, credentials, and permissions.

Usage:
    python validate_environment.py
"""

import sys
import subprocess

import boto3
from botocore.exceptions import ClientError, NoCredentialsError


class EnvironmentValidator:
    """Validates AWS development environment configuration."""

    def __init__(self):
        self.checks_passed = 0
        self.checks_failed = 0

    def run_all_checks(self):
        """Execute all validation checks."""
        print("AWS Development Environment Validation")
        print("=" * 60)
        print()

        self.check_python_version()
        self.check_aws_cli()
        self.check_boto3()
        self.check_credentials()
        self.check_default_region()
        self.check_basic_permissions()
        self.print_summary()

    def check_python_version(self):
        """Verify Python version meets requirements."""
        print("Checking Python version...")
        version = sys.version_info
        if version.major == 3 and version.minor >= 8:
            self._pass(f"Python {version.major}.{version.minor}.{version.micro}")
        else:
            self._fail(
                f"Python {version.major}.{version.minor} (3.8+ required)",
                "Upgrade Python to version 3.8 or higher"
            )

    def check_aws_cli(self):
        """Check if AWS CLI is installed and get version."""
        print("\nChecking AWS CLI installation...")
        try:
            result = subprocess.run(
                ['aws', '--version'],
                capture_output=True, text=True, check=True
            )
            version = result.stdout.strip()
            if 'aws-cli/2' in version:
                self._pass(f"AWS CLI v2: {version}")
            else:
                self._warn(f"AWS CLI v1: {version}",
                           "Consider upgrading to AWS CLI v2")
        except (subprocess.CalledProcessError, FileNotFoundError):
            self._fail("AWS CLI not installed",
                       "Install from https://aws.amazon.com/cli/")

    def check_boto3(self):
        """Check if Boto3 is installed."""
        print("\nChecking Boto3 installation...")
        try:
            import boto3
            self._pass(f"Boto3 version {boto3.__version__}")
        except ImportError:
            self._fail("Boto3 not installed", "Run: pip install boto3")

    def check_credentials(self):
        """Verify AWS credentials are configured."""
        print("\nChecking AWS credentials...")
        try:
            sts = boto3.client('sts')
            identity = sts.get_caller_identity()
            account = identity['Account']
            username = identity['Arn'].split('/')[-1]
            self._pass(f"Authenticated as '{username}' on account {account}")
        except NoCredentialsError:
            self._fail("No AWS credentials configured", "Run: aws configure")
        except ClientError as e:
            self._fail(f"Invalid credentials: {e}",
                       "Check your access keys in ~/.aws/credentials")

    def check_default_region(self):
        """Check if default region is configured."""
        print("\nChecking default region...")
        session = boto3.Session()
        region = session.region_name
        if region:
            self._pass(f"Default region: {region}")
        else:
            self._warn("No default region set",
                       "Run: aws configure set region us-east-1")

    def check_basic_permissions(self):
        """Test basic AWS service permissions."""
        print("\nChecking AWS service permissions...")
        services = [
            ('S3', 's3', 'list_buckets'),
            ('EC2', 'ec2', 'describe_instances'),
            ('IAM', 'iam', 'get_user')
        ]
        for service_name, service_id, operation in services:
            try:
                client = boto3.client(service_id)
                getattr(client, operation)()
                self._pass(f"{service_name} access verified")
            except ClientError as e:
                code = e.response['Error']['Code']
                if code in ('AccessDenied', 'AccessDeniedException'):
                    self._warn(f"{service_name} access denied",
                               f"Add {service_name} permissions to your IAM user")
                else:
                    self._fail(f"{service_name} error: {code}",
                               "Check your configuration and network")

    def _pass(self, message):
        print(f"  [OK] {message}")
        self.checks_passed += 1

    def _fail(self, message, suggestion):
        print(f"  [FAIL] {message}")
        print(f"     -> {suggestion}")
        self.checks_failed += 1

    def _warn(self, message, suggestion):
        print(f"  [WARN] {message}")
        print(f"     -> {suggestion}")
        self.checks_passed += 1

    def print_summary(self):
        total = self.checks_passed + self.checks_failed
        print("\n" + "=" * 60)
        print(f"SUMMARY: {self.checks_passed}/{total} checks passed, "
              f"{self.checks_failed} failed")
        print("=" * 60)
        if self.checks_failed == 0:
            print("\nYour AWS development environment is fully configured.")
        else:
            print("\nSome checks failed. Address the issues above.")


if __name__ == "__main__":
    validator = EnvironmentValidator()
    validator.run_all_checks()