How to Easily Manage Terraform State Files Using Remote Backends

Prerequisites

This tutorial builds on concepts from my previous Terraform article. If you haven’t read it yet, please start here: Link

If you've just finished creating your first S3 bucket with Terraform, you might be wondering: "How does Terraform remember what it created?" That's where the state file comes in. In this tutorial, we'll explore what state files are, why they matter, and how to manage them properly using a remote backend.

How Terraform Actually Works

Think of Terraform like a home organiser. Every time you ask it to organise your room, it:

1. Takes a photo of how your room currently looks (current state)

2. Compares it with how you want it to look (your configuration)

3. Only moves or changes things that are different

Here's how this process works:

The key player here is the state file - it's Terraform's memory of what actually exists in AWS right now.

What is a State File?

The state file (terraform.tfstate) is a JSON file that Terraform creates automatically. Think of it as Terraform's notebook where it writes down everything it knows about your infrastructure.

What's Inside the State File?

When you created your S3 bucket in the previous tutorial, Terraform stored information like:

• The bucket name: ijas-tf-test-bucket-12345

• The bucket's AWS region

• All the tags you applied

• Internal AWS identifiers

⚠️ Important: The state file can contain sensitive information like passwords and secret keys. This is why you should never commit it to Git or share it publicly!

The Problem with Local State Files

When you ran terraform apply in your S3 tutorial, Terraform created a terraform.tfstate file in your project folder. This is called a local state file.

Why Local State is Problematic

Real-world scenario:

• You create an S3 bucket (state file on your laptop)

• Your teammate modifies it (state file on their laptop)

• Now you both have different "memories" of what exists

• Next time either of you runs Terraform, everything breaks!

State File Best Practices

Before we dive into remote backends, here are the golden rules:

1. Never edit state file manually - Let Terraform handle it

2. Store remotely - Not on your local file system

3. Enable state locking - Prevent concurrent modifications

4. Backup regularly - Protect against accidental loss

5. Separate environments - Different state files for dev/staging/prod

6. Restrict access - It contains sensitive data

7. Encrypt - Both at rest and in transit

Enter Remote Backend: The Solution

A remote backend is like moving from a personal notebook to a shared online document that everyone can access, but with strict rules about who can edit it and when.

Benefits of Remote Backend

AWS Remote Backend with S3

For AWS users, the most common remote backend uses:

• S3 Bucket: To store the state file

• S3 Native State Locking: To prevent conflicts (new in Terraform 1.10!)

What Changed in Terraform 1.10?

The Old Way (Before Terraform 1.10):

You needed two AWS services:

• S3 for storing the state

• DynamoDB for managing locks

The New Way (Terraform 1.10+):

Everything happens in S3! No DynamoDB needed.

How S3 Remote Backend Works

1. When you run terraform apply, Terraform tries to create a lock file in S3

2. S3 uses "conditional writes" with the If-None-Match header

3. If the lock file already exists, S3 rejects the creation = lock acquired by someone else

4. If the lock file doesn't exist, S3 creates it = you got the lock!

5. When done, Terraform deletes the lock file (or it becomes a delete marker with versioning)

Setting Up Remote Backend: Step-by-Step

Step 1: Create the State Bucket

First, we need an S3 bucket specifically for storing our state files. Choose a unique name (S3 bucket names must be globally unique across all AWS accounts).

• Go to AWS Console → S3 → Create Bucket

• Set a unique Bucket name, like infra-with-ijas-terraform-state

• Choose Region: ap-south-1 (or your preferred region)

• Enable versioning (recommended for backup)

• Enable encryption

Why versioning? Think of it as an unlimited undo button for your state file!

Step 2: Configure Your Terraform Files

Now let's update our S3 bucket example to use the remote backend:

terraform {
  # Configure remote backend
  backend "s3" {
    bucket       = "infra-with-ijas-terraform-state"  # Your state bucket
    key          = "dev/terraform.tfstate"            # Path within bucket
    region       = "ap-south-1"                       # Bucket region
    encrypt      = true                               # Enable encryption
    use_lockfile = true                               # Enable S3 native locking
  }
  
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

# Configure the AWS Provider
provider "aws" {
  region = "ap-south-1"
}

# Create S3 Bucket (same as before)
resource "aws_s3_bucket" "my_first_bucket" {
  bucket = "ijas-tf-test-bucket-12345"

  tags = {
    Name        = "My bucket 2.0"
    Environment = "Dev"
  }
}

Let's break down the backend configuration:

• bucket: The name of your state storage bucket

• key: The file path within the bucket (using dev/ helps organise different environments)

• region: Where your bucket lives

• encrypt: Always set to true to protect sensitive data

• use_lockfile: Set to true to enable S3 native state locking

Step 3: Initialise and Apply

# Optional: Create an alias for convenience
alias tf="terraform"

# Initialize Terraform (this migrates state to S3)
terraform init

# Preview changes
terraform plan

# Apply changes
terraform apply

# Or skip confirmation
terraform apply --auto-approve

What happens during terraform init with a remote backend?

Step 4: Verify Remote State

After applying, check your S3 bucket. You should see:

infra-with-ijas-terraform-state/
  └── dev/
      └── terraform.tfstate

You can also verify using AWS CLI:

aws s3 ls s3:/<your-tf-remote-state>/dev/

Working with State Files: Essential Commands

Now that your state is remote, here are commands you'll use regularly:

List All Resources

terraform state list

# Output
# aws_s3_bucket.my_first_bucket

This shows all resources Terraform is managing. Think of it as a table of contents.

Show Detailed Resource Information

terraform state show aws_s3_bucket.my_first_bucket

This displays everything Terraform knows about that specific resource - like looking at a detailed page in the notebook.

State Commands Reference

# List all resources in state
terraform state list

# Show detailed state information for a resource
terraform state show <resource_name>

# Remove resource from state (without destroying it in AWS)
terraform state rm <resource_name>

# Move resource to different state address (rename)
terraform state mv <source> <destination>

# Pull current state and display as JSON
terraform state pull

# Syntax: terraform import <resource_address> <aws_resource_id>
terraform import aws_s3_bucket.my_first_bucket ijas-tf-test-bucket-12345

Cleaning Up

When you're done with this tutorial:

# Destroy the resources Terraform created
terraform destroy

Optional: You can also delete the state bucket manually:

# Via AWS Console
# Or via CLI
aws s3 rb s3://infra-with-ijas-terraform-state --force

What's Next?

Now that you understand state management, you're ready to:

• Create more complex infrastructure with multiple resources

• Work safely in teams

• Organise your code into modules

• Implement proper environment separation

The remote backend is your safety net - it ensures everyone on your team is working with the same source of truth, and that no two people accidentally step on each other's toes!