Every CloudWatch Agent tutorial follows the same script: install the package, drop in a config, run the start command, done. Then you check the console an hour later and there are no logs. You SSH back in, find nothing obviously wrong, and start the slow Google spiral.

I’ve done that spiral more times than I’d like to admit. The setup itself is five commands. The reasons it silently fails are what nobody writes about. So here is a different kind of post — gotchas first, setup second.

Where it actually breaks

1. The IAM policy is attached, but to the wrong thing

This is the most common failure mode and the most embarrassing one once you realise what happened. You attach CloudWatchAgentServerPolicy to your IAM user, congratulate yourself, and move on. The agent runs as the EC2 instance, not as you. It needs the policy attached to the instance profile — the role that the EC2 instance itself assumes.

The symptom is buried in the agent’s own log:

AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/MyInstanceRole/i-0abc...
is not authorized to perform: logs:CreateLogGroup

To check what role the instance is actually using, SSH in and run:

aws sts get-caller-identity

If the Arn doesn’t reference a role with CloudWatchAgentServerPolicy attached, that’s your problem. Fix it in the EC2 console under Actions → Security → Modify IAM role, then restart the agent.

2. The agent can’t read your log files

The agent runs as root by default, which mostly hides this problem. The moment you set run_as_user: cwagent in the config — a perfectly reasonable thing to do for least-privilege reasons — file permissions start mattering.

Application logs are often 640 root:root or owned by a service user like myapp:myapp. The cwagent user can’t read those, and the failure is silent. The agent reports itself healthy, but the file simply never gets picked up.

Check it the boring way:

sudo -u cwagent cat /var/log/myapp/app.log

If that fails, the agent fails. Two fixes, in order of preference:

  1. ACL the file directory: sudo setfacl -R -m u:cwagent:rX /var/log/myapp — surgical, doesn’t change ownership.
  2. Run the agent as root: remove run_as_user from the config. Simple, but you give up a useful security boundary.

I default to ACLs unless I have a reason not to.

3. Wildcards and placeholders don’t match what you think

Two sub-gotchas live here.

Log rotation breaks the obvious glob. file_path: /var/log/myapp/*.log matches app.log but not app.log.1, app.log.2, or app.log.2026-05-20. If you rotate frequently and your agent restarts during a rotation, you can lose a window of logs. Either match the rotated form explicitly:

"file_path": "/var/log/myapp/*.log*"

…or configure logrotate to use copytruncate so the original file stays in place.

Placeholder names are case-sensitive. Valid placeholders for log_stream_name are {instance_id}, {hostname}, {ip_address}, and {local_hostname}. {InstanceId} looks right and fails silently — your log stream literally becomes the string {InstanceId}. Lowercase, snake_case. Always.

4. The agent is “running” but the config never loaded

The status command lies in a specific way:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status

It tells you the agent process is up. It does not tell you whether the config you tried to apply was valid. A trailing comma, a mistyped key, a wrong value type — the fetch-config step rejects the config, and the agent keeps running with whatever was loaded before. Often nothing.

The real error lives in a file almost no tutorial mentions:

sudo cat /opt/aws/amazon-cloudwatch-agent/logs/configuration-validation.log

This file is gold. It tells you exactly which line of your JSON the agent choked on. Bookmark the path.

The other useful command is the agent’s own translator, which converts the friendly JSON config into the underlying TOML the agent actually consumes. If translation fails, the config never reaches the running agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config -m ec2 -s \
  -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

Run this every time you change the config. The output is verbose but immediately tells you whether the config was accepted.

5. Retention defaults to “Never Expire”

This one isn’t a bug. It’s a slow money leak.

retention_in_days in the agent config is only respected when the agent creates the log group. If the group already exists — because you tested with a previous config, or it was created by Terraform, or anything else — the agent silently ignores the retention setting.

Default retention on a manually-created log group is “Never Expire.” A chatty app and three months of forgetting can quietly add a real line item to your AWS bill.

Set retention explicitly, separately, after the group exists:

aws logs put-retention-policy \
  --log-group-name /ec2/myapp \
  --retention-in-days 14

Or, better, manage it in your IaC layer so retention is enforced uniformly across every log group rather than per-config. Terraform’s aws_cloudwatch_log_group with retention_in_days is one line and worth the discipline.

The setup, now that you know what can go wrong

Same five steps you’ve seen elsewhere — but now you know what each one is silently betting on.

1. Attach the IAM policy to the instance role (not your user — see gotcha one):

EC2 console → instance → Actions → Security → Modify IAM role → attach a role with CloudWatchAgentServerPolicy.

2. Install the agent:

Amazon Linux 2 / 2023:

sudo yum install -y amazon-cloudwatch-agent

Ubuntu / Debian:

wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

3. Write the config to /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json:

{
  "agent": {
    "run_as_user": "root"
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/myapp/*.log*",
            "log_group_name": "/ec2/myapp",
            "log_stream_name": "{instance_id}",
            "timezone": "UTC",
            "retention_in_days": 14
          }
        ]
      }
    }
  }
}

Note *.log* (gotcha three), lowercase {instance_id} (also three), and root to sidestep gotcha two. Swap to cwagent once you’ve set the ACLs.

4. Apply the config and start the agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config -m ec2 -s \
  -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

Enable on boot:

sudo systemctl enable amazon-cloudwatch-agent

5. Verify against the gotchas, not against the happy path.

The checklist to paste into your runbook

When logs aren’t showing up, walk this top to bottom:

# Is the agent process even running?
sudo systemctl status amazon-cloudwatch-agent

# What does the agent itself think? (the file almost no one reads)
sudo tail -50 /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

# Did the config validate? (the other file almost no one reads)
sudo cat /opt/aws/amazon-cloudwatch-agent/logs/configuration-validation.log

# What IAM identity is the instance actually using?
aws sts get-caller-identity

# Can that identity actually talk to CloudWatch Logs?
aws logs describe-log-groups --region <your-region> --max-items 1

# Can the agent user read your log files?
sudo -u cwagent cat /var/log/myapp/app.log | head -1
# (skip this if running as root)

# Does the log group exist, and what's its retention?
aws logs describe-log-groups --log-group-name-prefix /ec2/myapp

Nine times out of ten, one of those commands will surface the problem in under a minute. The two log files in particular — amazon-cloudwatch-agent.log and configuration-validation.log — answer the vast majority of “why isn’t this working” questions, and they are the single most under-mentioned thing in the entire CloudWatch Agent ecosystem.

Closing thoughts

The CloudWatch Agent isn’t hard to set up. It’s hard to debug, because every failure mode is silent. Wrong IAM, wrong permissions, wrong glob, wrong config, wrong retention. None of them throw a loud error. All of them leave you staring at an empty log group wondering what you missed.

The fix isn’t a better tutorial. The fix is knowing where the agent writes its own complaints — and checking there first, before you open a new browser tab.