Sending Spring Boot Application Logs to AWS CloudWatch

Monitoring application logs is a crucial part of application development and maintenance. In this blog post, we’ll walk you through the process of sending Spring Boot application logs to AWS CloudWatch, Amazon’s monitoring and observability service. This is particularly useful for distributed applications where log management can be challenging.

Pre-requisites

Before proceeding, make sure your EC2 instance has an IAM role attached to it with the necessary permissions to create and write to CloudWatch Logs. Your policy should look like the following:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Action":[
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents",
            "logs:DescribeLogStreams"
         ],
         "Effect":"Allow",
         "Resource":[
            "arn:aws:logs:*:*:*"
         ]
      }
   ]
}

This policy enables your application running on the EC2 instance to interact with CloudWatch Logs, allowing it to create log groups and streams, and put log events.

Setting Up the Dependencies

First, you need to add a couple of dependencies in your pom.xml file: cloudwatchlogs from the AWS SDK and logback-classic which provides logging services. Include the following:

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>cloudwatchlogs</artifactId>
    <version>2.20.26</version>
</dependency>
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.5.6</version>
</dependency>

Configuring the logback.xml File

Once the dependencies are set up, define your logging configuration in the logback.xml file. Create an appender named CLOUDWATCH that refers to our custom CloudWatch appender class.

<configuration>
    <appender name="CLOUDWATCH" class="com.your.package.CloudWatchAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>INFO</level>
        </filter>
    </appender>

    <root level="info">
        <appender-ref ref="CLOUDWATCH" />
    </root>
</configuration>

Creating the CloudWatch Appender

The CloudWatchAppender class, which is a part of our package, is where the magic happens. This class extends UnsynchronizedAppenderBase<ILoggingEvent>, part of the Logback Classic module, which enables us to append logging events to our desired output - in this case, AWS CloudWatch.

This class communicates with CloudWatch, sending logs directly from our Spring Boot application. It manages a queue of log events, and once certain conditions are met, it flushes these events to AWS CloudWatch.

package com.your.package;

import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.UnsynchronizedAppenderBase;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.cloudwatchlogs.CloudWatchLogsClient;
import software.amazon.awssdk.services.cloudwatchlogs.model.DescribeLogStreamsRequest;
import software.amazon.awssdk.services.cloudwatchlogs.model.DescribeLogStreamsResponse;
import software.amazon.awssdk.services.cloudwatchlogs.model.InputLogEvent;
import software.amazon.awssdk.services.cloudwatchlogs.model.PutLogEventsRequest;

import java.util.LinkedList;
import java.util.Queue;

public class CloudWatchAppender extends UnsynchronizedAppenderBase<ILoggingEvent> {
    private CloudWatchLogsClient client;
    private String logGroupName;
    private String logStreamName;

    private Queue<InputLogEvent> eventQueue;

    public CloudWatchAppender() {
        logGroupName = "LOG-GROUP-NAME-IN-CLOUDWATCH";
        logStreamName = "LOG-STREAM-NAME-IN-CLOUDWATCH";

        client = CloudWatchLogsClient.builder()
                .region("CLOUDWATCH-REGION")
                .build();
        eventQueue = new LinkedList<>();
    }

    @Override
    protected void append(ILoggingEvent event) {
        // Construct the log message
        InputLogEvent logEvent = InputLogEvent.builder()
                .message(event.getLevel().levelStr + " " + event.getFormattedMessage())
                .timestamp(event.getTimeStamp())
                .build();

        // Add event to the queue
        eventQueue.add(logEvent);

        // Flush queue if it has more than 10 events - Prod
        /*
        if (eventQueue.size() >= 10) {
            flushEvents();
        }
        */
        // Flush queue - Dev
        flushEvents();
    }

    private void flushEvents() {
        // Retrieve the existing log events
        DescribeLogStreamsResponse describeLogStreamsResponse = client.describeLogStreams(DescribeLogStreamsRequest.builder()
                .logGroupName(logGroupName)
                .logStreamNamePrefix(logStreamName)
                .build());

        String sequenceToken = describeLogStreamsResponse.logStreams().get(0).uploadSequenceToken();

        // Batch up the next 10 events
        LinkedList<InputLogEvent> logEventsBatch = new LinkedList<>();
        while (!eventQueue.isEmpty() && logEventsBatch.size() < 10) {
            logEventsBatch.add(eventQueue.poll());
        }

        // Check if logEventsBatch is empty
        if (logEventsBatch.isEmpty()) {
            return; // Skip the API call if there are no log events
        }

        // Put the log events into the CloudWatch stream
        PutLogEventsRequest putLogEventsRequest = PutLogEventsRequest.builder()
                .logGroupName(logGroupName)
                .logStreamName(logStreamName)
                .logEvents(logEventsBatch)
                .sequenceToken(sequenceToken)
                .build();

        client.putLogEvents(putLogEventsRequest);
    }

    @Override
    public void stop() {
        // Flush any remaining events before stopping
        flushEvents();

        // Clean up the AWS CloudWatchLogs client
        client.close();

        super.stop();
    }
}

Please ensure to replace "LOG-GROUP-NAME-IN-CLOUDWATCH", "LOG-STREAM-NAME-IN-CLOUDWATCH", and "CLOUDWATCH-REGION" with your actual CloudWatch Log Group Name, Log Stream Name, and the AWS CloudWatch region respectively.

How to Set Up AWS Credentials for Local Development Using IntelliJ IDEA

When developing locally, you need to configure your AWS credentials in IntelliJ IDEA. This is achieved by setting environment variables in the “Run/Debug Configurations” dialog.

Here are the steps:

1. In the IntelliJ IDEA menu, navigate to Run > Edit Configurations....
2. In the “Run/Debug Configurations” dialog that pops up, select the configuration for your application. Next to Build and run click Modify options -> 'Environment variables' . Paste aws credentials in below format -

AWS_ACCESS_KEY_ID=*****;AWS_SECRET_ACCESS_KEY=*****

3. Click ‘Apply’ and then‘OK’ to close all dialog boxes and save your changes.

This configuration will only apply to the current run/debug configuration. Each time you run or debug your application using this configuration, these environment variables will be set, allowing your application to authenticate with AWS.

Wrapping Up

This post showed you how to send Spring Boot application logs directly to AWS CloudWatch using Logback Classic and AWS SDK’s CloudWatch Logs. This is a powerful setup that can help you centralize your logs, making it easier to monitor application events and troubleshoot issues. We hope you found this helpful as you continue building your Spring Boot applications.

Happy coding!