Building an Application Log Analytics Platform with Amazon S3 Tables: Cost Optimization by Migrating from CloudWatch Logs

Manato Takai Posted on May 31 • Originally published at kaminashi-developer.hatenablog.jp Building an Application Log Analytics Platform with Amazon S3 Tables: Cost Optimization by Migrating from CloudWatch Logs # architecture # aws # infrastructure # monitoring Introduction Our team had been using CloudWatch Logs as the log storage layer for our identity management system, but as the service grew, the associated cost became a concern. This article describes how we built and migrated to a log storage architecture using Amazon S3 Tables, referred to here as S3 Tables, and the cost optimization results we achieved. Rising CloudWatch Logs Costs as the Service Grew Amazon CloudWatch Logs, referred to here as CloudWatch Logs, is an AWS service for storing and analyzing logs. In many common architectures, application logs are stored directly in CloudWatch Logs. When an issue occurs, teams query logs using CloudWatch Logs Insights and configure alerts for specific error logs in combination with CloudWatch Alarms. Our identity management platform followed a similar pattern. Logs from ECS were delivered to CloudWatch Logs through a log router, and we used Amazon Managed Service for Grafana, referred to here as Grafana, for day-to-day development and operations. As our services grew, the identity management platform, which handles authentication and authorization requests across all services, saw a corresponding increase in request volume and feature scope. As a result, logging costs grew to a level that could no longer be ignored. We had made steady improvements through weekly reviews by removing unnecessary logs and consolidating others, but these efforts alone were not enough to achieve substantial cost optimization. We therefore began considering a more fundamental solution. The main costs incurred during normal CloudWatch Logs operations can be grouped into three categories: log ingestion, storage, and queries. As the pricing table also shows, log ingestion is the p