Log Ingestion

Content

4.4.1 Check resource
4.4.2 Test Lambda Shipper
4.4.3 CloudWatch Agent Research

Objective

Collect logs from the system through CloudWatch and deliver them into the processing pipeline.

Architecture

In the ingestion pipeline, CloudWatch acts as the log collection layer. Lambda Shipper receives these logs and forwards them to SQS for further processing. Struct

Architecture Description

In this system, logs are generated by the application and sent to CloudWatch. CloudWatch then ingests the logs and triggers Lambda Shipper for processing. Lambda Shipper forwards the data to SQS, where it continues through the processing pipeline.

Role of Lambda Shipper

  • Receive logs from CloudWatch
  • Process and forward log data
  • Send logs into the processing pipeline via SQS

Result

Logs are automatically ingested and transferred into the processing system.