Week 7 Worklog

Week 7 Objectives:

  • Refactor the entire project to an advanced Serverless architecture to resolve bottleneck issues.
  • Implement a Message Queue mechanism to coordinate stable log data flows.
  • Optimize storage structure and Big Data queries on S3 using Hive Partitioning and Amazon Athena.

Tasks implemented this week:

No.TaskStart DateEnd DateResource
1- Research: Data Routing architecture and advantages of SQS FIFO in ensuring data ordering.
- Project:
  + Transition data flow to use SQS FIFO as a central processing buffer.		04/20/202604/20/2026AWS Documentation
2- Research: Alert Debounce logic and controlling alert states based on timestamps.
- Project:
  + Build an algorithm to suppress repeated SNS alerts based on time-status queries from DynamoDB.		04/21/202604/21/2026AWS Documentation
3- Research: Hive Partitioning mechanisms and advanced DDL commands in Amazon Athena.
- Project:
  + S3 Storage Upgrade: Deploy Hive Partitioning (year/month/day/appid).
  + Synchronize data partitions using Athena.		04/22/202604/22/2026AWS Documentation
4- Research: Lambda integration with HTTP API Gateway and authentication for Client Dashboard.
- Project:
  + Review integration flow between API Gateway and Lambda for data retrieval.		04/23/202604/23/2026AWS Documentation
5- Research: DynamoDB Time to Live (TTL) and S3 Lifecycle Management.
- Project:
  + Configure S3 Lifecycle Rules; create DynamoDB tables with TTL mechanism.
  + Package Python source code and set up SQS as an Event Trigger for Lambda.		04/24/202604/24/2026AWS Documentation

Key Achievements in Week 7:

1. Advanced System Architecture

  • Successfully transitioned from a traditional log collection model to a distributed Serverless architecture, utilizing SQS FIFO queues to guarantee message ordering and system load capacity.
  • Mastered JSON parsing within the Lambda Consumer to simultaneously distribute data into cold storage (S3) and hot storage (DynamoDB).

2. Alert Management Optimization

  • Completely resolved the email “bombing” issue using an Alert Debounce algorithm, allowing for the control of SNS alert frequency within a 15-minute window.

3. Large-Scale Data Management

  • Established a Hive Partitioning structure on S3, which optimized storage space and significantly increased query speeds for Amazon Athena.
  • Successfully implemented a Time to Live (TTL) mechanism on DynamoDB to automatically clean up status logs, maintaining table performance and reducing costs.

4. Infrastructure Deployment & Integration

  • Successfully deployed and connected event flows between core services (SQS, Lambda, S3, DynamoDB, SNS) through the configuration of strict and secure IAM Role policies.

Knowledge Gained:

  • Technical Expertise: Gained a deep understanding of the differences between synchronous and asynchronous architectures in log processing. Mastered Hive data partitioning techniques for handling Big Data on the Cloud platform.
  • Soft Skills: Developed advanced Python source code analysis capabilities. Honed an architectural optimization mindset to meet real-world performance and cost requirements.