DynamoDB to Athena using DynamoDB stream+Kinesis Firehose

Introduction

We will describe how to set up a data streaming pipeline from AWS DynamoDB to Athena using various AWS services.

This solution will result in a replication of the DynamoDB in Athena.

Architecture Overview

The solution will involve the following components:

1. DynamoDB: The source database where Jargonic tables reside.

2. DynamoDB Stream: A stream attached to the DynamoDB table that captures changes to the data.

3. Lambda: A serverless function that processes DynamoDB stream records and transforms them for further processing.

4. Lambda Role: IAM role that grants necessary permissions to the Lambda function.

5. Kinesis Firehose: Streams processed data from Lambda to an S3 bucket.

6. Kinesis Firehose Role: IAM role that grants necessary permissions to Kinesis Firehose.

7. S3 Bucket: Stores the streamed data in a structured format.

8. Glue Table: Catalogs the data in S3, making it queryable by Athena.

9. Athena: Allows SQL queries on data stored in the Glue table.