Short technology and team post on Gad and data technology and practices.
Introduction We will explain how we set up a data streaming pipeline from Kinesis Data stream to Athena using various AWS services. This solution will result in a presentation of the events written to the stream as records in a table.. Architecture Overview The solution will involve the following components: Events: Usage of aiOla app …
Kinesis Data stream+Kinesis Firehose – UI events Read More »
Introduction We will describe how to set up a data streaming pipeline from AWS DynamoDB to Athena using various AWS services. This solution will result in a replication of the DynamoDB in Athena. Architecture Overview The solution will involve the following components: DynamoDB: The source database where Jargonic tables reside. DynamoDB Stream: A stream attached …
DynamoDB to Athena using DynamoDB stream+Kinesis Firehose Read More »
We streams the data from DynamoDB to S3, using Kinesis, Kinesis FireHose and formatting Lambda, And use Airflow and DBT to create Iceberg tables on top of it and query it via Athena. The initial load is being done via federation, we are using federation to perform data checks and validation as part of the …