We use Kinesis Data Firehose as the consumer in this use case, with AWS Lambda as the record transformer, because our target storage is Amazon Redshift, which is supported by Kinesis Data Firehose. Other options include Amazon Kinesis Data Analytics with Flink, Amazon EMR with Spark streaming, Kinesis Data Firehose, or a custom application based on Kinesis consumer library. Lambda supports multiple programming languages, and for our use case, we use Python 3.8. It’s a computing service that runs code in response to events and automatically manages the computing resources required by that code. Lambda is an event-driven, serverless computing platform provided by AWS.In our use case, our target storage layer is Amazon Redshift, so Kinesis Data Firehose fits great to simplify the solution. It can also batch, compress, transform, and encrypt the data before loading it, minimizing the amount of storage used at the destination and increasing security. It’s a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration. It can capture, transform, and load streaming data into Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon OpenSearch Service, and Splunk, enabling near-real-time analytics with existing business intelligence (BI) tools and dashboards you’re already using today. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |