It is widely known that Simple Queue Service, or SQS, is commonly used to decouple applications and processes tasks asynchronously. This means we usually have data producer then SQS and then data consumer. Producer sends messages to SQS queue, while…
Data ingestion layer is the entry point of a data platform. That means it receives data from external providers, devices, applications or even internal systems and brings it into the platform in controlled way. In AWS ecosystem this layer is…
One of the most common pattern being used within distributed systems on AWS is fan-out. Simple definition for fan-out would be when one event happens multiple independent systems need to react to it. Simple example could be when a user…
When you need very low latency data access, Redis is usually one of the first storage that comes to mind. However traditional Redis is often used as a cache, which means the source of truth still lives somewhere else. Amazon…
When building event-driven architectures and data streaming workloads on AWS, two services often appear in the same conversation: Amazon Kinesis Data Streams and Amazon MSK, which is AWS managed Apache Kafka. At first glance, they solve a similar problem. Producers…
Often when we hear that app would need login, it often sounds as simple feature: a user opens application, enters credentials and gets access. But in real AWS applications, especially enterprise applications, login is usually only the first part of…
When API Gateway invokes a Lambda function, the simplest setup is to point the API directly to the Lambda function. Since we often deploy new version of Lambdas as they change over time, every time we deploy Lambda can publish…
User-uploaded files are a common security risk. If an application allows users to upload PDFs, images, or any documents into S3, those files should be scanned before they are trusted by downstream services. Instead of building a custom pipeline with…
When working with a Terraform VPC Lambda setup, you will not always be creating AWS accounts and networking from scratch. In many projects, the client already has an AWS Landing Zone in place, with much of the foundational infrastructure and…
Sometimes we just need frontend to simply place files into S3, so routing every upload through API Gateway and maybe Lambda adds extra latency, extra cost and extra code to maintain. Also there is API upload size limit of 10MB.…
