It is widely known that Simple Queue Service, or SQS, is commonly used to decouple applications and processes tasks asynchronously. This means we usually have data producer then SQS and then data consumer. Producer sends messages to SQS queue, while another component receives and processes them.
However there are three concepts important when working with AWS SQS polling:
- Short polling
- Long polling
- Batching (Lambda batching)
Those are related, but they solve different problems:
- Polling -> controls how messages are retrieved from SQS
- Batching -> controls how retrieved messages are grouped before usually Lambda is invoked
How consumers receive messages from SQS
SQS does not actively push messages, instead the consumer calls the ReceiveMessage API and asks SQS whether messages are available for consuming. For example, an airport parking application running on ECS might repeatedly make requests to SQS: “do you have any messages?”.
At this point there are to valiable options how SQS will respond:
- Short polling (default SQS behaviour)
- Long polling
Important thing is that polling is configured on SQS level so that AWS manages the polling. You can check here more details. Also, polling works the same regardless if you pick FIFO queue or not.
Short polling
Short polling is default when the WaitTimeSeconds value is set to 0 seconds. When consumer makes a ReceiveMessage request, SQS checks a subset of the servers which physically store message for the queue and it immediately returns a response. This means two things:
- The response may be empty when the queue contains no messages.
- The response may occasionally be empty even when messages exist on servers that were not checked.
The second one we call false empty response and it can occur when consumer for example check only server that currently doesn’t have data so consumer might receive an empty response even though there are messages within SQS queue. But since we have short polling immediately after consumer will send another request to find them.
The frequent polling or sometimes an empty or low-traffic queue may generate many unnecessary API requests which costs.
Long polling
Long polling is enabled when the WaitTimeSeconds value is greater than 0 seconds. The maximum wait time is 20 seconds. Instead of immediately returning an empty response, SQS keeps the ReceiveMessage request open until:
- At least one message become available for consuming.
- The configured wait time expires.
Long polling also checks all SQS servers rather than subset of them which reduces both:
- Empty responses when no messages exist.
- False empty responses when messages exist but short polling does not find them.
For example, suppose a consumer starts a long-polling requests to check for new airport parking reservations with a wait time of 20 seconds.
0 seconds -> Consumer calls ReceiveMessage API
10 seconds -> A message arrives to SQS queue
10 seconds -> SQS immediately returns the messageSQS doesn’t wait for the remaining 10 seconds. The configured value is the maximum wait time. If there aren’t new airport parking reservations (no message arrive within 20 seconds), SQS simply returns empty response and the consumer can make new request.
When to use short vs long polling
Long polling is usually better for SQS consumers where we do not have time critical solutions or low traffic systems. If amount of message is low or inconsistent without a long polling we would have hundreds of requests asking for work even when the queue is empty. With long polling, each request can remain open for defined amount of time (up to 20 seconds), reducing empty responses and unnecessary SQS API calls.
Short polling should be used when you need an immediate response and when queue is usually busy so it is very unlikely to have empty responses. An example could be high-traffic order queue where messages are almost always available and consumer should have them
Batching SQS messages for Lambda?
When SQS is configured as a Lambda trigger, it may look like SQS directly invokes the function. In reality, Lambda creates an event source mapping that polls the queue on your behalf.
When messages become available, the event source mapping collects them into a batch and invokes the Lambda function.
Terraform example of event source mapping:
resource "aws_lambda_event_source_mapping" "sqs_trigger" {
event_source_arn = aws_sqs_queue.example.arn
function_name = aws_lambda_function.consumer.arn
batch_size = 50
maximum_batching_window_in_seconds = 10
function_response_types = [
"ReportBatchItemFailures"
]
enabled = true
}This configuration means:
- Lambda can receive up to 50 messages per invocation.
- The event source mapping may wait up to 10 seconds while collecting a larger batch.
ReportBatchItemFailuresallows only failed messages to be retried instead of the entire batch.
Lambda is invoked when one of the following happens first:
- 50 messages are collected;
- The batching window expires;
- The maximum payload size is reached.
The function may therefore receive fewer than 50 messages when traffic is low.
After successful processing, Lambda deletes the messages from the queue. The function itself does not call ReceiveMessage, because AWS manages the polling through the event source mapping.
Polling vs batching
Polling controls how messages are retrieved from SQS. Lambda batching controls how the messages retrieved by the event source mapping are grouped before the function is invoked. Batching is also related to Lambda concurrency feel free to check more on that topic here.
For custom consumers running on ECS, EC2, or EKS, short and long polling are direct application-level decisions. For SQS-to-Lambda integrations, the main settings you usually configure are the batch size and maximum batching window.
