AWS Lambda request limit
In this blog post, I will explain the AWS Lambda request limit of 1 MB and how it affects your serverless applications. I will also show some examples of how to handle large payloads with Lambda functions and provide some references for further reading.
AWS Lambda is a service that lets you run code without provisioning or managing servers. You can use Lambda to create serverless applications that respond to events such as HTTP requests, database changes, or messages from other services. However, Lambda has some limits on the amount of compute and storage resources that you can use to run and store functions. These limits are either hard or soft, meaning that they cannot be changed or they can be increased upon request.
One of these limits is the request payload size, which is the amount of data that you can send or receive from your Lambda function. The request payload size depends on how you invoke your Lambda function:
- If you invoke your Lambda function synchronously, such as through an API Gateway or an Application Load Balancer (ALB), the request payload size is limited to 6 MB for both the request and the response. This means that you can only send and receive up to 6 MB of data in each invocation.
- If you invoke your Lambda function asynchronously, such as through an S3 event or an SNS message, the request payload size is limited to 256 KB for the request and 6 MB for the response. This means that you can only send up to 256 KB of data in each invocation, but you can receive up to 6 MB of data in the response.
- If you invoke your Lambda function using a container image, the request payload size is limited to 16 KB for both the request and the response. This means that you can only send and receive up to 16 KB of data in each invocation.
However, there is an additional limit for ALB-Lambda integrations: the request body size. The request body is the part of the HTTP request that contains the data that you want to send to your Lambda function. For example, if you want to upload a file or submit a form, the file or form data would be in the request body. The request body size is limited to 1 MB for ALB-Lambda integrations, regardless of whether you invoke your Lambda function synchronously or asynchronously . This means that you can only send up to 1 MB of data in the request body when using an ALB-Lambda integration.
Why is there a 1 MB request body limit for ALB-Lambda integrations?
The reason for this limit is not clear, but it may be related to how ALB handles HTTP requests and responses. ALB is a service that distributes incoming traffic across multiple targets, such as EC2 instances or Lambda functions. ALB supports both HTTP/1.x and HTTP/2 protocols, which have different ways of encoding and transferring data.
HTTP/1.x uses a text-based format for headers and a binary format for bodies. Headers are key-value pairs that provide metadata about the request or response, such as the content type, content length, or cookies. Bodies are optional and contain the actual data that you want to send or receive, such as a file or a JSON object. Headers and bodies are separated by a blank line and are sent sequentially over a single TCP connection.
HTTP/2 uses a binary format for both headers and bodies. Headers are compressed and encoded into frames, which are small chunks of data that have a fixed length and a type. Bodies are also encoded into frames, but they can be split into multiple frames and interleaved with other frames over multiple TCP connections.
ALB supports both protocols by converting between them as needed. For example, if a client sends an HTTP/2 request to ALB, ALB will convert it into an HTTP/1.x request before sending it to a target. Conversely, if a target sends an HTTP/1.x response to ALB, ALB will convert it into an HTTP/2 response before sending it back to the client.
This conversion process may introduce some overhead and complexity when dealing with large payloads. For example, if a client sends a large file in an HTTP/2 request body, ALB will have to split it into multiple frames, compress and encode each frame, and then reassemble them into an HTTP/1.x request body before sending it to a target. Similarly, if a target sends a large file in an HTTP/1.x response body, ALB will have to decompress and decode each frame, and then reassemble them into an HTTP/2 response body before sending it back to the client.
To reduce this overhead and complexity, ALB may impose a limit on the size of the request body that it can handle. This limit may be 1 MB, which is the same as the limit for the request line and the single header in HTTP/1.x. By limiting the request body size, ALB may avoid having to split, compress, encode, and reassemble large payloads, and thus improve its performance and reliability.
How to handle large payloads with Lambda functions?
If you need to send or receive large payloads with your Lambda functions, you may encounter some challenges due to the request payload size and the request body size limits. However, there are some possible solutions that you can use to overcome these challenges:
- Use a different service or protocol to invoke your Lambda function. For example, instead of using an ALB-Lambda integration, you can use an API Gateway-Lambda integration, which has a higher request payload size limit of 10 MB for both the request and the response. Alternatively, you can use a WebSocket protocol, which allows you to send and receive data in real-time without any size limits.
- Use compression or encoding techniques to reduce the size of your payload. For example, you can use gzip or base64 to compress or encode your data before sending it to your Lambda function, and then decompress or decode it inside your function. However, this may increase the CPU and memory usage of your function, and thus affect its performance and cost.
- Use streaming or chunking techniques to split your payload into smaller pieces. For example, you can use multipart/form-data or chunked transfer encoding to send your data in multiple parts or chunks over a single HTTP connection. Your Lambda function can then process each part or chunk as it arrives, without having to wait for the entire payload. However, this may require some additional logic and error handling in your function code.
- Use external storage services to store and retrieve your payload. For example, you can use S3 or EFS to upload or download your data from a bucket or a file system. Your Lambda function can then access your data using an S3 event or an EFS mount point. However, this may introduce some latency and cost for transferring and storing your data.
References:
Comentarios