When should I paginate my REST API response?

Im currently having an issue with my production deployment setup on AWS.
I have an EC2 instance hosting NGINX/PHP/Laravel 5.1, that sits behind an AWS ELB.
One of my RESTAPIs returns a response that amounts to 1.2 to 1.6 MB, in a specific scenario (In most general scenarios, its returns much smaller response sizes), and the calls times-out.
The following error message gets logged

upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxxxxxxxxxx, server: xxxxxxxxxxx, request: "POST /api/event/gallery HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.0-fpm.sock", host: "xxxxxxxxxxx"

1.2-1.6mb does not appear to be a large amount of response size to explain the timeout.

I did confirm that my actual PHP call (at the bottom of the call stack) returns within 20 seconds.

Subsequent investigations revealed that the AWS ELB has a timeout of 60 sec, and probably closes the connection.

My AWS EC2 instance is of type m4.large. And Im concerned about why a production setup with proven components like AWS ELB, NGINX, will have trouble handling 1.x mb of data efficiently. Its likely that Im doing something wrong, or I have not configured the overall deployment system correctly. Please help with your suggestions.

Or perhaps I’m being too optimistic, and should move to a paginate my response to cater to large response sizes. But then again, I dont want to paginate without understanding the limits, or clearly knowing when I should. What is the max size/limit for a REST API response, beyond which I should consider pagination?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s