azure - Web activity throws overlimit error when calling rest api

Question

My ADF pipeline has a lookup activity which uses a sql query to get data from a table and passes it to a web activity which posts the JSON to an API (azure app service). When the query gets 1000 rows it works fine but when I try over 5000 rows the web activity returns the error.

"errorCode": "2001",
"message": "The length of execution ouput is over limit (around 1M currently). ",
"failureType": "UserError",

When I post the 5000 rows to the API using postman it works fine. Any idea what this error means and how to resolve it?

Thanks in advance.

Answer 1

As you found, web activity has times out limitation for 1 minute. Also, based on the above error The length of execution output is over limit (around 1M currently)., web activity also has output size limitation for 1 MB.

You could find the limitation rules here and some of the them could be adjusted if you ask for Contact Support.

If nothing they can do,I provide you with a workaround that you could use ForEach Activity. Maybe you need to use paging query for your rest api and return a limited number of the data each time. Then query your data in loop until the number of return data is lower than threshold value.