Request Pagination
Configuring requests for data that span multiple pages.
Last updated
Configuring requests for data that span multiple pages.
Last updated
Integration APIs commonly enforce limits for how many results can be returned in a single request. When you need to access more results, the API will provide a value in the response to use in your next request to get the next page of results.
Our native Integration Actions use pagination automatically, but Request and Integration Request steps can also be configured to page across multiple requests.
Request steps with Pagination will wait until all requests for data have been completed before proceeding in the workflow.
Before configuring Pagination, press Test Step () in your Request to make sure that you have Test Data to work with. This will be required to select fields from the output to configure subsequent, paginated requests.
After confirming that this step has Test Data, turn on the Pagination switch (below Authorization) for the Request step.
Pagination requires you to configure 3 values:
Output: Select the field from Test Data that represents a single page of results. Output must be an Array-type value, which will be aggregated across the paginated requests.
Page Token: Select a field from Test Data to use as a parameter for subsequent requests.
Stop When: Specify a condition for when request pagination should end. For example, ".nextPageToken Is null".
Finally, you'll need to use the Page Token value in the Request step configuration to make paginated requests fetch the next page of available data.
For example, if the API expects you to pass the Page Token as a query parameter called cursor, you can add this parameter to your request with the value {{self.pageToken}}.
In test mode, Request steps will only attempt to page once after the initial request. Verify that your request is paging correctly by checking the output array.
Requests that use Pagination will automatically attempt to retry requests that fail due to rate limits with an exponential backoff strategy.
If pagination cannot continue, then the step will provide the partial set of results that it successfully requested alongside the final error that occurred before the Request step.
Example output:
To handle errors from Request steps using Pagination, you can use a Conditional step to check for the presence of the error field before continuing the workflow.
Each page fetched in a Request step with Pagination configured will consume 1 task. For example, a Request that uses 150 requests to fully page through your user's data will consume 150 tasks.
You can configure your request to use larger page sizes (as allowed by the integration) to reduce the number of tasks that your Request step consumes.
As an example of configuring Request Pagination, we'll implement a workflow that uses Slack's API to get all channels from a Slack workspace.
The API endpoint we will use is conversations.list (see Slack documentation).
Slack allows for a maximum of 1000 channels to be retrieved at a time, but recommends no more than 200:
This method uses cursor-based pagination to make it easier to incrementally collect information. To begin pagination, specify a
limitvalue under1000. We recommend no more than200results at a time.Responses will include a top-level
response_metadataattribute containing anext_cursorvalue. By using this value as acursorparameter in a subsequent request, along withlimit, you may navigate through the collection page by virtual page.
Start by creating a Slack Request step in a workflow and configuring it to use conversations.list in the URL.
Click Test Step to see the output of this step and confirm that you see Slack channels from your connected Preview account.
Next, turn on Pagination. You will see 3 new fields appear to configure: Output, Page Token, and Stop When.
Set Output to the array of channel results by clicking on the Output field and using {{ to open the variable menu. Select the response.body.channels field from Test Data.
Set Page Token to the field that will be used as the "cursor" field of the request. Slack returns the result in response.response_metadata.next_cursor, as mentioned in their documentation quoted above.
Set Stop When to "response.response_metadata.next_cursor does not exist". This means that we will stop paging for additional results when Slack does not send us a next_cursor value in the last response.
According to the Slack documentation quoted above, we can request the next page with the cursor query parameter. Add cursor to the URL Parameters and use {{ to open the variable menu and set the value to Page Token.
Test the step again to verify that pagination is working as intended. Any errors that occur during pagination will appear in Test Data.
You can test your Request Pagination by clicking Test Step () after configuring the above options.
