Pagination
Most resource types in Cognite Data Fusion (CDF) can be paginated, as indicated by the nextCursor
field in the response. Pass the value of nextCursor
as the cursor to get the next page of limit results. Note that all parameters except cursor
have to stay the same.
A request can return fewer results than its limits, as long as there is a nextCursor
that indicates that there are more data to fetch.
Simple cursor example
HTTP GET /assets?limit=1
{
"nextCursor": "8ZiApWzGe5RnTAE1N5SABLDNv7GKkUGiVUyUjzNsDvM",
"items": [
{
"name": "23-TE-96116-04",
"parentId": 3117826349444493,
"description": "VRD - PH 1STSTGGEAR THRUST BRG OUT",
"metadata": {
"ELC_STATUS_ID": "1211",
"RES_ID": "525283"
},
"id": 702630644612,
"createdTime": 0,
"lastUpdatedTime": 0,
"rootId": 6687602007296940
}
]
}
Getting the next page of assets.
When you send more parameters to /assets
, make sure that you also include them in all further requests in addition to the cursor
parameter.
HTTP GET /assets?limit=1&cursor=8ZiApWzGe5RnTAE1N5SABLDNv7GKkUGiVUyUjzNsDvM
Parallel retrieval
If you download many resources (for example, events with a particular filter), it can take a long time to paginate through millions of records. CDF supports parallel retrieval through the partition
parameter. The parameter has the format m/n
, where n
is the number of partitions to split the entire data set into.
Example
This example downloads an entire data set in parallel by splitting it into 10 partitions and doing the following requests with m
running from 1 to 10:
HTTP GET /events?type=WORKORDER&partition=1/10
HTTP GET /events?type=WORKORDER&partition=2/10
HTTP GET /events?type=WORKORDER&partition=3/10
Each of the results looks similar to this:
{
"items": [
{
"startTime": 1266130800000,
"endTime": 1284328234000,
"type": "Workorder",
"subtype": "VAL",
"assetIds": [4650652196144007],
"source": "cdf-tenant",
"id": 206950949774808,
"lastUpdatedTime": 1539151642457,
"createdTime": 1533747616899
}
],
"nextCursor": "bU-qc7X1jKMUbXgOYdONu2kRvpsoc60v9qh0Votm4MT0LaH0J0VVhVLoorSh8j_j"
}
Follow the nextCursor
in each response until each partition is drained.
Make sure that you send the filter parameters to each request that follows.
Datapoint pagination
Datapoints, including aggregates, can use pagination similar to other endpoints. The first datapoint on the second page will follow immediately after the last datapoint on the first page.
The main difference to other endpoints is that you will receive a nextCursor
for each time series that has more data and that the nextCursor
fields are inside the elements of the items
array. In further requests, you must combine each cursor alongside its corresponding time series.
If the other parameters are equal, you may combine (time series, cursor)
pairs from different requests.
You may also drop the time series you've completed and increase the limit on the remaining time series. Requests with smaller time series with a higher limit will be more effective than paging.
Datapoint cursors are opaque and not designed to be meaningful to users. The cursors aren't encrypted but shouldn't be reverse-engineered to construct a new cursor to supply to the API.
Cursors are valid for a reasonable time after the original API call, also if they've been used in a follow-up request. Still, we don't guarantee a minimum validity time, and you shouldn't store cursors to use them after a protracted time.
Cursors from different requests don't affect each other, and you can concurrently submit distinct series of cursor-based requests even if they cover the same period.
In general, distinct API calls aren't atomic with each other, also not when using cursors. The cursor doesn't "remember" the state of the data points at the time of the original request.
An API call that provides a cursor that was returned by a previous call will only see points with a timestamp after the last timestamp in the previous call. However, the second call might see some data points that didn't exist at the time of the first call and might not see some data points that did exist at the time of the first call. Also, the second call may let you see a partial result from an unrelated, concurrent request. If the unrelated request adds or deletes data points in a range that straddles the timestamp that the cursor points to, the second cursor-based request might see the result of the other request even though the first request that produced the cursor didn't.