Retrieve all new results since the last retrieval

To ensure efficient handling of large datasets, the Dynamic Query API supports data pagination. By specifying parameters such as page size, users can retrieve data in manageable chunks, reducing response times and minimizing server load. This approach is particularly useful for applications requiring real-time data processing or displaying and even exporting results incrementally.

Prerequisites

API authentication

1. Initial query and results

In order to fetch all resources available for an entity containing more or less complex criteria, the search can be used. Below, a simple example retrieve the first page of data, with a page size (maximum number of resources returned in the response) of 5, sorting ascending by executionTime, which is one of the fields available for the diagnosticResult entity.

CODE

curl -X POST "[YOUR_DYNAMIC_QUERY_URL]/api/v1/search/diagnosticResult" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer [YOUR_ACCESS_TOKEN]" \
     --data '{"pageSize": 5, "sort": [{"type": "ASC", "field": "executionTime"}], "searchAfter": null}'

With the search service, the response body is a JSON structure composed by different attributes. In the example below were used the following ones.

pageSize: The amount of resources to be provided in each page.
sort: An array of Objects with two fields that are type ("ASC" | "DESC") and field.
searchAfter: Essential field in order to provide context for pagination to the API. For the first call, it is set to null

Other parameters can be given, but they are out of scope of this guide.

In the example above, you are using field executionTime of the diagnostiResult entity for sorting. This can be interesting if you want to sort the data to highlight its chronological aspect.

However, other strategies exist. Another one would consist in sorting with field last_update_date, which appears on the majority of the entities served by the Dynamic Query API. Using that field with "DESC" sorting, you can traverse all data from the most up to date the least up to date. That can be very useful to maintains your data regularly up to date.

2. First response while fetching data

The request above will return a JSON response similar to the one below, which was simplified for learning purposes (the real one contains 4 additional results, that were replaced by "..." below).

CODE

{
   "data":[
      {
         "audioCompatibility":{
            "jitter":31.8,
            "downloadSpeed":5505398.700649675,
            "latency":325,
            "uploadSpeed":36647082.83220174
         },
         "ipAddress":"XX.XX.XX.XXX",
         "executionTime":1732793103687,
         "webSocketCompatibility":{
            "enabled":true
         },
         "tenantId":"2001",
         "browserCompatibility":{
            "result":{
               "listed":true,
               "accepted":true
            },
            "os":{
               "name":"mac os",
               "version":"10.15.7"
            },
            "browser":{
               "name":"chrome",
               "version":"129.0.0.0"
            },
            "userAgentString":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
         },
         "audio":true,
         "additionalData":{
            "country":"Luxembourg",
            "workstation_name":"A workstation name",
            "school_name":"A school name"
         },
         "last_update_date":1732793103703,
         "_id":"eMOEcpMB5KMcFSTmdc9X"
      },
      ...
   ],
   "totalResults":47,
   "lastId":[
      1733202910967,
      6,
      "p9S1AwEYbnNhLXFhLWRpYWdub3N0aWMtcmVzdWx0FkI3NlRLZGVEU002YW11NVQ4bVdKUGcAFlhiMWJFWHFHVGl5ZHBPckVRQTI3d0EAAAAAAApYi0EWeWNqODlQWEFTZXFrSW9KcVgxd2lkQQABFkI3NlRLZGVEU002YW11NVQ4bVdKUGcAAA=="
   ]
}

To start pagination and look for additional pages, identify in the totalResults field of the response indicating 47, more data is available.

2. Additional request to fetch more data

In order to find updated data (since a previous request), a request similar to the previous one can be executed. First, identify the lastId field returned in the initial request above.

CODE

...
"lastId":[1733202910967,6,"p9S1AwEYbnNhLXFhLWRpYWdub3N0aWMtcmVzdWx0FkI3NlRLZGVEU002YW11NVQ4bVdKUGcAFlhiMWJFWHFHVGl5ZHBPckVRQTI3d0EAAAAAAApYi0EWeWNqODlQWEFTZXFrSW9KcVgxd2lkQQABFkI3NlRLZGVEU002YW11NVQ4bVdKUGcAAA=="]
...

That information is required to provide context to the Dynamic Query API for subsequent search requests. In order to retrieve the next page of data, the following request can be executed:

CODE

curl -X POST "[YOUR_DYNAMIC_QUERY_URL]/api/v1/search/diagnosticResult" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer [YOUR_ACCESS_TOKEN]" \
     --data '{"pageSize": 5, "sort": [{"type": "ASC", "field": "executionTime"}], "searchAfter": [1733202910967,6,"p9S1AwEYbnNhLXFhLWRpYWdub3N0aWMtcmVzdWx0FkI3NlRLZGVEU002YW11NVQ4bVdKUGcAFlhiMWJFWHFHVGl5ZHBPckVRQTI3d0EAAAAAAApYi0EWeWNqODlQWEFTZXFrSW9KcVgxd2lkQQABFkI3NlRLZGVEU002YW11NVQ4bVdKUGcAAA=="]}'

You will then receive the 5 next resources, and a new value for the lastId field of your response, to be used in your next request as the value of the searchAfter parameter, until the end of all available pages.

3. Last request while fetching data

After chaining subsequent calls, you will eventually reach the end of the pagination and be returned an empty set of data, as in the example response below:

CODE

{"data":[],"totalResults":47,"lastId":["p9S1AwEYbnNhLXFhLWRpYWdub3N0aWMtcmVzdWx0FkI3NlRLZGVEU002YW11NVQ4bVdKUGcAFlhiMWJFWHFHVGl5ZHBPckVRQTI3d0EAAAAAAApYkFwWeWNqODlQWEFTZXFrSW9KcVgxd2lkQQABFkI3NlRLZGVEU002YW11NVQ4bVdKUGcAAA=="]}