Fetching and counting dynamic query resources

1. Introduction

The Dynamic Query REST API provides a powerful and flexible interface for querying data from your system. Designed to facilitate efficient data retrieval, this API supports a variety of features, including entity discovery, field listing, data pagination, and advanced filtering. This document introduces these core functionalities to help you make the most of the Dynamic Query API.

2. Prerequisites

View the following pages:

3. Listing available entities

Before fetching data from Dynamic Query, it is important to know what kind of entities you can retrieve from it. A specific listing service exist for that purpose. The Dynamic Query API allows users to retrieve a comprehensive list of available entities within the system. Entities represent distinct data objects or categories, such as sessions or results. By querying the endpoint dedicated to entity listing, users can dynamically explore the data structures available for querying and analysis.

Bash
curl -X GET "[YOUR_DYNAMIC_QUERY_URL]/api/v1/entity/list" \
     -H "Authorization: Bearer [YOUR_ACCESS_TOKEN]"

The response will contain all the available types of entities available in Dynamic Query. The name property values will be used to consume data from specific entities, such as "portalSessions" or "portalUserGroups".

JSON
[
   {
      "name":"portalSessions",
      "description":"Holds information on portal sessions"
   },
   {
      "name":"portalUserGroups",
      "description":"Holds information on portal user groups"
   },
   ...
   {
      "name":"diagnosticResult",
      "description":"Holds information on diagnostic results"
   },
   ...
]

4. Listing available entity fields

Once an entity is identified, the API provides functionality to list all available fields for that specific entity. This feature enables users to understand the structure and attributes of an entity, facilitating precise query formulation. Field metadata includes details such as field name, and potential usage constraints.

You can get the fields associated to a given entity using its [ENTITY_NAME] by executing the request example below.

Bash
curl -X GET "[YOUR_DYNAMIC_QUERY_URL]/api/v1/entity/[ENTITY_NAME]" \
     -H "Authorization: Bearer [YOUR_ACCESS_TOKEN]"

Below, an example JSON response returned when [ENTITY_NAME] is diagnosticResult.

JSON
[
   {
      "fieldName":"additionalData",
      "fieldType":"object",
      "canAggregate":false,
      "canAccumulate":false
   },
   {
      "fieldName":"audio",
      "fieldType":"boolean",
      "canAggregate":false,
      "canAccumulate":false
   },
   {
      "fieldName":"audioCompatibility.downloadSpeed",
      "fieldType":"float",
      "canAggregate":true,
      "canAccumulate":true
   },
   ...
]

5. Fetching data using pagination

To ensure efficient handling of large datasets, the Dynamic Query API supports data pagination. By specifying parameters such as page size, users can retrieve data in manageable chunks, reducing response times and minimizing server load. This approach is particularly useful for applications requiring real-time data processing or displaying and even exporting results incrementally.

5.1 Initial request for fetching data

In order to fetch all resources available for an entity given more or less complex criteria, the search can be used. Below, a simple example retrieve the first page of data, with a page size (maximum number of resources returned in the response) of 5, sorting ascending by executionTime, which is one of the fields available for the diagnosticResult entity.

Bash
curl -X POST "[YOUR_DYNAMIC_QUERY_URL]/api/v1/search/diagnosticResult" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer [YOUR_ACCESS_TOKEN]" \
     --data '{"pageSize": 5, "sort": [{"type": "ASC", "field": "executionTime"}], "searchAfter": null}'

With the search service, the response body is a JSON structure composed by different attributes. In the example below were used the following ones.

  • pageSize: The amount of resources to be provided in each page.

  • sort: An array of Objects with two fields that are type ("ASC" | "DESC") and field.

  • searchAfter: Essential field in order to provide context for pagination to the API. For the first call, it is set to null

Other parameters can be given, but they are out of scope of this simple guide.

In the example above, you are using field executionTime of the diagnostiResult entity for sorting. This can be interesting if you want to sort the data to highlight its chronological aspect.

However, many other strategies exist. Another one would consist in sorting with field last_update_date, which appears on the majority of the entities served by the Dynamic Query API. Using that field with "DESC" sorting, you can traverse all data from the most up to date the least up to date. That can be very useful to maintains your data regularly up to date.

5.2 First response while fetching data

The request above will return a JSON response similar to the one below, that was simplified for learning purpose (the real one contains 4 additional results, that were replaced by "..." below).

JSON
{
   "data":[
      {
         "audioCompatibility":{
            "jitter":31.8,
            "downloadSpeed":5505398.700649675,
            "latency":325,
            "uploadSpeed":36647082.83220174
         },
         "ipAddress":"XX.XX.XX.XXX",
         "executionTime":1732793103687,
         "webSocketCompatibility":{
            "enabled":true
         },
         "tenantId":"2001",
         "browserCompatibility":{
            "result":{
               "listed":true,
               "accepted":true
            },
            "os":{
               "name":"mac os",
               "version":"10.15.7"
            },
            "browser":{
               "name":"chrome",
               "version":"129.0.0.0"
            },
            "userAgentString":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
         },
         "audio":true,
         "additionalData":{
            "country":"Luxembourg",
            "workstation_name":"A workstation name",
            "school_name":"A school name"
         },
         "last_update_date":1732793103703,
         "_id":"eMOEcpMB5KMcFSTmdc9X"
      },
      ...
   ],
   "totalResults":47,
   "lastId":[
      1733202910967,
      6,
      "p9S1AwEYbnNhLXFhLWRpYWdub3N0aWMtcmVzdWx0FkI3NlRLZGVEU002YW11NVQ4bVdKUGcAFlhiMWJFWHFHVGl5ZHBPckVRQTI3d0EAAAAAAApYi0EWeWNqODlQWEFTZXFrSW9KcVgxd2lkQQABFkI3NlRLZGVEU002YW11NVQ4bVdKUGcAAA=="
   ]
}

As this guide focuses on describing how to fetch all the data for a given result, it is time for you to start pagination and look for additional pages. As you can see in the totalResults field of the response indicating 47, more data is available.

5.3 Additional request to fetch more data

In order to find the rest of data, a request similar to the previous one can be executed. First, pay attention to the lastId field returned in the initial request above.

JSON
...
"lastId":[1733202910967,6,"p9S1AwEYbnNhLXFhLWRpYWdub3N0aWMtcmVzdWx0FkI3NlRLZGVEU002YW11NVQ4bVdKUGcAFlhiMWJFWHFHVGl5ZHBPckVRQTI3d0EAAAAAAApYi0EWeWNqODlQWEFTZXFrSW9KcVgxd2lkQQABFkI3NlRLZGVEU002YW11NVQ4bVdKUGcAAA=="]
...

That information is required to provide context to the Dynamic Query API for subsequent search requests. In order to retrieve the next page of data, the following request can be executed:

Bash
curl -X POST "[YOUR_DYNAMIC_QUERY_URL]/api/v1/search/diagnosticResult" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer [YOUR_ACCESS_TOKEN]" \
     --data '{"pageSize": 5, "sort": [{"type": "ASC", "field": "executionTime"}], "searchAfter": [1733202910967,6,"p9S1AwEYbnNhLXFhLWRpYWdub3N0aWMtcmVzdWx0FkI3NlRLZGVEU002YW11NVQ4bVdKUGcAFlhiMWJFWHFHVGl5ZHBPckVRQTI3d0EAAAAAAApYi0EWeWNqODlQWEFTZXFrSW9KcVgxd2lkQQABFkI3NlRLZGVEU002YW11NVQ4bVdKUGcAAA=="]}'

You will then receive the 5 next resources, and a new value for the lastId field of your response, to be used in your next request as the value of the searchAfter parameter, until the end of all available pages.

5.4 Last request while fetching data

After chaining subsequent calls, you will realize you are at the end of the pagination by being returned an empty set of data, as in the example response below:

JSON
{"data":[],"totalResults":47,"lastId":["p9S1AwEYbnNhLXFhLWRpYWdub3N0aWMtcmVzdWx0FkI3NlRLZGVEU002YW11NVQ4bVdKUGcAFlhiMWJFWHFHVGl5ZHBPckVRQTI3d0EAAAAAAApYkFwWeWNqODlQWEFTZXFrSW9KcVgxd2lkQQABFkI3NlRLZGVEU002YW11NVQ4bVdKUGcAAA=="]}

6. Filtering on range

It can be interesting to retrieve data produced and/or updated during a specific time frame e.g. a computer-based assessment campaign, an academic year, …, etc.

To do so, you can use a special type of filter: range. The example below explains how to fetch all data that was produced/updated by filtering data on the last_update_date field of the diagnosticResult entity, from (using fromValue property of the filter) epoch 1734652801000 to (using toValue property of the filter) epoch 1734825601000.

For an easier reading, the example below retrieves data from to . Epochs are Unix timestamps in milliseconds.

Bash
curl -X POST "[YOUR_DYNAMIC_QUERY_URL]/api/v1/search/diagnosticResult" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer [YOUR_ACCESS_TOKEN]" \
     --data '{"pageSize": 5, "sort": [{"type": "DESC", "field": "last_update_date"}], "filters": [{"type": "range", "field": "last_update_date", "fromValue": 1734652801000, "toValue": 1734825601000}], "searchAfter": null}'

As a response are received the first 5 results over a total of 6. You can fetch the rest of the data using the usual fetching techniques described earlier in this document.

JSON
{
   "data":[
      {
         "audioCompatibility":{
            "jitter":26.6,
            "downloadSpeed":672253.2973621103,
            "latency":175,
            "uploadSpeed":36564196.59517168
         },
         "ipAddress":"XX.XX.XXX.XXX",
         "executionTime":1734724266554,
         "webSocketCompatibility":{
            "enabled":true
         },
         "tenantId":"2001",
         "browserCompatibility":{
            "result":{
               "listed":true,
               "accepted":true
            },
            "os":{
               "name":"mac os",
               "version":"10.15.7"
            },
            "browser":{
               "name":"chrome",
               "version":"131.0.0.0"
            },
            "userAgentString":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
         },
         "audio":true,
         "additionalData":{
            "country":"Albania",
            "workstation_name":"afsdfdsf",
            "school_name":"afsdfdsfd"
         },
         "last_update_date":1734724266568,
         "_id":"tcif5ZMB5KMcFSTmqqNJ"
      },
      ...
   ],
   "totalResults":6,
   "lastId":[
      1734722572575,
      14,
      "p9S1AwEYbnNhLXFhLWRpYWdub3N0aWMtcmVzdWx0FkI3NlRLZGVEU002YW11NVQ4bVdKUGcAFlhiMWJFWHFHVGl5ZHBPckVRQTI3d0EAAAAAAApaoisWeWNqODlQWEFTZXFrSW9KcVgxd2lkQQABFkI3NlRLZGVEU002YW11NVQ4bVdKUGcAAA=="
   ]
}

7. Counting

You can count resources available for a given entity by calling the metric endpoint. It works in a similar way to the search endpoints with the difference that it will not return resources data, but only counting information.

7.1 Simple counting

Without providing any filtering information in the request body, you can get the total count of resources for a give entity. Below, an example with the diagnosticResult entity.

Bash
curl -X POST "https://[YOUR_DYNAMIC_QUERY_URL]/api/v1/metric/diagnosticResult" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer [YOUR_ACCESS_TOKEN]"

As a response, you will receive the total count of resources for the diagnosticResult entity.

JSON
{
   "total":69
}

7.2 Counting by range

In order to maximize the potential of the metric endpoint, you can use filters to limit the counting scope and get more precise information. The following example counts the amount of diagnosticResult resources corresponding to diagnostic processes that were executed on from 00:00:01 GMT to 23:59:59 GMT.

Bash
curl -X POST "https://[YOUR_DYNAMIC_QUERY_URL]/api/v1/metric/diagnosticResult" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer [YOUR_ACCESS_TOKEN]" \
     --data '{"filters": [{"type": "range", "field": "executionTime", "fromValue": 1736121601000, "toValue": 1736207999000}]}'

As a result, the counting gets applied to a more limited scope, resulting as expected in a smaller final value.

JSON
{
   "total":34
}

7.3 Counting by search terms

In order to quickly find resources by property value, you can use the search using terms. In the following examples, we are searching for the portalUserSessions records having for sessionName property the “My Session Name” value.

curl -X POST "[YOUR_DYNAMIC_QUERY_URL]/api/v1/search/portalUserSessions" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer [YOUR_ACCESS_TOKEN]" \
     --data '{"filters": [{"type": "terms", "field": "sessionName", "values": ["[YOUR_SESSION_NAME]"]}]}'

As a result, the records that could be returned by such a request:

{
  "data": [
    {
      "testRunnerReviewSettings": {
        "enabled": false
      },
      "testRunnerSecurityPlugins": {
        "preventRightClickAndKeyboardShortcuts": true,
        "requireFullscreen": true
      },
      "timezone": "Europe/Luxembourg",
      "groupId": "f521906d-b23a-4a96-916d-b64bb733ceff",
      "login": "OAT-DQ-TUTORIAL-USER-1",
      "deliveryExecutionStatus": "initial",
      "groupRole": "TEST_TAKER",
      "deliveryId": "3a0fe9a5b300",
      "isPublic": false,
      "reportSettings": {
        "studentReport": false
      },
      "sessionActive": true,
      "last_update_date": 1736765106493,
      "sessionName": "OAT-DQ-TUTORIAL-SESSION",
      "testRunnerNotification": {
        "onSessionCreate": false,
        "onResultAvailable": false
      },
      "active": true,
      "sessionId": "f6e7c1bf-ba47-4ed7-bf37-c70e690d8f88",
      "groupName": "OAT-DQ-TUTORIAL-GROUP",
      "groupDescription": "OAT-DQ-TUTORIAL-SESSION for educational purpose",
      "gradesNotificationStartDate": null,
      "gradingRequired": false,
      "deliveryGradingStatus": "graded",
      "tenantId": "2001",
      "name": "John Doe",
      "groupActive": true,
      "deliveryName": "OAT-DQ-TUTORIAL-USER-DELIVERY-LABEL",
      "_id": "f6e7c1bf-ba47-4ed7-bf37-c70e690d8f88OAT-DQ-TUTORIAL-USER-1"
    },
    ...
  ],
  "totalResults": 50,
  "lastId": [
    284580
  ]
}