Units
The Units in Cognite Data Fusion (CDF) associate a set of units of measurement with data stored in other CDF resources, such as time series and data models.
Concepts in unit management
To use Units in Cognite Data Fusion, you need to understand the following concepts:
- Quantity: A specific, measurable property or characteristic of a physical system or substance. For example, temperature, pressure, mass, and length.
- Unit: A standard amount or value of a particular quantity by which other amounts of the same quantity can be measured or expressed. For example,
°Cand°Fare used for temperature, andbarandpsiare used for pressure. - Unit system: A collection of default units for given quantities. Examples include the
SIunit system ( The default value isKfor temperature andPascalfor pressure) and theImperialunit system ( The default value is°Ffor temperature andpsifor pressure).
About units
A list of standard industry-relevant units is available on every Cognite Data Fusion project. You can extend the units per project with custom units.
Each unit contains the following attributes:
externalId: A unique identifier structured as{quantity}:{unit}(for example,temperature:deg_c), following the snake_case convention.name: The primary name of the unit, such asDEG_C.longName: A descriptive name for the unit, likedegree Celsius.symbol: The symbol represents the unit, like°C.aliasNames: An array of alternative names or aliases for the unit.quantity: The physical quantity the unit measures likeTemperature.conversion: An object containing conversion factors (multiplierandoffset) for unit conversion.source: The primary source of the unit's definition, typically a standard organization likequdt.org.sourceReference: A URL reference to the external source for the unit's definition.
For more information on how to work with units, visit the Units API documentation.
Unit management in time series
Time series has a free text unit field that stores a reference to the unit as specified in the source system where the data has been read from. With the addition of the unit catalog in Cognite Data Fusion, a new field has been added to time series called unitExternalId.
You can assign the unitExternalId field to any unit available in the unit catalog. You can do this assignment when you create or update time series. See the example below for the creation of time series with the new unitExternalId field in the Python SDK:
client.time_series.create(
time_series = [
TimeSeries(
external_id="test_unit_ts_1",
name="Temperature Sensor 1",
unit="deg C",
unit_external_id="temperature:deg_c",
description="Temperature measured in degrees Celsius"
),
TimeSeries(
external_id="test_unit_ts_2",
name="Temperature Sensor 2",
unit="F",
unit_external_id="temperature:deg_f",
description="Temperature measured in degrees Fahrenheit"
)
]
)
Once a time series is created (or updated) with units from the catalog, it's possible to filter both on unitExternalId and unitQuantity.
When you update unitExternalId on a time series, it will not automatically convert the ingested data points. The unitExternalId provides typing information and enables unit conversion when querying data points. Make sure that you ingest the data in the correct unit.
The example below shows valid time series queries in the Python SDK:
# List time series assigned to degree Fahrenheit
client.time_series.list(
unit_external_id="temperature:deg_f"
)
# List time series assigned to units of quantity Temperature
client.time_series.list(
unit_quantity="Temperature"
)
The aggregate endpoints also support unitExternalId and unitQuantity, so you can get counts of the time series according to units and quantities.
The example below shows valid time series aggregations in the Python SDK.
# Get the number of unique unitExternalIds associated with time series
client.time_series.aggregate_cardinality_values(
property="unitExternalId"
)
# Get the number of unique unitQuantities associated with time series
client.time_series.aggregate_cardinality_values(
property="unitQuantity"
)
# Get the count per unitExternalIds associated with time series
client.time_series.aggregate_unique_values(
property="unitExternalId"
)
# Get the count per unitQuantities associated with time series
client.time_series.aggregate_unique_values(
property="unitQuantity"
)
For time series with associated units from the catalog, you can query data points in a compatible unit (units from the same quantity). The example below displays the data point queries with unit conversion in the Python SDK.
client.time_series.data.retrieve(
external_id=[
{
"external_id": "test_unit_ts_1",
"target_unit": "temperature:deg_c"
},
{
"external_id": "test_unit_ts_3",
"target_unit": "length:m"
}
],
start="2w-ago",
end="now"
)
You can specify a target unit system instead of specific units. The API will target the default unit for the allocated unit system. The example below shows the data points queries with unit conversion to a unit system in the Python SDK:
client.time_series.data.retrieve(
external_id=["test_unit_ts_1", "test_unit_ts_3"],
target_unit_system="SI",
start="2w-ago",
end="now"
)
See Time Series API documentation for more information about the time series endpoints.
Unit management in data models
You can associate float properties with specific units from the unit catalog within data models. This relationship occurs when you create or update a container. The example below demonstrates creating a container with float properties assigned to a specific unit.
payload = {
"items": [
{
"space": "pumps_space",
"externalId": "PumpSpecificationsContainer",
"name": "PumpSpecifications",
"description": "Container storing pump specifications",
"usedFor": "node",
"properties": {
"maxPressure": {
"nullable": True,
"description": "Maximum Pump Pressure",
"name": "maxPressure",
"type": {
"type": "float64",
"unit": {
"externalId": "pressure:bar",
"sourceUnit": "BAR"
}
},
},
"maxTemperature": {
"nullable": True,
"description": "Maximum Pump Temperature",
"name": "maxTemperature",
"type": {
"type": "float64",
"unit": {
"externalId": "temperature:deg_c"
}
}
},
"rotationConfigurations": {
"nullable": True,
"description": "Rotation Configurations",
"name": "rotationConfigurations",
"type": {
"type": "float64",
"unit": {
"externalId": "angular_velocity:rev-per-min"
},
"list": True
}
},
}
}
]
}
res = client.post(
url=f"/api/v1/projects/{client.config.project}/models/containers",
json=payload
)
Assigning unit.externalId and unit.sourceUnit to a float-type container property is optional. While unit.externalId must correspond to an entry in the unit catalog, unit.sourceUnit is a free text string where users can store the original name or alias used for the unit in the source system.
Updating the unit.externalId on a container property will not automatically convert the values ingested to the container. The unit.externalId provides typing information and enables unit conversion when querying the data. Make sure you ingest data in the correct unit.
Data modeling service (DMS) is the recommended way to manage industrial data models. Creating data models using the Data modeling extension for GraphQL (DML) is also possible.
The equivalent representation in DML for this container definition would be:
type PumpSpecifications {
"Maximum Pump Pressure"
maxPressure: Float @unit(externalId: "pressure:bar", sourceUnit: "BAR")
"Maximum Pump Temperature"
maxTemperature: Float @unit(externalId: "temperature:deg_c")
"Rotation Configurations"
rotationConfigurations: [Float64] @unit(externalId: "angular_velocity:rev-per-min")
}
Note: The @unit directive associates a property with a specific unit. (This works precisely like assigning units using DMS; the externalId argument must correspond to an entry in the unit catalog.)
See the Data modeling extension for GraphQL (DML) documentation for more information on using DML.
After you create the container, creating or updating views to map container properties is straightforward. For example, to map properties from the PumpSpecificationsContainer to a PumpSpecificationsView, see the Python code below:
payload = {
"items": [
{
"externalId": "PumpSpecificationsView",
"space": "pumps_space",
"name": "PumpSpecifications",
"description": "View pointing to pump specifications",
"version": "1",
"properties": {
"maxPressure": {
"name": "maxPressure",
"description": "Maximum Pump Pressure",
"container": {
"type": "container",
"space": "pumps_space",
"externalId": "PumpSpecificationsContainer"
},
"containerPropertyIdentifier": "maxPressure"
},
"maxTemperature": {
"name": "maxTemperature",
"description": "Maximum Pump Temperature",
"container": {
"type": "container",
"space": "pumps_space",
"externalId": "PumpSpecificationsContainer"
},
"containerPropertyIdentifier": "maxTemperature"
},
"rotationConfigurations": {
"name": "rotationConfigurations",
"description": "Rotation Configurations",
"container": {
"type": "container",
"space": "pumps_space",
"externalId": "PumpSpecificationsContainer"
},
"containerPropertyIdentifier": "rotationConfigurations"
},
}
}
]
}
res = client.post(
url=f"/api/v1/projects/{client.config.project}/models/views",
json=payload
)
The view definition doesn't contain unit details. The unit information is stored in the container definition and automatically inherited by the view.
Once you populate the container with data, querying the view or container will result in unit-aware data. Since the unit is fixed per property, make sure that you ingest the data in the correct unit. The example below shows populating the PumpSpecificationsView with sample data:
data = pd.DataFrame(
data={
"maxPressure": [35.2, 12.15, 23.5, 7.88],
"maxTemperature": [155.2, 65.4, 188.1, 203.5],
"rotationConfigurations": [
[3600, 2400, 1800],
[2400, 1800],
[3600, 2400, 1800],
[3600, 2400],
]
},
index=["P-3501", "P-3502", "P-3503", "P-3504"]
)
payload = {
"items": [
{
"instanceType": "node",
"space": "pumps_space",
"externalId": idx,
"sources": [
{
"source": {
"type": "view",
"space": "pumps_space",
"externalId": "PumpSpecificationsView",
"version": "1"
},
"properties": {
"maxPressure": row["maxPressure"],
"maxTemperature": row["maxTemperature"],
"rotationConfigurations": row["rotationConfigurations"]
}
}
]
}
for idx, row in data.iterrows()
]
}
res = client.post(
url=f"/api/v1/projects/{client.config.project}/models/instances",
json=payload
)
When querying data, set includeTyping to true to retrieve information about the units associated with all properties included in the response.
To add unit conversion into a response, specify either unit.externalId or unit.unitSystemName for each property requiring conversion within the targetUnits argument. When you set includeTyping to true, the typing information will reflect the units of the data included in the request. For example, if maxPressure was stored as pressure:bar, a request to convert to pressure:pa will result in both the data and typing information being in pressure:pa.
When applying a filter to a property included in targetUnits, the filter will be unit-aware, ensuring that the value passed on the filter aligns with the unit of the data. For instance, if maxPressure was stored as pressure:bar and a request to convert to pressure:pa with a filter greater than or equal to 3520000.0 will filter the data to instances where maxPressure is greater than or equal to 3520000.0 Pascal.
Converting values across various units may introduce a minor conversion error in the data. For example, when you convert 1.69999999999 meters to centimeters, it results in 170 centimeters. This discrepancy becomes particularly evident with the application of filters. For instance, when applying a filter for height less than 170 cm, one might anticipate results including the original height value of 1.69999999999, but due to rounding errors in conversion, these results might get excluded.
Be aware of the potential ambiguities when setting target units with filters. For instance, consider a scenario where two different views map to the same property, propertyA, but each view uses a separate unit for the property. When you apply a filter on propertyA and reference the property through its container (for example, [space, containerId, property]), the system cannot determine the correct source and, therefore, the appropriate targetUnit for the filter. To avoid this ambiguity, explicitly reference propertyA in the filter by specifying the view (for example, [space, viewId/viewVersion, property]).
All the data modeling query endpoints are updated to support typing information, unit conversion, and unit-aware filtering.
The following section will explain some examples of valid queries.
List instances with unit conversion and unit-aware filtering
The example below shows a list instance with unit conversion and unit-aware filtering.
payload = {
"sources": [
{
"source": {
"type": "view",
"space": "pumps_space",
"externalId": "PumpSpecificationsView",
"version": "1"
},
"targetUnits": [
{
"property": "maxPressure",
"unit": {
"externalId": "pressure:pa"
}
},
{
"property": "maxTemperature",
"unit": {
"unitSystemName": "SI"
}
},
{
"property": "rotationConfigurations",
"unit": {
"unitSystemName": "SI"
}
},
]
},
],
"filter": {
"range": {
"property": [
"pumps_space",
"PumpSpecificationsView/1",
"maxPressure"
],
"gte": 3520000.0
}
},
"instanceType": "node",
"includeTyping": True,
"limit": 1000
}
res = client.post(
url=f"/api/v1/projects/{client.config.project}/models/instances/list",
json=payload
)
Query instances with unit conversion and unit-aware filtering
The example below shows a query instance with unit conversion and unit-aware filtering.
payload = {
"select": {
"pumpsHighPressure": {
"sources": [
{
"source": {
"type": "view",
"space": "pumps_space",
"externalId": "PumpSpecifications",
"version": "1"
},
"properties": [
"maxPressure",
"maxTemperature",
"rotationConfigurations"
],
"targetUnits": [
{
"property": "maxPressure",
"unit": {
"externalId": "pressure:pa"
}
},
{
"property": "maxTemperature",
"unit": {
"unitSystemName": "SI"
}
},
{
"property": "rotationConfigurations",
"unit": {
"unitSystemName": "SI"
}
}
]
}
]
}
},
"with": {
"pumpsHighPressure": {
"limit": 1000,
"nodes": {
"filter": {
"range": {
"property": [
"pumps_space",
"PumpSpecificationsContainer",
"maxPressure"
],
"gte": 3520000.0
}
}
}
}
},
"includeTyping": True
}
res = client.post(
url=f"/api/v1/projects/{client.config.project}/models/instances/query",
json=payload
)
Aggregate data across nodes/edges with unit conversion and unit-aware filtering
The example below shows aggregate data across nodes/edges with unit conversion and unit-aware filtering.
payload = {
"aggregates": [
{"avg": {"property": "maxPressure"}},
{"avg": {"property": "maxTemperature"}},
{"count": {"property": "rotationConfigurations"}}
],
"filter": {
"range": {
"property": [
"pumps_space",
"PumpSpecificationsView/1",
"maxPressure"
],
"gte": 3520000.0
}
},
"instanceType": "node",
"view": {
"type": "view",
"space": "pumps_space",
"externalId": "PumpSpecifications",
"version": "1",
},
"targetUnits": [
{
"property": "maxPressure",
"unit": {
"externalId": "pressure:pa"
}
},
{
"property": "maxTemperature",
"unit": {
"unitSystemName": "SI"
}
},
{
"property": "rotationConfigurations",
"unit": {
"unitSystemName": "SI"
}
}
],
"includeTyping": True,
"limit": 100,
}
res = client.post(
url=f"/api/v1/projects/{client.config.project}/models/instances/aggregate", json=payload
)
Query parameter values are not unit-aware and are considered in the same unit as the underlying property. No unit conversion will occur if you provide a filter property through query parameters.
Apart from querying data using the DMS endpoints, use GraphQL to fetch data from containers and time series data points. The GraphQL API is updated to support unit conversion and unit-aware filtering. See the below examples below for some valid queries.
For more information on data modeling endpoints, see the Data models API documentation.
List query with unit conversion and unit-aware filtering
The example below shows a list query with unit conversion and unit-aware filtering.
query MyQuery {
# filter on maxPressure -> only 1 entry has maxPressure >= 3520000.0 Pa
listPumpSpecifications(filter: { maxPressure: { gte: 3520000.0 } }) {
items {
# return maxPressure and convert its values and the filter to Pa
maxPressure(targetUnit: "pressure:pa")
# return maxTemperature and convert its values to the SI unitSystem
maxTemperature(targetUnitSystem: "SI")
# return rotationConfigurations and convert its values to the SI unitSystem
rotationConfigurations(targetUnitSystem: "SI")
}
# the units which are associated with the values returned in the request
unitInfo {
maxPressure {
# externalId of the unit
unitExternalId
# free text unit alias
sourceUnit
}
maxTemperature {
unitExternalId
}
rotationConfigurations {
unitExternalId
}
}
}
}
Aggregate query with unit conversion and filtering
The example below shows an aggregate query with unit conversion and filtering.
query MyQuery {
# filter on maxPressure -> only 1 entry has maxPressure >= 3520000.0 Pa
aggregatePumpSpecifications(filter: {maxPressure: {gte: 3520000.0}}) {
items {
avg {
maxPressure(targetUnit: "pressure:pa")
maxTemperature(targetUnitSystem: "SI")
}
count {
rotationConfigurations
}
}
unitInfo {
avg {
maxPressure {
unitExternalId
sourceUnit
}
maxTemperature {
sourceUnit
unitExternalId
}
rotationConfigurations {
sourceUnit
unitExternalId
}
}
}
}
}
Time series data points query with unit conversion
The example below shows time series data points query with unit conversion.
query MyQuery($startTime: Int64!, $endTime: Int64!) {
# select only one instance based on the externalId
listSensorList( filter: { externalId: { eq: "TI-FORNEBU-02" } } ) {
items {
# externalId of the node
externalId
# property included in the view
name
sensor {
# externalId of the "sensor" time series
externalId
# name of the "sensor" time series
name
# free text unit assigned to the "sensor" time series
unit
# unitExternalId of the "sensor" time series where the data is stored
unitExternalId
getDataPoints(targetUnitSystem: "SI", start: $startTime, end: $endTime, granularity: "1h", aggregates: AVERAGE ) {
# unitExternalId of the datapoints returned in the request
unitExternalId
# datapoint items
items {
timestamp
average
}
}
}
}
}
}
For this query, we are using the following view definition:
type SensorList @view(version: "1") {
name: String
sensor: TimeSeries
}
Note: You need to define the variables startTime and endTime before executing the query.