Device integration using MQTT

Besides REST, Cumulocity supports MQTT as native protocol.

Overview

Note: MQTT support is currently in beta. We encourage device integrators to provide feedback through Stackoverflow or the support web site.

The MQTT implementation of Cumulocity provides the following benefits:

  • Multi-tenancy support: A single endpoint serves multiple tenants.
  • Device identity management: Devices authenticate using device-specific credentials.
  • Device registration: Non-personalized devices can be deployed by pairing them with Cumulocity tenants.
  • Device management: Rich, pre-defined device management payload formats to enable out-of-the-box management of millions of devices.
  • Standard IoT payload formats: Pre-defined payload formats to support IoT sensor readings, alarm management, remote control and device hierarchies.
  • Custom payload formats: Additional payload formats can be added.
  • Minimum traffic overhead.
  • Processing modes: Control whether data is persisted in Cumulocity database, transiently passed to real-time processing, processed using quiescent mode which ensures that real-time notifications are disabled or is processed using CEP mode that ensures data is transiently sent to real-time processing engine only with real-time notifications disabled.
  • Full bi-directional communication.
  • MQTT over WebSockets support.
  • TLS support.
  • Full horizontal scalability.

The MQTT section is structured as follows:

  • Hello MQTT provides an easy introduction to the Cumulocity MQTT protocol using a popular MQTT client.
  • Hello MQTT C provides an introduction on how to use the MQTT client in C with Cumulocity using pre-defined messages (called "static templates").
  • Hello MQTT C# provides an introduction on how to use the C# MQTT client with Cumulocity using pre-defined messages (called "static templates").
  • Hello MQTT Java provides an introduction on how to use the Java MQTT client with Cumulocity using pre-defined messages (called "static templates").
  • Hello MQTT Javascript provides an introduction on how to use the Javascript MQTT client with Cumulocity using pre-defined messages (called "static templates").
  • Hello MQTT Python provides an introduction on how to use the Python MQTT client with Cumulocity using pre-defined messages (called "static templates").
  • MQTT implementation gives a detailed reference of protocol-level aspects in the Cumulocity implementation of MQTT.
  • Device integration walks you through the process of interfacing devices with Cumulocity through MQTT.
  • SmartREST 1.0 defines the SmartREST 1.0 payload format for MQTT for easy portability of existing SmartREST devices.
  • SmartREST 2.0 defines the SmartREST 2.0 payload format for MQTT with improvements in usability and traffic usage.
  • MQTT static templates provides a reference of pre-defined payload formats that you can use straight away.
  • Handling IDs describes how IDs are handled in the Cumulocity MQTT protocol.

This section does not describe the basics of MQTT communication. If you are unfamiliar with MQTT, we recommend consulting one of the numerous introductions in the Internet. Some references can be found on the MQTT web site.

Info: For all MQTT connections to our platform, the maximum accepted payload size is 8092 bytes(8KB).

Hello MQTT

In this tutorial, you will learn how to use MQTT with Cumulocity using pre-defined messages (called "static templates").

Prerequisites

In order to follow this tutorial, check the following prerequisites:

  • You have a valid tenant, a user and a password in order to access Cumulocity.
  • You have installed MQTTBox or a similar MQTT tool.

Info: The screenshots in the tutorial use MQTTBox. Other tools may look slightly different.

Talking MQTT

Configuring the MQTT connection

To configure the MQTT connection, you need to pass the following connection parameters (see the screenshot below).

  • Hostname: For this example, use TCP and Port 1883. You also need to pass the URL to your tenant (e.g. mqtt.cumulocity.com).
  • Client ID: You can use the "Generate a random ID" button (most tools will offer such a button) or fill one in yourself. This ID will be linked to your device. To reconnect to the same device, use the same ID.
  • Username: You need to enter your tenant and username separated by a slash (tenant/username). You can use the same user that you use for logging into Cumulocity for this example.
  • Password: The password of the user.

Example MQTTBox Configuration

The other configurations like "clean session" are not important for this example. You can change them to your needs. After clicking Save changes, you will see a screen similar to the following screenshot.

MQTTBox Established Connection

If there is a blue button on the top bar with a label "Not Connected", verify your configuration (especially username and password). If the button is green, you successfully established an MQTT connection to Cumulocity.

Sending data

All MQTT "publish messages" in this tutorial will be sent to the topic "s/us". This is the topic used for Cumulocity's pre-provided static templates.

MQTTBox Publish Message

Creating the device

The first message sent will create our device. Although the static templates support automatic device creation, in this example we will create the device manually. The template "100" will create a new device. It can be used with two optional parameters (deviceName, deviceType).

100,My first MQTT device,c8y_MQTTdevice

Afterwards, you will find this device in the Device Management application as a new device. If you switch to the "Identity" tab of the device you will notice that there was an identity created automatically to link the device to the MQTT ClientId.

Besides the name and the type, the device is empty so master data needs to be added.

You can use multiple static templates per publishing separated by a line break (one template per row). This feature is used to set the hardware and the required interval for the device in a single published message.

The hardware can be set with the template "110". It can take 3 parameters (serialNumber, model, revision). Optional parameters in static templates can be left empty if you don't want to set them. For the hardware all parameters are optional.

The required interval can be set with the template "117" and just takes a single parameter (the interval in minutes).

110,,MQTT test model,1.2.3
117,10

After a reload of the "Info" page of your device in Device Management, you should see the information we just added.

Creating measurements

Now the device has some master data and we can start sending some measurements. There are a couple of measurements that can be created directly by using a static template:

  • 210: Signal strength measurement
  • 211: Temperature measurement
  • 212: Battery measurement

The temperature and battery measurement just take the value and time as parameters. For the signal strength, you can pass two values (RSSI and BER).

Passing timestamps in the Cumulocity MQTT implementation is always optional. If you don't pass them along, the server will automatically create a timestamp with the current server time.

We will make use of this feature in this example. Also, if you don't set the last parameters you do not need to enter the remaining commas.

210,-87
211,24
212,95

Besides the measurements above we can also use the template "200" to create a more custom measurement. It will take the measurement fragment, series, value, unit and time as its parameters.

200,myCustomTemperatureMeasurement,fahrenheit,75.2,F

After a reload in the Device Management application you should see 4 graphs with the newly added measurements in the "Measurements" tab of your device.

Creating alarms

In the next step, we want to create some alarms for this device. There are 4 templates to create alarms for the 4 alarm severities:

  • 301: CRITICAL
  • 302: MAJOR
  • 303: MINOR
  • 304: WARNING

Each of them note a type (which is mandatory), a text and a time (both optional).

301,gpio_critical,There is a GPIO alarm
304,simple_warning

The alarm list of your device should now contain one critical alarm and one warning. Note that we did not set any text for the warning therefore it was created with a default alarm text.

With the next step, we want to clear the critical alarm again. To achieve this we use the template "306" which refers to the type of the alarm that should be cleared.

306,gpio_critical

Afterward, the critical alarm should be cleared. Note that you did not have to handle any alarm IDs with the MQTT implementation. Cumulocity will take over this part so that the device communication can be as easy as possible.

Creating events

Next, we want to create some location events for the device. You can use the link http://www.latlong.net/ to get the latitude and longitude for your city if you want.

The template "401" lets you create location events and takes latitude, longitude, altitude, accuracy and the time as parameters, but for now we will just use the first two.

401,51.227741,6.773456

In the Device Management application, you can see one event in the event list but the location has not been updated. This is because on REST these are different requests. Instead of the template "401" you can use the template "402" in MQTT. It works exactly the same as "401" but additionally it also updates the position of the device itself.

402,51.227741,6.773456

Now you should see both the "Location" and the "Tracking" tab in the device with the "Location" tab having the same latitude longitude as the last location event.

Receiving data

So far we have only used MQTT to send data from the client to the server. Now we want to send data from the server to the client.

To achieve that, we first need to subscribe to the responsible topic. We will do two subscriptions:

  • s/ds : This will subscribe to the static operation templates for the device
  • s/e : This will subscribe to an error topic that can be used for debugging

You can enter both topics after another in the "Subscribe" field and click Subscribe. The QoS selection does not matter for this example.

Afterward, your MQTTBox should look like this:

MQTTBox Subscribed Topics

Receiving operations

At the current state the UI does not show any tabs for operations. Up to this point it was unknown what exactly the device supports. But the list of supported operations can be modified with the template "114". A list of supported operations can be added here.

We will add support for the configuration and shell.

114,c8y_Command,c8y_Configuration

After reloading the UI, the two new tabs will appear ("Configuration" and "Shell").

We can now create a shell command from the UI and click Execute.

In the MQTTBox you should now have received a new message for the s/ds subscription.

MQTTBox Received Operation

The "511" is indicating what kind of operation we received (in this case c8y_Command). This will be followed by the deviceIdentifier to locate the device with the dedicated operation. This is required if you have a hierarchy with multiple children. In that case, you need to know for which of the children the operation was dedicated. Finally, you have the operation specific parameters which in the case of c8y_Command is only the command text.

After receiving the operation we can start EXECUTING it to initiate the client's handling the operation. Similar to changing the status of an alarm you can add the type of operation to the template.

501,c8y_Command

After finishing the handling the operation can be set to successful with the template "503".

Besides the operation type, this operation can also take additional parameters based on what kind of operation it was. For the c8y_Command we can return a result.

503,c8y_Command,Everything went fine
Learning from errors

The topic s/e can help you debugging in case something went wrong.

If we for example try to send

999,I made this up

we can see a message on the topic because the template "999" is unknown.

40,999,No static template for this message id

Hello MQTT C

In this tutorial, you will learn how to use MQTT client in C with Cumulocity using pre-defined messages (called "static templates").

Prerequisites

In order to follow this tutorial, check the following prerequisites:

  • You have a valid tenant, a user, and a password in order to access Cumulocity.
  • Verify that you have a gcc compiler installed:

    gcc --version
    gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
    Copyright (C) 2015 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    
  • Download, compile and install the MQTT C Paho Client. You will find more details about Paho on the Paho website.

Developing the "Hello, MQTT world!" client

To develop a very simple "Hello, world!" MQTT client for Cumulocity, you need to

  • create the application,
  • build and run the application.

Creating the application

Create a source file, for example "hello_mqtt.c" with the following content:

#include "stdlib.h"
#include "string.h"
#include "unistd.h"
#include "MQTTClient.h"

#define ADDRESS     "<<serverUrl>>"
#define CLIENTID    "<<clientId>>"

void publish(MQTTClient client, char* topic, char* payload) {
    MQTTClient_message pubmsg = MQTTClient_message_initializer;
    pubmsg.payload = payload;
    pubmsg.payloadlen = strlen(pubmsg.payload);
    pubmsg.qos = 2;
    pubmsg.retained = 0;
    MQTTClient_deliveryToken token;
    MQTTClient_publishMessage(client, topic, &pubmsg, &token);
    MQTTClient_waitForCompletion(client, token, 1000L);
    printf("Message '%s' with delivery token %d delivered\n", payload, token);
}

int on_message(void *context, char *topicName, int topicLen, MQTTClient_message *message) {
    char* payload = message->payload;
    printf("Received operation %s\n", payload);
    MQTTClient_freeMessage(&message);
    MQTTClient_free(topicName);
    return 1;
}

int main(int argc, char* argv[]) {
    MQTTClient client;
    MQTTClient_create(&client, ADDRESS, CLIENTID, MQTTCLIENT_PERSISTENCE_NONE, NULL);
    MQTTClient_connectOptions conn_opts = MQTTClient_connectOptions_initializer;
    conn_opts.username = "<<tenant>>/<<username>>";
    conn_opts.password = "<<password>>";

    MQTTClient_setCallbacks(client, NULL, NULL, on_message, NULL);

    int rc;
    if ((rc = MQTTClient_connect(client, &conn_opts)) != MQTTCLIENT_SUCCESS) {
        printf("Failed to connect, return code %d\n", rc);
        exit(-1);
    }
    //create device
    publish(client, "s/us", "100,C MQTT,c8y_MQTTDevice");
    //set hardware information
    publish(client, "s/us", "110,S123456789,MQTT test model,Rev0.1");
    //listen for operation
    MQTTClient_subscribe(client, "s/ds", 0);

    for (;;) {
        //send temperature measurement
        publish(client, "s/us", "211,25");
        sleep(3);
    }
    MQTTClient_disconnect(client, 1000);
    MQTTClient_destroy(&client);
    return rc;
}

Replace "<<clientId>>", "<<serverUrl>>", "<<tenant>>", "<<username>>", "<<password>>" with your data.

The Cumulocity MQTT protocol supports both unsecured TCP and secured SSL connections (e.g. tcp://mqtt.cumulocity.com:1883 or ssl://mqtt.cumulocity.com:8883), so as the "<<serverUrl>>" you can pick the one which fits for you. When using SSL remember to configure MQTTClient_SSLOptions and set it in the MQTTClient_connectOptions.

What does the code in "main" do?

  • Configure an MQTT connection.
  • Register a on_message callback function which will print incoming messages.
  • Connect with Cumulocity via an MQTT protocol.
  • Create a new device with aC MQTT name and ac8y_MQTTDevice type.
  • Update the device hardware information by putting a S123456789 serial, a MQTT test model model and a Rev0.1 revision.
  • Subscribe to the static operation templates for the device - this will result in a on_message method call every time a new operation is created.
  • Send temperature measurement every 3 seconds.

What does the code in "publish" do?

  • Create a new MQTT message and set a payload.
  • Publish message via MQTT protocol.
  • Wait maximum 1 second for a message delivered ACK from the server.

Note that the subscription is established after the device creation, otherwise if there is no device for a given clientId the server will not accept it.

Building and running the application

To build:

gcc hello_mqtt.c -o hello_mqtt -lpaho-mqtt3c

To run:

./hello_mqtt
Message '100,C MQTT,c8y_MQTTDevice' with delivery token 1 delivered
...

After starting the application you should see a new device in the Device Management application in the device list. Additionally, if there will be a new operation created for this device, (for example c8y_Restart) information about it will be printed to the console.

Improving the agent

Now that you have done your first step, check out the section Hello MQTT to learn more about Cumulocity MQTT and improve your application.

Hello MQTT Java

In this tutorial, you will learn how to use the Java MQTT client with Cumulocity using pre-defined messages (called "static templates").

Prerequisites

In order to follow this tutorial, check the following prerequisites:

  • You have a valid tenant, a user, and a password in order to access Cumulocity.
  • Verify that you have Maven 3 installed with Java 7:

    mvn -v
    Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T17:41:47+01:00)
    Maven home: /home/schm/development/devtools/apache-maven-3.3.9
    Java version: 1.7.0_80, vendor: Oracle Corporation
    Java home: /usr/lib/jvm/java-7-oracle/jre
    Default locale: pl_PL, platform encoding: UTF-8
    OS name: "linux", version: "4.4.0-66-generic", arch: "amd64", family: "unix"
    

Maven can be downloaded from the Maven website.

Developing the "Hello, MQTT world!" client

To develop a very simple "Hello, world!" MQTT client for Cumulocity, you need to

  • create a Maven project,
  • add a dependency to the MQTT Java client library to the Maven pom.xml (in this example we will use Paho Java Client),
  • create a Java application,
  • build and run the Java application.

Creating a Maven project

To create a plain Java project with Maven, run

mvn archetype:generate -DgroupId=c8y.example -DartifactId=hello-mqtt-java -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

This will create a folder "hello-mqtt-java" in the current directory with a skeleton structure for your project.

Adding the MQTT Java client library

Edit the "pom.xml" in the "hello-mqtt-java" folder. Add a dependency to the MQTT Paho Java Client.

<dependency>
    <groupId>org.eclipse.paho</groupId>
    <artifactId>org.eclipse.paho.client.mqttv3</artifactId>
    <version>1.1.0</version>
</dependency>

Creating a Java application

Edit the "App.java" file in the folder "hello-mqtt-java/src/main/java/c8y/example" with the following content:

package c8y.example;

import org.eclipse.paho.client.mqttv3.*;
import java.util.concurrent.*;

public class App {

    public static void main(String[] args) throws Exception {
        final String clientId = "<<clientId>>";
        final String serverURI = "<<serverUrl>>";

        //configure MQTT connection
        final MqttConnectOptions options = new MqttConnectOptions();
        options.setUserName("<<tenant>>/<<username>>");
        options.setPassword("<<password>>".toCharArray());
        final MqttClient client = new MqttClient(serverURI, clientId, null);

        //connect to the Cumulocity
        client.connect(options);

        //create device
        client.publish("s/us", "100,My MQTT device,c8y_MQTTDevice".getBytes(), 2, false);

        //set hardware information
        client.publish("s/us", "110,S123456789,MQTT test model,Rev0.1".getBytes(), 2, false);

        //listen for operation
        client.subscribe("s/ds", new IMqttMessageListener() {
            public void messageArrived(final String topic, final MqttMessage message) throws Exception {
                final String payload = new String(message.getPayload());
                System.out.println("Received operation " + payload);
                if (payload.startsWith("510")) {
                    System.out.println("Simulating device restart...");
                    client.publish("s/us", "501,c8y_Restart".getBytes(), 2, false);
                    System.out.println("...restarting...");
                    Thread.sleep(TimeUnit.SECONDS.toMillis(1));
                    client.publish("s/us", "503,c8y_Restart".getBytes(), 2, false);
                    System.out.println("...done...");
                }
            }
        });

        Executors.newSingleThreadScheduledExecutor().scheduleWithFixedDelay(new Runnable() {
            public void run() {
                try {
                    //send temperature measurement
                    client.publish("s/us", new MqttMessage("211,25".getBytes()));
                } catch (MqttException e) {
                    e.printStackTrace();
                }
            }
        }, 1, 3, TimeUnit.SECONDS);
    }
}

Replace "<<clientId>>", "<<serverUrl>>", "<<tenant>>", "<<username>>", and "<<password>>" with your data.

Cumulocity MQTT protocol supports both unsecured TCP and secured SSL connections (e.g. tcp://mqtt.cumulocity.com:1883 or ssl://mqtt.cumulocity.com:8883), so as the "<<serverUrl>>" you can pick the one which fits for you.

What does the code in "main" do?

  • Configure the MQTT connection.
  • Connect with Cumulocity via a MQTT protocol.
  • Create a new device with a My MQTT device name and a c8y_MQTTDevice type.
  • Update the device hardware information by putting a S123456789 serial, a MQTT test model model and a Rev0.1 revision.
  • Subscribe to the static operation templates for the device, print all received operations to the console and in case of a c8y_Restart operation simulate device restart.
  • Create a new thread which sends temperature measurement every 3 seconds.

Note that the subscription is established after the device creation, otherwise if there is no device for a given clientId the server will not accept it.

Building and running the application

To build:

cd hello-mqtt-java
mvn clean install
...
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hello-mqtt-java ---
[INFO] Building jar: /home/schm/Pulpit/hello-mqtt-java/target/hello-mqtt-java-1.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hello-mqtt-java ---
[INFO] Installing /home/schm/Pulpit/hello-mqtt-java/target/hello-mqtt-java-1.0-SNAPSHOT.jar to /home/schm/.m2/repository/c8y/example/hello-mqtt-java/1.0-SNAPSHOT/hello-mqtt-java-1.0-SNAPSHOT.jar
[INFO] Installing /home/schm/Pulpit/hello-mqtt-java/pom.xml to /home/schm/.m2/repository/c8y/example/hello-mqtt-java/1.0-SNAPSHOT/hello-mqtt-java-1.0-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.642 s
[INFO] Finished at: 2017-03-14T09:16:25+01:00
[INFO] Final Memory: 14M/301M
[INFO] ------------------------------------------------------------------------

To run:

mvn exec:java -Dexec.mainClass="c8y.example.App"
...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building hello-mqtt-java 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ hello-mqtt-java ---
Received operation 510,123456789

After starting the application you should see a new device in the Device Management application in the device list. Additionally if there will be a new operation created for this device, (for example c8y_Restart) information about it will be printed to the console.

Improving the agent

Now that you have done your first step, check out the Section Hello MQTT to learn more about Cumulocity MQTT and improve your application.

Hello MQTT JavaScript

In this tutorial, you will learn how to use the JavaScript MQTT client with Cumulocity using pre-defined messages (called "static templates").

Prerequisites

In order to follow this tutorial, check the following prerequisites:

  • You have a valid tenant, a user, and a password in order to access Cumulocity.

Developing the "Hello, MQTT world!" client

To develop a very simple "Hello, world!" MQTT client for Cumulocity, you need to

  • create a JavaScript application,
  • run the application.

Creating a JavaScript application

Create an HTML file, for example "hello_mqtt_js.html" with the following content:

<html>
<head>
    <meta http-equiv="Content-Type" content="text/html;charset=utf-8"></meta>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js" type="text/javascript"></script>
    <script type="text/javascript">
        var undeliveredMessages = []           
        var temperature = 25

        var client = new Paho.MQTT.Client("<<serverUrl>>", "<<clientId>>");
        client.onMessageArrived = onMessageArrived;
        client.onMessageDelivered = onMessageDelivered;

        function onMessageArrived(message) {
            log('Received operation "' + message.payloadString + '"');
            if (message.payloadString.indexOf("510") == 0) {
                log("Simulating device restart...");
                publish("s/us", "501,c8y_Restart");
                log("...restarting...");
                setTimeout(function() {
                    publish("s/us", "503,c8y_Restart");
                }, 1000);
                log("...done...");
            }
        }

        function onMessageDelivered(message) {
            log('Message "' + message.payloadString + '" delivered');
            var undeliveredMessage = undeliveredMessages.pop();
            if (undeliveredMessage.onMessageDeliveredCallback) {
                undeliveredMessage.onMessageDeliveredCallback();
            }
        }

        function createDevice() {
            //create device
            publish("s/us", "100,JS MQTT,c8y_MQTTDevice", function() {
                //set hardware information
                publish("s/us", "110,S123456789,MQTT test model,Rev0.1", function() {
                    publish('s/us', '114,c8y_Restart', function() { 
                        log('Enable restart operation support') 
                        //listen for operation
                        client.subscribe("s/ds");
                    })

                    //send temperature measurement
                    setInterval(function() {
                        publish("s/us", '211,'+temperature);
                        temperature += 0.5 - Math.random()
                    }, 3000);
                });
            });
        }

        function publish(topic, message, onMessageDeliveredCallback) {
            message = new Paho.MQTT.Message(message);
            message.destinationName = topic;
            message.qos = 2;
            undeliveredMessages.push({
                message: message,
                onMessageDeliveredCallback: onMessageDeliveredCallback
            });
            client.send(message);
        }

        function init() {
            client.connect({
                userName: "<<tenant>>/<<username>>",
                password: "<<password>>",
                onSuccess: createDevice
            });
        }

        function log(message) {
            document.getElementById('logger').insertAdjacentHTML('beforeend', '<div>' + message + '</div>');
        }
    </script>
</head>
<body onload="init();"><div id="logger"></div></body>
</html>

Replace <<clientId>>, <<serverUrl>>, <<tenant>>, <<username>>, and <<password>> with your valid data.

The Cumulocity MQTT protocol supports both unsecured TCP and also secured SSL connections (e.g. ws://mqtt.cumulocity.com/mqtt or wss://mqtt.cumulocity.com/mqtt), so as <<serverUrl>> select the one which fits for you.

What does the code do?

  • Configure the MQTT connection.
  • Register onMessageArrived a callback function which will print all incoming messages and in case of c8y_Restart operation it will simulate a device restart.
  • Register onMessageDelivered a callback function which will be called after a publish message has been delivered.
  • After the page is fully loaded, call init function which connects with Cumulocity via a MQTT protocol.
  • When the connection is established call a createDevice function.
  • Create a new device with a JS MQTT name and a c8y_MQTTDevice type.
  • Update the device hardware information by putting a S123456789 serial, a MQTT test model model and a Rev0.1 revision.
  • Subscribe to the static operation templates for the device - this will result in onMessageArrived method call every time a new operation is created.
  • Send temperature measurement every 3 seconds.

Note that subscription is established after the device creation, otherwise if there is no device for a given clientId the server will not accept it.

Running the application

Open "hello_mqtt_js.html" in a browser. You should see the new device in the Cumulocity application in the device list.

Additionally if there is a new operation created for this device (for example c8y_Restart), related information will be printed in the browser page.

Improving the agent

Now that you have done your first step, check out the section Hello MQTT to learn more about Cumulocity MQTT and improve your application.

Hello MQTT Python

In this tutorial, you will learn how to use the Python MQTT client with Cumulocity using pre-defined messages (called "static templates").

Prerequisites

In order to follow this tutorial, check the following prerequisites:

  • You have a valid tenant, a user, and a password in order to access Cumulocity.
  • Verify that you have Python installed:

    python --version
    Python 2.7.12
    

    Python can be downloaded from www.python.org.

  • Install the Paho client:

        pip install paho-mqtt
    

Developing the "Hello, MQTT world!" client

To develop a very simple "Hello, world!" MQTT client for Cumulocity, you need to

  • create a python script,
  • Run the script.

Creating a python script

Create a script file, for example "hello_mqtt.py" with the following content:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import paho.mqtt.client as mqtt
import time, threading, ssl

receivedMessages = []

def on_message(client, userdata, message):
  print("Received operation " + str(message.payload))
  if (message.payload.startswith("510")):
     print("Simulating device restart...")
     publish("s/us", "501,c8y_Restart");
     print("...restarting...")
     time.sleep(1)
     publish("s/us", "503,c8y_Restart");
     print("...done...")

def sendMeasurements():
  try:
    print("Sending temperature measurement")
    publish("s/us", "211,25");
    thread = threading.Timer(3, sendMeasurements)
    thread.daemon=True
    thread.start()
    while True: time.sleep(100)
  except (KeyboardInterrupt, SystemExit):
    print 'Received keyboard interrupt, quitting ...'


def publish(topic, message, waitForAck = False):
  mid = client.publish(topic, message, 2)[1]
  if (waitForAck):
    while mid not in receivedMessages:
      time.sleep(0.25)

def on_publish(client, userdata, mid):
  receivedMessages.append(mid)

client = mqtt.Client(client_id="<<clientId>>")
client.username_pw_set("<<tenant>>/<<username>>", "<<password>>")
client.on_message = on_message
client.on_publish = on_publish

client.connect("<<serverHost>>", 1883)
client.loop_start()
publish("s/us", "100,Python MQTT,c8y_MQTTDevice", True)
publish("s/us", "110,S123456789,MQTT test model,Rev0.1")
client.subscribe("s/ds")
sendMeasurements()

Replace "<<clientId>>", "<<serverHost>>", "<<tenant>>", "<<username>>", "<<password>>" with your data.

Cumulocity MQTT protocol supports both unsecured TCP and secured SSL connections, so when configuring a port remember to use the correct one. No matter which connection type you choose your "<<serverHost>>" should stay the same (like mqtt.cumulocity.com).

The above example uses a TCP connection. If you would like to use an SSL connection, remember the proper configuration in the Paho MQTT client. Further information can be found in www.eclipse.org.

What does the script do?

  • Configure a MQTT connection.
  • Register an on_message callback function which will print incoming messages and in case of a c8y_Restart operation will simulate a device restart.
  • Register an on_publish callback function which will be called after a publish message has been delivered.
  • Connect with Cumulocity via the MQTT protocol.
  • Create a new device with a Python MQTT name and a c8y_MQTTDevice type.
  • Update the device hardware information by putting a S123456789 serial, a MQTT test model model and a Rev0.1 revision.
  • Subscribe to the static operation templates for the device - this will result in an on_message method call every time a new operation is created.
  • Call the sendMeasurements method which sends a temperature measurement every 3 seconds.

What does the publish message do?

  • Publish a given message about the given topic via MQTT.
  • When publishing the message it uses QoS 2. So to be sure that the message was delivered it will wait for server ACK (until the on_publish method is called with the matching message ID).

Note that the subscription is established after the device creation, otherwise if there is no device for a given clientId the server will not accept it.

Running the script

To run the script just call:

python hello_mqtt.py 

After starting the application you should see a new device in the Device Management application in the device list. Additionally if there will be a new operation created for this device, (for example c8y_Restart) information about it will be printed to the console.

In the console you should see following output:

Sending temperature measurement
...

Improving the agent

Now that you have done your first step, check out the section Hello MQTT to learn more about Cumulocity MQTT and improve your application.

MQTT implementation

This section will list the implementation details for the MQTT protocol. The Cumulocity implementation supports MQTT Version 3.1.1.

Connecting via MQTT

Cumulocity supports MQTT both via TCP and WebSockets. As URL you use "mqtt.cumulocity.com".

Available ports:

TCP WebSockets
SSL 8883 443
no SSL 1883 80


Info: To use WebSockets you need to connect to the path "/mqtt" and follow the MQTT standard for WebSocket communication.

SmartREST Payload

The Cumulocity MQTT implementation uses SmartREST as a payload. SmartREST is a CSV-like message protocol that uses templates on the server side to create data in Cumulocity.

Info: For all MQTT connections to our platform the maximum accepted payload size is 8092 bytes.

SmartREST basics

A SmartREST message is a single row in which each parameter is separated by comma. The first parameter is an ID that defines the message. You can send multiple messages in a single publish by using a line break between messages.

SmartREST escaping

The following characters within a parameter need to be enclosed in double quotes:

  • Comma (,)
  • Line break (\n)
  • Carriage return (\r)
  • Double quotes (")

Additionally, each double quote within the parameter needs to be escaped with a backslash ().

The same escaping rules apply to messages that will be send from the server to the client.

Publish example:

100,"This value, needs escaping",This value does not need escaping

Subscribe example:

511,myDeviceSerial,"execute this\nand this\nand \"this\""

Device hierarchies

MQTT sessions are linked to a single device but this device can have a freely configurable device hierarchy below it.

All children require a unique ID defined when creating the device. We recommend using a combination of the unique ID of the root device and an unique ID within the hierarchy.

To create data for a child instead of the root device the unique ID of the child is added as another section in the topic (e.g. "s/us/myChildDeviceIdentifier").

The client will automatically receive operations for every child in the hierarchy by subscribing to the respective topic. It is not required to subscribe for each child.

Every operation received will contain the template ID followed by the ID of the device/child for which the operation was created (followed by other operation parameters).

MQTT features

MQTT authentication

MQTT supports setting a username and a password. To connect to Cumulocity the MQTT username needs to include both tenant and username in the format "tenant/username".

MQTT ClientId

The MQTT ClientId is a field to uniquely identify each connected client. The Cumulocity implementation also uses the ClientId to link the client directly to a device. Therefore the following format should be used for the ClientId:

"connectionType:deviceIdentifier:defaultTemplateIdentifier"

Field Mandatory Description
connectionType NO Indication of connection type default: d (device)
deviceIdentifier YES A unique identifier for your device e.g. IMEI, Serial number, ...
defaultTemplateIdentifier NO Check the SmartREST section for more information about template identifiers

For the simplest version of a client the MQTT clientId can just be the deviceIdentfier. It will automatically be interpreted as device connection.

Example ClientIds:

mySerialNumber
d:mySerialNumber
d:mySerialNumber:myDefaultTemplate

The uniqueness of the MQTT ClientId is determined only by the deviceIdentifier. Therefore from the above examples only one client can be connected at the same time.

MQTT Quality of Service

The Cumulocity implementation supports all 3 levels of MQTT QoS

  • QoS 0: At most once
  • QoS 1: At least once
  • QoS 2: Exactly once

For subscriptions to the operation or error topics, we will deliver all messages in the QoS which the client defined when subscribing to the topic.

MQTT clean session

MQTT clients can set the clean session flag to "0" (false). This will ensure that in case the client disconnects your subscription will still work and when you reconnect the client will receive the missed messages.

Info: Cumulocity requires clean session to be set to "1" (true). Currently we cannot guarantee that disabling clean session will work reliably, hence we recommend to always enable clean session.

MQTT retained flag

In the current Cumulocity implementation, subscriptions to topics where devices publish data are not allowed. Publishing data with the retained flag on this topic is allowed but has no practical difference to sending it without the flag. Messages published by Cumulocity like operations and errors do not contain the retained flag.

MQTT last will

In MQTT, the "last will" is a message that is specified at connection time and that is executed when the client loses the connection. For example, using

400,c8y_ConnectionEvent,"Device connection was lost."

as last will message and "s/us" as last will topic raises an event whenever the device loses the connection.

The execution of the "last will" updates the device availability.

Debugging

To support developers during development you can subscribe to the topic s/e. On this topic the device can retrieve debug and error messages that occur during a publish from the device.

Info: This topic is purely designed to support the development of clients. It is not recommended to always subscribe to this channel as the messages are verbose and can significantly increase the data usage. Also you should not use this topic to trigger actions of the device based on what you receive on the topic. It is not a response channel.

Device integration

Overview

The basic life cycle for integrating devices into Cumulocity is discussed in Interfacing devices.

In this section, we will show how this life cycle can be managed using the MQTT implementation.

The life cycle consists of two phases, a startup phase and a cycle phase.

The startup phase can be as short as just checking the credentials:

  • Step 0: Request device credentials, if they have not been requested yet.
  • Step 1: Ensure that the device exists.
  • Step 2: Ensure that the device children exist.
  • Step 3: Subscribe to the topics.

The cycle phase consists of two kinds of actions:

MQTT phases

Startup phase

Step 0: Request device credentials

In Cumulocity, every MQTT connection needs to be authenticated. You can use the device credentials topics in the MQTT implementation to generate new credentials for a device.

Once the device retrieved the credentials it needs to store them locally for further connections.

To establish connection you need to configure the following connection parameters:

  • Host: <your_cumulocity_url>
  • User: /
  • Password: Your cumulocity password

For more information, refer to the Hello MQTT section.

The process works as follows:

  • Cumulocity assumes each device to have some form of unique ID. A good device identifier may be the MAC address of the network adapter, the IMEI of a mobile device or a hardware serial number.
  • When you take a new device into use, you enter this unique ID into "Device registration" in the Device Management application in Cumulocity and start the device.
  • The device will use this ID as part of the MQTT ClientId and static user credentials that can be enquired from support@cumulocity.com.
  • The device subscribes to the topic s/dcr.
  • The device starts publishing continuous empty messages on the topic s/ucr to notify the server that it is ready to retrieve credentials.
  • Next, you should accept the connection from the device in the Device Registration page.
  • When the device sends the next empty message it should receive credentials in the following format:
    70,<tenant>,<username>,<password>
    

After receiving the credentials the device can close the MQTT connection and create a new one with the received credentials.

Step 1: Verify device

As MQTT supports an automatic device creation if the client sends data and there is no device present, this step is only required if you want to create the device manually.

The device creation can be achieved by the static template 100. This template can be blindly used on every boot of the device as it will only create the device if it is not already present.

The device will automatically be linked to the ID the client uses with its MQTT ClientId.

Info:The topic used for Cumulocity's pre-provided static templates is "s/us".

100,Device Name,Device Type

Step 2: Verify children

Like the root device also children of it are covered by the automatic device creation.

For handling this step manually you can send the static template 101 for creating a child device. The template will only create the child if it does not already exist.

101,Unique Child ID,Child Name,Child Type

Step 3: Subscribe topics

If the device supports operations it should subscribe to all required topics (static templates and SmartREST 2.0).

Cycle phase

Step A: Send CSV data

While the device holds an active MQTT connection it can publish either on the topics for static templates or on the topics for a SmartREST template to send data to the server.

Based on the MQTT ClientId the physical device is directly connected to the device object in Cumulocity. Therefore any data you send is automatically connected to the device.

To send data to a child device, publish the data to the topics described in Device hierarchies.

Step B: Receive CSV operations

By subscribing to a topic the device automatically tells Cumulocity that it wants to receive operations. Any operation created will be automatically parsed using either the static templates or the templates the device defined.

SmartREST 1.0

This section describes how you can use your existing SmartREST 1.0 templates with MQTT.

Note, that SmartREST 1.0 was designed for HTTP request/response and also does not support the ID-less communication with MQTT. It only uses the MQTT connection to send exactly the same request as you would send using HTTP and therefore comes with some limitations as MQTT is not request/response.

The support for SmartREST 1.0 was added to ease transition if you have an existing implementation using it.

If you start a new device integration we highly recommend to use SmartREST 2.0.

For general information on SmartREST 1.0, refer to Using the REST interface > Using SmartREST in the Microservice SDK guide.

MQTT ClientId

Although you need to send the IDs in the body of each message with SmartREST 1.0, it is still important to connect with the correct MQTT ClientId.

The MQTT ClientId needs to match the externalId with type "c8y_Serial" of your device. It is used to assign the correct operations and responses.

Sending SmartREST 1.0

To send data to the server you can publish the same content as you would POST to the SmartREST endpoint /s.

The X-Id header is part of the topic the client needs to publish on.

Topic:

s/ul/<X-ID>;

Processing mode

Since Cumulocity SmartREST protocol supports TRANSIENT processing mode for avoiding storage of sent data in the database, publishing on MQTT t/ topic instead of s/ topic will only pass the data to real-time processing.

Topic:

t/ul/<X-ID>;

Cumulocity SmartREST protocol also supports QUIESCENT processing mode for avoiding real-time notifications by publishing on MQTT q/ topic instead of s/ topic. Currently, the QUIESCENT processing mode is applicable for measurements and events only.

Topic:

q/ul/<X-ID>;

Cumulocity SmartREST protocol also supports CEP processing mode for ensuring data is only sent to the real-time event processing engine with real-time notifications disabled by publishing on MQTT c/ topic instead of s/ topic. Currently, the CEP processing mode is applicable for measurements and events only.

Topic:

c/ul/<X-ID>;

Receiving SmartREST 1.0

If a template triggers a response template the returning message will be published by the server on the following topic.

Topic:

s/dl/<X-ID>;

This topic can be subscribed by the client.

Receiving operations

SmartREST 1.0 via HTTP offers the /devicecontrol/notifications endpoint to listen to realtime operations. You can receive the same content on the following MQTT topic.

Topic:

s/ol/<X-ID>;

Limitations

MQTT currently does not support request/response. Therefore if you send a request on the publish topic and receive a response on the subscribe topic the client cannot securely match that they belong together.

You can counter this limitation by designing the templates in a way that you never need to know what request triggered the response and the client automatically knows it by the messageId.

SmartREST 2.0

Overview

This section describes the SmartREST 2.0 payload format that can be used with the Cumulocity MQTT implementation.

SmartREST 2.0 was designed to make use of the MQTT protocol and therefore can reduce the payload even more than the SmartREST 1.0 via HTTP.

SmartREST 2.0 is only available via MQTT.

SmartREST 2.0 offers the following MQTT topics for the main communication:

To publish messages:

s/uc/<X-ID>

To publish messages in TRANSIENT mode:

t/uc/<X-ID>

To publish messages in QUIESCENT mode:

+q/uc/<X-ID>

To publish messages in CEP mode:

c/uc/<X-ID>

Refer to SmartREST > Processing mode in the Reference guide for more information about transient, quiescent & CEP data processing.

To subscribe for responses:

s/dc/<X-ID>

The topics for creating templates are described in Creating templates via MQTT.

Changes from SmartREST 1.0

In its base SmartREST 2.0 is like the previous version a CSV-like payload format that is backed by previously created templates to finally create the targeted JSON structure.

Several changes in the functionality have been made:

  • Templates no longer contain IDs of objects (instead, IDs will be resolved by e.g. MQTT ClientId)
  • Managed objects can be created and retrieved directly with external IDs
  • Creating request templates now uses JSON path (like response templates)
  • Support for lists in responses
  • Responses also return if only part of the patterns were found
  • Declaring a default X-Id for the connection

Supported templates

SmartREST 2.0 lets you create templates for the following matching HTTP methods:

API GET POST PUT
Inventory x x x
Alarm x x
Event x
Measurement x
Operation x

Additionally, you can create templates to return certain values from responses and operations.

Template collections

A template collection is a set of request and response templates that specify a device communication protocol. Each collection is referenced by a unique ID (called X-Id).

Creating templates via MQTT

Like in SmartREST 1.0 you need to pass all templates in a collection in one message. After the creation of a template collection it can no longer be modified through MQTT.

When creating templates the client needs to publish to the following topic:

s/ut/<X-ID>

To verify if a template collection exists the client can subscribe to the topic:

s/dt

When subscribed the client can send an empty message to the creation topic which will trigger a new message about the creation status of this X-ID.

Example

Empty publish to s/ut/myExistingTemplateCollection

20,myExistingTemplateCollection,<ID of collection>

Empty publish to s/ut/myNotExistingTemplateCollection

41,myNotExistingTemplateCollection

Request templates

A request template contains the following basic fields:

Field Data type Possible values Mandatory Description
messageId String YES Unique ID to reference the template within the collection
method String GET
PUT
POST
YES Whether to get, update or create data
api String INVENTORY
MEASUREMENT
ALARM
EVENT
OPERATION
YES Cumulocity API to be used
response Boolean true
false
NO Whether the request should trigger response templates. For GET templates by default true otherwise by default false
mandatoryValues List<String> YES Values for the mandatory fields on the API. The values depend on the API and method the template uses
customValues List<CustomValue> NO Custom values that should be added to the object

A request template lists all the fragments in the object structure (mandatory and custom) that should be added when creating or updating the data. It can set fixed values in the template that will then be replaced by the server. If it does not set the value in the template the value needs to be included in the publish message (this includes mandatoryValues).

Example

We create a template to create a measurement like this (measurements have two mandatory values - type and time):

// 10,msgId,api,method,response,type,time,custom1.path,custom1.type,custom1.value
10,999,POST,MEASUREMENT,,c8y_MyMeasurment,,c8y_MyMeasurement.M.value,NUMBER,

This template declares one additional custom property for the measurement. It leaves two fields empty in the template declaration (time and the custom property) so to use the template the client needs to send these two values

999,2016-06-22T17:03:14.000+02:00,25
// We can also use server time by leaving the time empty
999,,25

The following sections will get into more detail of how to create and use different templates.

GET templates

The GET templates for the inventory do not need any mandatory or custom values. Instead they use two different fields.

With SmartREST 2.0 you have the option to either get an object from inventory by its ID or by an externalId directly. Therefore instead of the fields mandatoryValues and customValues the following two fields are used:

Field Data type Possible values Mandatory Description
byId boolean true
false
YES Whether the GET should be executed by Cumulocity ID (=true) or externalId (=false)
externalIdType String NO Sets a fixed externalIdType if the template calls by externalId

This enables you to query inventory in 3 different ways:

By Cumulocity ID

// Creation:
10,999,GET,INVENTORY,,true
// Usage:
999,123456

By external ID with a fixed type in the template

// Creation:
10,999,GET,INVENTORY,,false,c8y_Serial
// Usage:
999,myDeviceImei

By external ID without fixed type in the template

// Creation:
10,999,GET,INVENTORY,,false
// Usage:
999,c8y_Serial,myDeviceImei
POST templates

The POST templates require a different set of mandatory values based on the API:

API mandatory values
MEASUREMENT type, time
EVENT type, text, time
ALARM type, text, status, severity, time
INVENTORY externalIdType

This results in the following minimal template creations:

// Creation:
10,100,POST,MEASUREMENT,false,c8y_CustomMeasurement,,
10,101,POST,EVENT,,c8y_CustomEvent,mytext,,
10,102,POST,ALARM,,c8y_CustomAlarm,mytext,ACTIVE,MAJOR,
// Usage:
100
101
102

Creating data on the inventory optionally includes the creation of an externalId for that object. This is controlled by the mandatory value externalIdType.

Important: All POST Inventory templates start with the value of the externalId after the msgId. Leaving this column empty will result in not creating an external ID.

// Creation:
10,100,POST,INVENTORY,,c8y_MySerial
10,101,POST,INVENTORY,,
// Usage:
// Create object with externalId c8y_MySerial/myImei
100,myImei
// Create object with externalId c8y_MySerial/myImei
101,myImei,c8y_MySerial
// This message will result in not creating an external ID
101,,c8y_MySerial
PUT templates

The PUT templates for inventory follow the same logic like the GET templates with the addition that you can also use custom values for PUT.

// Creation:
// 10,msgId,method,api,response,byId,externalIdTyoe,custom1.path,custom1.type,custom1.value
10,999,PUT,INVENTORY,,false,c8y_Serial,c8y_MyCustomValue,STRING,
// Usage:
999,myDeviceImei,myValue

The PUT template for alarms uses the type of the alarm to find the alarm to update. It will first check the ACTIVE alarms and if there is no ACTIVE alarm it will check the ACKNOWLEDGED alarms.

// Creation:
// 10,msgId,method,api,response,type,custom1.path,custom1.type,custom1.value
10,999,PUT,ALARM,,c8y_MyCustomAlarm,status,ALARMSTATUS
// Usage:
999,FAILED

The PUT templates for operations use the fragment of the operation to find the operation. It will first check the EXECUTING operations and if there is no EXECUTING operation it will check the PENDING operations.

// Creation:
// 10,msgId,method,api,response,fragment,custom1.path,custom1.type,custom1.value
10,999,PUT,OPERATION,,c8y_MyOperation,status,OPERATIONSTATUS,SUCCESSFUL,c8y_Fragment.val,NUMBER,
// Usage:
999,24
Adding custom properties

All POST and PUT values enable you to add custom properties to the results of the templates.

A single custom property requires you to add the following three values to your template creation:

field description
path A JsonPath for the value that should be set
type An optional data type of the value. Default: STRING
value The value to be set. Leaving this field empty requires the client to send the value when using the template
type description
STRING The default type. No additional verification of the value
DATE A time stamp in the ISO 8601 format. Using date and not sending a time stamp results in the use of server time
NUMBER An integer or number with decimal places
INTEGER An integer
UNSIGNED An integer (only positive)
FLAG An empty map (e.g. c8y_IsDevice: {}). The client does not need to send anything for this value
SEVERITY A severity of an alarm. Used to update the severity field of alarms
ALARMSTATUS A status of an alarm. Used to update the status field of alarms
OPERATIONSTATUS A status of an operation. Used to update the status field of operations
Examples

Template for clearing an alarm with an additional custom property

// Creation:
10,999,PUT,ALARM,,c8y_MyCustomALarm,status,ALARMSTATUS,CLEARED,c8y_CustomFragment.reason,STRING,
// Usage:
999,Device resolved alarm on its own

Template for creating a custom measurement

// Creation:
10,999,POST,MEASUREMENT,,c8y_CustomMeasurement,,c8y_CustomMeasurement.custom.value,NUMBER,,c8y_CustomMeasurement.custom.unit,STRING,X
// Usage:
999,30.6

Template for updating a property in the device

// Creation:
10,999,PUT,INVENTORY,,false,c8y_Serial,c8y_MyCustomValue,STRING,
// Usage:
999,myDeviceImei,updatedValue

Response templates

The SmartREST 2.0 response templates use the same structure as in SmartREST 1.0.

Field Data type Mandatory Description
messageId String YES Unique ID to reference the template within the collection
base String NO A JsonPath prefix that all patterns will use
condition String NO A JsonPath that needs to exist in the object to use the pattern
pattern List<String> YES A list of JsonPath that will be extracted from the object and returned to the device

Response will be used for every operation and for any request template that defines the response field with true. In each case the server will try every registered response template so there might be multiple response lines for a single operation or request.

SmartREST 2.0 will always return a response template if the condition is true (or no condition was defined). Patterns that did not resolve will be returned as empty string.

You should make use of the condition field to control when response templates should be returned.

Examples

Querying data from the device object

Device object:

{
  "id": "12345",
  "type": "myMqttDevice",
  "c8y_IsDevice": {},
  "c8y_Configuration": "val1=1\nval2=2"
}

Template creation:

10,999,GET,INVENTORY,,true
11,888,,c8y_IsDevice,type,c8y_Test,c8y_Configuration

Client publishes:

999,12345

Client receives:

888,myMqttDevice,,"val1=1\nval2=2"


Parsing custom operations

Operation object:

{
  "id": "12345",
  "deviceId": "67890",
  "agentId": "67890",
  "status": "PENDING",
  "c8y_CustomConfiguration": {
    "val1": "1",
    "val2": "2",
    "customValues": ["a", "b", "c"]
  },
  "description": "Send custom configuration"
}

Template creation:

11,111,c8y_CustomConfiguration,deviceId,val1,val2,customValues[*]
11,222,,deviceId,c8y_CustomConfiguration.val1,c8y_CustomConfiguration.val2
11,333,,deviceId,val1,val2,customValues[*]
11,444,c8y_CustomConfiguration,c8y_CustomConfiguration.val3,val1,val2,customValues[*]

Client receives (assuming the ClientId is "myMqttTestDevice"):

111,myMqttTestDevice,1,2,a,b,c
222,myMqttTestDevice,1,2
333,myMqttTestDevice,,,

The template 444 is not returned as the condition does not match the operation.


Querying data from the device object containing key with multiple objects

Device object:

{
  "id": "12345",
  "name": "test",
  "type": "c8y_MQTTdevice",
  "c8y_IsDevice": {},
  "myList": [
      {
          "pid": 12345,
          "type": "test"
      },
      {
          "pid": 123456,
          "type": "test2"
      }
  ]
}

Template creation:

10,999,GET,INVENTORY,,true
11,888,,,"$.myList[*].type"

Client publishes:

999,12345

Client receives:

888,test,test2

Using a default collection

Having the X-ID as part of the topic gives you the freedom to easily use multiple template collections but adds additional bytes for every message. If the device anyways uses mostly (or completely) a single collection it makes sense to specify this collection as you default collection. With a default collection specified the client can use special topics which don't require the X-ID and instead the server will use the X-ID previously specified. The topics are "s/ud" for publishing and "s/dd" for subscribing.

You can specify one X-ID within your MQTT ClientID (see MQTT implementation). Your MQTT ClientID could look like this:

d:myDeviceSerial:myDefaultTemplateXID

Info: If you use a default X-ID in the ClientId you need to include the "d:" at the beginning to specify that the client is a device.

It is not required that the default template exists at the time of establishing the MQTT connection (it will be verified once the client uses it).

MQTT Static templates

Overview

To ease device integration Cumulocity already supports a number of static templates that can be used by any client without the need for creating own templates.

These templates focus on the most commonly used messages for device management purposes.

To use the templates listed below you need to publish the messages to the topic "s/us" ("t/us" for transient processing of published content, "q/us" for quiescent processing of published content or "c/us" for CEP processing of published content, see SmartREST > Processing mode in the Reference guide for further information.

To receive operations with the static templates you need to subscribe to the topic "s/ds".

Automatic device creation

The topic for static templates supports an automatic creation of devices. Whenever there is no child associated with your MQTT ClientID and you send data, Cumulocity will automatically create a device for the MQTT ClientID. If you want to create the device on your own your first message must be the device creation. In this case Cumulocity will create the device from the template.

The automatic creation of devices is also supported for 1st level child devices. For child devices on a deeper level you must use the template for creating a child device by sending it to the topic of the child device under which you want to place the new child.

Handling none mandatory parameters

If a parameter is not declared as mandatory the client can send an empty string in that place.

Example:

100,,myType

Tailing commas are not required. The two lines below result in the same message.

100,,
100

Publish templates

The following templates can be used to publish data on the topics "s/us" as well as "t/us". For more information about t/ topic for transient data processing, refer to SmartRest > Processing mode in the Reference guide.

Inventory templates (1xx)

Device creation (100)

Will create a new device for the serial number in the inventory if not yet existing. An externalId for the device with type “c8y_Serial” and the device identifier of the MQTT clientId as value will be created

Position Parameter Mandatory Default
1 device name NO MQTT Device <serialNumber>
2 device type NO c8y_MQTTDevice

Example

100,myDevice,myType
Child device creation (101)

Will create a new child device for the current device. The newly created object will be added as child device. Additionally an externaId for the child will be created with type “c8y_Serial” and the value a combination of the serial of the root device and the unique child ID.

Position Parameter Mandatory Default
1 unique child ID YES
2 device name NO MQTT Device <serialNumber>
3 device type NO c8y_MQTTChildDevice

Example

101,uniqueChildId,myChildDevice,myChildType
Get child devices (105)

Will trigger the sending of child devices of the device.

Example

105
Configure Hardware (110)

Will update the hardware properties of the device.

Position Parameter Mandatory
1 serialNumber NO
2 model NO
3 revision NO

Example

110,1234567890,myModel,1.2.3
Configure Mobile (111)

Will update the mobile properties of the device.

Position Parameter Mandatory
1imei NO
2 iccid NO
3 imsi NO
4 mcc NO
5 mnc NO
6 lac NO
7 cellId NO

Example

111,1234567890,,54353
Configure Position (112)

Will update the position properties of the device.

Position Parameter Mandatory
1 latitude NO
2 longitude NO
3 altitude NO
4 accuracy NO

Example

112,50.323423,6.423423
Set Configuration (113)

Will update the configuration properties of the device.

Position Parameter Mandatory
1 configuration NO

Example

113,"€val1=1\nval2=2"
Set supported operations (114)

Will set the supported operations of the device.

Position Parameter Mandatory
1... List of supported operations

Example

114,c8y_Restart,c8y_Configuration,c8y_SoftwareList
Set firmware (115)

Will set the firmware installed on the device.

Position Parameter Mandatory
1 name NO
2 version NO
3 url NO

Example

115,firmwareName,firmwareVersion,firmwareUrl
Set software list (116)

Will set the list of software installed on the device.

Position Parameter Mandatory
1... List of 3 values per software NO
1.1 name NO
1.2 version NO
1.3 url NO

Example

116,software1,version1,url1,software2,,url2,software3,version3
Set required availability (117)

Will set the required interval for availability monitoring. It will only set the value if does not exist. Values entered e.g. through UI are not overwritten.

Position Parameter Mandatory
1 Required interval NO

Example

117,60

Measurement templates (2xx)

Create custom measurement (200)

Will create a measurement with given fragment and series.

Position Parameter Mandatory Default
1 fragment YES
2 series YES
3 value YES
4 unit NO
5 time NO Current server time

Example

200,c8y_Temperature,T,25
Create signal strength measurement (210)

Will create a measurement of type c8y_SignalStrength.

Position Parameter Mandatory Default
1 rssi value YES, if 2 not set
2 ber value YES, if 1 not set
3 time NO Current server time

Example

210,-90,23,2016-06-22T17:03:14.000+02:00
Create temperature measurement (211)

Will create a measurement of type c8y_TemperatureMeasurement.

Position Parameter Mandatory Default
1 temperature value YES
2 time NO Current server time

Example

211,25,2016-06-22T17:03:14.000+02:00
Create battery measurement (212)

Will create a measurement of type c8y_Battery.

Position Parameter Mandatory Default
1 battery value YES
2 time NO Current server time

Example

212,95,2016-06-22T17:03:14.000+02:00

Alarm templates (3xx)

Create CRITICAL alarm (301)

Will create a CRITICAL alarm.

Position Parameter Mandatory Default
1 type YES
2 text NO Alarm of type <alarmType> raised
3 time NO Current server time

Example

301,c8y_TemperatureAlarm
Create MAJOR alarm (302)

Will create a MAJOR alarm.

Position Parameter Mandatory Default
1 type YES
2 text NO Alarm of type <alarmType> raised
3 time NO Current server time

Example

302,c8y_TemperatureAlarm,”This is an alarm”
Create MINOR alarm (303)

Will create a MINOR alarm.

Position Parameter Mandatory Default
1 type YES
2 text NO Alarm of type <alarmType> raised
3 time NO Current server time

Example

303,c8y_TemperatureAlarm
Create WARNING alarm (304)

Will create a WARNING alarm.

Position Parameter Mandatory Default
1 type YES
2 text NO Alarm of type <alarmType> raised
3 time NO Current server time

Example

304,c8y_TemperatureAlarm,,2013-06-22T17:03:14.000+02:00
Update severity of existing alarm (305)

Will change the severity of an existing alarm.

Position Parameter Mandatory
1 type YES
2 severity YES

Example

305,c8y_TemperatureAlarm,CRITICAL
Clear existing alarm (306)

Will clear an existing alarm.

Position Parameter Mandatory
1 type YES

Example

306,c8y_TemperatureAlarm

Event templates (4xx)

Create basic event (400)

Will create an event of given type and text.

Position Parameter Mandatory Default
1 type YES
2 text YES
3 time NO Current server time

Example

400,c8y_MyEvent,"€Something was triggered"
Create location update event (401)

Will create typical location update event containing c8y_Position.

Position Parameter Mandatory Default
1 latitude NO
2 longitude NO
3 altitude NO
4 accuracy NO
5 time NO Current server time

Example

401,51.151977,6.95173,67
Create location update event with device update (402)

Will create typical location update event containing c8y_Position. Additionally the device will be updated with the same c8y_Position fragment.

Position Parameter Mandatory Default
1 latitude NO
2 longitude NO
3 altitude NO
4 accuracy NO
5 time NO Current server time

Example

402,51.151977,6.95173,67

Operation templates (5xx)

Get PENDING operations (500)

Will trigger the sending of all PENDING operations for the agent.

Example:

500
Set operation to EXECUTING (501)

Will set the oldest PENDING operation with given fragment to EXECUTING.

Position Parameter Mandatory
1 fragment YES

Example

501,c8y_Restart
Set operation to FAILED (502)

Will set the oldest EXECUTING operation with given fragment to FAILED.

Position Parameter Mandatory
1 fragment YES
2 failureReason NO

Example

502,c8y_Restart,"Could not restart"
Set operation to SUCCESSFUL (503)

Will set the oldest EXECUTING operation with given fragment to SUCCESSFUL. It enables the device to send additional parameters that trigger additional steps based on the type of operation send as fragment (see section Updating operations).

Position Parameter Mandatory
1 fragment YES
2... parameters NO

Example

503,c8y_Restart

Subscribe templates

The client can receive the following templates when subscribing to "s/ds".

Inventory templates (1xx)

Get children of device (106)

Lists all children of the device.

Position Parameter
1... child

Example

106,child1,child2,child3

Operation templates (5xx)

All operation responses have the same base structure leading with the message ID followed by the ID of either the root device or a child which should handle the operation.

Restart (510)

Tells the device to restart.

Example

510,DeviceSerial
Command (511)

Tells the device to run the command send in the operation.

Position Parameter
1 Command text

Example

511,DeviceSerial,execute this
Configuration (513)

Tells the device to set the configuration send in the operation.

Position Parameter
1 configuration

Example

513,DeviceSerial,"val1=1\nval2=2"
Firmware (515)

Tells the device to install the firmware from the url.

Position Parameter
1 firmware name
2 firmware version
3 url

Example

515,DeviceSerial,myFimrware,1.0,http://www.my.url
Software list (516)

Tells the device to install the software send in the operation.

Position Parameter
1... List of 3 values per software
1.1 name
1.2 version
1.3 url

Example

516,DeviceSerial,softwareA,1.0,url1,softwareB,2.0,url2
Measurement request operation (517)

Tells the device to send the measurements specified by the request name.

Position Parameter
1 request name

Example

517,DeviceSerial,LOGA
Relay (518)

Tells the device to either open or close the relay.

Position Parameter
1 Relay state

Example

518,DeviceSerial,OPEN
RelayArray (519)

Tells the device to either open or close the relays in the array.

Position Parameter
1... List of relay state

Example

519,DeviceSerial,OPEN,CLOSE,CLOSE,OPEN
Upload configuration file (520)

The current configuration is uploaded from Cumulocity to the device.

Example

520,DeviceSerial
Download configuration file (521)

Tells the device to download a configuration file from the url.

Position Parameter
1 url

Example

521,DeviceSerial,http://www.my.url
Logfile request (522)

Tells the device to upload a log file for the given parameters.

Position Parameter
1 Log file name
2 Start date
3 End date
4 Search text
5 Maximum lines

Example

522,DeviceSerial,logfileA,2013-06-22T17:03:14.000+02:00,2013-06-22T18:03:14.000+02:00,ERROR,1000
Communication mode (523)

Tells the device to change the communication mode.

Position Parameter
1 mode

Example

523,DeviceSerial,SMS

Updating operations

When using the template to set an operation to state SUCCESSFUL it supports sending additional parameters to trigger additional calls on the server. The table below shows the operations that support this feature and what will be done with the parameters.

Fragment Parameters Action triggered
c8y_Command result Result will be added to operation
c8y_RelayArray relay states Device object will be updated with the states
c8y_CommunicationMode no parameter needed Device object will be updated with the mode
c8y_LogfileRequest file url File url will be added to operation
c8y_DownloadConfigFile (optional) timestamp Device object will be updated with the ID of the configuration dump and the timestamp (or server time)

Handling IDs

Concept of ID-less communication

The MQTT implementation of Cumulocity is specifically designed for device communication and therefore tries to remove as much unnecessary logic from the client side.

Using the REST (or SmartREST) protocol requires to know the ID of every object, alarm and operation you want to update.

Therefore a client needs to keep state of these IDs. For example, if it creates an alarm it needs to know the ID of the alarm so it can clear it afterwards. With the MQTT implementation we want to reduce the logic required on the device to do such actions and move the logic to the server.

Example 1: ID of the device

To update data in the device object via REST you need to know the ID of the device object. Also this ID is required for every other data that needs to be associated with the device e.g. the source in a measurement, alarm or event.

To remove the necessity of persisting the ID on the device, Cumulocity offers the identity API where you can link external IDs (e.g. a serial number) to the object so you can query the ID any time.

Therefore a typical device start looks like this:

REST Receiving Device ID

With MQTT we automatically use the identity API with the MQTT clientId. This removes the necessity to tell the ID to the device and because the client sends also the other data on this connection we can associate every measurement, alarm, event, ... with the correct device.

MQTT Automatically Resolve ID

Example 2: ID of alarms

When a client creates an alarm using the REST API it needs to ensure that it gets the ID of the alarm that was generated by Cumulocity in return.

The client will need this ID to later update the alarm for example to status CLEARED if the alarm is not active anymore.

REST Handling Alarms

In Cumulocity, a device can only have a single alarm per type in status ACTIVE. If it creates another alarm with the same type it will get de-duplicated.

Therefore the MQTT implementation uses the type of an alarm as identifier. The client only needs to send which type of alarm has been resolved and the server will find the correct alarm object.

MQTT Handling Alarms

JSON via MQTT

This section describes the JSON payload format that can be used with the Cumulocity MQTT implementation.

Compared to SmartREST 2.0, which only works with fixed templates, JSON support for MQTT was designed to combine the payload flexibility of our REST API with the low protocol overhead of MQTT.

The SmartREST way should still be the preferred way if it is important to reduce your payload to the minimum (mobile traffic, low capability device).

Topic structure

The topic structure in JSON MQTT is quite similar to the REST endpoints. The main difference is in the additional action part which is included in the topic.

To publish messages:

<api>/<resource>/<action>/<id>

To publish messages in TRANSIENT mode:

t/<api>/<resource>/<action>/<id>

To publish messages in QUIESCENT mode:

q/<api>/<resource>/<action>/<id>

To publish messages in CEP mode:

c/<api>/<resource>/<action>/<id>

Refer to Processing mode for more information about transient, quiescent and CEP data processing.

Topic actions

The action in the topic corresponds to the HTTP methods combined with the content-type header.

The following actions are available:

  • create - corresponds to HTTP POST
  • createBulk - corresponds to HTTP POST with the content-type header value set to collection media type, for example application/vnd.com.nsn.cumulocity.measurementCollection+json;charset=UTF-8;ver=0.9
  • update - corresponds to HTTP PUT
  • delete - corresponds to HTTP DELETE

Supported endpoint

The current JSON MQTT implementation does not cover all SmartREST 2.0 operations, so for example the whole device bootstrap process has to be done using SmartREST 2.0.

The following endpoints and actions are supported:

Endpoint create createBulk update delete
event/events x x x x
alarm/alarms x x x
measurement/measurements x x x
inventory/managedObjects x x
inventory/managedObjects/<DeviceID>/childDevices x  

If the operation is not supported a proper error message will be send to the error topic.

For all of the above endpoints you can use the same payload like in the REST API. The only difference is in the source field - in REST this field is mandatory while for JSON MQTT there is no need to set the device ID here. The source device ID will automatically be resolved based on the MQTT client ID. This value will always be used, no matter if there already is something defined there.

Examples

Create new event

Publish message on topic /event/events/create with payload:

  {
    "type": "TestEvent",
    "text": "sensor was triggered",
    "time": "2014-03-03T12:03:27.845Z"
  }

Create many events

Publish message on topic /event/events/createBulk with payload:

  {
    "events": [
      {
        "type": "TestEvent1",
        "text": "sensor was triggered",
        "time": "2014-03-03T12:03:27.845Z"
      },
      {
        "type": "TestEvent2",
        "text": "sensor was triggered",
        "time": "2014-03-04T12:03:27.845Z"
      }
    ]
  }

Update event

Publish message on topic /event/events/update/<event_id> with payload:

  {
    "text": "new text"
  }

Delete event

Publish message on topic /event/events/delete/<event_id> with empty payload.

Error handling

To subscribe for errors related to the JSON MQTT implementation use the error topic. In case of invalid payload, wrong topic or any other exception, a notification will be published on this topic.

The payload is in JSON format. Besides a standard error message it also contains a message ID which helps the client in finding out which exact message was failing.

Example payload:

{
  "error": "undefined/validationError",
  "message": "Following mandatory fields should be included: severity,text,time",
  "messageId": 3
}

Receiving operations

A notification client can subscribe to the devicecontrol/notifications topic to receive notifications of newly created operations. Initially upon subscription all operations which are not yet forwarded will be published.

Additionally, it contains an External ID so the client can identify for which child the operation is executed.

Example notification:

{
  "agentId": "1",
  "creationTime": "2018-05-17T07:33:15.555Z",
  "delivery": {
    "log": [

    ],
    "status": "PENDING",
    "time": "2018-05-17T07:33:15.575Z"
  },
  "deviceId": "2",
  "id": "123",
  "status": "PENDING",
  "c8y_Command": {
    "text": "Do something"
  },
  "description": "Execute shell command",
  "externalSource": {
    "externalId": "3",
    "type": "c8y_Serial"
  }
}