Real-time processing

Cumulocity allows developers and power users to run real-time IoT business logic inside Cumulocity based on a high-level real-time processing language.

This section introduces the basic concepts of real-time processing and shows how you can develop your own functional business logic at Cumulocity.

What is real-time processing in Cumulocity IoT?

On top of Cumulocity IoT you can use the Apama streaming analytics engine to define business operations for immediate processing of incoming data from devices or other data sources. These user-defined operations can for example alert applications of new incoming data, create new operations based on the received data (such as sending an alarm when a threshold for a sensor is exceeded), or trigger operations on devices. The operation logic is implemented in Apama's Event Processing Language (EPL).

Apama's Event Processing Language covers statements, which are organized into actions and monitors. These can be deployed one file at a time, where a file may contain multiple monitors and event definitions. Monitor files can be edited with Software AG Designer - an Eclipse-based development environment, and can be deployed as Cumulocity applications, see Administration > Own applications in the User guide.

For further information on using Apama's Event Processing Language in Cumulocity IoT refer to Using Apama Event Processing Language below and to the Analytics guide.

For more information about the interfaces for real-time processing also see Real-time Statements in the Reference guide.

Important: All new Cumulocity installations will use the Apama CEP engine. Using the Esper CEP engine is still supported for older installations but will no longer be provided for new installations and not be invested into in the future. For information on using the deprecated CEL functionality based on Esper refer to the CEL Analytics guide.

What are the benefits of using real-time processing?

Cumulocity's real-time processing feature has the following benefits:

  • React instantly to events from remote sensors.
  • Develop highly interactive IoT applications.
  • Run IoT use cases directly inside Cumulocity without software development and leave the hosting and management to Cumulocity.
  • Validate, normalize and derive data according to your own business rules across different device makes.
  • Trigger automated remote control actions based on events.
  • Use powerful, stream-oriented business logic, like time windows and joins.
  • Reduce the cost of online tracking devices by preselecting data necessary for long-term storage.

Using Apama Event Processing Language (EPL)

Overview

Apama Event Processing Language has a syntax similar to Java. In addition to simple flow control statements such as if, while, for, users can write listeners with the on keyword to react to events.

Apama EPL is documented in the Apama documentation.

As an example, the following statement listens for new temperature sensor readings ranging above a particular temperature:

 on all Measurement(type = "c8y_TemperatureMeasurement") as e {
          if e.measurements.getOrDefault("c8y_TemperatureMeasurement").getOrDefault("T").value > 100.0 {
              send Alarm("", "c8y_TemperatureAlert", e.source, e.time, "Temperature too high",
                  "CRITICAL", "", 1, new dictionary<string,any>) to Alarm.CHANNEL;
          }
}

Here, Measurement is a pre-defined event containing the measurements. In this example, "e" is the "Measurement" event, the listener is filtering for measurements which are "c8y_TemperatureMeasurement" and the property is "c8y_TemperatureMeasurement.T". "value" is in degrees Celsius of a temperature sensor (see the sensor library).

Listeners such as the above should be placed in a monitor in the onload statement, and the file will need to contain using statements for the types used by the listener - for most of the Cumulocity events, these are in the package com.apama.cumulocity. The full list is provided below - for the sake of brevity, we will omit these from further examples:

using com.apama.cumulocity.ManagedObject;
using com.apama.cumulocity.Operation;
using com.apama.cumulocity.Event;
using com.apama.cumulocity.Alarm;
using com.apama.cumulocity.Error;
using com.apama.cumulocity.FindAlarm;
using com.apama.cumulocity.FindAlarmResponse;
using com.apama.cumulocity.FindAlarmResponseAck;
using com.apama.cumulocity.FindManagedObject;
using com.apama.cumulocity.FindManagedObjectResponse;
using com.apama.cumulocity.FindManagedObjectResponseAck;
using com.apama.cumulocity.Measurement;
using com.apama.cumulocity.MeasurementValue;
using com.apama.cumulocity.RequestAllDevices;
using com.apama.cumulocity.RequestAllDevicesComplete;
using com.apama.cumulocity.SendEmail;
using com.apama.cumulocity.SendSMS;
using com.apama.cumulocity.SendSpeech;
using com.apama.cumulocity.SMSResourceReference;
using com.apama.cumulocity.SMSResponse;
using com.apama.util.AnyExtractor;
using com.apama.correlator.timeformat.TimeFormat;
using com.softwareag.connectivity.httpclient.HttpOptions;
using com.softwareag.connectivity.httpclient.Request;
using com.softwareag.connectivity.httpclient.RequestType;
using com.softwareag.connectivity.httpclient.Response;
using com.apama.cumulocity.Util;

monitor ListenForHighTemperatures {
    action onload() {
        on all Measurement(type = "c8y_TemperatureMeasurement") as e {
            if e.measurements.getOrDefault("c8y_TemperatureMeasurement").getOrDefault("T").value > 100.0 {
                // handle the measurement
        }
    }
}
}

How can I create derived data from EPL?

To create new Alarm or Operation objects, create an instance of the relevant event type and use the send statement to send it to the relevant channel (defined with a constant on the event type). Assume that an alarm should be generated immediately if the temperature of a sensor exceeds a defined value. This is done with the following statement:

on all Measurement(type = "c8y_TemperatureMeasurement") as e {

if e.measurements.getOrDefault("c8y_TemperatureMeasurement").getOrDefault("T").value > 100.0 {

send Event("", "c8y_TemperatureAlert", e.source, e.time, "Temperature too high", {"severity":<any> "CRITICAL"} ) to Event.CHANNEL;

}

}

Technically, this statement produces a new "Event" event each time a temperature sensor reads more than 100 degrees Celsius and sends it to Cumulocity.

How can I control devices from EPL?

Remote control with EPL is done by sending DeviceOperation events. Remote operations are targeted to a specific device. The following example illustrates switching a relay based on temperature readings:

on all Measurement(type = "c8y_TemperatureMeasurement") as e {
    if
e.measurements.getOrDefault("c8y_TemperatureMeasurement").getOrDefault("T").value >
100.0 {
    send Operation("", e.source, "PENDING", {"c8y_Relay":<any>
{"relayState":"CLOSED"} }) to Operation.CHANNEL;
    }    
} 
  • e.source is a placeholder for the ID of the heating that should be triggered.

  • The params field (the last field) defines the nested content of the operation a "c8y_Relay" that has relayState set to "CLOSED"; note the top-level fields must be dictionary<string, any>, thus the use of the cast operations.

How can I query data from EPL?

It may be required to query information from the Cumulocity database as part of the ongoing event processing. This is supported by sending events and using listeners to wait for responses. Here is an example that shows how to summarize total sales for vending machines every hour. The sales report data created after a purchase is retrieved from the Cumulocity database.

using com.apama.aggregates.count; 

monitor SalesReport {
    event SalesReport {
        Event e;
        ManagedObject customer;
    }
    event SalesOutput {
        integer count;
        string customerId;
    }

action onload() {
    monitor.subscribe(Measurement.CHANNEL);
    on all Event() as e {
        integer reqId := integer.getUnique();
        monitor.subscribe(FindManagedObjectResponse.CHANNEL);
        send FindManagedObject(reqId, "", {"childAssetId":e.source}) to FindManagedObject.CHANNEL;

        on FindManagedObjectResponseAck(reqId = reqId) {
            monitor.unsubscribe(FindManagedObjectResponse.CHANNEL);
        }
        on all FindManagedObjectResponse(reqId = reqId) as d and not FindManagedObjectResponseAck(reqId = reqId) {
            route SalesReport(e, d.managedObject);
        }
    }

    from sr in all SalesReport() within 3600.0  every 3600.0
        group by sr.customer.id
        select SalesOutput(count(), sr.customer.id) as sales {
        send Measurement("", "total_cust_trx", "customer_trx_counterId",
            currentTime, {"total_cust_trx":{
                "total":MeasurementValue(sales.count.toFloat(), "COUNT", new dictionary<string,any>)}
            }, {"customer_id":<any> sales.customerId}) to Measurement.CREATE_CHANNEL;
    }
}    
}

Above we create event definitions. These hold the SalesReport (the Event and ManagedObject that identifies a sale) and the information we want to derive from a set of sales: the count and customerId. We listen for Event objects, and send a FindManagedObject request to look up the ManagedObject that the event came from. These SalesReport objects are sent, via the route statement, into a stream query. The stream query fires every hour (3,600 seconds) and selects an aggregate of the sales data per customer, and sends a new Measurement representing the sales for that vending machine.

How is real-time processing implemented in Cumulocity?

There are two processing modes for API requests in Cumulocity: persistent and temporary. The "persistent" mode is the default: It will store data in the Cumulocity database as well as send the data to the real-time engine. After both is done, Cumulocity returns the result of the request.

Data marked as "temporary" is not stored into Cumulocity's database but just handled by the real-time engine. This saves on storage and processing cost for example when tracking devices in real-time without requiring data to be stored.

The "temporary" mode will only send the data to the real-time engine and immediately return asynchronously and not store it in Cumulocity's database. This mode saves storage and processing costs and is useful for example when tracking devices in real-time without requiring data to be stored.

CEP architecture

Examples

Assume that location updates from cars should be monitored every second while the car is driving, but only be stored once in a minute into the database for reporting purposes. This is done using the following Apama statement:

monitor SendEveryMinute {
    dictionary<string, Event> latestUpdates;
    action onload() {
    monitor.subscribe(Measurement.CHANNEL);
        on all Event() as e {
            if e.params.hasKey("c8y_LocationUpdate") {
                latestUpdates[e.source] := e;
            }
        }
        on all wait(60.0) {
            Event e;
            for e in latestUpdates.values() {
                send e to Event.CHANNEL;
            }
            latestUpdates.clear();
        }
    }
}

Another option is to output only every 60th update.

monitor SendEverySixtyEvents {
    event UpdateAndCount {
        Event latest;
        integer count;
    }
    dictionary<string, UpdateAndCount > latestUpdates;
    action onload() {
    monitor.subscribe(Measurement.CHANNEL);
        on all Event() as e {
            if e.params.hasKey("c8y_LocationUpdate") {
                UpdateAndCount updateCount := latestUpdates.getOrAddDefault(e.source);
                updateCount.latest := e;
                integer count := updateCount.count + 1;
                if count = 60 {
                    send e to Event.CHANNEL;
                    latestUpdates.remove(e.source);
                } else {
                    updateCount.count := count;
                }
            }
        }
    }
}