Eclipse Ditto™

Today the Eclipse Ditto team proudly presents its first release 0.8.0. Please have a look at the 0.8.0 release notes for a more detailled information on the release. The main changes and new features since the latest milestone 0.8.0-M3 are: ... [More] Support Eclipse Hono’s command&control in Ditto connectivity several smaller bugfixes Artifacts The new Java artifacts have been published at the Eclipse Maven repository as well as Maven central. The Docker images have been pushed to Docker Hub: eclipse/ditto-policies eclipse/ditto-things eclipse/ditto-things-search eclipse/ditto-gateway eclipse/ditto-connectivity eclipse/ditto-concierge – The Eclipse Ditto team [Less]

Today the Eclipse Ditto team proudly presents its first release 0.8.0. Please have a look at the 0.8.0 release notes for a more detailed information on the release. The main changes and new features since the latest milestone 0.8.0-M3 are: ... [More] Support Eclipse Hono’s command&control in Ditto connectivity several smaller bugfixes Artifacts The new Java artifacts have been published at the Eclipse Maven repository as well as Maven central. The Docker images have been pushed to Docker Hub: eclipse/ditto-policies eclipse/ditto-things eclipse/ditto-things-search eclipse/ditto-gateway eclipse/ditto-connectivity eclipse/ditto-concierge – The Eclipse Ditto team [Less]

Preparing the upcoming first release 0.8.0 of Eclipse Ditto, this milestone is a last checkpoint to ensure that the release will be performed smoothly. Therefore, this milestone release primarily focuses on stabilization. Have a look at the ... [More] Milestone 0.8.0-M3 release notes for what changed in detail. Warning: If you want to upgrade an existing Ditto installation, you’ll have to execute a small database migration - see release notes. The main changes and new features are speed up of search index creation applying enforcement of messages received via connections (e.g. from Eclipse Hono) copying already existing policies when creating things Artifacts The new Java artifacts have been published at the Eclipse Maven repository as well as Maven central. The Docker images have been pushed to Docker Hub: eclipse/ditto-policies eclipse/ditto-things eclipse/ditto-things-search eclipse/ditto-gateway eclipse/ditto-connectivity eclipse/ditto-concierge – The Eclipse Ditto team [Less]

Eclipse Ditto can now connect to MQTT 3.1.1 brokers. Perfect timing to happily welcome a new colleague to our team behind Eclipse Ditto: David. In order to get familiar with Ditto and the development with digital twins, David was assigned with one ... [More] of his first tasks to get his hands on one of our newest features, MQTT connectivity which was released recently with milestone 0.8.0-M2. On his journey into digital twin land he made a great example with an ESP8266 powered board connected via an MQTT broker to Ditto and published it to the Eclipse Ditto examples repository: This example is about how to communicate between device and solution in a two way pattern through Ditto using MQTT. This means we will add a policy, a thing and a MQTT connection to Ditto. When Ditto ist set up and working, we will create real world device (“octopus board”) and connect it to it’s digital twin via MQTT. At the end, we will create a basic frontend webapp. The webapp will automatically connect to Ditto when you type in your credentials, automatically pull your things and show them in a list. You can create, modify and delete devices in the webapp and if there is a real world device connected to the thing in the list, you can send it a command message to control any feature on it. The list of things will always be up-to-date when you listen to server-sent-events, which you can activate easily with pressing a button. Source: https://github.com/eclipse/ditto-examples We will use an Octopus-board with an ESP8266 on it. It has several sensors built in, but for simplicity we will just use it’s temperature and altitude sensor. To show the functionality of Eclipse Ditto messages, we will switch on/off a LED on the Octopus-board through it. If you have any wishes, improvements, are missing something or just want to get in touch with us, you can use one of our feedback channels. – The Eclipse Ditto team [Less]

Brace yourself, Eclipse Ditto is preparing for its first release 0.8.0. We are happy to announce our next milestone towards that goal. Have a look at the Milestone 0.8.0-M2 release notes for what changed in detail. The main changes and new ... [More] features are enforcement of max. entity size of twins and messages added MQTT support connecting to MQTT 3.1.1 brokers subscribing to changes based on filters conditional requests at all APIs Artifacts Unfortunately, we had some problems during the milestone release build causing that the released artifacts have version 0.8.0-M2b instead of 0.8.0-M2. The new Java artifacts have been published at the Eclipse Maven repository as well as Maven central. The Docker images have been pushed to Docker Hub: eclipse/ditto-policies eclipse/ditto-things eclipse/ditto-things-search eclipse/ditto-gateway eclipse/ditto-connectivity eclipse/ditto-concierge – The Eclipse Ditto team [Less]

The connectivity service supercharged Ditto’s flexibility in integrating with other services. It’s such a great feature to let the other connected services know about thing updates and property changes. Even the direct exchange with real-world assets ... [More] became more flexible through the multi-protocol support. But with a steady increase in connected devices, those messages easily sum up to a huge number. Note: In order to simplify a little, we here use the term message as synonym for both Ditto signals and messages. Also, not every consuming application needs to know everything that’s going on. In fact, the only use case that requires processing of every message is logging. Therefore most of the times an application waits for a specific message to trigger a specific action. So all other messages are discarded unused. This adds a lot of unnecessary overhead both to the message transport capabilities and the processing of messages at the receiving end. But what if you could avoid receiving those messages at all. Well, you can! This is exactly what selective push notifications do: Configurable message filters that are applied to Ditto’s publishing connection before anything goes on the line. They can help you with a lot of problems in a bunch of scenarios: Bandwidth limitations: The amount of occurring events is too large and/or frequent to be delivered via the available channels. With selective message filters, you can mute the noise in your event stream. Information hiding: Let consuming services only know what they need to know. Message filters allow you to control all published content in great detail. Specialized notifications: A specific event filter can be used to set a value thresholds or a status-change trigger. This removes the burden of implementing filter logic on the application side. Event routing: Create multiple connections with Ditto’s connectivity service and route your events through those aligned with your requirements. All by specifying appropriate filters for your connection targets. The following diagram visualizes this context: With the upcoming Ditto release 0.8.0-M2, those filters are available for the following endpoints: WebSocket Server-Sent Events (SSE) All supported connectivity protocols (AMQP 0.9.1, AMQP 1.0 / Eclipse Hono, MQTT) You can use a basic namespace filter on the following topics: Twin events Live events Live messages Live commands This filter is a comma-separated list of selected namespaces. It only allows messages related to one of the given namespaces. Furthermore, there is an additional RQL filter for an advanced description of twin and live events. Powered by the mighty syntax of Ditto’s search API it allows configuring the selected events in the same manner as you search for things. Check out the documentation for more information on options and configuration. A simple example Imagine you have a flat with multiple environmental sensors: Some measure temperature, some humidity and some both. This information can be useful for different applications. In our case, a smart thermostat uses the sensor data to control the indoor climate and there is also a fire alarm installed that detects fires by abnormal high measured temperatures The following figure displays this setting: So let’s start with the prerequisites. You need: A running Ditto instance with a valid user (You can follow our Hello World example to create one). This example uses dittos default user on a locally running instance. A tool for executing HTTP requests (e.g. Ditto’s Swagger API, cURL, Postman). We use this to create our twins and simulate the sensors. A modern browser supporting WebSockets. This example uses websocket.org as a websocket client. The site will tell you if your browser supports the WebSocket protocol. We will mock our applications this way. The digital twins First we configure our sensors digital twins: A temperature sensor curl -X PUT -u 'ditto:ditto' --header 'Content-Type: application/json' -d \ '{ "features": { "environmentSensor": { "properties": { "temperature": 0.0 } } } }' \ 'http://localhost:8080/api/2/things/org.eclipse.ditto%3ATemperatureSensor' A humidity sensor curl -X PUT -u 'ditto:ditto' --header 'Content-Type: application/json' -d \ '{ "features": { "environmentSensor": { "properties": { "humidity": 0 } } } }' \ 'http://localhost:8080/api/2/things/org.eclipse.ditto%3AHumiditySensor' A combined temperature and humidity sensor curl -X PUT -u 'ditto:ditto' --header 'Content-Type: application/json' -d \ '{ "features": { "environmentSensor": { "properties": { "temperature": 0.0, "humidity": 0 } } } }' \ 'http://localhost:8080/api/2/things/org.eclipse.ditto%3ATemperatureAndHumiditySensor' And finally, a teapot curl -X PUT -u 'ditto:ditto' --header 'Content-Type: application/json' -d \ '{}' \ 'http://localhost:8080/api/2/things/org.eclipse.ditto%3ATeapot' Mocking the consuming applications Open your browser on https://websocket.org/echo.html. This site allows you to connect with any WebSocket endpoint and supports simple sending and receiving of messages. The interface is shown below: Enter Ditto’s WebSocket endpoint with user credentials ws://ditto:ditto@localhost:8080/ws/2 and hit the Connect button. The log output should confirm the action by printing a simple CONNECTED. This means the socket is open and you’re able to receive messages from Ditto. But first, you should let Ditto know in what kind of messages you’re interested. This interest differs for both of the example applications: The thermostat app only needs to know every humidity and temperature report so you can define a filter for change events on twins having those properties: START-SEND-EVENTS?filter=or(exists(features/environmentSensor/properties/temperature),exists(features/environmentSensor/properties/humidity)) Paste it into the Message input and use the Send button to post it. Ditto should acknowledge with a START-SEND-EVENTS:ACK. That’s it for our thermostat app, let’s proceed to the fire alarm. Open https://websocket.org/echo.html again in a separate tab and repeat the connection process. But instead of consuming all temperature and humidity reports, we only want to be notified when a specific temperature threshold is exceeded. 90°C seems to be a solid value for this: START-SEND-EVENTS?filter=gt(features/environmentSensor/properties/temperature,90) After receiving Ditto’s acknowledgment, you’re done with the configuration. Report mocked sensor values to Ditto Use Ditto’s HTTP API to send mocked data on behalf of our sensors. First report a new humidity value for the humidity sensor: curl -X PUT -u 'ditto:ditto' --header 'Content-Type: application/json' -d \ '55' \ 'http://localhost:8080/api/2/things/org.eclipse.ditto%3AHumiditySensor/features/environmentSensor/properties/humidity' Now check both websocket.org tabs. The thermostat tab should have received an event with the reported value while nothing happened in the alarm tab. Continue with some temperature data from another sensor: curl -X PUT -u 'ditto:ditto' --header 'Content-Type: application/json' -d \ '23' \ 'http://localhost:8080/api/2/things/org.eclipse.ditto%3ATemperatureAndHumiditySensor/features/environmentSensor/properties/temperature' The value change should be reported to the thermostat, but still no events for the alarm tab. Finally it’s time to start a fire. Report a very high temperature for the third sensor: curl -X PUT -u 'ditto:ditto' --header 'Content-Type: application/json' -d \ '120' \ 'http://localhost:8080/api/2/things/org.eclipse.ditto%3ATemperatureSensor/features/environmentSensor/properties/temperature' Now both applications should have received the reported data, and the fire alarm can use this event to (virtually) trigger its bell. But what about the teapot? Let him declare his identity by setting a personal message: curl -X PUT -u 'ditto:ditto' --header 'Content-Type: application/json' -d \ '{ "properties": { "message": "I'\''m a teapot" } }' \ 'http://localhost:8080/api/2/things/org.eclipse.ditto%3ATeapot/features/status' Unfortunately, no one cares and this no one is notified about that change. We do hope that you care about this feature, we think it’s really awesome. – The Eclipse Ditto team [Less]

Even during the summer break the Ditto team worked hard in order to provide the next milestone release. Here it is: Milestone 0.8.0-M1. Have a look at the Milestone 0.8.0-M1 release notes for what changed in detail and why there was a version bump ... [More] from 0.3.0-M2 to 0.8.0-M1. The main changes and new features are security enhancement by making some of Ditto’s headers not settable from the outside report application metrics to Prometheus automatically form a cluster when running in Kubernetes improvement of Ditto’s things-service memory consumption stabilization of the connectivity to AMQP 1.0 and 0.9.1 Artifacts The new Java artifacts have been published at the Eclipse Maven repository as well as Maven central. The Docker images have been pushed to Docker Hub: eclipse/ditto-policies eclipse/ditto-things eclipse/ditto-things-search eclipse/ditto-gateway eclipse/ditto-connectivity eclipse/ditto-concierge – The Eclipse Ditto team [Less]

Today we, the Eclipse Ditto team, are happy to announce our next milestone 0.3.0-M2. The main changes are improvement of Ditto’s cluster performance with many managed Things a new Ditto service ditto-concierge was added for this ... [More] improved cluster bootstrapping based on DNS with the potential to easy plugin other mechanism (e.g. for Kubernetes) Have a look at the Milestone 0.3.0-M2 release notes for a detailed description of what changed. Artifacts The new Java artifacts have been published at the Eclipse Maven repository as well as Maven central. The Docker images have been pushed to Docker Hub: eclipse/ditto-policies eclipse/ditto-things eclipse/ditto-things-search eclipse/ditto-gateway eclipse/ditto-connectivity eclipse/ditto-concierge – The Eclipse Ditto team [Less]

With the recently released Ditto milestone 0.3.0-M1 the connectivity to AMQP 1.0 endpoints can now be established in a durable and stable way (including failovers, etc.). That means Ditto now is ready to be connected to Eclipse Hono’s “northbound” ... [More] API which is provided via AMQP 1.0. By doing so it is for example possible to receive Hono telemetry messages (see heading “Northbound Operations”) which a device demo-device connected to the “southbound” of Hono sends via HTTP or MQTT (the currently available protocol adapters of Hono) in Ditto. When received, the payload can be translated into a format Ditto understands in order to update the digital twin of the demo-device device and provide API access to the twin, e.g. via HTTP or WebSocket. This blog post walks through the steps required to connect Ditto and Hono by adding a connection between the Hono and Ditto sandboxes at hono.eclipse.org ditto.eclipse.org Scenario The following graphic illustrates the scenario: Scenario for providing a digital twin in Ditto of a device connected via Hono Let’s assume for this tutorial that we have a device (e.g. containing a sensor) demo-device which is capable of measuring temperature and humidity. This device sends the sensor telemetry data every 5 minutes via MQTT into the cloud in either of the following formats: { "temp": 23.42, "hum": 44.42 } { "temp": 23.42 } { "hum": 44.42 } We want to create a digital twin for this device in order to access the device’s sensor data as API via Eclipse Ditto. Steps in Hono The steps in order to get started with Eclipse Hono are documented in the Hono getting started and in a new Blog post about using multi-tenancy in Eclipse Hono. We show them very briefly here as well but in order to comprehend what and why we are doing what we do please consult the Hono documentation. Create a tenant First of all, create a new Hono tenant (we chose the tenant name org.eclipse.ditto): $ curl -X POST -i -H 'Content-Type: application/json' -d '{"tenant-id": "org.eclipse.ditto"}' http://hono.eclipse.org:28080/tenant Register a device Register a new device in Hono (we chose the device-id demo-device): $ curl -X POST -i -H 'Content-Type: application/json' -d '{"device-id": "demo-device"}' http://hono.eclipse.org:28080/registration/org.eclipse.ditto Add a device credential In order for the device being able to send telemetry it needs to authenticate. For that we will need to add a credential for that device in Hono. We choose the hashed-password type: $ PWD_HASH=$(echo -n 'demo-device-password' | openssl dgst -binary -sha512 | base64 -w 0) $ curl -X POST -i -H 'Content-Type: application/json' -d '{ "device-id": "demo-device", "type": "hashed-password", "auth-id": "demo-device-auth", "secrets": [{ "hash-function" : "sha-512", "pwd-hash": "'$PWD_HASH'" }] }' http://hono.eclipse.org:28080/credentials/org.eclipse.ditto Publish data You are now able to publish telemetry (or also event) data via the Hono HTTP adapter: $ curl -X POST -i -u [email protected]:demo-device-password -H 'Content-Type: application/json' -d '{"temp": 23.07}' http://hono.eclipse.org:8080/telemetry $ curl -X POST -i -u [email protected]:demo-device-password -H 'Content-Type: application/json' -d '{"hum": 45.85}' http://hono.eclipse.org:8080/telemetry However as there is not yet a consumer listening for the messages, the Hono HTTP adapter will for example return an error code 503 - Service unavailable when publishing a telemetry message. Alternatively you can also publish telemetry data via MQTT: $ mosquitto_pub -u '[email protected]' -P demo-device-password -t telemetry -m '{"temp": 23.07}' $ mosquitto_pub -u '[email protected]' -P demo-device-password -t telemetry -m '{"hum": 45.85}' In the following steps we will register the missing consumer in Ditto by creating a connection to the Hono tenant in Ditto’s connectivity. Steps in Ditto We want to create a digital twin of the device connected to Eclipse Hono in order to access its latest reported state via the Ditto HTTP API, in order to be able to find it in a population of digital twins or in order to be notified about changed via an API optimized for the web. Create a digital twin The first step is to create a skeleton for the digital twin by creating a Ditto Thing. Notice that we authenticate with the sandbox user demo5 - a default Policy is implicitly created so that only that user may read+write the created Thing. $ curl -X PUT -i -u demo5:demo -H 'Content-Type: application/json' -d '{ "attributes": { "location": "Germany" }, "features": { "temperature": { "properties": { "value": null } }, "humidity": { "properties": { "value": null } } } }' https://ditto.eclipse.org/api/2/things/org.eclipse.ditto:demo-device Make sure the digital twin was created: $ curl -i -u demo5:demo https://ditto.eclipse.org/api/2/things/org.eclipse.ditto:demo-device Create a connection to Hono Ditto recently added support for managing connections to foreign endpoints (currently to AMQP 1.0 or to AMQP 0.9.1). As Hono provides an AMQP 1.0 endpoint, a connection can be added in Ditto which connects to Hono and acts as a “northbound” consumer. The following configuration for the connection has to be applied: AMQP 1.0 hostname: hono.eclipse.org AMQP 1.0 port: 15672 username: consumer@HONO password: verysecret sources: telemetry/org.eclipse.ditto event/org.eclipse.ditto Test the connection Send the following “test connection” command via HTTP in order to test if the Ditto sandbox can connect to the Hono one. $ curl -X POST -i -u devops:devopsPw1! -H 'Content-Type: application/json' -d '{ "targetActorSelection": "/system/sharding/connection", "headers": { "aggregate": false }, "piggybackCommand": { "type": "connectivity.commands:testConnection", "connection": { "id": "hono-sandbox-connection-1", "connectionType": "amqp-10", "connectionStatus": "open", "uri": "amqp://consumer%40HONO:[email protected]:15672", "failoverEnabled": true, "sources": [{ "addresses": [ "telemetry/org.eclipse.ditto", "event/org.eclipse.ditto" ], "authorizationContext": ["nginx:demo5"] }] } } }' https://ditto.eclipse.org/devops/piggyback/connectivity?timeout=8000 The result should be: { "?": { "-1": { "type": "connectivity.responses:testConnection", "status": 200, "connectionId": "hono-sandbox-connection-1", "testResult": "ditto-cluster=Success(successfully connected + initialized mapper)" } } } Great, it looks like with the provided credentials we can connect to the Hono sandbox. Define a payload mapping In the scenario we described the payloads our device sends via MQTT. As those JSON payloads are missing some information required for Ditto to map it to a Ditto Protocol message Ditto uses for updating the digital twin, we have to configure a payload mapping in order to add the missing information. Whenever one of the 3 following messages arrives at Ditto’s consumer, a payload mapping should be performed: { "temp": 23.42, "hum": 44.42 } { "temp": 23.42 } { "hum": 44.42 } A JavaScript based mapping which exactly does this could look like this: function mapToDittoProtocolMsg( headers, textPayload, bytePayload, contentType ) { if (contentType !== "application/json") { return null; // only handle messages with content-type application/json } var jsonData = JSON.parse(textPayload); var temperature = jsonData.temp; var humidity = jsonData.hum; var path; var value; if (temperature != null && humidity != null) { path = "/features"; value = { temperature: { properties: { value: temperature } }, humidity: { properties: { value: humidity } } }; } else if (temperature != null) { path = "/features/temperature/properties/value"; value = temperature; } else if (humidity != null) { path = "/features/humidity/properties/value"; value = humidity; } if (!path || !value) { return null; } return Ditto.buildDittoProtocolMsg( "org.eclipse.ditto", // the namespace we use headers["device_id"], // Hono sets the authenticated device-id in this header "things", // it is a Thing entity we want to update "twin", // we want to update the twin "commands", "modify", // command = modify path, headers, // copy all headers as Ditto headers value ); } In order to add this script to the connection we want to create, the newlines have to be replaced by \n so that the script fits in a single line JSON string and the " characters have to be replaced with \": "function mapToDittoProtocolMsg(\n headers,\n textPayload,\n bytePayload,\n contentType\n) {\n\n if (contentType !== \"application/json\") {\n return null;\n }\n\n var jsonData = JSON.parse(textPayload);\n var temperature = jsonData.temp;\n var humidity = jsonData.hum;\n \n var path;\n var value;\n if (temperature != null && humidity != null) {\n path = \"/features\";\n value = {\n temperature: {\n properties: {\n value: temperature\n }\n },\n humidity: {\n properties: {\n value: humidity\n }\n }\n };\n } else if (temperature != null) {\n path = \"/features/temperature/properties/value\";\n value = temperature;\n } else if (humidity != null) {\n path = \"/features/humidity/properties/value\";\n value = humidity;\n }\n \n if (!path || !value) {\n return null;\n }\n\n return Ditto.buildDittoProtocolMsg(\n \"org.eclipse.ditto\",\n headers[\"device_id\"],\n \"things\",\n \"twin\",\n \"commands\",\n \"modify\",\n path,\n headers,\n value\n );\n}" Create the connection We use the payload of the previous “test connection” command and add the JavaScript mapping script from above in order to specify the “create connection” command, which we will use to create the connection between Eclipse Hono and Ditto: $ curl -X POST -i -u devops:devopsPw1! -H 'Content-Type: application/json' -d '{ "targetActorSelection": "/system/sharding/connection", "headers": { "aggregate": false }, "piggybackCommand": { "type": "connectivity.commands:createConnection", "connection": { "id": "hono-sandbox-connection-1", "connectionType": "amqp-10", "connectionStatus": "open", "uri": "amqp://consumer@HONO:[email protected]:15672", "failoverEnabled": true, "sources": [{ "addresses": [ "telemetry/org.eclipse.ditto", "event/org.eclipse.ditto" ], "authorizationContext": ["nginx:demo5"], }], "mappingContext": { "mappingEngine": "JavaScript", "options": { "incomingScript": "function mapToDittoProtocolMsg(\n headers,\n textPayload,\n bytePayload,\n contentType\n) {\n\n if (contentType !== \"application/json\") {\n return null;\n }\n\n var jsonData = JSON.parse(textPayload);\n var temperature = jsonData.temp;\n var humidity = jsonData.hum;\n \n var path;\n var value;\n if (temperature != null && humidity != null) {\n path = \"/features\";\n value = {\n temperature: {\n properties: {\n value: temperature\n }\n },\n humidity: {\n properties: {\n value: humidity\n }\n }\n };\n } else if (temperature != null) {\n path = \"/features/temperature/properties/value\";\n value = temperature;\n } else if (humidity != null) {\n path = \"/features/humidity/properties/value\";\n value = humidity;\n }\n \n if (!path || !value) {\n return null;\n }\n\n return Ditto.buildDittoProtocolMsg(\n \"org.eclipse.ditto\",\n headers[\"device_id\"],\n \"things\",\n \"twin\",\n \"commands\",\n \"modify\",\n path,\n headers,\n value\n );\n}" } } } } }' https://ditto.eclipse.org/devops/piggyback/connectivity?timeout=8000 When establishing the connection + parsing the JavaScript worked, we get a success result as HTTP response again, otherwise an error message would be returned. Retrieve connection metrics After the connection was created, we can use the following command in order to retrieve the current connection status and some metrics about how many messages were consumed: $ curl -X POST -i -u devops:devopsPw1! -H 'Content-Type: application/json' -d '{ "targetActorSelection": "/system/sharding/connection", "headers": { "aggregate": false }, "piggybackCommand": { "type": "connectivity.commands:retrieveConnectionMetrics", "connectionId": "hono-sandbox-connection-1" } }' https://ditto.eclipse.org/devops/piggyback/connectivity?timeout=8000 The result looks like this: { "?": { "-1": { "type": "connectivity.responses:aggregatedResponse", "status": 200, "connectionId": "hono-sandbox-connection-1", "responsesType": "connectivity.responses:retrieveConnectionMetrics", "responses": { "1": { "type": "connectivity.responses:retrieveConnectionMetrics", "status": 200, "connectionId": "hono-sandbox-connection-1", "connectionMetrics": { "connectionStatus": "open", "connectionStatusDetails": "Connected at 2018-04-30T12:43:13.050Z", "inConnectionStatusSince": "2018-04-30T12:43:13.050Z", "clientState": "CONNECTED", "sourcesMetrics": [ { "addressMetrics": { "event/org.eclipse.ditto-0": { "status": "open", "statusDetails": "Started at 2018-04-30T12:43:13.038Z", "messageCount": 0 }, "telemetry/org.eclipse.ditto-0": { "status": "open", "statusDetails": "Started at 2018-04-30T12:43:13.039Z", "messageCount": 2, "lastMessageAt": "2018-04-30T12:51:12.537Z" } }, "consumedMessages": 2 } ], "targetsMetrics": [] } } } } } } Test the integration Whenever the device now sends telemetry in its own JSON format the message count of the connection metrics in Ditto should be increased by one the digital twin with the Thing ID org.eclipse.ditto:demo-device should receive the updated value which is also reflected at the twin’s HTTP endpoint https://ditto.eclipse.org/api/2/things/org.eclipse.ditto:demo-device Verify that by simulate sending telemetry using the Hono HTTP adapter: $ curl -X POST -i -u [email protected]:demo-device-password -H 'Content-Type: application/json' -d '{"temp": 14.51}' http://hono.eclipse.org:8080/telemetry $ curl -X POST -i -u [email protected]:demo-device-password -H 'Content-Type: application/json' -d '{"hum": 52.17}' http://hono.eclipse.org:8080/telemetry $ curl -X POST -i -u [email protected]:demo-device-password -H 'Content-Type: application/json' -d '{"temp": 23.07, "hum": 45.85}' http://hono.eclipse.org:8080/telemetry Try it out for yourself and give us (the Ditto and the Hono teams) feedback what you like or what could be improved. – The Eclipse Ditto team [Less]

With the recently released Ditto milestone 0.3.0-M1 the connectivity to AMQP 1.0 endpoints can now be established in a durable and stable way (including failovers, etc.). That means Ditto now is ready to be connected to Eclipse Hono’s “northbound” ... [More] API which is provided via AMQP 1.0. By doing so it is for example possible to receive Hono telemetry messages (see heading “Northbound Operations”) which a device demo-device connected to the “southbound” of Hono sends via HTTP or MQTT (the currently available protocol adapters of Hono) in Ditto. When received, the payload can be translated into a format Ditto understands in order to update the digital twin of the demo-device device and provide API access to the twin, e.g. via HTTP or WebSocket. This blog post walks through the steps required to connect Ditto and Hono by adding a connection between the Hono and Ditto sandboxes at hono.eclipse.org ditto.eclipse.org Scenario The following graphic illustrates the scenario: Scenario for providing a digital twin in Ditto of a device connected via Hono Let’s assume for this tutorial that we have a device (e.g. containing a sensor) demo-device which is capable of measuring temperature and humidity. This device sends the sensor telemetry data every 5 minutes via MQTT into the cloud in either of the following formats: { "temp": 23.42, "hum": 44.42 } { "temp": 23.42 } { "hum": 44.42 } We want to create a digital twin for this device in order to access the device’s sensor data as API via Eclipse Ditto. Steps in Hono The steps in order to get started with Eclipse Hono are documented in the Hono getting started and in a new Blog post about using multi-tenancy in Eclipse Hono. We show them very briefly here as well but in order to comprehend what and why we are doing what we do please consult the Hono documentation. Create a tenant First of all, create a new Hono tenant (we chose the tenant name org.eclipse.ditto): $ curl -X POST -i -H 'Content-Type: application/json' -d '{"tenant-id": "org.eclipse.ditto"}' http://hono.eclipse.org:28080/tenant Register a device Register a new device in Hono (we chose the device-id demo-device): $ curl -X POST -i -H 'Content-Type: application/json' -d '{"device-id": "demo-device"}' http://hono.eclipse.org:28080/registration/org.eclipse.ditto Add a device credential In order for the device being able to send telemetry it needs to authenticate. For that we will need to add a credential for that device in Hono. We choose the hashed-password type: $ PWD_HASH=$(echo -n 'demo-device-password' | openssl dgst -binary -sha512 | base64 -w 0) $ curl -X POST -i -H 'Content-Type: application/json' -d '{ "device-id": "demo-device", "type": "hashed-password", "auth-id": "demo-device-auth", "secrets": [{ "hash-function" : "sha-512", "pwd-hash": "'$PWD_HASH'" }] }' http://hono.eclipse.org:28080/credentials/org.eclipse.ditto Publish data You are now able to publish telemetry (or also event) data via the Hono HTTP adapter: $ curl -X POST -i -u [email protected]:demo-device-password -H 'Content-Type: application/json' -d '{"temp": 23.07}' http://hono.eclipse.org:8080/telemetry $ curl -X POST -i -u [email protected]:demo-device-password -H 'Content-Type: application/json' -d '{"hum": 45.85}' http://hono.eclipse.org:8080/telemetry However as there is not yet a consumer listening for the messages, the Hono HTTP adapter will for example return an error code 503 - Service unavailable when publishing a telemetry message. Alternatively you can also publish telemetry data via MQTT: $ mosquitto_pub -u '[email protected]' -P demo-device-password -t telemetry -m '{"temp": 23.07}' $ mosquitto_pub -u '[email protected]' -P demo-device-password -t telemetry -m '{"hum": 45.85}' In the following steps we will register the missing consumer in Ditto by creating a connection to the Hono tenant in Ditto’s connectivity. Steps in Ditto We want to create a digital twin of the device connected to Eclipse Hono in order to access its latest reported state via the Ditto HTTP API, in order to be able to find it in a population of digital twins or in order to be notified about changed via an API optimized for the web. Create a digital twin The first step is to create a skeleton for the digital twin by creating a Ditto Thing. Notice that we authenticate with the sandbox user demo5 - a default Policy is implicitly created so that only that user may read+write the created Thing. $ curl -X PUT -i -u demo5:demo -H 'Content-Type: application/json' -d '{ "attributes": { "location": "Germany" }, "features": { "temperature": { "properties": { "value": null } }, "humidity": { "properties": { "value": null } } } }' https://ditto.eclipse.org/api/2/things/org.eclipse.ditto:demo-device Make sure the digital twin was created: $ curl -i -u demo5:demo https://ditto.eclipse.org/api/2/things/org.eclipse.ditto:demo-device Create a connection to Hono Ditto recently added support for managing connections to foreign endpoints (currently to AMQP 1.0 or to AMQP 0.9.1). As Hono provides an AMQP 1.0 endpoint, a connection can be added in Ditto which connects to Hono and acts as a “northbound” consumer. The following configuration for the connection has to be applied: AMQP 1.0 hostname: hono.eclipse.org AMQP 1.0 port: 15672 username: consumer@HONO password: verysecret sources: telemetry/org.eclipse.ditto event/org.eclipse.ditto Test the connection Send the following “test connection” command via HTTP in order to test if the Ditto sandbox can connect to the Hono one. $ curl -X POST -i -u devops:devopsPw1! -H 'Content-Type: application/json' -d '{ "targetActorSelection": "/system/sharding/connection", "headers": { "aggregate": false }, "piggybackCommand": { "type": "connectivity.commands:testConnection", "connection": { "id": "hono-sandbox-connection-1", "connectionType": "amqp-10", "connectionStatus": "open", "uri": "amqp://consumer%40HONO:[email protected]:15672", "failoverEnabled": true, "sources": [{ "addresses": [ "telemetry/org.eclipse.ditto", "event/org.eclipse.ditto" ], "authorizationContext": ["nginx:demo5"] }] } } }' https://ditto.eclipse.org/devops/piggyback/connectivity?timeout=8000 The result should be: { "?": { "-1": { "type": "connectivity.responses:testConnection", "status": 200, "connectionId": "hono-sandbox-connection-1", "testResult": "ditto-cluster=Success(successfully connected + initialized mapper)" } } } Great, it looks like with the provided credentials we can connect to the Hono sandbox. Define a payload mapping In the scenario we described the payloads our device sends via MQTT. As those JSON payloads are missing some information required for Ditto to map it to a Ditto Protocol message Ditto uses for updating the digital twin, we have to configure a payload mapping in order to add the missing information. Whenever one of the 3 following messages arrives at Ditto’s consumer, a payload mapping should be performed: { "temp": 23.42, "hum": 44.42 } { "temp": 23.42 } { "hum": 44.42 } A JavaScript based mapping which exactly does this could look like this: function mapToDittoProtocolMsg( headers, textPayload, bytePayload, contentType ) { if (contentType !== "application/json") { return null; // only handle messages with content-type application/json } var jsonData = JSON.parse(textPayload); var temperature = jsonData.temp; var humidity = jsonData.hum; var path; var value; if (temperature != null && humidity != null) { path = "/features"; value = { temperature: { properties: { value: temperature } }, humidity: { properties: { value: humidity } } }; } else if (temperature != null) { path = "/features/temperature/properties/value"; value = temperature; } else if (humidity != null) { path = "/features/humidity/properties/value"; value = humidity; } if (!path || !value) { return null; } return Ditto.buildDittoProtocolMsg( "org.eclipse.ditto", // the namespace we use headers["device_id"], // Hono sets the authenticated device-id in this header "things", // it is a Thing entity we want to update "twin", // we want to update the twin "commands", "modify", // command = modify path, headers, // copy all headers as Ditto headers value ); } In order to add this script to the connection we want to create, the newlines have to be replaced by \n so that the script fits in a single line JSON string and the " characters have to be replaced with \": "function mapToDittoProtocolMsg(\n headers,\n textPayload,\n bytePayload,\n contentType\n) {\n\n if (contentType !== \"application/json\") {\n return null;\n }\n\n var jsonData = JSON.parse(textPayload);\n var temperature = jsonData.temp;\n var humidity = jsonData.hum;\n \n var path;\n var value;\n if (temperature != null && humidity != null) {\n path = \"/features\";\n value = {\n temperature: {\n properties: {\n value: temperature\n }\n },\n humidity: {\n properties: {\n value: humidity\n }\n }\n };\n } else if (temperature != null) {\n path = \"/features/temperature/properties/value\";\n value = temperature;\n } else if (humidity != null) {\n path = \"/features/humidity/properties/value\";\n value = humidity;\n }\n \n if (!path || !value) {\n return null;\n }\n\n return Ditto.buildDittoProtocolMsg(\n \"org.eclipse.ditto\",\n headers[\"device_id\"],\n \"things\",\n \"twin\",\n \"commands\",\n \"modify\",\n path,\n headers,\n value\n );\n}" Create the connection We use the payload of the previous “test connection” command and add the JavaScript mapping script from above in order to specify the “create connection” command, which we will use to create the connection between Eclipse Hono and Ditto: $ curl -X POST -i -u devops:devopsPw1! -H 'Content-Type: application/json' -d '{ "targetActorSelection": "/system/sharding/connection", "headers": { "aggregate": false }, "piggybackCommand": { "type": "connectivity.commands:createConnection", "connection": { "id": "hono-sandbox-connection-1", "connectionType": "amqp-10", "connectionStatus": "open", "uri": "amqp://consumer%40HONO:[email protected]:15672", "failoverEnabled": true, "sources": [{ "addresses": [ "telemetry/org.eclipse.ditto", "event/org.eclipse.ditto" ], "authorizationContext": ["nginx:demo5"] }], "mappingContext": { "mappingEngine": "JavaScript", "options": { "incomingScript": "function mapToDittoProtocolMsg(\n headers,\n textPayload,\n bytePayload,\n contentType\n) {\n\n if (contentType !== \"application/json\") {\n return null;\n }\n\n var jsonData = JSON.parse(textPayload);\n var temperature = jsonData.temp;\n var humidity = jsonData.hum;\n \n var path;\n var value;\n if (temperature != null && humidity != null) {\n path = \"/features\";\n value = {\n temperature: {\n properties: {\n value: temperature\n }\n },\n humidity: {\n properties: {\n value: humidity\n }\n }\n };\n } else if (temperature != null) {\n path = \"/features/temperature/properties/value\";\n value = temperature;\n } else if (humidity != null) {\n path = \"/features/humidity/properties/value\";\n value = humidity;\n }\n \n if (!path || !value) {\n return null;\n }\n\n return Ditto.buildDittoProtocolMsg(\n \"org.eclipse.ditto\",\n headers[\"device_id\"],\n \"things\",\n \"twin\",\n \"commands\",\n \"modify\",\n path,\n headers,\n value\n );\n}" } } } } }' https://ditto.eclipse.org/devops/piggyback/connectivity?timeout=8000 When establishing the connection + parsing the JavaScript worked, we get a success result as HTTP response again, otherwise an error message would be returned. Retrieve connection metrics After the connection was created, we can use the following command in order to retrieve the current connection status and some metrics about how many messages were consumed: $ curl -X POST -i -u devops:devopsPw1! -H 'Content-Type: application/json' -d '{ "targetActorSelection": "/system/sharding/connection", "headers": { "aggregate": false }, "piggybackCommand": { "type": "connectivity.commands:retrieveConnectionMetrics", "connectionId": "hono-sandbox-connection-1" } }' https://ditto.eclipse.org/devops/piggyback/connectivity?timeout=8000 The result looks like this: { "?": { "?": { "type": "connectivity.responses:retrieveConnectionMetrics", "status": 200, "connectionId": "hono-sandbox-connection-1", "containsFailures": false, "connectionMetrics": { "inbound": { "consumed": { "success": { "PT1M": 2, "PT1H": 2, "PT24H": 2, "lastMessageAt": "2019-02-06T09:37:28.416Z" }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "mapped": { "success": { "PT1M": 2, "PT1H": 2, "PT24H": 2, "lastMessageAt": "2019-02-06T09:37:28.422Z" }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "dropped": { "success": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "enforced": { "success": { "PT1M": 2, "PT1H": 2, "PT24H": 2, "lastMessageAt": "2019-02-06T09:37:28.422Z" }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } } }, "outbound": { "dispatched": { "success": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null }, "failure": { "PT1M": 2, "PT1H": 2, "PT24H": 2, "lastMessageAt": "2019-02-06T09:37:28.439Z" } }, "filtered": { "success": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "mapped": { "success": { "PT1M": 2, "PT1H": 2, "PT24H": 2, "lastMessageAt": "2019-02-06T09:37:28.443Z" }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "dropped": { "success": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "published": { "success": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } } } }, "sourceMetrics": { "addressMetrics": { "event/org.eclipse.ditto": { "consumed": { "success": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "mapped": { "success": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "dropped": { "success": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "enforced": { "success": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } } }, "telemetry/org.eclipse.ditto": { "consumed": { "success": { "PT1M": 2, "PT1H": 2, "PT24H": 2, "lastMessageAt": "2019-02-06T09:37:28.416Z" }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "mapped": { "success": { "PT1M": 2, "PT1H": 2, "PT24H": 2, "lastMessageAt": "2019-02-06T09:37:28.422Z" }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "dropped": { "success": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "enforced": { "success": { "PT1M": 2, "PT1H": 2, "PT24H": 2, "lastMessageAt": "2019-02-06T09:37:28.422Z" }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } } } } }, "targetMetrics": { "addressMetrics": { "_responses": { "dispatched": { "success": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null }, "failure": { "PT1M": 2, "PT1H": 2, "PT24H": 2, "lastMessageAt": "2019-02-06T09:37:28.439Z" } }, "filtered": { "success": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "mapped": { "success": { "PT1M": 2, "PT1H": 2, "PT24H": 2, "lastMessageAt": "2019-02-06T09:37:28.443Z" }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "dropped": { "success": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } }, "published": { "success": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null }, "failure": { "PT1M": 0, "PT1H": 0, "PT24H": 0, "lastMessageAt": null } } } } } } } } Test the integration Whenever the device now sends telemetry in its own JSON format the message count of the connection metrics in Ditto should be increased by one the digital twin with the Thing ID org.eclipse.ditto:demo-device should receive the updated value which is also reflected at the twin’s HTTP endpoint https://ditto.eclipse.org/api/2/things/org.eclipse.ditto:demo-device Verify that by simulate sending telemetry using the Hono HTTP adapter: $ curl -X POST -i -u [email protected]:demo-device-password -H 'Content-Type: application/json' -d '{"temp": 14.51}' http://hono.eclipse.org:8080/telemetry $ curl -X POST -i -u [email protected]:demo-device-password -H 'Content-Type: application/json' -d '{"hum": 52.17}' http://hono.eclipse.org:8080/telemetry $ curl -X POST -i -u [email protected]:demo-device-password -H 'Content-Type: application/json' -d '{"temp": 23.07, "hum": 45.85}' http://hono.eclipse.org:8080/telemetry Try it out for yourself and give us (the Ditto and the Hono teams) feedback what you like or what could be improved. – The Eclipse Ditto team [Less]