Eclipse Ditto™

Motivation Ditto 1.2.0 introduced at-least-once delivery via acknowledgement requests. It increased coupling between the publisher and the subscriber of signals in that the subscriber is no longer at the liberty to filter for signals it is ... [More] interested in. Instead, the subscriber must consume all signals in order to fulfill acknowledgement requests and prevent endless redelivery. To combat the problem, Ditto 1.4.0 made acknowledgement labels unique and introduced the requirement to manage declared acknowledgements, identifying of each subscriber. It is now possible for Ditto to issue weak acknowledgements on behalf of the subscriber whenever it decides to not consume a signal. That allows subscribers to configure RQL and namespace filters freely without causing any futile redelivery. Note: Weak acknowledgements are available since Ditto 1.5.0. What it is A weak acknowledgement is issued by Ditto for any acknowledgement request that will not be fulfilled now or ever without configuration change. A weak acknowledgement is identified by the header ditto-weak-ack: true. The status code of weak acknowledgements is 200 OK; it signifies that any redelivery is not to be made on their account. A weak acknowledgement may look like this in Ditto protocol: { "topic": "com.acme/xdk_53/things/twin/acks/my-mqtt-connection:my-mqtt-topic", "headers": { "ditto-weak-ack": true }, "path": "/", "value": "Acknowledgement was issued automatically, because the subscriber is not authorized to receive the signal.", "status": 200 } How it works Since Ditto 1.4.0, subscribers of twin events or live signals are required to declare unique acknowledgement labels they are allowed to send. The labels of acknowledgement requests are then identifying the intended subscribers. If the intended subscriber exists but does not receive the signal for non-transient reasons, Ditto issues a weak acknowledgement for that subscriber. Such reasons may be: The intended subscriber is not authorized to receive the signal by policy; The intended subscriber did not subscribe for the signal type (twin event, live command, live event or live message); The intended subscriber filtered the signal out by its namespace or RQL filter; The intended subscriber dropped the signal because its payload mapper produced nothing. Limitation The distributed nature of cluster pub/sub means that weak acknowledgements are not always issued correctly. They are only eventually correct in the sense that some time after a change to the publisher-subscriber pair, the issued weak acknowledgements will reflect the change. Such changes include: Opening and closing of Websocket or other connections acting as the subscriber; Subscribing and unsubscribing for different signal types via Websocket; Modification of connections via the connectivity API; Migration of a connection from one Ditto cluster member to another due to load balancing. Feedback? Please get in touch if you have feedback or questions towards this new concept of weak acknowledgements. – The Eclipse Ditto team [Less]

Desired feature properties added to things model With the release of Eclipse Ditto version 1.5.0 desired feature properties are introduced to the things model for API versions later than 1. The desired properties for features are added on the same ... [More] level of the model as the feature properties and can reflect for example feature property updates ,which are intended, but not yet applied. Further logics for desired feature properties might be implemented in future Ditto versions. A fully-fledged JSON representation of a feature with desired properties is shown below: { "lamp": { "definition": [ "com.mycompany.fb:Lamp:1.0.0" ], "properties": { "configuration": { "on": true, "location": { "longitude": 34.052235, "latitude": -118.243683 } }, "status": { "on": false, "color": { "red": 128, "green": 255, "blue": 0 } } }, "desiredProperties": { "configuration": { "on": false } } } } Operations on desired feature properties CRUD operations You can create multiple desired properties of a feature or just single ones. You can retrieve all desired properties of a feature or just single ones. You can modify all desired properties of a feature or just single ones. You can delete all desired properties of a feature or just single ones. Search You can search for things with specific desired properties with RQL-functions. You can search for things, which have existent desired properties for a feature. Get notified on changes You can receive events for changes done to the desired properties of things you’re authorized to read. You can enrich and filter the events you want to receive, for changes done to the desired properties. Executing CRUD operations on desired feature properties CRUD operations can be executed either via the Ditto HTTP API versions later than 1 or via ditto-protocol messages. Possible CRUD operations for desired feature properties via ditto-protocol: Retrieve all desired properties of a feature via ditto-protocol Retrieve a single desired property of a feature via ditto-protocol Create/Modify all desired properties of a feature via ditto-protocol Create/Modify a single desired property of a feature via ditto-protocol Delete all desired properties of a feature via ditto-protocol Delete a single desired property of a feature via ditto-protocol Using the ditto-client to manage desired feature properties The desired feature properties can also be retrieved, modified and deleted via the Ditto Java Client. At the time of publishing of this blog post (Ditto Java Client version 1.5.0), no special CRUD operations for desired feature properties are implemented in the client. Thus, the operations have to be executed via creating ditto-protocol messages manually in the client. Example for creating/modifying desired feature properties of a thing via the ditto-client: final Adaptable modifyFeatureDesiredProperties = Adaptable.newBuilder(TopicPath.newBuilder(ThingId.of("com.mycompany.fb:Car:1.0.0")) .things() .twin() .commands() .modify() .build()) .withPayload(Payload.newBuilder( JsonPointer.of("/features/lamp/desiredProperties")) .withValue(JsonObject.newBuilder().set("on", false).build()) .build()).build(); client.sendDittoProtocol(modifyFeatureDesiredProperties).whenComplete(((adaptable, throwable) -> { if (throwable != null) { LOGGER.error("Received error while sending ModifyFeatureDesiredProperties: '{}' ", throwable.toString()); } else { LOGGER.info("Received response for ModifyFeatureDesiredProperties: '{}'", adaptable); } })); – The Eclipse Ditto team [Less]

Desired feature properties added to things model With the upcoming release of Eclipse Ditto version 1.5.0 desired feature properties are introduced to the things model for API versions later than 1. The desired properties for features are added on ... [More] the same level of the model as the feature properties and can reflect for example feature property updates ,which are intended, but not yet applied. Note: Further logics for desired feature properties might be implemented in future Ditto versions. A fully-fledged JSON representation of a feature with desired properties is shown below: { "lamp": { "definition": [ "com.mycompany.fb:Lamp:1.0.0" ], "properties": { "configuration": { "on": true, "location": { "longitude": 34.052235, "latitude": -118.243683 } }, "status": { "on": false, "color": { "red": 128, "green": 255, "blue": 0 } } }, "desiredProperties": { "configuration": { "on": false } } } } Operations on desired feature properties CRUD operations You can create multiple desired properties of a feature or just single ones. You can retrieve all desired properties of a feature or just single ones. You can modify all desired properties of a feature or just single ones. You can delete all desired properties of a feature or just single ones. Search You can search for things with specific desired properties with RQL-functions. You can search for things, which have existent desired properties for a feature. Get notified on changes You can receive events for changes done to the desired properties of things you’re authorized to read. You can enrich and filter the events you want to receive, for changes done to the desired properties. Executing CRUD operations on desired feature properties CRUD operations can be executed either via the Ditto HTTP API versions later than 1 or via ditto-protocol messages. Possible CRUD operations for desired feature properties via ditto-protocol: Retrieve all desired properties of a feature via ditto-protocol Retrieve a single desired property of a feature via ditto-protocol Create/Modify all desired properties of a feature via ditto-protocol Create/Modify a single desired property of a feature via ditto-protocol Delete all desired properties of a feature via ditto-protocol Delete a single desired property of a feature via ditto-protocol Using the ditto-client to manage desired feature properties The desired feature properties can also be retrieved, modified and deleted via the Ditto Java Client. With the upcoming (Ditto Java Client version 1.5.0), no special CRUD operations for desired feature properties are implemented in the client. Thus, the operations have to be executed via creating ditto-protocol messages manually in the client. Example for creating/modifying desired feature properties of a thing via the ditto-client: final Adaptable modifyFeatureDesiredProperties = Adaptable.newBuilder(TopicPath.newBuilder(ThingId.of("com.mycompany.fb:Car:1.0.0")) .things() .twin() .commands() .modify() .build()) .withPayload(Payload.newBuilder( JsonPointer.of("/features/lamp/desiredProperties")) .withValue(JsonObject.newBuilder().set("on", false).build()) .build()).build(); client.sendDittoProtocol(modifyFeatureDesiredProperties).whenComplete(((adaptable, throwable) -> { if (throwable != null) { LOGGER.error("Received error while sending ModifyFeatureDesiredProperties: '{}' ", throwable.toString()); } else { LOGGER.info("Received response for ModifyFeatureDesiredProperties: '{}'", adaptable); } })); Feedback? Please get in touch if you have feedback or questions towards this new concept of desired properties. – The Eclipse Ditto team [Less]

Desired feature properties added to things model With the upcoming release of Eclipse Ditto version 1.5.0 desired feature properties are introduced to the things model for API versions later than 1. The desired properties for features are added on ... [More] the same level of the model as the feature properties and can reflect for example feature property updates ,which are intended, but not yet applied. Note: Further logics for desired feature properties might be implemented in future Ditto versions. A fully-fledged JSON representation of a feature with desired properties is shown below: { "lamp": { "definition": [ "com.mycompany.fb:Lamp:1.0.0" ], "properties": { "configuration": { "on": true, "location": { "longitude": 34.052235, "latitude": -118.243683 } }, "status": { "on": false, "color": { "red": 128, "green": 255, "blue": 0 } } }, "desiredProperties": { "configuration": { "on": false } } } } Operations on desired feature properties CRUD operations You can create multiple desired properties of a feature or just single ones. You can retrieve all desired properties of a feature or just single ones. You can modify all desired properties of a feature or just single ones. You can delete all desired properties of a feature or just single ones. Search You can search for things with specific desired properties with RQL-functions. You can search for things, which have existent desired properties for a feature. Get notified on changes You can receive events for changes done to the desired properties of things you’re authorized to read. You can enrich and filter the events you want to receive, for changes done to the desired properties. Executing CRUD operations on desired feature properties CRUD operations can be executed either via the Ditto HTTP API versions later than 1 or via ditto-protocol messages. Possible CRUD operations for desired feature properties via ditto-protocol: Retrieve all desired properties of a feature via ditto-protocol Retrieve a single desired property of a feature via ditto-protocol Create/Modify all desired properties of a feature via ditto-protocol Create/Modify a single desired property of a feature via ditto-protocol Delete all desired properties of a feature via ditto-protocol Delete a single desired property of a feature via ditto-protocol Using the ditto-client to manage desired feature properties The desired feature properties can also be retrieved, modified and deleted via the Ditto Java Client. With the upcoming (Ditto Java Client version 1.5.0), no special CRUD operations for desired feature properties are implemented in the client. Thus, the operations have to be executed via creating ditto-protocol messages manually in the client. Example for creating/modifying desired feature properties of a thing via the ditto-client: final Adaptable modifyFeatureDesiredProperties = Adaptable.newBuilder(TopicPath.newBuilder(ThingId.of("com.mycompany.fb:Car:1.0.0")) .things() .twin() .commands() .modify() .build()) .withPayload(Payload.newBuilder( JsonPointer.of("/features/lamp/desiredProperties")) .withValue(JsonObject.newBuilder().set("on", false).build()) .build()).build(); client.sendDittoProtocol(modifyFeatureDesiredProperties).whenComplete(((adaptable, throwable) -> { if (throwable != null) { LOGGER.error("Received error while sending ModifyFeatureDesiredProperties: '{}' ", throwable.toString()); } else { LOGGER.info("Received response for ModifyFeatureDesiredProperties: '{}'", adaptable); } })); Feedback? Please get in touch if you have feedback or questions towards this new concept of desired properties. – The Eclipse Ditto team [Less]

Today, the Ditto team is happy to announce the next feature update of Ditto 1.x: Eclipse Ditto 1.4.0 1.4.0 focuses on: Declaration of acknowledgement labels unique to each subscriber Please have a look at the 1.4.0 release notes for a more ... [More] detailed information on the release. Also, some bugs were fixed which are not backported to Ditto 1.3.0 - it is recommended to update to Ditto 1.4.0 right away and skip 1.3.0. Artifacts The new Java artifacts have been published at the Eclipse Maven repository as well as Maven central. Also the Ditto Java client’s artifacts were published to Maven central. The Docker images have been pushed to Docker Hub: eclipse/ditto-policies eclipse/ditto-things eclipse/ditto-things-search eclipse/ditto-gateway eclipse/ditto-connectivity eclipse/ditto-concierge Kubernetes ready: Helm chart In order to run Eclipse Ditto in a Kubernetes environment, best rely on the official Helm chart and deploy Ditto via the Helm package manager. – The Eclipse Ditto team [Less]

Today, the Ditto team is happy to announce the next feature update of Ditto 1.x: Eclipse Ditto 1.4.0 1.4.0 focuses on: Declaration of acknowledgement labels unique to each subscriber Please have a look at the 1.4.0 release notes for a more ... [More] detailed information on the release. Also, some bugs were fixed which are not backported to Ditto 1.3.0 - it is recommended to update to Ditto 1.4.0 right away and skip 1.3.0. Artifacts The new Java artifacts have been published at the Eclipse Maven repository as well as Maven central. Also the Ditto Java client’s artifacts were published to Maven central. The Docker images have been pushed to Docker Hub: eclipse/ditto-policies eclipse/ditto-things eclipse/ditto-things-search eclipse/ditto-gateway eclipse/ditto-connectivity eclipse/ditto-concierge Kubernetes ready: Helm chart In order to run Eclipse Ditto in a Kubernetes environment, best rely on the official Helm chart and deploy Ditto via the Helm package manager. – The Eclipse Ditto team [Less]

E2E acknowledgement using Eclipse Ditto By adding the new acknowledgements feature to Ditto, it is now possible to provide an end to end QoS (quality of service) with level 1. Previously the connectivity service of Ditto did accept all incoming ... [More] messages immediately as soon as it received the message. This behaviour can now be controlled by a Ditto header called requested-acks. In this blog post I will provide a few examples for how to use this new feature in the following scenarios: Device publishes a change of its state and doesn’t care about whether the event regarding this change will be received by any subscriber or not. This is the fire and forget scenario. Device publishes a change of its state and wants to be sure it is persisted in Ditto. This is the processed in Ditto scenario. Device publishes a change of its state and wants to be sure an application connected to Eclipse Ditto will receive the event regarding this change of the state. This is the notification scenario. Device publishes a change of its state and wants to be notified when a subscriber was not able to receive the event regarding this change. This is the tracked notification scenario. Device sends a live message and wants to be sure that it will eventually receive a response. This is the long running live message scenario. Device sends a live message and wants to either receive the response within a given timeout or never. This is the asking for required information scenario. Scenarios The following scenarios all share the same context: a device sends data/messages via a device connectivity layer (e.g. Eclipse Hono or an MQTT broker) to Ditto Ditto’s connectivity service consumes the Ditto Protocol message forwards the message into the Ditto cluster to be processed is responsible for technically acknowledging/settling the consumed message at the device connectivity layer / broker the strategy of when this is done and with which outcome is handled by the Ditto headers mentioned in the scenarios Although all scenarios are based on the device sending something (e.g. telemetry data or a message), the E2E acknowledgement can also be used the other way around when e.g. a backend application sends something to a device. Scenario 1 - Fire and Forget This is the simplest scenario of all, since the change can be published in a fire and forget semantics. In this scenario the device will send the modification command containing the headers: response-required=false requested-acks=[] Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/twin/commands/modify", "headers": { "response-required": false, "requested-acks": [] }, "path": "/features/lightSwitch/properties/status", "value": "on" } For this case the connectivity service will immediately acknowledge the incoming message at the messaging system and then continues to process the command. It doesn’t matter if the command could be processed successfully or if any subscriber received an event for this change. Scenario 2 - Processed in Ditto For this scenario the device wants to be sure its change will be properly persisted in Ditto. The command needs to define the following headers: response-required=false requested-acks=["twin-persisted"] Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/twin/commands/modify", "headers": { "response-required": false, "requested-acks": ["twin-persisted"] }, "path": "/features/lightSwitch/properties/status", "value": "on" } For this case the connectivity service will wait until the modification will be properly persisted in Ditto before acknowledging the incoming message at the messaging system. If the device published this command for example via an AMQP broker (with a QoS 1 “at least once” semantic), this will cause the broker to redeliver the command to Ditto if the acknowledgment fails. Please be aware that a redelivery will only be requested for the following error status codes: 408 (Request timed out) 424 (Dependency Failure) All kinds of 5xx status codes In this scenario it does matter if the command could be processed successfully, but it’s still not relevant if any subscriber received an event for this change. Scenario 3 - Notification scenario For this scenario the device wants to be sure another system will be notified about the change of its state. This could be for example an alarming system which wants to be sure a backend application receives the information that the alarm was triggered. A prerequisite for this is that any kind of connection exists that publishes the event to the backend application and declares a user defined acknowledgement label. This can be a WebSocket session or any kind of Ditto Connection types which can be found here. For this example we expect the event to be forwarded by an HTTP connection which declared the following acknowledgement label as issued acknowledgement of the target: d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed. The UUID prefix in this case is the ID of the HTTP connection and thebackend-process part is a custom label, defined by the user. The command needs to define the following headers: response-required=false requested-acks=["d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed"] timeout=30s (optional. Default is 10s.) Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/twin/commands/modify", "headers": { "response-required": false, "requested-acks": ["d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed"], "timeout": "30s" }, "path": "/features/alarm/properties/status", "value": "on" } For this case the connectivity service will wait until the HTTP request, which forwards the event regarding the thing change, will be finished before acknowledging the incoming message at the messaging system. The status code of the HTTP response will in this case determine if the message will be acknowledged successfully or not and if a redelivery will be requested or not. All kinds of 2xx status codes will lead to a successful acknowledgement at the messaging system. All other status codes will lead to a failed acknowledgement at the messaging system and for the following status codes a redelivery will be requested: 408 (Request timed out) 424 (Dependency Failure) All kinds of 5xx status codes In this scenario it is ensured a specified subscriber will receive an event for this change. Scenario 4 - Tracked notification scenario For this scenario the device wants to know when a system could not be notified about the change of its state. This could be for example an alarming system which wants to be sure a backend application receives the information, or if not: tries to send an SMS as notification. This scenario is mostly like scenario 3, but needs to set the response-required header to true and it’s required to configure the reply-target of the source to also expect “nack” responses. Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/twin/commands/modify", "headers": { "response-required": true, "requested-acks": ["d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed"] }, "path": "/features/alarm/properties/status", "value": "on" } In this case the device will receive an acknowledgement response containing the status code and payload of the response of the backend application. Based on this the device can decide how to handle the situation. It is suggested to publish the modification command with QoS 0 (“at most once” semantics) in this case because the device handles the result of the E2E acknowledgement. With QoS 1 (“at least once” semantics) brokers would usually redeliver the message to Ditto. If the HTTP endpoint of the backend application responds with the following response: headers: "content-type": "application/json" body: { "errorCode": "notification.smartphone.failed", "message": "Could not notify smartphone." } status code: 424 The response received at the device would look like this: { "topic": "org.eclipse.ditto/my-thing/things/twin/acks/d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed", "headers": { "response-required": false, "requested-acks": ["d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed"], "timeout": "30s", "content-type": "application/json" }, "path": "/", "value": { "errorCode": "notification.smartphone.failed", "message": "Could not notify smartphone." }, "status": 424 } Scenario 5 - Long running live message scenario For this scenario the device is going to ask for an information which it needs eventually. Let’s say the device asks for the endpoint where it should download the new firmware from. It’s not required that this response arrives within a given time. It’s just required to eventually arrive at the device and after it was received the device can signal the user, that it is ready to download the firmware. The headers of the live message should have the following values: response-required=true requested-acks=["live-response"] Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/live/messages/firmware", "headers": { "response-required": true, "requested-acks": ["live-response"], "content-type": "text/plain", "timeout": "5s" }, "path": "/outbox/messages/firmware", "value": "firmware.url.query" } By requesting the acknowledgement live-response the connectivity service will wait until the response for the live message arrived for 5s before acknowledging the incoming message at the broker and will request a redelivery if the response did not arrive within this timeout. This will repeat until either the broker discards the message or the response arrives in the specified timeout. That way it is guaranteed that the device will eventually receive the response. Scenario 6 - Asking for required information scenario For this scenario the device is going to ask for an information which it needs right now to proceed with its current task. Let’s say the device asks if it should allow a car with a license plate it detected to drive on the property by opening the barrier. It could be possible to ask for that information, so the barrier opens automatically, but providing a fallback mechanism like entering a code directly at the device if this response does not arrive within time. The headers of the live message should have the following values: response-required=true requested-acks=[] Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/live/messages/car-enter", "headers": { "response-required": true, "requested-acks": [], "content-type": "text/plain", "timeout": "5s" }, "path": "/outbox/messages/car-enter", "value": "FN IB 1337" } By requesting explicitly not requesting any acknowledgement but still requiring a response, the connectivity service will immediately acknowledge the incoming message at the broker. The device will then either receive the response within the specified timeout or never. So the device can provide its alternative options to open the barrier after 5 seconds. We embrace your feedback I hope I could demonstrate the power of the new acknowledgement feature properly and could make it clear how it can be used. Maybe you did recognize some of your use cases in the given examples or maybe you have another use case which can or cannot be solved by this feature. We would love to get your feedback. – The Eclipse Ditto team [Less]

E2E acknowledgement using Eclipse Ditto By adding the new acknowledgements feature to Ditto, it is now possible to provide an end to end QoS (quality of service) with level 1. Previously the connectivity service of Ditto did accept all incoming ... [More] messages immediately as soon as it received the message. This behaviour can now be controlled by a Ditto header called requested-acks. In this blog post I will provide a few examples for how to use this new feature in the following scenarios: Device publishes a change of its state and doesn’t care about whether the event regarding this change will be received by any subscriber or not. This is the fire and forget scenario. Device publishes a change of its state and wants to be sure it is persisted in Ditto. This is the processed in Ditto scenario. Device publishes a change of its state and wants to be sure an application connected to Eclipse Ditto will receive the event regarding this change of the state. This is the notification scenario. Device publishes a change of its state and wants to be notified when a subscriber was not able to receive the event regarding this change. This is the tracked notification scenario. Device sends a live message and wants to be sure that it will eventually receive a response. This is the long running live message scenario. Device sends a live message and wants to either receive the response within a given timeout or never. This is the asking for required information scenario. Scenarios The following scenarios all share the same context: a device sends data/messages via a device connectivity layer (e.g. Eclipse Hono or an MQTT broker) to Ditto Ditto’s connectivity service consumes the Ditto Protocol message forwards the message into the Ditto cluster to be processed is responsible for technically acknowledging/settling the consumed message at the device connectivity layer / broker the strategy of when this is done and with which outcome is handled by the Ditto headers mentioned in the scenarios Although all scenarios are based on the device sending something (e.g. telemetry data or a message), the E2E acknowledgement can also be used the other way around when e.g. a backend application sends something to a device. Scenario 1 - Fire and Forget This is the simplest scenario of all, since the change can be published in a fire and forget semantics. In this scenario the device will send the modification command containing the headers: response-required=false requested-acks=[] Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/twin/commands/modify", "headers": { "response-required": false, "requested-acks": [] }, "path": "/features/lightSwitch/properties/status", "value": "on" } For this case the connectivity service will immediately acknowledge the incoming message at the messaging system and then continues to process the command. It doesn’t matter if the command could be processed successfully or if any subscriber received an event for this change. Scenario 2 - Processed in Ditto For this scenario the device wants to be sure its change will be properly persisted in Ditto. The command needs to define the following headers: response-required=false requested-acks=["twin-persisted"] Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/twin/commands/modify", "headers": { "response-required": false, "requested-acks": ["twin-persisted"] }, "path": "/features/lightSwitch/properties/status", "value": "on" } For this case the connectivity service will wait until the modification will be properly persisted in Ditto before acknowledging the incoming message at the messaging system. If the device published this command for example via an AMQP broker (with a QoS 1 “at least once” semantic), this will cause the broker to redeliver the command to Ditto if the acknowledgment fails. Please be aware that a redelivery will only be requested for the following error status codes: 408 (Request timed out) 424 (Dependency Failure) All kinds of 5xx status codes In this scenario it does matter if the command could be processed successfully, but it’s still not relevant if any subscriber received an event for this change. Scenario 3 - Notification scenario For this scenario the device wants to be sure another system will be notified about the change of its state. This could be for example an alarming system which wants to be sure a backend application receives the information that the alarm was triggered. A prerequisite for this is that any kind of connection exists that publishes the event to the backend application and declares a user defined acknowledgement label. This can be a WebSocket session or any kind of Ditto Connection types which can be found here. For this example we expect the event to be forwarded by an HTTP connection which declared the following acknowledgement label as issued acknowledgement of the target: d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed. The UUID prefix in this case is the ID of the HTTP connection and thebackend-process part is a custom label, defined by the user. The command needs to define the following headers: response-required=false requested-acks=["d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed"] timeout=30s (optional. Default is 10s.) Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/twin/commands/modify", "headers": { "response-required": false, "requested-acks": ["d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed"], "timeout": "30s" }, "path": "/features/alarm/properties/status", "value": "on" } For this case the connectivity service will wait until the HTTP request, which forwards the event regarding the thing change, will be finished before acknowledging the incoming message at the messaging system. The status code of the HTTP response will in this case determine if the message will be acknowledged successfully or not and if a redelivery will be requested or not. All kinds of 2xx status codes will lead to a successful acknowledgement at the messaging system. All other status codes will lead to a failed acknowledgement at the messaging system and for the following status codes a redelivery will be requested: 408 (Request timed out) 424 (Dependency Failure) All kinds of 5xx status codes In this scenario it is ensured a specified subscriber will receive an event for this change. Scenario 4 - Tracked notification scenario For this scenario the device wants to know when a system could not be notified about the change of its state. This could be for example an alarming system which wants to be sure a backend application receives the information, or if not: tries to send an SMS as notification. This scenario is mostly like scenario 3, but needs to set the response-required header to true and it’s required to configure the reply-target of the source to also expect “nack” responses. Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/twin/commands/modify", "headers": { "response-required": true, "requested-acks": ["d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed"] }, "path": "/features/alarm/properties/status", "value": "on" } In this case the device will receive an acknowledgement response containing the status code and payload of the response of the backend application. Based on this the device can decide how to handle the situation. It is suggested to publish the modification command with QoS 0 (“at most once” semantics) in this case because the device handles the result of the E2E acknowledgement. With QoS 1 (“at least once” semantics) brokers would usually redeliver the message to Ditto. If the HTTP endpoint of the backend application responds with the following response: headers: "content-type": "application/json" body: { "errorCode": "notification.smartphone.failed", "message": "Could not notify smartphone." } status code: 424 The response received at the device would look like this: { "topic": "org.eclipse.ditto/my-thing/things/twin/acks/d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed", "headers": { "response-required": false, "requested-acks": ["d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed"], "timeout": "30s", "content-type": "application/json" }, "path": "/", "value": { "errorCode": "notification.smartphone.failed", "message": "Could not notify smartphone." }, "status": 424 } Scenario 5 - Long running live message scenario For this scenario the device is going to ask for an information which it needs eventually. Let’s say the device asks for the endpoint where it should download the new firmware from. It’s not required that this response arrives within a given time. It’s just required to eventually arrive at the device and after it was received the device can signal the user, that it is ready to download the firmware. The headers of the live message should have the following values: response-required=true requested-acks=["live-response"] Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/live/messages/firmware", "headers": { "response-required": true, "requested-acks": ["live-response"], "content-type": "text/plain", "timeout": "5s" }, "path": "/outbox/messages/firmware", "value": "firmware.url.query" } By requesting the acknowledgement live-response the connectivity service will wait until the response for the live message arrived for 5s before acknowledging the incoming message at the broker and will request a redelivery if the response did not arrive within this timeout. This will repeat until either the broker discards the message or the response arrives in the specified timeout. That way it is guaranteed that the device will eventually receive the response. Scenario 6 - Asking for required information scenario For this scenario the device is going to ask for an information which it needs right now to proceed with its current task. Let’s say the device asks if it should allow a car with a license plate it detected to drive on the property by opening the barrier. It could be possible to ask for that information, so the barrier opens automatically, but providing a fallback mechanism like entering a code directly at the device if this response does not arrive within time. The headers of the live message should have the following values: response-required=true requested-acks=[] Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/live/messages/car-enter", "headers": { "response-required": true, "requested-acks": [], "content-type": "text/plain", "timeout": "5s" }, "path": "/outbox/messages/car-enter", "value": "FN IB 1337" } By requesting explicitly not requesting any acknowledgement but still requiring a response, the connectivity service will immediately acknowledge the incoming message at the broker. The device will then either receive the response within the specified timeout or never. So the device can provide its alternative options to open the barrier after 5 seconds. We embrace your feedback I hope I could demonstrate the power of the new acknowledgement feature properly and could make it clear how it can be used. Maybe you did recognize some of your use cases in the given examples or maybe you have another use case which can or cannot be solved by this feature. We would love to get your feedback. – The Eclipse Ditto team [Less]

E2E acknowledgement using Eclipse Ditto By adding the new acknowledgements feature to Ditto, it is now possible to provide an end to end QoS (quality of service) with level 1. Previously the connectivity service of Ditto did accept all incoming ... [More] messages immediately as soon as it received the message. This behaviour can now be controlled by a Ditto header called requested-acks. In this blog post I will provide a few examples for how to use this new feature in the following scenarios: Device publishes a change of its state and doesn’t care about whether the event regarding this change will be received by any subscriber or not. This is the fire and forget scenario. Device publishes a change of its state and wants to be sure it is persisted in Ditto. This is the processed in Ditto scenario. Device publishes a change of its state and wants to be sure an application connected to Eclipse Ditto will receive the event regarding this change of the state. This is the notification scenario. Device publishes a change of its state and wants to be notified when a subscriber was not able to receive the event regarding this change. This is the tracked notification scenario. Device sends a live message and wants to be sure that it will eventually receive a response. This is the long running live message scenario. Device sends a live message and wants to either receive the response within a given timeout or never. This is the asking for required information scenario. Scenarios The following scenarios all share the same context: a device sends data/messages via a device connectivity layer (e.g. Eclipse Hono or an MQTT broker) to Ditto Ditto’s connectivity service consumes the Ditto Protocol message forwards the message into the Ditto cluster to be processed is responsible for technically acknowledging/settling the consumed message at the device connectivity layer / broker the strategy of when this is done and with which outcome is handled by the Ditto headers mentioned in the scenarios Although all scenarios are based on the device sending something (e.g. telemetry data or a message), the E2E acknowledgement can also be used the other way around when e.g. a backend application sends something to a device. Scenario 1 - Fire and Forget This is the simplest scenario of all, since the change can be published in a fire and forget semantics. In this scenario the device will send the modification command containing the headers: response-required=false requested-acks=[] Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/twin/commands/modify", "headers": { "response-required": false, "requested-acks": [] }, "path": "/features/lightSwitch/properties/status", "value": "on" } For this case the connectivity service will immediately acknowledge the incoming message at the messaging system and then continues to process the command. It doesn’t matter if the command could be processed successfully or if any subscriber received an event for this change. Scenario 2 - Processed in Ditto For this scenario the device wants to be sure its change will be properly persisted in Ditto. The command needs to define the following headers: response-required=false requested-acks=["twin-persisted"] Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/twin/commands/modify", "headers": { "response-required": false, "requested-acks": ["twin-persisted"] }, "path": "/features/lightSwitch/properties/status", "value": "on" } For this case the connectivity service will wait until the modification will be properly persisted in Ditto before acknowledging the incoming message at the messaging system. If the device published this command for example via an AMQP broker (with a QoS 1 “at least once” semantic), this will cause the broker to redeliver the command to Ditto if the acknowledgment fails. Please be aware that a redelivery will only be requested for the following error status codes: 408 (Request timed out) 424 (Dependency Failure) All kinds of 5xx status codes In this scenario it does matter if the command could be processed successfully, but it’s still not relevant if any subscriber received an event for this change. Scenario 3 - Notification scenario For this scenario the device wants to be sure another system will be notified about the change of its state. This could be for example an alarming system which wants to be sure a backend application receives the information that the alarm was triggered. A prerequisite for this is that any kind of connection exists that publishes the event to the backend application and declares a user defined acknowledgement label. This can be a WebSocket session or any kind of Ditto Connection types which can be found here. For this example we expect the event to be forwarded by an HTTP connection which declared the following acknowledgement label as issued acknowledgement of the target: d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed. The UUID prefix in this case is the ID of the HTTP connection and thebackend-process part is a custom label, defined by the user. The command needs to define the following headers: response-required=false requested-acks=["d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed"] timeout=30s (optional. Default is 10s.) Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/twin/commands/modify", "headers": { "response-required": false, "requested-acks": ["d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed"], "timeout": "30s" }, "path": "/features/alarm/properties/status", "value": "on" } For this case the connectivity service will wait until the HTTP request, which forwards the event regarding the thing change, will be finished before acknowledging the incoming message at the messaging system. The status code of the HTTP response will in this case determine if the message will be acknowledged successfully or not and if a redelivery will be requested or not. All kinds of 2xx status codes will lead to a successful acknowledgement at the messaging system. All other status codes will lead to a failed acknowledgement at the messaging system and for the following status codes a redelivery will be requested: 408 (Request timed out) 424 (Dependency Failure) All kinds of 5xx status codes In this scenario it is ensured a specified subscriber will receive an event for this change. Scenario 4 - Tracked notification scenario For this scenario the device wants to know when a system could not be notified about the change of its state. This could be for example an alarming system which wants to be sure a backend application receives the information, or if not: tries to send an SMS as notification. This scenario is mostly like scenario 3, but needs to set the response-required header to true and it’s required to configure the reply-target of the source to also expect “nack” responses. Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/twin/commands/modify", "headers": { "response-required": true, "requested-acks": ["d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed"] }, "path": "/features/alarm/properties/status", "value": "on" } In this case the device will receive an acknowledgement response containing the status code and payload of the response of the backend application. Based on this the device can decide how to handle the situation. It is suggested to publish the modification command with QoS 0 (“at most once” semantics) in this case because the device handles the result of the E2E acknowledgement. With QoS 1 (“at least once” semantics) brokers would usually redeliver the message to Ditto. If the HTTP endpoint of the backend application responds with the following response: headers: "content-type": "application/json" body: { "errorCode": "notification.smartphone.failed", "message": "Could not notify smartphone." } status code: 424 The response received at the device would look like this: { "topic": "org.eclipse.ditto/my-thing/things/twin/acks/d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed", "headers": { "response-required": false, "requested-acks": ["d45d4522-142e-4057-ae87-8969343a3ddc:backend-processed"], "timeout": "30s", "content-type": "application/json" }, "path": "/", "value": { "errorCode": "notification.smartphone.failed", "message": "Could not notify smartphone." }, "status": 424 } Scenario 5 - Long running live message scenario For this scenario the device is going to ask for an information which it needs eventually. Let’s say the device asks for the endpoint where it should download the new firmware from. It’s not required that this response arrives within a given time. It’s just required to eventually arrive at the device and after it was received the device can signal the user, that it is ready to download the firmware. The headers of the live message should have the following values: response-required=true requested-acks=["live-response"] Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/live/messages/firmware", "headers": { "response-required": true, "requested-acks": ["live-response"], "content-type": "text/plain", "timeout": "5s" }, "path": "/outbox/messages/firmware", "value": "firmware.url.query" } By requesting the acknowledgement live-response the connectivity service will wait until the response for the live message arrived for 5s before acknowledging the incoming message at the broker and will request a redelivery if the response did not arrive within this timeout. This will repeat until either the broker discards the message or the response arrives in the specified timeout. That way it is guaranteed that the device will eventually receive the response. Scenario 6 - Asking for required information scenario For this scenario the device is going to ask for an information which it needs right now to proceed with its current task. Let’s say the device asks if it should allow a car with a license plate it detected to drive on the property by opening the barrier. It could be possible to ask for that information, so the barrier opens automatically, but providing a fallback mechanism like entering a code directly at the device if this response does not arrive within time. The headers of the live message should have the following values: response-required=true requested-acks=[] Example Ditto Protocol message: { "topic": "org.eclipse.ditto/my-thing/things/live/messages/car-enter", "headers": { "response-required": true, "requested-acks": [], "content-type": "text/plain", "timeout": "5s" }, "path": "/outbox/messages/car-enter", "value": "FN IB 1337" } By requesting explicitly not requesting any acknowledgement but still requiring a response, the connectivity service will immediately acknowledge the incoming message at the broker. The device will then either receive the response within the specified timeout or never. So the device can provide its alternative options to open the barrier after 5 seconds. We embrace your feedback I hope I could demonstrate the power of the new acknowledgement feature properly and could make it clear how it can be used. Maybe you did recognize some of your use cases in the given examples or maybe you have another use case which can or cannot be solved by this feature. We would love to get your feedback. – The Eclipse Ditto team [Less]

Before Ditto Java Client 1.3.0, a client object connects to a configured Ditto back-end during its creation. // create a client object and block until it connects to the Ditto back-end. final DittoClient client = ... [More] DittoClients.newInstance(messagingProvider); There are several problems with the approach. The calling thread blocks waiting for IO, namely the authentication process and establishment of a websocket. If the client is configured to reconnect, then an incorrect end-ponit configuration makes the factory method block forever. If the client is not configured to reconnect, then the factory method will throw an exception. But it is not possible to give the client reference to the exception handler, since the client creation did not complete. Consequently the exception handler has no simple way to free all resources allocated for the client. 1.3.0 addresses these problems by introducing an asynchronous client creation interface. public final class DittoClients { public static DisconnectedDittoClient newDisconnectedInstance(MessagingProvider mp); } public interface DisconnectedDittoClient { CompletionStage<DittoClient> connect(); void destroy(); } The method DittoClients.newDisconnectedInstance(MessagingProvider) creates a DisconnectedDittoClient object. The DisconnectedDittoClient has references to all resources allocated for the client and can free them via the destroy() method. The DisconnectedDittoClient object offers no method to interact with the Ditto API. By calling connect(), one obtains a future that yields a familiar DittoClient object upon completion. One might use the asynchronous client creation interface thus: final DisconnectedDittoClient disconnectedClient = DittoClients.newDisconnectedInstance(messagingProvider); disconnectedClient.connect() .thenAccept(this::startUsingDittoClient) .exceptionally(exception -> { this.handleConnectionFailure(exception); disconnectedClient.destroy(); return null; }); The asynchronous client creation interface has the following advantages. The calling thread does not block. Even if configured to reconnect, the user can receive connection errors via the connection error handler in MessagingConfiguration and shut down the client at will. When initial reconnection is disabled, the method DisconnectedDittoClient.connect() returns a failed future on connection error. It is possible to reference the DittoDisconnectedClient object in the future’s error handler, where the client can be destroyed. – The Eclipse Ditto team [Less]