Microservices Basics Training – 5. Debezium with Outbox Pattern (optional)

Docs: 5.1 Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

Source Code

The source-code used for this lab is in the folder /code/debezium

Online Repository Link: https://github.com/acend/microservices-lab/tree/main/code/debezium .

Task 5.1.1 - Getting your workspace ready

If you still have a previous lab running, stop it with docker-compose down --remove-orphans.
Open your IDE or favorite tool and switch to the referenced source folder.

Docs: 5.2 Docker Environment

Mon, 01 Jan 0001 00:00:00 +0000

Environment Setup

With completing this section the environment is configured. However, in the next few steps we will build the order microservice and its container. After this section you will not be able to start the environment as this container is still missing. You will be able to start it later.

Changes to the docker-compose file

The setup for this lab is similar as the previous. However, we use Kafka Connect with Debezium PostgreSQL connector to write messages to Kafka.

Kafka Connect with Debezium

For Kafka Connect a pre-built docker container based on the Kafka Strimzi image will be used.

The following was added to the docker-compose.yaml file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


 connect:
 image: quay.io/acend/microservices-lab-debezium-connect:latest
 hostname: connect
 container_name: debezium-connect
 ports:
 - 8083:8083
 networks:
 - debezium
 depends_on:
 - kafka
 - order-db
 - stock-db
 - zookeeper
 command: /opt/kafka/kafka_connect_run.sh
 environment:
 - KAFKA_CONNECT_BOOTSTRAP_SERVERS=kafka:9092
 - |
 KAFKA_CONNECT_CONFIGURATION=
 key.converter=org.apache.kafka.connect.json.JsonConverter
 value.converter=org.apache.kafka.connect.json.JsonConverter
 key.converter.schemas.enable=false
 value.converter.schemas.enable=false
 group.id=connect-debezium
 offset.storage.topic=connect-debezium-offsets
 config.storage.topic=connect-debezium-configs
 status.storage.topic=connect-debezium-status
 offset.storage.replication.factor=1
 config.storage.replication.factor=1
 status.storage.replication.factor=1
 consumer.interceptor.classes=io.opentracing.contrib.kafka.TracingConsumerInterceptor
 producer.interceptor.classes=io.opentracing.contrib.kafka.TracingProducerInterceptor
 - KAFKA_CONNECT_METRICS_ENABLED=true
 - JAEGER_SERVICE_NAME=debezium-connect
 - JAEGER_AGENT_HOST=jaeger
 - STRIMZI_TRACING=jaeger

Database docker image

The Debezium PostgreSQL Connector needs to access the WAL log. We therefore switch to the quay.io/debezium/postgres:13 image maintained by Debezium. This image provides us the configurations and permissions needed to access the WAL log.

The following changes have been made in the docker-compose.yaml file:

order-db: changed image to quay.io/debezium/postgres:13
stock-db: changed image to quay.io/debezium/postgres:13

Task 5.2.1 - Inspecting the docker environment

Have a look at the /code/debezium/docker folder and get familiar with the docker environment specified in docker-compose.yaml.

Docs: 5.3 Microservices Setup

Mon, 01 Jan 0001 00:00:00 +0000

In this stage we will have a look at the setup and configuration of the Order microservice.

Task 5.3.1 - Inspect Dependencies

To simplify the use of the Outbox pattern the following dependencies have already been added to the pom.xml.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


<dependencies>
 <dependency>
 <groupId>io.debezium</groupId>
 <artifactId>debezium-quarkus-outbox</artifactId>
 <version>${debezium.version}</version>
 </dependency>
 <dependency>
 <groupId>io.quarkus</groupId>
 <artifactId>quarkus-smallrye-reactive-messaging-kafka</artifactId>
 </dependency>
 <dependency>
 <groupId>io.opentracing.contrib</groupId>
 <artifactId>opentracing-kafka-client</artifactId>
 </dependency>
 ...
</dependencies>

Dependency	Description
`quarkus-smallrye-reactive-messaging-kafka`	Reactive Messaging extensions to receive and write Messages to or from Apache Kafka.
`debezium-quarkus-outbox`	Provides a reusable, configurable component that facilitates the use of the Outbox pattern in conjunction with debezium.
`opentracing-kafka-client`	Injects and reads tracing information from Headers of an Apache Kafka message.

full source of pom.xml

Debezium Quarkus Outbox Extension

This quarkus extension provides us with the ExportedEvent interface. For every different event in our application we will implement this interface to create our custom event.

The application will then be able to emit the event using the javax.enterprise.event.Event class. The extension will also take care of writing the emitted events to our outboxevent database table.

Task 5.3.2 - Inspect application properties

The configurations for the Reactive-Messaging and the Outbox Pattern are located in the quarkus configuration file application.properties.

Have a look at the Quarkus Debezium Outbox pattern configuration.

1
2
3
4
5
6
7
8


quarkus.debezium-outbox.table-name=outboxevent
quarkus.debezium-outbox.id.name=id
quarkus.debezium-outbox.aggregate-id.name=aggregateid
quarkus.debezium-outbox.aggregate-type.name=aggregatetype
quarkus.debezium-outbox.type.name=type
quarkus.debezium-outbox.timestamp.name=timestamp
quarkus.debezium-outbox.payload.name=payload
quarkus.debezium-outbox.remove-after-insert=false

This configuration will define the table and field names of our outbox table. For further details, please refer to the Debezium Outbox Quarkus Extension documentation.

Please take a note of the configuration quarkus.debezium-outbox.remove-after-insert=false (default true). With disabling this our events stay in the outbox table forever. In theory there is no need to keep these persisted. Debezium will get the create event (row insert) from the WAL log. The provided EventRouter is only sensible to create events. For debugging we have set this to false.

Now have a look at the reactive messaging configuration.

1
2
3
4
5
6
7


mp.messaging.incoming.stock.topics=stock-stockcomplete-events,stock-stockincomplete-events
mp.messaging.incoming.stock.group.id=order-service
mp.messaging.incoming.stock.connector=smallrye-kafka
mp.messaging.incoming.stock.bootstrap.servers=kafka:9092
mp.messaging.incoming.stock.key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
mp.messaging.incoming.stock.value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
mp.messaging.incoming.stock.interceptor.classes=io.opentracing.contrib.kafka.TracingConsumerInterceptor

With this configuration our application will use a Kafka connector and will provide access to the messages using an incoming channel named stock. This channel will contain events from both topics stock-stockcomplete-events and stock-stockincomplete-events.

To get more details about the concepts of Connectors and Channels you may head over to the SmallRye Reactive Messaging Documentation

Have a look at the snippet below which is used do consume the messages

1
2
3
4


@Incoming("stock")
public CompletionStage<Void> onMessage(KafkaRecord<String, String> message) {
 // implementation
}

If you want to know more about reactive messaging in quarkus have a look at the Quarkus Reactive Messaging Guide .

Docs: 5.4 Preparing the Database

Mon, 01 Jan 0001 00:00:00 +0000

Since we had a brief look at the configuration we are now going to define the outboxevent table. We will also create an event history table consumedevent to ignore events which have already been processed (idempotency).

Outbox Event Table

The EventRouter used by the Debezium Connector expects the outboxevent table to have a certain layout.

In our application configuration application.properties we mostly stuck with the defaults. Therefore, our outboxevent table should look like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


$ admin=# \d outboxevent;
 Table "outboxevent"
 Column | Type | Collation | Nullable | Default
--------------------+-----------------------------+-----------+----------+---------
 id | uuid | | not null |
 aggregatetype | character varying(64) | | not null |
 aggregateid | character varying(64) | | not null |
 type | character varying(32) | | not null |
 timestamp | timestamp without time zone | | not null |
 tracingspancontext | character varying(255) | | |
 payload | text | | |

Indexes:
 "outboxevent_pkey" PRIMARY KEY, btree (id)

Task 5.4.1 - Create outbox table with Liquibase

Limited lab duration

This step is already implemented. For a better understanding we encourage you to have a quick read but you may also skip it.

Change the resources/db/changeLog.xml to include the specification for the outboxevent table. Create the table according the description above.

Task Hint

Your configuration for the outboxevent table should look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


 <createTable tableName="outboxevent">
 <column name="id" type="UUID">
 <constraints primaryKey="true" nullable="false"/>
 </column>
 <column name="aggregatetype" type="varchar(64)">
 <constraints nullable="false"/>
 </column>
 <column name="aggregateid" type="varchar(64)">
 <constraints nullable="false"/>
 </column>
 <column name="type" type="varchar(32)">
 <constraints nullable="false"/>
 </column>
 <column name="timestamp" type="timestamp">
 <constraints nullable="false"/>
 </column>
 <column name="tracingspancontext" type="varchar(255)">
 <constraints nullable="true"/>
 </column>
 <column name="payload" type="text">
 <constraints nullable="true"/>
 </column>
 </createTable>

full source changeLog.xml

Task 5.4.2 - Create table for event history

Limited lab duration

This step is already implemented. For a better understanding we encourage you to have a quick read but you may also skip it.

Since we want to track the events which were already processed we create a table consumedevent. In this table we are going to store the event id and a received timestamp. We can then use this to check if an incoming event has already been processed by just looking up the event id. This will provide us with idempotency for events which are delivered more than once.

As in the previous task, create a table with the following structure in the liquibase config changeLog.xml.

1
2
3
4
5
6
7
8
9


$ admin=# \d consumedevent;
 Table "consumedevent"
 Column | Type | Collation | Nullable | Default
----------+-----------------------------+-----------+----------+---------
 id | uuid | | not null |
 received | timestamp without time zone | | not null |

Indexes:
 "consumedevent_pkey" PRIMARY KEY, btree (id)

Task Hint

Your configuration for the consumedevent table should look like this:

1
2
3
4
5
6
7
8


 <createTable tableName="consumedevent">
 <column name="id" type="UUID">
 <constraints primaryKey="true" nullable="false"/>
 </column>
 <column name="received" type="timestamp">
 <constraints nullable="false"/>
 </column>
 </createTable>

full source changeLog.xml

Docs: 5.5 Define Events

Mon, 01 Jan 0001 00:00:00 +0000

In the previous section we have configured the tables for our event handling. Now it’s time to define our actual events for the application.

Event overview

We will build the following event flow. Keep in mind that messages are not directly written to Kafka.

Define Events

To define our events we have to implement ExportedEvent. This event implementation is responsible to fill up our outboxevent table. The definition of the values is as follows:

Field	Description
`id`	unique id. Can be used by consumers to detect duplicates. Should be generated when creating the event Object.
`aggregatetype`	the origin of a given event. It could be the microservice name or for example in case of domain-driven design the origin domain. For example “order” or “stock”.
`aggregateid`	contains the id affected by a given event. In case of the `aggregatetype` order this could be the order id. This id is used as the key for Kafka messages. Only messages with the same key are guaranteed to be processed in order as they are placed in the same partition of a kafka topic.
`type`	the actual type of the event. For example “OrderCreated” or “StockComplete”. We will use this information to trigger specific event handlers on the consumer side.
`payload`	in our example a JSON structure which contains the event body. For example the actual Order with Article id and Amount ordered.

Task 5.5.1 - Implement OrderCreated Event

Given the information above. Complete the ch.puzzle.mm.debezium.event.entity.OrderCreatedEvent event implementation.
- Set the required fields accordingly (AGGREGATE_TYPE and EVENT_TYPE)

For help, here’s an excerpt of the order outboxevent table. For simplicity the timestamp is omitted.

1
2
3
4
5
6
7


admin=# select id, aggregatetype, aggregateid, type, payload from outboxevent;
 id | aggregatetype | aggregateid | type | payload
-----------+---------------+-------------+----------------+---------------------------------------------------------
 bed8bd... | Order | 100000 | OrderCreated | {"orderId":100000,"items":[{"articleId":1,"amount":9}]}
 28e777... | Order | 100002 | OrderCreated | {"orderId":100002,"items":[{"articleId":1,"amount":3}]}
 e09acc... | Order | 100000 | OrderCancelled | {"orderId":100000,"items":[{"articleId":1,"amount":9}]}
 3709f2... | Order | 100004 | OrderCreated | {"orderId":100004,"items":[{"articleId":1,"amount":3}]}

Task Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60


public class OrderCreatedEvent implements ExportedEvent<String, JsonNode> {

 private static ObjectMapper mapper = new ObjectMapper();

 private static final String AGGREGATE_TYPE = "Order";
 private static final String EVENT_TYPE = "OrderCreated";

 private final UUID id;
 private final Long orderId;
 private final JsonNode jsonNode;
 private final Instant timestamp;

 public OrderCreatedEvent(Instant created, ShopOrder shopOrder) {
 this.id = UUID.randomUUID();
 this.orderId = shopOrder.id;
 this.timestamp = created;
 this.jsonNode = asJson(shopOrder);
 }

 @Override
 public String getAggregateId() {
 return String.valueOf(orderId);
 }

 @Override
 public String getAggregateType() {
 return AGGREGATE_TYPE;
 }

 @Override
 public String getType() {
 return EVENT_TYPE;
 }

 @Override
 public Instant getTimestamp() {
 return timestamp;
 }

 @Override
 public JsonNode getPayload() {
 return jsonNode;
 }

 public ObjectNode asJson(ShopOrder order) {
 ObjectNode asJson = mapper.createObjectNode()
 .put("orderId", order.id);

 ArrayNode items = asJson.putArray("items");

 for (ArticleOrder article : order.getArticleOrders()) {
 items.add(mapper.createObjectNode()
 .put("articleId", article.getArticleId())
 .put("amount", article.getAmount())
 );
 }

 return asJson;
 }
}

Docs: 5.6 Event Handling

Mon, 01 Jan 0001 00:00:00 +0000

In the previous section we defined our events. Now let’s have a look at the application logic to fire and consume these.

Event overview

We will build the following event flow. Keep in mind that messages are not directly written to Kafka.

Creating Orders

New orders are received by requests to our RESTful API /shop-orders. They are handled by the ch.puzzle.mm.debezium.order.boundary.ShopOrderResource.

The required POST body corresponds to the ShopOrderDTO class.

1
2
3
4
5
6
7
8


{
 "articleOrders" : [
 {
 "articleId" : 1,
 "amount" : 3
 }
 ]
}

Task 5.6.1 - Implement order creation and fire event

The actual work for creating an order is done by method createOrder in ch.puzzle.mm.debezium.order.control.ShopOrderService class.

1
2
3
4
5
6
7
8
9


public ShopOrder createOrder(ShopOrderDTO shopOrderDTO) {
 // TODO: implementation - create ArticleOrder list

 // TODO: implementation - create new shopOrder

 // TODO: fire OrderCreatedEvent

 return new ShopOrder();
}

Finish implementation of the createOrder Method:

Create list of ArticleOrder entities from article list in ShopOrderDto
Create a new ShopOrder
- status: Set to ShopOrderStatus.NEW
- articleOrders: Set created list of ArticleOrder
- persist the order
Fire the OrderCreatedEvent event
Return the newly created order

Hint list of ArticleOrder creation

1
2
3


 List<ArticleOrder> articleOrders = shopOrderDTO.articleOrders.stream()
 .map(s -> new ArticleOrder(s.articleId, s.amount))
 .collect(Collectors.toList());

Hint ShopOrder creation

1
2
3
4
5


// store order to shopOrder table
ShopOrder shopOrder = new ShopOrder();
shopOrder.setStatus(ShopOrderStatus.NEW);
shopOrder.setArticleOrders(articleOrders);
shopOrder.persist();

Hint event firing

Fire events with

1
2


// fire event (outbox table)
event.fire(new OrderCreatedEvent(Instant.now(), shopOrder));

Complete Task Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


@ApplicationScoped
public class ShopOrderService {

 @Inject
 Event<ExportedEvent<?, ?>> event;

 public ShopOrder createOrder(ShopOrderDTO shopOrderDTO) {
 List<ArticleOrder> articleOrders = shopOrderDTO.articleOrders.stream().map(s -> new ArticleOrder(s.articleId, s.amount)).collect(Collectors.toList());

 // store order to shopOrder table
 ShopOrder shopOrder = new ShopOrder();
 shopOrder.setStatus(ShopOrderStatus.NEW);
 shopOrder.setArticleOrders(articleOrders);
 shopOrder.persist();

 // fire event (outbox table)
 event.fire(new OrderCreatedEvent(Instant.now(), shopOrder));

 return shopOrder;
 }

 // ...
}

Cancelling Order

In this lab you can delete orders by POST to /shop-orders/{id}/status. For simplicity cancelling is only allowed if the order is in state completed (stock have been deducted).

Task 5.6.2 - Implement order cancellation and fire event

Limited lab duration

This step is already implemented. For a better understanding we encourage you to have a quick read but you may also skip it.

The actual work for cancelling an order is done by cancelOrder method in ch.puzzle.mm.debezium.order.control.ShopOrderService class.

Fetch the ShopOrder by orderId (already done)
Check Status (already done)
Set status of order to ShopOrderStatus.CANCELLED
Fire the OrderCancelledEvent event
Return the modified order

Complete Task Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


@ApplicationScoped
public class ShopOrderService {

 @Inject
 Event<ExportedEvent<?, ?>> event;

 // ...

 public ShopOrder cancelOrder(long orderId) {
 ShopOrder order = ShopOrder.getByIdOrThrow(orderId);
 if (order.getStatus().canCancel()) {
 order.setStatus(ShopOrderStatus.CANCELLED);
 event.fire(new OrderCancelledEvent(Instant.now(), order));
 return order;
 } else {
 throw new IllegalStateException("Cannot cancel Order " + orderId);
 }
 }
}

Consuming Stock Events

The following configuration defines the incoming stock channel containing the messages from the Kafka topics stock-stockcomplete-events and stock-stockincomplete-events

1

mp.messaging.incoming.stock.topics=stock-stockcomplete-events,stock-stockincomplete-events

Our event consumer class is the ch.puzzle.mm.debezium.event.boundary.KafkaEventConsumer.

Task 5.6.3 - Consuming Kafka messages

Consume incoming events in KafkaEventConsumer and delegate the processing to the StockEventHandler.

Finish implementation of onMessage in ch.puzzle.mm.debezium.event.boundary.KafkaEventConsumer

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


public CompletionStage<Void> onMessage(KafkaRecord<String, String> message) {
 return CompletableFuture.runAsync(() -> {
 try (final Scope span = tracer.buildSpan("handle-stock-message")
 .asChildOf(TracingKafkaUtils.extractSpanContext(message.getHeaders(), tracer))
 .startActive(true)) {

 // TODO: implementation - read id, eventType from headers

 // TODO: implementation - delegate message to StockEventHandler
 } catch (Exception e) {
 logger.error("Error while preparing articlestock", e);
 throw e;
 }
 }).thenRun(message::ack);
}

Annotate method onMessage as handler for the incoming stock channel with @Incoming(...).
Read the Kafka Headers id and eventType from the message using getHeadersAsString.
Inject the StockEventHandler and delegate processing to onStockEvent of StockEventHandler

Incoming definition Hint

Channels (incoming or outgoing) can be annotated on method level with @Incoming.

1
2


@Incoming("stock")
public CompletionStage<Void> onMessage(KafkaRecord<String, String> message) { }

Read Kafka Headers Hint

Use the provided method getHeaderAsString.

1
2


String eventId = getHeaderAsString(message, "id");
String eventType = getHeaderAsString(message, "eventType");

Calling StockEventHandler Hint

1
2
3
4
5
6


stockEventHandler.onStockEvent(
 UUID.fromString(eventId),
 eventType,
 message.getKey(),
 message.getPayload(),
 message.getTimestamp());

Complete Task Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


@Incoming("stock")
public CompletionStage<Void> onMessage(KafkaRecord<String, String> message) {
 return CompletableFuture.runAsync(() -> {
 try (final Scope span = tracer.buildSpan("handle-stock-message").asChildOf(TracingKafkaUtils.extractSpanContext(message.getHeaders(), tracer)).startActive(true)) {
 logger.debug("Kafka message with key = {} arrived", message.getKey());

 logHeaders(message);

 String eventId = getHeaderAsString(message, "id");
 String eventType = getHeaderAsString(message, "eventType");

 stockEventHandler.onStockEvent(
 UUID.fromString(eventId),
 eventType,
 message.getKey(),
 message.getPayload(),
 message.getTimestamp()
 );
 } catch (Exception e) {
 logger.error("Error while preparing articlestock", e);
 throw e;
 }
 }).thenRun(message::ack);
}

Task 5.6.4 - Creating an event log

Limited lab duration

This step is already implemented. For a better understanding we encourage you to have a quick read but you may also skip it.

The EventLog class uses the ConsumedEvent entity to keep track of processed events.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


@ApplicationScoped
public class EventLog {

 private static final Logger logger = LoggerFactory.getLogger(EventLog.class);

 public void processed(UUID eventId) {
 // TODO: implementation - store
 }

 public boolean alreadyProcessed(UUID eventId) {
 // TODO: implementation - check exists
 return false;
 }
}

Transaction management

Ensure that processed requires a transaction by using Transactional.TxType.MANDATORY
Ensure that alreadyProcessed requires a transaction by using Transactional.TxType.MANDATORY

Transaction management Hint

Adding the @Transactional annotation ensures that methods throw an exception if there isn’t an existing transaction context. Annotate both methods with:

1

@Transactional(value = Transactional.TxType.MANDATORY)

Remember that we use the Panache extension and that our entities provide methods to easily query database records. You can get more details about using the Active Record Pattern in Quarkus in the Simplified Hibernate ORM with Panache Guide .

Implement both EventLog methods.

Implement method processed to write a new ConsumedEvent with the eventId and the current timestamp to the database.
Implement method alreadyProcessed to check the database for having a record with the given eventId.

Complete EventLog Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


@Traced
@ApplicationScoped
public class EventLog {

 private static final Logger logger = LoggerFactory.getLogger(EventLog.class);

 @Transactional(value = Transactional.TxType.MANDATORY)
 public void processed(UUID eventId) {
 ConsumedEvent.persist(new ConsumedEvent(eventId, Instant.now()));
 }

 @Transactional(value = Transactional.TxType.MANDATORY)
 public boolean alreadyProcessed(UUID eventId) {
 logger.info("Looking for event with id {} in message log", eventId);
 return ConsumedEvent.findByIdOptional(eventId).isPresent();
 }
}

Task 5.6.5 - Processing stock events

Implement the event handling in StockEventHandler

Check if message is already processed by using the EventLog.
- If so, skip the message.
Depending on eventType call the corresponding method in ShopOrderService
- StockComplete: deserialize the event and call onStockCompleteEvent
- StockIncomplete: deserialize the event and call onStockIncompleteEvent
- If none of them: log warning about an unknown event.
Register the event as processed using the EventLog

StockEventHandler Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


public void onStockEvent(UUID eventId, String eventType, String key, String event, Instant ts) {
 if (eventLog.alreadyProcessed(eventId)) {
 logger.info("Event with id {} was already processed, ignore.", eventId);
 return;
 }

 logger.info("Received '{}' event {} - OrderId: {}, ts: '{}'", eventType, eventId, key, ts);
 if (eventType.equalsIgnoreCase("StockComplete")) {
 shopOrderService.onStockCompleteEvent(deserialize(event));
 } else if (eventType.equalsIgnoreCase("StockIncomplete")) {
 shopOrderService.onStockIncompleteEvent(deserialize(event));
 } else {
 logger.warn("Ignoring unknown event '{}'", eventType);
 }

 eventLog.processed(eventId);
}

Task 5.6.6 - Complete the order management

For the order to be managed completely we have to implement the methods onStockCompleteEvent and onStockIncompleteEvent in our ShopOrderService.

Implement the stock complete event

Find ShopOrder in database using the orderId from the event
Set status to ShopOrderStatus.COMPLETED

OnStockCompleteEvent Hint

1
2
3
4
5


public void onStockCompleteEvent(ShopOrderStockResponse stockComplete) {
 ShopOrder.findByIdOptional(stockComplete.orderId).ifPresent(o -> {
 ((ShopOrder)o).setStatus(ShopOrderStatus.COMPLETED);
 });
}

Implement the stock incomplete event

Find ShopOrder in database using the orderId from the event
Set status to ShopOrderStatus.STOCK_INCOMPLETE

OnStockIncompleteEvent Hint

1
2
3
4
5


public void onStockIncompleteEvent(ShopOrderStockResponse stockIncomplete) {
 ShopOrder.findByIdOptional(stockIncomplete.orderId).ifPresent(o -> {
 ((ShopOrder) o).setStatus(ShopOrderStatus.STOCK_INCOMPLETE);
 });
}

Docs: 5.7 Running the Lab

Mon, 01 Jan 0001 00:00:00 +0000

In the previous section we implemented the microservice. Let’s build and run it.

Building

The debezium-order microservice provides a multistage Dockerfile.multistage.jvm for building the java application and creating a docker container.

Task 5.7.1 - Build docker container

Docker Version

Depending on your docker version you have to specify the output format. If you get a warning that there is no output specified for docker-container driver just add -o type=docker to your command line.

Change to base directory of this lab <PROJECT_ROOT>/code/debezium and run docker build.

1

docker build -f debezium-order/src/main/docker/Dockerfile.multistage.jvm -t microservices-lab-debezium-order:latest debezium-order

Running

Task 5.7.2 - Starting with Docker Compose

Enter a shell and change your directory to the <PROJECT_ROOT>/code/debezium/docker folder.

If you have any previous lab running stop these containers

1

docker-compose down --remove-orphans

Start the microservices with

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


docker-compose up -d
Creating debezium-grafana ... done
Creating debezium-order-db ... done
Creating debezium-prometheus ... done
Creating debezium-jaeger ... done
Creating debezium-alertmanager ... done
Creating debezium-stock-db ... done
Creating debezium-zookeeper ... done
Creating debezium-stock ... done
Creating debezium-order ... done
Creating debezium-kafka ... done
Creating debezium-connect ... done
Creating debezium-viewer ... done

Docs: 5.8 Using the Lab

Mon, 01 Jan 0001 00:00:00 +0000

We have finished our work on the environment and our order microservice. We are now ready to setup our debezium connectors.

Debezium Connectors

With starting the environment we have also started Kafka Connect. We need to configure our Debezium PostgreSQL Connector to listen for events on the order-db and stock-db database.

Task 5.8.2 - Configuring Debezium Connectors

Configuring the Debezium PostgreSQL Connector for the order-db database.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


curl --request POST \
 --url http://localhost:8083/connectors \
 --header 'Accept: application/json' \
 --header 'Content-Type: application/json' \
 --data '{
 "name": "order-events",
 "config": {
 "connector.class" : "io.debezium.connector.postgresql.PostgresConnector",
 "tasks.max" : "1",
 "database.hostname" : "order-db",
 "database.port" : "5432",
 "database.user" : "admin",
 "database.password" : "1234",
 "database.dbname" : "admin",
 "database.server.name" : "order-db",
 "schema.include.list" : "public",
 "table.include.list" : "public.outboxevent",
 "tombstones.on.delete" : "false",
 "transforms" : "outbox,topiccase",
 "transforms.outbox.type" : "io.debezium.transforms.outbox.EventRouter",
 "transforms.outbox.route.by.field" : "type",
 "transforms.outbox.route.topic.replacement" : "order-${routedByValue}-events",
 "transforms.outbox.table.field.event.timestamp" : "timestamp",
 "transforms.outbox.table.fields.additional.placement" : "type:header:eventType",
 "transforms.topiccase.type" : "ch.puzzle.mm.debezium.connect.smt.ChangeTopicCase",
 "transforms.topiccase.toCase" : "lower"
 }
}'

Configuring the Debezium PostgreSQL Connector for the stock-db database.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


curl --request POST \
 --url http://localhost:8083/connectors \
 --header 'Accept: application/json' \
 --header 'Content-Type: application/json' \
 --data '{
 "name": "stock-events",
 "config": {
 "connector.class" : "io.debezium.connector.postgresql.PostgresConnector",
 "tasks.max" : "1",
 "database.hostname" : "stock-db",
 "database.port" : "5432",
 "database.user" : "admin",
 "database.password" : "1234",
 "database.dbname" : "admin",
 "database.server.name" : "stock-db",
 "schema.include.list" : "public",
 "table.include.list" : "public.outboxevent",
 "tombstones.on.delete" : "false",
 "transforms" : "outbox,topiccase",
 "transforms.outbox.type" : "io.debezium.transforms.outbox.EventRouter",
 "transforms.outbox.route.by.field" : "type",
 "transforms.outbox.route.topic.replacement" : "stock-${routedByValue}-events",
 "transforms.outbox.table.field.event.timestamp" : "timestamp",
 "transforms.outbox.table.fields.additional.placement" : "type:header:eventType",
 "transforms.topiccase.type" : "ch.puzzle.mm.debezium.connect.smt.ChangeTopicCase",
 "transforms.topiccase.toCase" : "lower"
 }
}'

As you can see in the configuration of the connectors we specify our connection information and our tables with the table.include.list and our transforms. We use the EventRouter transformation which handles our outboxevent table.

Check if the connectors are created successfully

1
2
3
4
5
6
7
8


curl --request GET \
 --url http://127.0.0.1:8083/connectors \
 --header 'Accept: application/json'

[
 "order-events",
 "stock-events"
]

Sample Requests

We provide a requests/api.http file containing all requests needed for this lab. This file is compatible with IntelliJ IDEA and also includes the curl version for the requests.

Get article stock information

1

curl --request GET --url http://localhost:8081/article-stocks

Get order information

1

curl --request GET --url http://localhost:8080/shop-orders

Create new order

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


curl --request POST \
 --url http://localhost:8080/shop-orders \
 --header 'Content-Type: application/json' \
 --data '{
 "articleOrders" : [
 {
 "articleId" : 1,
 "amount" : 1
 }
 ]
}'

Task 5.8.3 - Using the application

There is a Viewer Microservice running which provides an overview of events in the Kafka Topics. It uses websockets and shows newly arriving events.

Feel free to test the following scenarios:

Order Article 1 with quantity 9 (e.g. orderId: 100000)
Order Article 1 with quantity 3

What do you expect? Was the second order rejected?

What happens if you proceed like this?

Cancel OrderId 100000
Order Article 1 with quantity 3

What do you expect? Was the second order rejected?

What happens if you proceed like this?

Delete the connectors (order-events and stock-events)
Order Article 1 with quantity 3
Order Article 1 with quantity 3
Order Article 1 with quantity 3
Recreate the connectors (order-events and stock-events)

Were there any errors when the connectors were down? Do the microservices catch up after recreating?

What does the Jaeger Tracing show?