Microservices Basics Training – Labs

Docs: 2.1 Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

Source Code

The source-code used for this lab is in the folder /code/monolith

Online Repository Link: https://github.com/acend/microservices-lab/tree/main/code/monolith .

Task 2.1.1 - Getting your workspace ready

We will only inspect files in this lab.

Open your IDE or favorite tool and switch to the referenced source folder.

Docs: 2.2 Docker Environment

Mon, 01 Jan 0001 00:00:00 +0000

Docker Compose

Our setup for the application is quite simple. It consists of the Quarkus service application and its database application-db. Further we have three more containers setup. We use jaeger as our tracing, prometheus to scrape the metrics and grafana to visualize them in dashboards.

All configuration for the docker environment is located in the folder docker. The folder docker/config is used to configure prometheus and grafana.

Host mapped container ports

Starting up the docker-compose will create a bridge network. We mapped most container ports to the docker host. This provides easy access for debugging like attaching the database to your IDE.

However, the most important ports of the containers are:

Container	Internal Port	Host Port	Purpose
`application`	8080	8080	Main application port
`application-db`	5432	5432	Database port
`jaeger`	16686	16686	Jaeger UI
`jaeger`	14268	14268	Port used to send traces from our Quarkus applications to Jaeger
`prometheus`	9090	9090	Prometheus service port (includes UI)
`grafana`	3000	3000	Grafana UI

Have a look at Jaeger Port Details if you need more information about the available Jaeger ports.

Task 2.2.1 - Inspecting the docker environment

Have a look at the /code/monolith/docker folder and get familiar with the docker environment specified in docker-compose.yaml

Docs: 2.3 Application Structure

Mon, 01 Jan 0001 00:00:00 +0000

Code Structure

The application in this lab is structured as follows:

monolith-application: Base directory for code or artifacts belonging to the application container
monolith-application/src/main/docker: Dockerfiles to build application and container
monolith-application/src/main/java/ch/puzzle/mm/monolith/: Location where most of the java application code resides
monolith-application/src/main/resources/db: contains the database migration scripts

The java code is structured by domains and the sub folders boundary, control and entity. Beside domains there are also helpers. For example the package monkey which contains a basic ChaosMonkey to inject errors.

Domains

In the application we have the following domains:

article: Static article management
order: Handles the management of the order and initiates the stock management
stock: Represents a stock management and handles the amount of items in stock

The most important classes in the application are:

order/boundary/ShopOrderResource.java: Provides the RESTful API endpoint for creating a new order
order/control/ShopOrderService.java: Handles the effective creation of the order and triggers the stock management
stock/control/ArticleStockService.java: Handles the effective management of the items in stock

Task 2.3.1 - Inspecting application structure

Have a look at the /code/monolith folder and get familiar with the application structure.

Docs: 2.4 Using the Lab

Mon, 01 Jan 0001 00:00:00 +0000

Starting the Lab

Open a shell and change your directory to the /code/monolith/docker folder.

Start the microservices with

1
2
3
4
5
6
7
8


$ docker-compose up -d
Creating network "docker_monolith" with driver "bridge"
Creating monolith-jaeger ... done
Creating monolith-grafana ... done
Creating monolith-alertmanager ... done
Creating monolith-application-db ... done
Creating monolith-prometheus ... done
Creating monolith-application ... done

Now your environment is up and running.

Using the Lab

Available Endpoints

Endpoint	Container	Purpose
http://localhost:8080/shop-orders	application	Return and create new orders
http://localhost:8080/article-stocks	application	List article stock count
http://localhost:8080/chaos-monkey	application	Lets you add a ChaosMonkey

Sample Requests

In the folder requests within the monolith lab folder there is an IntelliJ IDEA compatible HTTP Request file named api.http. You may run the requests directly from this file or build your own requests.

Get the article stock information

1

curl --request GET --url http://localhost:8080/article-stocks

Get order information

1

curl --request GET --url http://localhost:8080/shop-orders

Create new order

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


curl --request POST \
 --url http://localhost:8080/shop-orders \
 --header 'Content-Type: application/json' \
 --data '{
 "articleOrders" : [
 {
 "articleId" : 1,
 "amount" : 1
 }
 ]
}'

Task 2.4.1 - Testing scenario

Run the following queries:

Check the article-stock count
Create an order for one item of article 1
Check the orders and article-stock again

Answer the questions:

Is the article-stock count correct?
What happens if you are running out of stock?

ChaosMonkey

There is a basic implementation of a ChaosMonkey. It will let you inject errors to methods annotated with @ChaosMonkey. You may configure the ChaosMonkey using the provided RESTful API /chaos-monkey.

The following error types are available:

Error Type	Executed	Description
RateLimit	Before method	Rate limit in calls/sec. Default: `Long.MAX_VALUE`
Latency	After method	Issues a Thread.sleep() in ms. Default: 0
ErrorRate	After method	Lets you specify a percentage of Errors thrown (0 to 1). Default: 0
Exception	After method	Always triggers an exception if enabled. Default: `false`

Sample request for configuring the ChaosMonkey which will throw an exception:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


curl --request POST \
 --url http://127.0.0.1:8080/chaos-monkey \
 --header 'Content-Type: application/json' \
 --data '[
 {
 "clazzName": "ArticleStockService",
 "methodName": "orderArticles",
 "enabled": true,
 "errorRate": 0.0,
 "latencyMs": 0,
 "permitsPerSec": 9.223372036854776E18,
 "rateLimiterType": "BLOCK",
 "throwException": true
 }
]'

Task 2.4.2 - Injecting an error

Enable a ChaosMonkey for the method orderArticles in the ArticleStockService.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


curl --request POST \
 --url http://127.0.0.1:8080/chaos-monkey \
 --header 'Content-Type: application/json' \
 --data '[
 {
 "clazzName": "ArticleStockService",
 "methodName": "orderArticles",
 "enabled": true,
 "throwException": true
 }
]'

This will throw an InternalServerErrorException after execution of the method orderArticles.

Ask yourself what to expect if an exception is thrown?

Issue the request from the Task 2.4.1.

What did you observe? Is there any difference?
Is the data still consistent?
What’s the role of a transaction in this case?

Task Hint

Since the application throws an InternalServerErrorException the order is not created and the stock count remains the same.

This is due to the fact that the code runs in a single transaction. Throwing a RuntimeException from an active transaction triggers a rollback of the transaction.

Docs: 2.5 Metrics (optional)

Mon, 01 Jan 0001 00:00:00 +0000

Optional Section

The following content is optional. If you have time, feel free to proceed.

Metrics with Prometheus and Grafana

The involved components for our metrics to be working are:

MicroProfile Metrics: Expose the application metrics as a RESTful Endpoint
Prometheus: Scrape the exposed metrics periodically
Grafana: Visualize the scraped metrics

Dependencies

Our application includes the following dependency in the pom.xml.

GroupId	ArtifactId	Description	Detailed information
`io.quarkus`	`quarkus-smallrye-metrics`	Exposes various metrics and statistics about the application	MicroProfile Metrics

Including this dependency is enough for our application to expose the metrics at /metrics

Metric Details

The exposed metrics are using the OpenMetrics Format format. Metrics can be put in three different scopes.

Scopes

base: metrics that all MicroProfile vendors have to provide. e.g. JVM metrics
vendor: vendor specific metrics
application: application-specific metrics

Source: github.com/eclipse/microprofile-metrics

More Details: MicroProfile Metrics - Scope

We will only generate metrics for the scope application.

There are various type of metrics available. The most used types are:

Counters: Counters are a metric that is used to keep an incremental or a decremental count.
Gauges: represent metrics that are sampled to obtain their value.
Meters: Meters are used to track throughput.
Timers: Timers are used to aggregate timing duration, in nanoseconds, and provide duration and throughput statistics.

MicroProfile Metrics provides these types as annotations @Counted, @Timed, @Metered and @Gauge.

Task 2.5.1 - Inspecting metrics

Start the environment (if not already running) and point your browser to http://localhost:8080/metrics

What do you get?
Can you identify the different scopes?
Can you locate some application specific metrics?

Task Hint

You will get all metrics in the OpenMetrics TEXT Format. Each metric is on a new line.

The different scopes can be seen by having a look at the first part of a metric (e.g. vendor_).

Defining metrics from code

The following snipped is part of the RESTful API endpoint ShopOrderResource.

1
2
3
4
5
6
7
8
9


@POST
@Transactional
@Counted(name = "monolith_order_create_request", absolute = true, description = "number of orders requested", tags = {"application=monolith", "resource=ShopOrderResource"})
@Timed(name = "monolith_order_create_timer", absolute = true, description = "timer for processing a order creation", tags = {"application=monolith", "resource=ShopOrderResource"})
public Response createShopOrder(ShopOrderDTO shopOrderDTO) {
 ShopOrder shopOrder = shopOrderService.createOrder(shopOrderDTO);
 registerSuccessfulOrder();
 return Response.ok(shopOrder).build();
}

full source ShopOrderResource.java

This code defines the following metrics:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


application_monolith_order_create_request_total{application="monolith",resource="ShopOrderResource"} 28.0
application_monolith_order_create_timer_rate_per_second{application="monolith",resource="ShopOrderResource"} 0.21598823940031425
application_monolith_order_create_timer_one_min_rate_per_second{application="monolith",resource="ShopOrderResource"} 0.3083022515809183
application_monolith_order_create_timer_five_min_rate_per_second{application="monolith",resource="ShopOrderResource"} 0.08583728986138126
application_monolith_order_create_timer_fifteen_min_rate_per_second{application="monolith",resource="ShopOrderResource"} 0.030253491344307742
application_monolith_order_create_timer_min_seconds{application="monolith",resource="ShopOrderResource"} 0.004008576
application_monolith_order_create_timer_max_seconds{application="monolith",resource="ShopOrderResource"} 0.194742506
application_monolith_order_create_timer_mean_seconds{application="monolith",resource="ShopOrderResource"} 0.020320811804737986
application_monolith_order_create_timer_stddev_seconds{application="monolith",resource="ShopOrderResource"} 0.044306979654618196
application_monolith_order_create_timer_seconds_count{application="monolith",resource="ShopOrderResource"} 28.0
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.5"} 0.008546583
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.75"} 0.01223613
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.95"} 0.190428873
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.98"} 0.194742506
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.99"} 0.194742506
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.999"} 0.194742506

Task 2.5.2 - Interpret metrics

Try to interpret the metrics from the listing above

How many orders have been requested?
How long does the creation of an order take?
Were the requests made recently or some time ago?
Were there peaks or were the requests evenly distributed over time?

Task Hint

A total of 28 orders created
The count on rate_per_second, one_min_rate_per_second are higher than fifteen_min_rate_per_second which indicates that most requests are seconds/minutes ago.
There seem to be peaks as the rates for second, minute, five and fifteen are not evenly distributed.

The graph may look like this

Task 2.5.3 - Inspect grafana dashboards

Head over to the Grafana UI at http://localhost:3001 . Login with admin/admin and inspect the two available dashboards.

SmallRye MicroProfile Metrics: Generated Dashboard containing base and vendor metrics.
Monolith Dashboard: Custom Dashboard containing some basic metrics from the application.

Try to change an existing visualization

Open the Monolith Dashboard
Edit “Order Create Timer seconds” by clicking the title and selecting edit
- This graph contains multiple quantiles in the same graph. display only the 0.75 quantile.
- Apply a filter to the metric by adding {quantile="0.75"} to the metrics query. This applies a filter to a tag named quantile with value 0.75

The modified query is using the Prometheus Query Language (PromQL). Check the Querying Prometheus page for more details.

Task Hint

Prometheus scraping

We know how our application exposes metrics. We have also seen dashboards with simple visualizations. Now have a look at the Prometheus configuration docker/config/prometheus/prometheus.yaml to see how the scraping of these metrics is configured.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# my global config
global:
 scrape_interval: 15s
 evaluation_interval: 15s

# omitted

scrape_configs:
 - job_name: 'prometheus'

 static_configs:
 - targets: ['localhost:9090']

 - job_name: 'application'
 static_configs:
 - targets: ['application:8080']

full source prometheus.yaml

This defines that Prometheus scrapes our application at port 8080. Note that the metrics_path is /metrics per default. Prometheus will therefore query http://application:8080/metrics every 15 seconds to get new metrics.

For more information about the Prometheus configuration head over to the Prometheus configuration site.

Docs: 2.6 Application Tracing (optional)

Mon, 01 Jan 0001 00:00:00 +0000

Optional Section

The following content is optional. If you have time, feel free to proceed.

OpenTracing with Jaeger Tracing

We use OpenTracing for the distributed tracing. Our tracing backend is Jaeger.

Dependencies

Our application includes the following dependencies in the pom.xml.

GroupId	ArtifactId	Description	Detailed information
`io.quarkus`	`quarkus-smallrye-opentracing`	MicroProfile OpenTracing implementation	MicroProfile OpenTracing
`io.opentracing.contrib`	`opentracing-jdbc`	Provides traces for database queries.

Configuration

The following configurations is required in the application.properties to setup the tracing:

1
2
3
4


quarkus.jaeger.endpoint=http://jaeger:14268/api/traces
quarkus.jaeger.service-name=monolith-application
quarkus.jaeger.sampler-type=const
quarkus.jaeger.sampler-param=1

endpoint: defines where we will send our metrics
service-name: the name of the current service
sampler-type: const defines a constant sampling
sampler-param: percentage of request which are sampled. 1 defines that we want to sample all requests.

Further we have to use the tracing enabled database driver.

1
2
3
4


quarkus.datasource.db-kind=postgresql
quarkus.datasource.jdbc.url=jdbc:tracing:postgresql://application-db:5432/admin
quarkus.datasource.jdbc.driver=io.opentracing.contrib.jdbc.TracingDriver
quarkus.hibernate-orm.dialect=org.hibernate.dialect.PostgreSQLDialect

Last we made the Quarkus log format to include the traceId and spanId in the logs. This is not necessary but will help troubleshooting.

1

quarkus.log.console.format=%d{HH:mm:ss} %-5p traceId=%X{traceId}, spanId=%X{spanId}, sampled=%X{sampled} [%c{2.}] (%t) %s%e%n

Configuration

The configuration listed above may slightly differ from the one used in the solution repository. This is because some properties are overridden using environment variables in the docker-compose.yaml file.

full source application.properties

Application code

Using the configuration made is basically everything which is needed to get traces from our application. By default all requests to RESTful endpoints are traced. Using the TracingDriver provides spans for all database queries.

If you want to customize the tracing you can annotate any class or method with the @Traced annotation. You may also disable the default tracing with @Traced(false). Further you may inject the Tracer to your application code. Using this tracer you can create custom spans in business methods or add tags to span.

This snipped is from class ShopOrderService.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


@ApplicationScoped
public class ShopOrderService {
 // ...

 @Inject
 Tracer tracer;

 @Traced(operationName = "createOrder")
 public ShopOrder createOrder(ShopOrderDTO shopOrderDTO) {
 ShopOrder shopOrder = new ShopOrder();
 // ...
 shopOrder.persist();

 tracer.activeSpan().setTag("order.id", shopOrder.id);

 return shopOrder;
 }
}

This will create a span for the method createOrder and name it accordingly. Using the injected Tracer we tag the span with the just created order.id.

full source ShopOrderService.java

Task 2.6.1 - Inspect the application traces

Start up your environment if not already running and point your browser to the Jaeger UI at http://localhost:16686/

Issue requests as described in section 2.4

Do you find traces for created orders? You may list all traces or use the tag filter order.id=X
How do traces look like if the operationName is not set?
Do you see the tag order.id on the span?
What happens if you inject an exception as described in section 2.4?

Task Hint

Traces from application code with no explicit name are named by their class and method. If an exception is thrown, there will be a tag error=true on the span. The Jaeger UI will flag such spans with an exclamation mark.

Docs: 3.1 Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

Source Code

The source-code used for this lab is in the folder /code/rest

Online Repository Link: https://github.com/acend/microservices-lab/tree/main/code/rest .

Task 3.1.1 - Getting your workspace ready

If you still have the previous lab running: stop it with docker-compose down.
Open your IDE and switch to the referenced source folder.

Docs: 3.2 Docker Environment

Mon, 01 Jan 0001 00:00:00 +0000

Environment Setup

With completing this section the environment is configured. However, in the next few steps we will build the order microservice and its container. After this section you will not be able to start the environment as this container is still missing. You will be able to start it later.

Changes to the docker-compose file

To split the monolith in two microservices we need to change our docker-compose.yaml file.

Microservice container specification

First we deleted the application and application-db specification which belonged to our monolith. Then we created a specification for the two microservices order and stock.

As we are building the order service by ourselves we reference a local container image. The stock service uses a pre-built image from the quay.io/acend registry.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


 order:
 image: microservices-lab-rest-order:latest
 hostname: order
 container_name: rest-order
 ports:
 - 8080:8080
 networks:
 - rest
 depends_on:
 - order-db
 environment:
 - QUARKUS_HTTP_PORT=8080
 - QUARKUS_DATASOURCE_JDBC_URL=jdbc:tracing:postgresql://order-db:5432/admin
 - APPLICATION_ARTICLESTOCK_API_URL=http://stock:8080/

 stock:
 image: quay.io/acend/microservices-lab-rest-stock:latest
 hostname: stock
 container_name: rest-stock
 ports:
 - 8081:8080
 networks:
 - rest
 depends_on:
 - stock-db
 environment:
 - QUARKUS_HTTP_PORT=8080
 - QUARKUS_DATASOURCE_JDBC_URL=jdbc:tracing:postgresql://stock-db:5432/admin

As both microservices need their own postgres database we added two databases to our docker-compose.yaml.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


 order-db:
 image: docker.io/postgres:13
 hostname: order-db
 container_name: rest-order-db
 volumes:
 - order-db-data:/var/lib/postgresql/data/
 networks:
 - rest
 environment:
 - POSTGRES_USER=admin
 - POSTGRES_PASSWORD=1234
 ports:
 - 5432:5432

 stock-db:
 image: docker.io/postgres:13
 hostname: stock-db
 container_name: rest-stock-db
 volumes:
 - stock-db-data:/var/lib/postgresql/data/
 networks:
 - rest
 environment:
 - POSTGRES_USER=admin
 - POSTGRES_PASSWORD=1234
 ports:
 - 5433:5432

Task 3.2.1 - Inspecting the docker environment

Have a look at the /code/rest/docker folder and get familiar with the docker environment specified in docker-compose.yaml.

Docs: 3.3 Building a Microservice

Mon, 01 Jan 0001 00:00:00 +0000

Stock Microservice

The order and stock microservice have a similar setup. Therefore, we will only work on the order microservice.

Order Microservice

We have already finished the specification of our environment in the docker compose file. Now we will write our order microservice.

As an overview this is the final structure of the order microservice at the end of the tasks in this section. You will be guided the implementation of these in the next steps.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


.
├── pom.xml
└── src
 └── main
    ├── docker
    ├── java
    │   └── ch
    │   └── puzzle
    │   └── mm
    │   └── rest
    │   ├── order
    │   │   ├── boundary
    │   │   │   └── ShopOrderResource.java # Order RESTful API
    │   │   ├── control
    │   │   │   └── ShopOrderService.java # Business code
    │   │   └── entity
    │   │   ├── ArticleOrderDTO.java # Entities and DTOs...
    │   │   ├── ArticleOrder.java
    │   │   ├── ShopOrderDTO.java
    │   │   ├── ShopOrder.java
    │   │   └── ShopOrderStatus.java
    │   └── stock
    │   └── boundary
    │   └── ArticleStockService.java # Stock REST-Client
    └── resources
    ├── application.properties # Application Properties
    ├── db
    │   └── changeLog.xml # Database migration
    └── META-INF
    └── resources
    └── index.html # Static webpage

Datasource and database schema

First we are going to setup the database connection and tables.

Task 3.3.1 - Setup database dependencies

Limited lab duration

This step is already implemented. For a better understanding we encourage you to have a quick read but you may also skip it.

Ensure the following dependencies are specified in the pom.xml.

GroupId	ArtifactId	Description	Detailed information
`io.quarkus`	`quarkus-hibernate-orm-panache`	Hibernate ORM with Panache support	Simplified Hibernate ORM with Panache
`io.quarkus`	`quarkus-liquibase`	Liquibase¹ Quarkus integration for database schema change management	Using Liquibase
`io.quarkus`	`quarkus-jdbc-postgresql`	JDBC PostgreSQL support for Quarkus	Datasources

Dependencies Task Hint

Ensure the following dependencies are in your pom.xml. Dependency quarkus-hibernate-orm-panache is already added.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


<dependencies>
 ...
 <!-- database dependencies -->
 <dependency>
 <groupId>io.quarkus</groupId>
 <artifactId>quarkus-hibernate-orm-panache</artifactId>
 </dependency>
 <dependency>
 <groupId>io.quarkus</groupId>
 <artifactId>quarkus-liquibase</artifactId>
 </dependency>
 <dependency>
 <groupId>io.quarkus</groupId>
 <artifactId>quarkus-jdbc-postgresql</artifactId>
 </dependency>
 ...
</dependencies>

After adding these dependencies, we are ready to configure quarkus to use the database. Add the following configuration to the application.properties file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# datasource
quarkus.datasource.db-kind = postgresql
quarkus.datasource.username = admin
quarkus.datasource.password = 1234
quarkus.datasource.jdbc.url = jdbc:tracing:postgresql://localhost:5432/admin
quarkus.datasource.jdbc.driver=io.opentracing.contrib.jdbc.TracingDriver
quarkus.hibernate-orm.dialect=org.hibernate.dialect.PostgreSQLDialect

# drop and create the database at startup (use `update` to only update the schema)
quarkus.hibernate-orm.database.generation = none

# liquibase properties
quarkus.liquibase.migrate-at-start=true
quarkus.liquibase.clean-at-start=true

This will defines our data source connection and specify that Liquibase should clean and migrate the database schema at start up.

Task 3.3.2 - Setup database schema

Limited lab duration

This step is already implemented. For a better understanding we encourage you to have a quick read but you may also skip it.

Let’s build the database schema with liquibase.

Create folder src/main/resources/db
Create file src/main/resources/db/changeLog.xml

You have to use the liquibase XML syntax to create the tables and sequence. You may find some details here:

For details about the supported data types in postgres, please check the PostgreSQL Data Types page.

You can use the following template:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
 xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext
 http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd
 http://www.liquibase.org/xml/ns/dbchangelog
 http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">

 <changeSet author="main" id="1">
 <!-- simple table -->
 <createTable tableName="TABLENAME">
 <column name="PKNAME" type="PKTYPE">
 <constraints primaryKey="true" nullable="false"/>
 </column>
 <column name="NAME" type="TYPE"/>
 </createTable>

 <!-- relation table -->
 <createTable tableName="TABLE1_TABLE2">
 <column name="TABLE1_id" type="TABLE1_PKTYPE">
 <constraints nullable="false" foreignKeyName="FK_TABLE1_NAME" referencedColumnNames="id"/>
 </column>
 <column name="TABLE2_id" type="TABLE2_PKTYPE">
 <constraints nullable="false" foreignKeyName="FK_TABLE2_NAME" referencedColumnNames="id"/>
 </column>
 </createTable>

 <createSequence sequenceName="SEQUENCE_NAME" startValue="STARTVALUE"/>
 </changeSet>
</databaseChangeLog>

Naming

Please be precise with the naming of tables and fields. If you name things differently you may run into trouble in later steps.

Create the shoporder table in changeLog.xml according to the details below. This table will store our orders.

1
2
3
4
5
6
7
8
9


admin=# \d shoporder;
 Table "shoporder"
 Column | Type | Collation | Nullable | Default
--------+------------------------+-----------+----------+---------
 id | bigint | | not null |
 status | character varying(255) | | |

Indexes:
 "shoporder_pkey" PRIMARY KEY, btree (id)

Create the articleorder table in changeLog.xml according to the details below. This table reflects the effective articles which have been ordered.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


admin=# \d articleorder;
 Table "articleorder"
 Column | Type | Collation | Nullable | Default
-----------+---------+-----------+----------+---------
 id | bigint | | not null |
 articleid | bigint | | |
 amount | numeric | | |

Indexes:
 "articleorder_pkey" PRIMARY KEY, btree (id)

Create the mapping table in changeLog.xml according to the details below. This table is used to build the mapping from shoporder to the articleorder. Define any needed foreignKeyName yourself.

1
2
3
4
5
6


admin=# \d shoporder_articleorder;
 Table "shoporder_articleorder"
 Column | Type | Collation | Nullable | Default
------------------+--------+-----------+----------+---------
 shoporder_id | bigint | | not null |
 articleorders_id | bigint | | not null |

Finally, create a hibernate sequence in changeLog.xml according to the details below.

1
2
3
4
5


admin=# \d hibernate_sequence;
 Sequence "hibernate_sequence"
 Type | Start | Minimum | Maximum | Increment | Cycles? | Cache
--------+--------+---------+---------------------+-----------+---------+-------
 bigint | 100000 | 1 | 9223372036854775807 | 1 | no | 1

Task Hint

Your changeLog.xml should look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
 xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext
 http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd
 http://www.liquibase.org/xml/ns/dbchangelog
 http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">

 <changeSet author="main" id="1">
 <createTable tableName="shoporder">
 <column name="id" type="bigint">
 <constraints primaryKey="true" nullable="false"/>
 </column>
 <column name="status" type="varchar(255)"/>
 </createTable>

 <createTable tableName="articleorder">
 <column name="id" type="bigint">
 <constraints primaryKey="true" nullable="false"/>
 </column>
 <column name="articleid" type="bigint"/>
 <column name="amount" type="number"/>
 </createTable>

 <createTable tableName="shoporder_articleorder">
 <column name="shoporder_id" type="bigint">
 <constraints nullable="false" foreignKeyName="shoporder_article_shoporder_fk" referencedColumnNames="id"/>
 </column>
 <column name="articleorders_id" type="bigint">
 <constraints nullable="false" foreignKeyName="shoporder_article_articleorder_fk" referencedColumnNames="id"/>
 </column>
 </createTable>

 <createSequence sequenceName="hibernate_sequence" startValue="100000"/>
 </changeSet>
</databaseChangeLog>

Entities and Data Transfer Objects (DTOs)

The next step is to define our Domain Entities and DTOs. For our order microservice we will need the following entities:

ShopOrder: The entity representing an order
ArticleOrder: The entity representing an ordered article and its order count
ShopOrderStatus: An enum representing the status of the order

We will trigger a new order using a RESTful API call to our order microservice. For this we will use DTOs instead of entities. Define the following DTOs:

ShopOrderDTO: A DTO to specify which articles have been ordered with this order.
ArticleOrderDTO: A DTO to specify the article id and the amount of items ordered.

Task 3.3.3 - Implement entities

Limited lab duration

This step is already implemented. For a better understanding we encourage you to have a quick read but you may also skip it.

Entity overview of the order microservice.

PanacheEntity

We are using the Quarkus Hibernate ORM with Panache extension. The classical approach with the EntityManager would work as well and might be required for complex use cases. However, in our simple implementation we will make use of Panache. For more information about Panache have a look at: Simplified Hibernate ORM with Panache

We will create our entities in src/main/java/ch/puzzle/mm/rest/order/entity.

Implement enum ShopOrderStatus
Implement entity ArticleOrder
- Implement the default Constructor and a Constructor with both fields as arguments.
Implement entity ShopOrder
- Annotate articleOrders with @OneToMany(cascade = CascadeType.ALL)
- Annotate status with @Enumerated(EnumType.STRING)
Do not forget to annotate the entities with @Entity and extending from PanacheEntity which provides us the id.

ShopOrderStatus enum Task Hint

1
2
3
4
5


public enum ShopOrderStatus {
 NEW,
 COMPLETED,
 INCOMPLETE,
}

ArticleOrder Entity Task Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


@Entity
public class ArticleOrder extends PanacheEntity {

 Long articleId;
 int amount;

 public ArticleOrder() {
 }

 public ArticleOrder(Long articleId, int amount) {
 this.articleId = articleId;
 this.amount = amount;
 }

 public Long getArticleId() {
 return articleId;
 }

 public void setArticleId(Long articleId) {
 this.articleId = articleId;
 }

 public int getAmount() {
 return amount;
 }

 public void setAmount(int amount) {
 this.amount = amount;
 }
}

ShopOrder Entity Task Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


@Entity
public class ShopOrder extends PanacheEntity {
 @OneToMany(cascade = CascadeType.ALL)
 private List<ArticleOrder> articleOrders;

 @Enumerated(EnumType.STRING)
 private ShopOrderStatus status;

 public ShopOrder() { }

 public List<ArticleOrder> getArticleOrders() {
 return articleOrders;
 }

 public void setArticleOrders(List<ArticleOrder> articles) {
 this.articleOrders = articles;
 }

 public ShopOrderStatus getStatus() {
 return status;
 }

 public void setStatus(ShopOrderStatus status) {
 this.status = status;
 }
}

Task 3.3.4 - Implement DTOs

Limited lab duration

This step is already implemented. For a better understanding we encourage you to have a quick read but you may also skip it.

The following JSON corresponds to the POST Request body which is received by the order microservice. It reflects both DTOs ShopOrderDTO and ArticleOrderDTO.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


{
 "articleOrders" : [
 {
 "articleId" : 1,
 "amount" : 1
 },
 {
 "articleId" : 2,
 "amount" : 3
 }
 ]
}

Implement both DTOs according to the given JSON. Put them in the same folder as the entities.

Implement ArticleOrderDTO
- set fields public, getters and setters are not needed
- articleId: use Long as type
- amount: use int as type
Implement ShopOrderDTO
- set field public, getters and setters are not needed
- articleOrders: use List as type

ArticleOrderDTO Task Hint

1
2
3
4


public class ArticleOrderDTO {
 public Long articleId;
 public int amount;
}

ShopOrderDTO Task Hint

1
2
3


public class ShopOrderDTO {
 public List<ArticleOrderDTO> articleOrders;
}

Building a Business Service for processing new orders

Processing of a new order will be handled by to the ShopOrderService.

Task 3.3.5 - Create service for order creation

Create the following ShopOrderService class in src/main/java/ch/puzzle/mm/rest/order/control

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


@ApplicationScoped
public class ShopOrderService {

 @Transactional
 public ShopOrder createOrder(ShopOrderDTO dto) {
 ShopOrder shopOrder = new ShopOrder();
 shopOrder.setStatus(ShopOrderStatus.NEW);

 // create order articles
 List<ArticleOrder> articleOrders = dto.articleOrders.stream()
 .map(s -> new ArticleOrder(s.articleId, s.amount))
 .collect(Collectors.toList());

 shopOrder.setArticleOrders(articleOrders);
 shopOrder.persist();

 return shopOrder;
 }
}

Building a Rest Client for the Stock Microservice

For calling our stock microservice we have to define a RESTful client. For a straightforward implementation we will use the MicroProfile:

MicroProfile Rest-Client²: Provides us a very simple way of creating rest clients by simply writing an interface using proper JAX-RS and MicroProfile annotations

Task 3.3.6 - Rest Client

Limited lab duration

This step is already implemented. For a better understanding we encourage you to have a quick read but you may also skip it.

Ensure the following dependencies are specified in the pom.xml.

GroupId	ArtifactId	Description	Detailed information
`io.quarkus`	`quarkus-rest-client`	Implementation of MicroProfile REST Client	Using the Rest Client

Dependencies Task Hint

The following dependencies have to be added.

1
2
3
4
5
6
7
8


<dependencies>
 ...
 <dependency>
 <groupId>io.quarkus</groupId>
 <artifactId>quarkus-rest-client</artifactId>
 </dependency>
 ...
</dependencies>

Task 3.3.7 - Define the Rest Client

Create the rest client by implementing a RestClient-Interface. Since this is a boundary service the interface goes in src/main/java/ch/puzzle/mm/rest/stock/boundary.

Create the interface ArticleStockService
Annotate the interface with @RegisterRestClient(configKey = "article-stock-api")
Annotate the interface with @Path("/article-stocks"). This matches the RESTful endpoint on the stock microservice.
Define the orderArticles method in ArticleStockService
- Use annotations like @POST, @Produces and @Consume as you would do in a RESTful endpoint.
- The method takes a List<ArticleOrderDTO> as argument

ArticleStockService Task Hint

1
2
3
4
5
6
7
8
9


@Path("/article-stocks")
@RegisterRestClient(configKey = "article-stock-api")
public interface ArticleStockService {

 @POST
 @Produces(MediaType.APPLICATION_JSON)
 @Consumes(MediaType.APPLICATION_JSON)
 Response orderArticles(List<ArticleOrderDTO> orders);
}

Use the following configuration in application.properties to define where our client will connect to

1
2
3
4


# ArticleStock API
application.articlestock.api.url: http://localhost:8081/
article-stock-api/mp-rest/url: ${application.articlestock.api.url}
article-stock-api/mp-rest/scope: javax.inject.Singleton

JSON RESTful Web Services

Our Microservice will provide a RESTful API to create and list existing orders. For this we need to create a JAX-RS resource.

The implementation is pretty straightforward. Define a ShopOrderResource in src/main/java/ch/puzzle/mm/rest/order/boundary endpoint and use the JAX-RS annotations. Below is a very simple implementation of a json rest service supporting GET and POST requests for http://<host>:<port>/shop-orders. For further details have a look at the Quarkus Guide Writing Json Rest Services .

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


@Path("/shop-orders")
@ApplicationScoped
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
public class ShopOrderResource {

 @GET
 public List<ShopOrder> list() {
 return null;
 }

 @POST
 @ChaosMonkey
 @Transactional
 public Response createShopOrder(ShopOrderDTO shopOrderDTO) {
 return null;
 }
}

Task 3.3.8 - JSON RESTful Resource

The ShopOrderResource is a boundary class and will be placed in src/main/java/ch/puzzle/mm/rest/order/boundary. Implement it according to the simple sniped above.

Inject the ArticleStockService to trigger the stock microservice
Inject the ShopOrderService to process the new order
- Annotate the ArticleStockService with @RestClient to tell Quarkus that this is a rest client.
Implement @GET which returns all ShopOrders
- Our entities are PanacheEntities and support the Active Record Pattern. They do provide handy static methods like listAll().
Implement @POST method to create a new order
- Annotate the method with @Transactional and @ChaosMonkey
- Use the injected ShopOrderService to create the order
- Use the injected ArticleStockService to trigger the stock handling
- After triggering the stock service, set the order status to COMPLETED
- Return the ShopOrder

GET Method Task Hint

1
2
3
4


@GET
public List<ShopOrder> listAll() {
 return ShopOrder.listAll();
}

POST Method Task Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


@POST
@ChaosMonkey
@Transactional
public Response createShopOrder(ShopOrderDTO shopOrderDTO) {
 // create ShopOrder locally
 ShopOrder shopOrder = shopOrderService.createOrder(shopOrderDTO);

 // call remote service
 articleStockService.orderArticles(shopOrderDTO.articleOrders);

 shopOrder.setStatus(ShopOrderStatus.COMPLETED);

 return Response.ok(shopOrder).build();
}

Complete Task Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


@ApplicationScoped
@Path("/shop-orders")
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
public class ShopOrderResource {

 @Inject
 @RestClient
 ArticleStockService articleStockService;

 @Inject
 ShopOrderService shopOrderService;

 @GET
 public List<ShopOrder> listAll() {
 return ShopOrder.listAll();
 }

 @POST
 @ChaosMonkey
 @Transactional
 public Response createShopOrder(ShopOrderDTO shopOrderDTO) {
 // create ShopOrder locally
 ShopOrder shopOrder = shopOrderService.createOrder(shopOrderDTO);

 // call remote service
 articleStockService.orderArticles(shopOrderDTO.articleOrders);

 shopOrder.setStatus(ShopOrderStatus.COMPLETED);

 return Response.ok(shopOrder).build();
 }
}

Liquibase: https://www.liquibase.org/ ↩︎
MicroProfile Rest Client: https://github.com/eclipse/microprofile-rest-client/blob/main/spec/src/main/asciidoc/microprofile-rest-client-spec.asciidoc ↩︎

Docs: 3.4 Running the Lab

Mon, 01 Jan 0001 00:00:00 +0000

In the previous section we implemented the microservice. Lets build and run it.

Building

The order microservice provides a multistage Dockerfile for building the java application and creating a docker container.

Task 3.4.1 - Build docker container

Docker Version

Depending on your docker version you have to specify the output format. If you get a warning that there is no output specified for docker-container driver just add -o type=docker to your command line.

Change to base directory of this lab <PROJECT_ROOT>/code/rest and run docker build.

1

docker build -f rest-order/src/main/docker/Dockerfile.multistage.jvm -t microservices-lab-rest-order:latest rest-order

Running

Task 3.4.2 - Starting with Docker Compose

Enter a shell and change your directory to the <PROJECT_ROOT>/code/rest/docker folder.

If you have any previous lab running stop these containers

1

docker-compose down --remove-orphans

Start the microservices with

1
2
3
4
5
6
7
8
9


$ docker-compose up -d
Starting rest-alertmanager ... done
Starting rest-jaeger ... done
Starting rest-grafana ... done
Starting rest-prometheus ... done
Starting rest-stock-db ... done
Starting rest-order-db ... done
Starting rest-stock ... done
Starting rest-order ... done

Docs: 3.5 Using the Lab

Mon, 01 Jan 0001 00:00:00 +0000

Available Endpoints

Endpoint	Microservice	Description
http://localhost:8080/shop-orders	Order	Return and create new orders
http://localhost:8080/chaos-monkey	Order	Inject errors for testing our application
http://localhost:8081/articles	Stock	Query article metadata
http://localhost:8081/article-stocks	Stock	List or change the article stock count
http://localhost:8081/chaos-monkey	Stock	Inject errors for testing our application

Order state

The order is expected to have the following states

State	Description
NEW	Order is created. Stock reservation has not happened.
COMPLETE	Order is created and the article-stock count has been decreased according to the article counts.
INCOMPLETE	Order is created but reservation could not complete.

Sample Requests

In the source folder <PROJECT_ROOT>/code/rest/requests there is an IntelliJ compatible HTTP Request file. You may run requests directly from IntelliJ or use it as reference for your requests.

Get article stock information

1

curl --request GET --url http://localhost:8081/article-stocks

Get order information

1

curl --request GET --url http://localhost:8080/shop-orders

Create new order

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


curl --request POST \
 --url http://localhost:8080/shop-orders \
 --header 'Content-Type: application/json' \
 --data '{
 "articleOrders" : [
 {
 "articleId" : 1,
 "amount" : 1
 }
 ]
}'

Task 3.5.1 - Testing scenario

Run the following queries:

Check the article-stock count
Create an order for one item of article 1
Check the orders and article-stock again

Answer the questions:

Is the article-stock count correct?
Do you see any problems with this implementation?
What happens if you are running out of stock?
What’s the role of a transaction in this case?

Task Hint

This implementation seems to be the most logical and straightforward implementation when we just split a monolith in two microservices and let them communicate with a simple RESTful API.

The happy-path seems to be working quite well. But the edge cases do not behave quite well. Running out of stock will return an exception from the stock service. The TransactionalInterceptor will be triggered by the exception and will issue a rollback.

In the best case our order is not created and the stock is also not reserved. But are we sure that the stock reservation has not been made? The exception might have occurred after the reservation on the stock service has been completed. Think about a simple timeout between these two services.

We could remove the Transactional annotation to not cover the rest call to our stock service. We could then have a look at the response code to determine if the operation was successful. But would that solve all problems? What if an exception occurred in the order microservice just after the stock reservation?

Task 3.5.2 - Injecting an error

Let us inject an error at the end of the createShopOrder method of the order microservice.

Use the chaos-monkey rest endpoint to inject an error with the following request:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


curl --request POST \
 --url http://localhost:8080/chaos-monkey \
 --header 'Content-Type: application/json' \
 --data '[
 {
 "clazzName": "ShopOrderResource",
 "methodName": "createShopOrder",
 "enabled": true,
 "throwException": true
 }
]'

This will throw an InternalServerErrorException after execution of the method createShopOrder.

Ask yourself what to expect if an exception is thrown?

Issue the request from the Task 3.5.1 but use article 3 for your order.

What did you observe?
Is there any difference?
Is the data still consistent between stock and order microservice?
Can you imagine why any inconsistency may happen?
What’s the role of a transaction in this case?

Task Hint

As expected, the reservation took place in the stock microservice. However, our local transaction in the order microservice was rolled back as there was an exception. Our Data is no inconsistent as the stock count was decreased but the order was not created.

Docs: 3.6 Distributed Transactions (optional)

Mon, 01 Jan 0001 00:00:00 +0000

Optional Section

The following content is optional. If you have time, feel free to proceed.

Implementation

Do not make the described changes below to your implementation.

The only code change needed is described in Task 3.6.2 and targets your docker-compose.yaml file.

In this section we explain the changes made to the order and stock microservice. At the end of the section you will be switching to the provided pre-built docker images which contain all these changes.

Both our microservices use local transactions to store their data. There are several cases where data may get out of sync between order and stock microservice.

Problem

Think of the following scenario.

New order is requested using the RESTful API
- Order microservice will create and store a new order
- Order microservice will trigger the stock microservice to change the amount of in-stock items.
- Stock microservice changes the stock count and returns successfully.
- Order microservice fails for any reason

Since the stock successfully applied the reservation, and the order microservice failed, this results in an inconsistent state.

MicroProfile LRA (Long Running Actions)

Motivation for Long Running Actions

The proposal introduces annotations and APIs for services to coordinate long running activities whilst still maintaining loose coupling and doing so in such a way as to guarantee a globally consistent outcome without the need to take locks on data. […]

Traditional techniques for guaranteeing consistency in distributed environments has focused on XA transactions where locks may be held for long periods thereby introducing strong coupling between services and decreasing concurrency to unacceptable levels. Additionally, if such a transaction aborts then valuable work which may be valid will be rolled back. In view of these issues an alternative approach is desirable. - github.com/eclipse/microprofile-lra

LRA Protocol Sequence - source: github.com/eclipse/microprofile-lra

The MicroProfile LRA is built for microservices with RESTful communication. Each microservice participating in a LRA will have to provide a compensate action which will be invoked if the transaction is cancelled. The life cycle of LRAs can be managed by annotating the JAX-RS resources with the following annotations.

Annotation	Description
`@LRA`	Controls the life cycle of an LRA.
`@Compensate`	Indicates that the method should be invoked if the LRA is cancelled.
`@Complete`	Indicates that the method should be invoked if the LRA is closed.
`@Forget`	Indicates that the method may release any resources that were allocated for this LRA.
`@Leave`	Indicates that this class is no longer interested in this LRA.
`@Status`	When the annotated method is invoked it should report the status.
`@AfterLRA`	When an LRA has reached a final state the annotated method is invoked.
Source: github.com/eclipse/microprofile-lra

See the basic sample below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


@Path("/")
@ApplicationScoped
public class SimpleLRAParticipant {
 @LRA(LRA.Type.REQUIRES_NEW)
 @Path("/cdi")
 @PUT
 public void doInTransaction(@HeaderParam(LRA_HTTP_CONTEXT_HEADER) URI lraId) {
 /*
 * Perform business actions in the context of the LRA identified by the
 * value in the injected JAX-RS header. This LRA was started just before
 * the method was entered (REQUIRES_NEW) and will be closed when the
 * method finishes at which point the completeWork method below will be
 * invoked.
 */
 }

 @Complete
 @Path("/complete")
 @PUT
 public Response completeWork(@HeaderParam(LRA_HTTP_CONTEXT_HEADER) URI lraId) {
 /*
 * Free up resources allocated in the context of the LRA identified by the
 * value in the injected JAX-RS header.
 *
 * Since there is no @Status method in this class, completeWork MUST be
 * idempotent and MUST return the status.
 */
 return Response.ok(ParticipantStatus.Completed.name()).build();
 }

 @Compensate
 @Path("/compensate")
 @PUT
 public Response compensateWork(@HeaderParam(LRA_HTTP_CONTEXT_HEADER) URI lraId) {
 /*
 * The LRA identified by the value in the injected JAX-RS header was
 * cancelled so the business logic should compensate for any actions
 * that have been performed while running in its context.
 *
 * Since there is no @Status method in this class, compensateWork MUST be
 * idempotent and MUST return the status
 */
 return Response.ok(ParticipantStatus.Compensated.name()).build();
 }
}

Source: github.com/eclipse/microprofile-lra

Handling Compensations

What does providing a compensation mean to our microservices?

Each operation like order creation is run in a LRA context. The LRA context is reflected by an associated lra Id. The lraId will be injected in the HTTP-Headers. The call for compensation will also contain this id. This means that we must be able to revert our database changes by using the lra id whenever a REST call is made to the method annotated with @Compensate.

Task 3.6.1 - How to track LRAs

Think of compensating an order, which changes are necessary to our entities that we can revert the change later on?

Task Hint

Lets assume the LRA with id lra-1 needs to be compensated.

To compensate the database changes we need to know:

Which order is created by lra-1
- If the order is compensated we set a different order status
Which articles were reserved by lra-1
- If the order is compensated we have to revert the stock deduction

For this to achieve a simple approach could be:

Store lra-1 which was used to create the order in the ShopOrder entity.
Track the ArticleOrder with the associated lra-1 in the stock microservice as a new table/entity articlestockchange

Database Changes

This leads to the following changes in our database schema for the order microservice.

1
2
3
4
5
6


<changeSet author="lra" id="2">
 <comment>Add LRA field to store associated lra-id</comment>
 <addColumn tableName="shoporder">
 <column name="lra" type="varchar(255)" />
 </addColumn>
</changeSet>

On the stock microservice we will add the following migration to create a new table articlestockchange.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


<createTable tableName="articlestockchange">
 <column name="id" type="bigint">
 <constraints primaryKey="true" nullable="false"/>
 </column>
 <column name="article_id" type="bigint">
 <constraints nullable="false" foreignKeyName="articlestockchange_article_fk" referencedColumnNames="id"/>
 </column>
 <column name="count" type="int"/>
 <column name="lra" type="varchar(255)"/>
</createTable>

LRA Coordinator

The LRA management will be made by an LRA coordinator. The Narayana Transaction Manager¹, which is the default JTA Transaction manager in Quarkus and WildFly Application servers, provides an implementation of an LRA Coordinator.

We will use this coordinator for our environment. Therefore, the following changes are made in the docker/docker-compose.yaml:

Add LRA Coordinator (do not forget to add the volume)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


 lra-coordinator:
 image: quay.io/jbosstm/lra-coordinator:latest
 hostname: lra-coordinator
 container_name: rest-lra-coordinator
 volumes:
 - lra-data:/opt/jboss
 networks:
 - rest
 ports:
 - 8090:8080
 environment:
 - LOG_LEVEL=DEBUG

# ... omitted

volumes:
 # ....
 lra-data:

Dependencies

Our application includes the following dependency in the pom.xml.

GroupId	ArtifactId	Description
`org.jboss.narayana.rts`	`narayana-lra`	Provides MicroProfile LRA implementation

Bridged Docker Network

The library uses the base url extracted form the request-uri to register itself at the LRA Coordinator. Our lab runs in a bridged docker network. Our requests are made to localhost with the mapped port of the docker host and are bridged to the container port inside the docker network.

This is why the order microservice would register itself as localhost if we call the order microservice with http://localhost:8080/shop-orders from the docker host.

To fix this environment specific issue we rewrite the host of the incoming requests from localhost to order. The registering at the LRA coordinator is therefore made correctly with order instead of localhost

To fix the docker network issue and specify the needed system variables we add the following to the order container specification:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


 order:
 # ... omitted
 environment:
 - JAVA_TOOL_OPTIONS=-Dlra.http.host=lra-coordinator -Dlra.http.port=8080
 - APPLICATION_REWRITE_BASE_ENABLED=true
 - APPLICATION_REWRITE_BASE_HOST=order

 stock:
 # ... omitted
 environment:
 - JAVA_TOOL_OPTIONS=-Dlra.http.host=lra-coordinator -Dlra.http.port=8080
 - APPLICATION_REWRITE_BASE_ENABLED=true
 - APPLICATION_REWRITE_BASE_HOST=stock

Using LRAs

Now we are going to change our code to make use of LRAs. We have to annotate the JAX-RS resources with the specific LRA-Annotations.

Changes in Order Microservice

We have to make the following changes to our ch.puzzle.mm.rest.order.boundary.ShopOrderResource

Annotate the POST method createShopOrder and specify the HeaderParams to be used to extract the LRA ID. The extracted LRA ID will be stored in the database to be able to complete or compensate the LRA.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


@POST
@ChaosMonkey
@LRA(value = LRA.Type.REQUIRES_NEW,
 cancelOn = {
 Response.Status.INTERNAL_SERVER_ERROR // cancel on a 500 code
 },
 cancelOnFamily = {
 Response.Status.Family.CLIENT_ERROR // cancel on any 4xx code
 })
public Response createShopOrder(@HeaderParam(LRA.LRA_HTTP_CONTEXT_HEADER) String lraUrl,
 ShopOrderDTO shopOrderDTO) {
 // extract lraId from headerstring
 String lraId = getLraId(lraUrl);

 // create ShopOrder locally
 ShopOrder shopOrder = shopOrderService.createOrder(shopOrderDTO, lraId);

 // call remote service
 articleStockService.orderArticles(shopOrderDTO.articleOrders);

 return Response.ok(shopOrder).build();
}

Implement @Complete method and set the COMPLETE status for this order. The LRA expects us to return the Status.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


@PUT
@Path("/complete")
@Complete
@Consumes(MediaType.TEXT_PLAIN)
@Transactional
public Response completeShopOrder(@HeaderParam(LRA.LRA_HTTP_CONTEXT_HEADER) String lraUrl) {
 // extract lraId from headerstring
 String lraId = getLraId(lraUrl);

 int updated = ShopOrder.setStatusByLra(ShopOrderStatus.COMPLETED, lraId);

 return Response.ok(ParticipantStatus.Completed.name()).build();
}

Implement @Compensate method and set the INCOMPLETE status. Also return the Status for the LRA.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


@PUT
@Path("/compensate")
@Compensate
@Consumes(MediaType.TEXT_PLAIN)
@Transactional
public Response compensateShopOrder(@HeaderParam(LRA.LRA_HTTP_CONTEXT_HEADER) String lraUrl) {
 // extract lraId from headerstring
 String lraId = getLraId(lraUrl);

 int updated = ShopOrder.setStatusByLra(ShopOrderStatus.INCOMPLETE, lraId);

 return Response.ok(ParticipantStatus.Compensated.name()).build();
}

Further you have to add a simple method to extract the LRA-id

1
2
3


String getLraId(String longRunningHeader) {
 return longRunningHeader.substring(longRunningHeader.lastIndexOf('/') + 1);
}

Complete Source of ShopOrderResource

We have also added method to the ShopOrder entity to change the ShopOrder by a given lraId. The entity also includes the lra field which we’ve been added to our database schema.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


@Entity
public class ShopOrder extends PanacheEntity {

 @NotNull
 @Column(unique=true, nullable = false)
 private String lra;

 // omitted

 public String getLra() {
 return lra;
 }

 public void setLra(String lra) {
 this.lra = lra;
 }

 public static int setStatusByLra(ShopOrderStatus status, String lra) {
 return update("status = :status where lra = :lra",
 Parameters.with("status", status).and("lra", lra));
 }
}

Complete Source of ShopOrder

Our ShopOrderService now takes an additional argument the lraId which is stored in the newly created field on the ShopOrder entity.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


@ApplicationScoped
public class ShopOrderService {

 @Transactional
 public ShopOrder createOrder(ShopOrderDTO dto, String lra) {
 // omitted

 // set lra id
 shopOrder.setLra(lra);

 // omitted
 }
}

Complete Source of ShopOrderService

At the end we switch our rest-client to the LRA-Enabled version for the stock service by changing the path to @Path("/lra/article-stocks").

Complete Source of ArticleStockService

Changes in Stock Microservice

Pre-built Container

The pre-built version of quay.io/acend/microservices-lab-rest-stock:latest supports both versions, the original one and the LRA enabled rest endpoint. That’s why there exists a second JAX-RS resource ArticleStockLraResource in the solution code.

We have to make the following changes to our ch.puzzle.mm.rest.stock.boundary.ArticleStockResource

Annotate the POST method createStockOrder and specify the HeaderParams to be used to extract the LRA ID. The extracted LRA ID will be stored in the database to be able to complete or compensate the LRA.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


@POST
@ChaosMonkey
@LRA(value = LRA.Type.MANDATORY, end = false)
public Response createStockOrder(@HeaderParam(LRA.LRA_HTTP_CONTEXT_HEADER) String lraUrl,
 List<ArticleOrderDTO> articles) {
 // extract lraId from headerstring
 String lraId = getLraId(lraUrl);

 // handle stock count
 articleStockService.orderArticles(articles, lraId);
}

Implement @Complete method to just return the LRA status.

1
2
3
4
5
6


@PUT
@Path("/complete")
@Complete
public Response completeShopOrder(@HeaderParam(LRA.LRA_HTTP_CONTEXT_HEADER) String lraUrl) {
 return Response.ok(ParticipantStatus.Completed.name()).build();
}

Implement @Compensate method which uses the stored LRA from the database and adds the reserved (decreased) amount of an article to the stock count.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


@PUT
@Path("/compensate")
@Compensate
public Response compensateShopOrder(@HeaderParam(LRA.LRA_HTTP_CONTEXT_HEADER) String lraUrl) {
 String lraId = getLraId(lraUrl);

 // find article stock change by LRA-ID and increment the stockcount.
 ArticleStockChange.findByLraId(lraId).forEach(asc -> {
 ArticleStock stock = ArticleStock.findByArticleId(asc.getArticle().id);
 int oldStockCount = stock.getCount();
 stock.setCount(stock.getCount() + asc.getCount());
 log.info("Compensating for Article {}: oldCount={}, release={}, newCount={}", stock.getArticle().id, oldStockCount, asc.getCount(), stock.getCount());
 });

 return Response.ok(ParticipantStatus.Compensated.name()).build();
}

Complete Source of ArticleStockLraResource

Changing to the LRA enabled version

To change to the LRA version, switch to the pre-built docker images.

Task 3.6.2 - Switching to the pre-built images

We provide a docker-compose.solution.yaml docker file which includes all changes necessary to use the LRA enabled version.

You may simply start the environment using this docker-compose file. Restart your docker environment with

1
2


docker-compose down --remove-orphans
docker-compose -f docker-compose.solution.yaml up -d

Source of docker-compose.solution.yaml

Task 3.6.3 - Rerun the tests from previous section

Issue the request from the Task 3.5.1.

What did you observe?
Is there any difference?
Is the data consistent between stock and order microservice?
What’s the role of a transaction in this case?
What’s the role of the LRA coordinator?

Use the chaos-monkey rest endpoint to inject an errors at the order microservice.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


curl --request POST \
 --url http://localhost:8080/chaos-monkey \
 --header 'Content-Type: application/json' \
 --data '[
 {
 "clazzName": "ShopOrderResource",
 "methodName": "createShopOrder",
 "enabled": true,
 "throwException": true
 }
]'

Rerun some tests with the monkey in place. What did happen?

You may delete the ChaosMonkey on the order microservice with

1
2
3


curl --request DELETE \
 --url http://127.0.0.1:8080/chaos-monkey/ShopOrderResource%23createShopOrder \
 --header 'Content-Type: application/json'

Now create a chaos-monkey at the stock microservice:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


curl --request POST \
 --url http://127.0.0.1:8081/chaos-monkey \
 --header 'Content-Type: application/json' \
 --data '[
 {
 "clazzName": "ArticleStockLraResource",
 "methodName": "createStockOrder",
 "enabled": true,
 "errorRate": 0,
 "latencyMs": 0,
 "permitsPerSec": 9.223372036854776E18,
 "rateLimiterType": "BLOCK",
 "throwException": true
 }
]'

Now rerun some tests with the monkey on the stock microservice.

Is your data still consistent?

Task Hint

Using LRAs the LRA coordinator tracks the state of the LRA. If an error is occurred it will notify every participant of the LRA about the error by triggering the method annotated with @Compensate. Since our microservices now track which entities have been created in which LRA contexts they are able to revert or compensate the changes.

You may see this clearly in the logs of the stock or order microservice.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


----------------------------
Create ArticleOrder: 0_ffffc0a86003_-1be006ed_5ff8d079_12
LRA Transaction Id: 0_ffffc0a86003_-1be006ed_5ff8d079_12
----------------------------
Article with id 1 processed. Stock oldCount=10, ordered=1, newCount=9

---------------------------
Compensate ArticleOrder
LRA Transaction Id: 0_ffffc0a86003_-1be006ed_5ff8d079_12
----------------------------
Compensating for Article 1: oldCount=9, release=1, newCount=10

Narayana Transaction Manager: https://narayana.io ↩︎

Docs: 4.1 Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

Source Code

The source-code used for this lab is in the folder /code/kafka

Online Repository Link: https://github.com/acend/microservices-lab/tree/main/code/kafka .

Task 4.1.1 - Getting your workspace ready

If you still have a previous lab running, stop it with docker-compose down.
Open your IDE and switch to the correct source-code location to start with.

Docs: 4.2 Docker Environment

Mon, 01 Jan 0001 00:00:00 +0000

Environment Setup

Changes to the docker-compose file

Since we are going to use Apache Kafka as a message broker, there are some changes to our docker-compose.yaml file.

Apache Kafka Broker

To use Apache Kafka the following was added to our docker-compose.yaml file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24



 zookeeper:
 image: quay.io/debezium/zookeeper:2.3
 hostname: zookeeper
 container_name: debezium-zookeeper
 networks:
 - kafka
 ports:
 - 2181:2181
 - 2888:2888
 - 3888:3888

 kafka:
 image: quay.io/acend/microservices-lab-debezium-kafka:latest
 hostname: kafka
 container_name: debezium-kafka
 ports:
 - 9092:9092
 networks:
 - kafka
 depends_on:
 - zookeeper
 environment:
 - ZOOKEEPER_CONNECT=zookeeper:2181

Kafka Viewer

Further a simple kafka-viewer microservice was added to the docker-compose.yaml. The kafka-viewer is a small web-UI which displays messages passing through our kafka topics.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


 viewer:
 image: quay.io/acend/microservices-lab-kafka-viewer:latest
 ports:
 - 8082:8080
 networks:
 - kafka
 depends_on:
 - kafka
 environment:
 - QUARKUS_HTTP_PORT=8080
 - MP_MESSAGING_INCOMING_KAFKAMESSAGES_TOPICS=shop-order-request,shop-order-confirmation,shop-order-compensation
 - MP_MESSAGING_INCOMING_KAFKAMESSAGES_KEY_DESERIALIZER=org.apache.kafka.common.serialization.StringDeserializer
 - MP_MESSAGING_INCOMING_KAFKAMESSAGES_VALUE_DESERIALIZER=org.apache.kafka.common.serialization.StringDeserializer

Microservice Configuration

To ensure your microservices will connect to the Kafka broker an additional property has been added to both microservices. The property KAFKA_BOOTSTRAP_SERVERS=kafka:9092 in the environment section will ensure that the microservices will connect to our kafka message broker.

Task 4.2.1 - Inspecting the docker environment

Have a look at the /code/kafka/docker folder and get familiar with the docker environment specified in docker-compose.yaml.

Docs: 4.3 Reactive Messaging

Mon, 01 Jan 0001 00:00:00 +0000

4.3.1: The Reactive Manifesto

“Today’s demands are simply not met by yesterday’s software architectures.” - The Reactive Manifesto

Old fashioned applications - often built as monoliths - struggle to meet the requirements of todays workload dimensions. Applications should become scalable, robust and easy to maintain. Similar to the twelve-factor-app manifesto there is a Reactive Manifesto. It describes requirements to tackle problems for modern enterprise solutions.

The Reactive Manifesto defines that reactive systems are:

Responsive
Resilient
Elastic
Message Driven

We have already learned basics of messaging in a microservice architecture. Time to take a next step and make our messaging reactive. The approach we have seen in the chapter before is completely legitimate and there is nothing wrong with this approach.

4.3.2: Reactive Messaging

In the last chapter we learned about basic messaging concepts and how two microservices can communicate with a message broker. In reactive messaging we connect channels directly to components. Instead of having a Thread running manually, we can annotate functions to bind them to events sent in a specific channel or data stream. This makes our code more readable and act in a reactive manner. Let’s look at an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10



/* [...] */

@Incoming("data-inbound-reactive")
@Outgoing("data-outbound-reactive")
public String streamProcess(String value) {
 return value.toUpperCase();
}

/* [...] */

If you read this example it’s pretty clear what is happening. We are connecting with a Connector to a Channel (Queue or Topic) we call “data-inbound-reactive” and define it as the inbound connector for this method. On the other side we connect the outcome of this method to the “data-outbound-reactive” stream. Whenever the “data-inobund-reactive” stream sends an message we perform a transformation to uppercase and return the value into the “data-outbound-reactive” channel. Simple as that!

4.3.2.1: Connectors

Connector can:

retrieve messages from a remote broker (inbound)
send messages to a remove broker (outbound)

A connector can, of course, implement both directions.

Inbound connectors are responsible for:

Getting messages from the remote broker,
Creating a Reactive Messaging Message associated with the retrieved message.
Potentially associating technical metadata with the message. This includes unmarshalling the payload.
Associating a acknowledgement callback to acknowledge the incoming message when the Reactive Messaging message is acknowledged.

Important: Reactive matters! The first step should follow the reactive streams principle: uses non-blocking technology, respects downstream requests.

Outbound connectors are responsible for:

Receiving Reactive Messaging Message and transform it into a structure understand by the remote broker. This includes marshalling the payload.
If the Message contains outbound metadata (metadata set during the processing to influence the outbound structure and routing), taking them into account.
Sending the message to the remote broker.
Acknowledging the Reactive Messaging Message when the broker has accepted / acknowledged the message.

4.3.2.2: Connectors Event Driven Architecture

With this reactive messaging approach we can build our applications on an event driven approach. Some interaction or trigger emits an event to a certain channel. Subscribers of this channel consume the message and react based on the event received. This loosens the coupling in our application and lowers the cohesion between logically seperated components.

In an event driven approach everything that happens in our application gets triggered by an event. Events in the event-driven software model describe what happens within a software system. If we imagine our application as a set of logically ordered processes every task in these processes gets triggered by such an event and might emit another new event. This concept is very intuitive to apply because it is a very natural way of thinking about how everyday things and tasks work.

Some typical patterns in event-driven architecture:

4.3.2.3: Connectors Event notification

In this approach, microservices emit events through channels to trigger behaviour or notify other components about the change of a state in the application. Notification events do not carry too much data and are very light weight. This results in a very effective and ressource friendly communication between the microservices.

4.3.2.4: Connectors Event-carried state transfer

Instead of only notifying about events this approach sends a payload as a message to another component containing every information needed to perform actions triggered by this event. This model comes very close to the typical RESTful approach and can be implemented very similar. Depending on the amount of data in the payload the network traffic might suffer under the amount of data transferred.

4.3.2.4: Connectors Event-sourcing

The goal of event-sourcing is to represent every change in a system’s state as an emitted event in chronological order. The event stream becomes the principle source of truth about the applications state. Changes in state, as sequences of events, are persisted in the event stream and can be ‘replayed’.

Docs: 4.4 Event Driven with Kafka

Mon, 01 Jan 0001 00:00:00 +0000

In this stage we will have a look at the setup of a microservice and we will implement the order microservice.

Stock Microservice

The order and stock microservice have a similar setup. Therefore, we will only work on the order microservice. If you have to make changes to the stock microservice it is clearly stated.

The tasks we need to complete are the following:

Update dependencies
Create and consume shop-order-request events
Create and consume shop-order-confirmation events
Create and consume shop-order-compensation events

Task 4.4.1 - Update dependencies

First we are going to declare the dependencies we are going to use in this chapter. Add the following dependencies to your pom.xml:

GroupId	ArtifactId	Description	Detailed information
`io.quarkus`	`quarkus-smallrye-reactive-messaging-kafka`	Reactive Messaging with Apache Kafka	Using Apache Kafka with Reactive Messaging
`io.quarkus`	`quarkus-smallrye-context-propagation`	Context Propagation in Quarkus	Context Propagation in Quarkus
`io.opentracing.contrib`	`opentracing-jdbc`	OpenTracing JDBC Instrumentation	OpenTracing JDBC Instrumentation
`io.opentracing.contrib`	`opentracing-kafka-client`	OpenTracing Kafka Client	OpenTracing Kafka Client

Dependencies Task Hint

The following dependencies have to be added.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


<dependencies>
 ...
 <dependency>
 <groupId>io.quarkus</groupId>
 <artifactId>quarkus-smallrye-reactive-messaging-kafka</artifactId>
 </dependency>
 <dependency>
 <groupId>io.quarkus</groupId>
 <artifactId>quarkus-smallrye-context-propagation</artifactId>
 </dependency>
 <dependency>
 <groupId>io.opentracing.contrib</groupId>
 <artifactId>opentracing-jdbc</artifactId>
 </dependency>
 <dependency>
 <groupId>io.opentracing.contrib</groupId>
 <artifactId>opentracing-kafka-client</artifactId>
 </dependency>
 ...
</dependencies>

Task 4.4.2 - Emit the first event

We are now going event driven. Let’s create our first message and emit an event whenever a new order comes in. We create a new @ApplicationScoped class ch.puzzle.mm.kafka.order.order.boundary.ShopOrderRequestProducer in the order microservice for emitting events to our shop-order-request topic. In order to to emit messages to a topic manually we have to define a channel. To send things (payload or Message) from imperative code to a specific channel you need to use: @Channel annotation on an Emitter type field. The @Channel lets you indicate to which channel you are going to send your payloads or messages. The Emitter is the object to use to send these payloads or messages.

In the ShopOrderRequestProducer class we define a @Channel("shop-order-request") on an Emitter<ShopOrderDTO> emitter. We then create a method which creates the requests and takes a ShopOrderDTO as a parameter. The method simply emits an event to the shop-order-request channel with emitter.send(Message.of(shopOrderDTO)).

Emit Event Task Hint

The class should look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


@ApplicationScoped
public class ShopOrderRequestProducer {

 @Inject
 @Channel("shop-order-request")
 Emitter<ShopOrderDTO> emitter;

 @Traced
 public void createRequest(ShopOrderDTO shopOrderDTO) {
 emitter.send(Message.of(shopOrderDTO));
 }
}

We need to update configurations to establish the communication to our message broker and define the connectors from the channels to the topics in our message broker:

Add the following properties to the application.properties file:

1
2
3
4
5
6
7


kafka.bootstrap.servers=kafka:9092

mp.messaging.outgoing.shop-order-request.connector=smallrye-kafka
mp.messaging.outgoing.shop-order-request.topic=shop-order-request
mp.messaging.outgoing.shop-order-request.key.serializer=io.quarkus.kafka.client.serialization.JsonbSerializer
mp.messaging.outgoing.shop-order-request.value.serializer=io.quarkus.kafka.client.serialization.JsonbSerializer
mp.messaging.outgoing.shop-order-request.incerceptor.classes=io.opentracing.contrib.kafka.TracingProducerInterceptor

In order to have our OpenTracing traces injected into the message headers we will need to add some code around the message emitter to make it work properly. Update your class to the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


@ApplicationScoped
public class ShopOrderRequestProducer {

 @Inject
 @Channel("shop-order-request")
 Emitter<ShopOrderDTO> emitter;

 @Inject
 Tracer tracer;

 @Traced
 public void createRequest(ShopOrderDTO shopOrderDTO) {
 HeadersMapInjectAdapter headersMapInjectAdapter = new HeadersMapInjectAdapter();
 try (Scope scope = tracer.buildSpan("create-request").startActive(true)) {
 tracer.inject(scope.span().context(), Format.Builtin.TEXT_MAP, headersMapInjectAdapter);
 OutgoingKafkaRecordMetadata<ShopOrderDTO> metadata = OutgoingKafkaRecordMetadata.<ShopOrderDTO>builder()
 .withKey(shopOrderDTO)
 .withTopic("shop-order-request")
 .withHeaders(headersMapInjectAdapter.getRecordHeaders())
 .build();
 emitter.send(Message.of(shopOrderDTO, Metadata.of(metadata)));
 }
 }
}

Now we can simply replace the REST call we made with help of the rest-client in the previous chapter with the emitting of the event by injecting the ShopOrderRequestProducer into our ShopOrderService and using its createRequest method to emit an event to the kafka topic. Please remove the update of the status flag as well from the ShopOrderService::createShopOrder. The updating of the status flag in the order should be done when we received the confirmation event from the stock microservice.

Emit event

The createOrder function could look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


 public ShopOrder createOrder(ShopOrderDTO shopOrderDTO) {
 List<ArticleOrder> articleOrders = shopOrderDTO.articleOrders.stream().map(s -> new ArticleOrder(s.articleId, s.amount)).collect(Collectors.toList());
 articleOrders.forEach(articleOrder -> articleOrder.persist());

 // store order to shopOrder table
 ShopOrder shopOrder = new ShopOrder();
 shopOrder.setStatus(ShopOrderStatus.NEW);
 shopOrder.setArticleOrders(articleOrders);
 shopOrder.persist();
 shopOrderDTO.id = shopOrder.id;

 // fire event
 shopOrderRequestProducer.createRequest(shopOrderDTO);

 return shopOrder;
 }

Task 4.4.3 - Consume the first event

When we create a valid shop-order-request to the topic, the stock microservice will handle the event and confirm the order in case of succession or compensate the order in case of failure. It then emits events to either the shop-order-confirmation topic or the shop-order-compensation topic. Let’s create our consumer for these two cases in the order microservice.

Create a class ch.puzzle.mm.kafka.order.order.boundary.ShopOrderConfirmationConsumer which will handle all incoming events in the shop-order-confirmation topic. Define a function CompletionStage<Void> consumeOrders(Message<ShopOrderDTO> message). We annotate the consumeOrders function with @Incoming("shop-order-confirmation") to indicate that incoming events in the shop-order-confirmation channel will be handled by this function. Inside the function we want our ShopOrderService to confirm the order, which updates the status to COMPLETED. Be careful though, the changing of the status will follow in a database commit which is a blocking operation. With quarkus reactive messaging blocking operations are only allowed in reactive context when executed from a worker thread. Spawning worker threads can be achieved by using the quarkus-smallrye-context-propagation class SmallRyeManagedExecutor which propagates the context properly into the new threads.

The consumer could look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17



@ApplicationScoped
@Traced
public class ShopOrderConfirmationConsumer {

 @Inject
 ShopOrderService shopOrderService;

 @Inject
 SmallRyeManagedExecutor executor;

 @Incoming("shop-order-confirmation")
 public CompletionStage<Void> consumeOrders(Message<ShopOrderDTO> message) {
 executor.runAsync(() -> shopOrderService.confirmOrder(message.getPayload().id));
 return message.ack();
 }
}

To connect the defined channel in the @Inbound annotation to the kafka broker we need to add a few lines to the application.properties:

1
2
3
4
5



mp.messaging.incoming.shop-order-confirmation.connector=smallrye-kafka
mp.messaging.incoming.shop-order-confirmation.topic=shop-order-confirmation
mp.messaging.incoming.shop-order-confirmation.value.deserializer=ch.puzzle.mm.kafka.order.order.boundary.ShopOrderDeserializer
mp.messaging.incoming.shop-order-confirmation.interceptor.classes=io.opentracing.contrib.kafka.TracingConsumerInterceptor

We add some boilerplate code around the consumer to make the propagation of the tracing headers work:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29



@ApplicationScoped
@Traced
public class ShopOrderConfirmationConsumer {

 private final Logger logger = LoggerFactory.getLogger(ShopOrderConfirmationConsumer.class.getName());

 @Inject
 ShopOrderService shopOrderService;

 @Inject
 Tracer tracer;

 @Inject
 SmallRyeManagedExecutor executor;

 @Incoming("shop-order-confirmation")
 public CompletionStage<Void> consumeOrders(Message<ShopOrderDTO> message) {
 Optional<IncomingKafkaRecordMetadata> metadata = message.getMetadata(IncomingKafkaRecordMetadata.class);
 if (metadata.isPresent()) {
 SpanContext extract = tracer.extract(Format.Builtin.TEXT_MAP, new HeadersMapExtractAdapter(metadata.get().getHeaders()));
 try (Scope scope = tracer.buildSpan("consume-confirmation").asChildOf(extract).startActive(true)) {
 executor.runAsync(() -> shopOrderService.confirmOrder(message.getPayload().id));
 return message.ack();
 }
 }
 return message.nack(new RuntimeException());
 }
}

Extend the ShopOrderService with the confirmOrder method:

1
2
3
4
5
6
7



 @Transactional
 public void confirmOrder(Long id) {
 ShopOrder shopOrder = ShopOrder.findById(id);
 shopOrder.setStatus(ShopOrderStatus.COMPLETED);
 shopOrder.persistAndFlush();
 }

Task 4.4.4 - Compensation

Now it is up to you to do the same for the shop-order-compensation events. Create a consumer listening on the “shop-order-compensation” topic to compensate your failed orders. Create a class ShopOrderCompensationConsumer in the same package as the ShopOrderConfirmationConsumer and update your application.properties to connect your defined connectors to the kafka broker.

Create a class ch.puzzle.mm.kafka.order.order.boundary.ShopOrderCompensationConsumer which will confirm and compensate orders.

Compensation consumer

The class should look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


@ApplicationScoped
@Traced
public class ShopOrderCompensationConsumer {

 @Inject
 ShopOrderService shopOrderService;

 @Inject
 Tracer tracer;

 @Inject
 SmallRyeManagedExecutor executor;

 @Incoming("shop-order-compensation")
 public CompletionStage<Void> consumeOrders(Message<ShopOrderDTO> message) {
 Optional<IncomingKafkaRecordMetadata> metadata = message.getMetadata(IncomingKafkaRecordMetadata.class);
 if (metadata.isPresent()) {
 SpanContext extract = tracer.extract(Format.Builtin.TEXT_MAP, new HeadersMapExtractAdapter(metadata.get().getHeaders()));
 try (Scope scope = tracer.buildSpan("consume-compensation").asChildOf(extract).startActive(true)) {
 compensateOrder(message.getPayload());
 return message.ack();
 }
 }
 return message.nack(new RuntimeException());
 }

 private void compensateOrder(ShopOrderDTO shopOrderDTO) {
 executor.runAsync(() -> shopOrderService.compensateOrder(shopOrderDTO.id));
 }
}

Do not forget to update your application.properties as well to connector your defined connector @Incoming("shop-order-compensation") to your kafka broker and topic.

1
2
3
4


mp.messaging.incoming.shop-order-compensation.connector=smallrye-kafka
mp.messaging.incoming.shop-order-compensation.topic=shop-order-compensation
mp.messaging.incoming.shop-order-compensation.value.deserializer=ch.puzzle.mm.kafka.order.order.boundary.ShopOrderDeserializer
mp.messaging.incoming.shop-order-compensation.interceptor.classes=io.opentracing.contrib.kafka.TracingConsumerInterceptor

Task 4.4.5 - Add business to your service

In the business parts of the ShopOrderService you will add logic to confirm and compensate your orders! Add the two following functions to your service:

ShopOrderService::compensateOrder: Find order with id id, set status to ShopOrderStatus.INCOMPLETE and save and flush.
ShopOrderService::confirmOrder: Find order with id id, set status to ShopOrderStatus.COMPLETED and save and flush.

Business hint

These two functions could look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


@Transactional
public void compensateOrder(Long id) {
 ShopOrder shopOrder = ShopOrder.findById(id);
 shopOrder.setStatus(ShopOrderStatus.INCOMPLETE);
 shopOrder.persistAndFlush();
}

@Transactional
public void confirmOrder(Long id) {
 ShopOrder shopOrder = ShopOrder.findById(id);
 shopOrder.setStatus(ShopOrderStatus.COMPLETED);
 shopOrder.persistAndFlush();
}

Docs: 4.5 Running the Lab

Mon, 01 Jan 0001 00:00:00 +0000

In the previous section we implemented the microservice. Let’s build and run it.

Building

The kafka-order microservice provides a multistage Dockerfile.multistage.jvm for building the java application and creating a docker container.

Task 4.5.4 - Build docker container

Docker Version

Change to base directory of this lab <PROJECT_ROOT>/code/kafka and run docker build.

1

docker build -f kafka-order/src/main/docker/Dockerfile.multistage.jvm -t microservices-lab-kafka-order:latest kafka-order

Running

Task 4.5.2 - Starting with Docker Compose

Enter a shell and change your directory to the <PROJECT_ROOT>/code/kafka/docker folder.

If you have any previous lab running stop these containers

1

docker-compose down --remove-orphans

Start the microservices with

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


docker-compose up -d
Starting docker_order_1 ... done
Starting docker_stock_1 ... done
Starting docker_viewer_1 ... done
Starting debezium-kafka ... done
Starting debezium-zookeeper ... done
Starting docker_prometheus_1 ... done
Starting docker_grafana_1 ... done
Starting docker_alertmanager_1 ... done
Starting docker_stock-db_1 ... done
Starting docker_order-db_1 ... done
Starting docker_jaeger_1 ... done

Docs: 4.6 Using the Lab

Mon, 01 Jan 0001 00:00:00 +0000

Test your microservice

If your microservices are all up and running, open two tabs in your browser. View the kafka-viewer microservice on http://localhost:8082 and open up the Jaeger UI on http://localhost:16686 . The kafka-viewer will open up an active connection to your kafka topics and show you messages passing through the topics. In the Jaeger UI you can follow your traces through the system.

Now fire a request against the order API to create a new shop order. Let’s do a POST request to our /shop-orders endpoint and create a shop order with 2 pieces of article 1 and 3 pieces of article 2.

Request Task Hint

Create the POST request with curl.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16



curl --request POST \
 --url http://localhost:8080/shop-orders \
 --header 'Content-Type: application/json' \
 --data '{
 "articleOrders": [
 {
 "articleId": 1,
 "amount": 2
 },
 {
 "articleId": 2,
 "amount": 3
 }
 ]
}'

Check your browser and see what insights the kafka-viewer and the Jaeger UI give us. You can clearly see where the event was passing through our system in the Jaeger UI. The trace passes from the order microservices’ ShopOrderResource REST resource to the shop-order-request topic on kafka (seen in the kafka viewer) with some content like:

1

{"articleOrders":[{"amount":2,"articleId":1},{"amount":3,"articleId":2}],"id":100017}

Will then be passed to the shop-order-confirmation topic to be again confirmed by the order microservice! Our workflow now works.

Task 4.6.1 - Follow your traces through the code

Can you follow the traced produced and observed in the Jaeger UI through your own code?

Task 4.6.2 - Compensation instead of confirmations

Can you try to get a compensation event to test your compensation workflow as well? Try to provoke an error to make the stock microservice deny the order!

Compensation Hint

Create the POST request similar to the one before, but let the article amounts go nuts! That way the stock microservice will throw a ArticleOutOfStockException.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16



curl --request POST \
 --url http://localhost:8080/shop-orders \
 --header 'Content-Type: application/json' \
 --data '{
 "articleOrders": [
 {
 "articleId": 1,
 "amount": 20
 },
 {
 "articleId": 2,
 "amount": 30
 }
 ]
}'

Task 4.6.3 - Observability test

Can you identify where the problem was in the previous request by just using our observability stack?

Observability Hint

Of course you can! Head over to the Jaeger UI and inspect the latest traces. You will directly see a trace with two errors. If you open the trace and inspect the spans you can investigate the error that happened and you will find the wanted ArticleOutOfStockException.

Task 4.6.4 - Enhance and Test observability (optional)

Optional Section

The following content is optional. If you have time, feel free to proceed.

Your task now is to enhance the observability in your order microservice. We would like to see metrics to see how many messages we produce and consume in the microservice. In order to do this you will add metrics to your microservice to track:

order_requests_produced
order_confirmation_consumed
order_compensation_consumed

Hint: In other labs you have already exposed custom metrics. Try to implement your own now!

Observability Hint 2

You can annotate your existing functions which handle incoming and outgoing messages with your custom metric like this:

1
2
3
4
5
6


 @Counted(name = "kafka_order_requested", absolute = true, description = "number of orders requested", tags = {"application=kafka-order"})
 @Traced
 public void createRequest(ShopOrderDTO shopOrderDTO) {
 ...
 }
 }

You can implement the consuming counterpart just the same!

Can you identify with your newly implemented metrics after rebuilding and rerunning your system how many messages were confirmed and compensated after some requests?

Docs: 5.1 Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

Source Code

The source-code used for this lab is in the folder /code/debezium

Online Repository Link: https://github.com/acend/microservices-lab/tree/main/code/debezium .

Task 5.1.1 - Getting your workspace ready

If you still have a previous lab running, stop it with docker-compose down --remove-orphans.
Open your IDE or favorite tool and switch to the referenced source folder.

Docs: 5.2 Docker Environment

Mon, 01 Jan 0001 00:00:00 +0000

Environment Setup

Changes to the docker-compose file

The setup for this lab is similar as the previous. However, we use Kafka Connect with Debezium PostgreSQL connector to write messages to Kafka.

Kafka Connect with Debezium

For Kafka Connect a pre-built docker container based on the Kafka Strimzi image will be used.

The following was added to the docker-compose.yaml file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


 connect:
 image: quay.io/acend/microservices-lab-debezium-connect:latest
 hostname: connect
 container_name: debezium-connect
 ports:
 - 8083:8083
 networks:
 - debezium
 depends_on:
 - kafka
 - order-db
 - stock-db
 - zookeeper
 command: /opt/kafka/kafka_connect_run.sh
 environment:
 - KAFKA_CONNECT_BOOTSTRAP_SERVERS=kafka:9092
 - |
 KAFKA_CONNECT_CONFIGURATION=
 key.converter=org.apache.kafka.connect.json.JsonConverter
 value.converter=org.apache.kafka.connect.json.JsonConverter
 key.converter.schemas.enable=false
 value.converter.schemas.enable=false
 group.id=connect-debezium
 offset.storage.topic=connect-debezium-offsets
 config.storage.topic=connect-debezium-configs
 status.storage.topic=connect-debezium-status
 offset.storage.replication.factor=1
 config.storage.replication.factor=1
 status.storage.replication.factor=1
 consumer.interceptor.classes=io.opentracing.contrib.kafka.TracingConsumerInterceptor
 producer.interceptor.classes=io.opentracing.contrib.kafka.TracingProducerInterceptor
 - KAFKA_CONNECT_METRICS_ENABLED=true
 - JAEGER_SERVICE_NAME=debezium-connect
 - JAEGER_AGENT_HOST=jaeger
 - STRIMZI_TRACING=jaeger

Database docker image

The Debezium PostgreSQL Connector needs to access the WAL log. We therefore switch to the quay.io/debezium/postgres:13 image maintained by Debezium. This image provides us the configurations and permissions needed to access the WAL log.

The following changes have been made in the docker-compose.yaml file:

order-db: changed image to quay.io/debezium/postgres:13
stock-db: changed image to quay.io/debezium/postgres:13

Task 5.2.1 - Inspecting the docker environment

Have a look at the /code/debezium/docker folder and get familiar with the docker environment specified in docker-compose.yaml.

Docs: 5.3 Microservices Setup

Mon, 01 Jan 0001 00:00:00 +0000

In this stage we will have a look at the setup and configuration of the Order microservice.

Task 5.3.1 - Inspect Dependencies

To simplify the use of the Outbox pattern the following dependencies have already been added to the pom.xml.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


<dependencies>
 <dependency>
 <groupId>io.debezium</groupId>
 <artifactId>debezium-quarkus-outbox</artifactId>
 <version>${debezium.version}</version>
 </dependency>
 <dependency>
 <groupId>io.quarkus</groupId>
 <artifactId>quarkus-smallrye-reactive-messaging-kafka</artifactId>
 </dependency>
 <dependency>
 <groupId>io.opentracing.contrib</groupId>
 <artifactId>opentracing-kafka-client</artifactId>
 </dependency>
 ...
</dependencies>

Dependency	Description
`quarkus-smallrye-reactive-messaging-kafka`	Reactive Messaging extensions to receive and write Messages to or from Apache Kafka.
`debezium-quarkus-outbox`	Provides a reusable, configurable component that facilitates the use of the Outbox pattern in conjunction with debezium.
`opentracing-kafka-client`	Injects and reads tracing information from Headers of an Apache Kafka message.

full source of pom.xml

Debezium Quarkus Outbox Extension

This quarkus extension provides us with the ExportedEvent interface. For every different event in our application we will implement this interface to create our custom event.

The application will then be able to emit the event using the javax.enterprise.event.Event class. The extension will also take care of writing the emitted events to our outboxevent database table.

Task 5.3.2 - Inspect application properties

The configurations for the Reactive-Messaging and the Outbox Pattern are located in the quarkus configuration file application.properties.

Have a look at the Quarkus Debezium Outbox pattern configuration.

1
2
3
4
5
6
7
8


quarkus.debezium-outbox.table-name=outboxevent
quarkus.debezium-outbox.id.name=id
quarkus.debezium-outbox.aggregate-id.name=aggregateid
quarkus.debezium-outbox.aggregate-type.name=aggregatetype
quarkus.debezium-outbox.type.name=type
quarkus.debezium-outbox.timestamp.name=timestamp
quarkus.debezium-outbox.payload.name=payload
quarkus.debezium-outbox.remove-after-insert=false

This configuration will define the table and field names of our outbox table. For further details, please refer to the Debezium Outbox Quarkus Extension documentation.

Please take a note of the configuration quarkus.debezium-outbox.remove-after-insert=false (default true). With disabling this our events stay in the outbox table forever. In theory there is no need to keep these persisted. Debezium will get the create event (row insert) from the WAL log. The provided EventRouter is only sensible to create events. For debugging we have set this to false.

Now have a look at the reactive messaging configuration.

1
2
3
4
5
6
7


mp.messaging.incoming.stock.topics=stock-stockcomplete-events,stock-stockincomplete-events
mp.messaging.incoming.stock.group.id=order-service
mp.messaging.incoming.stock.connector=smallrye-kafka
mp.messaging.incoming.stock.bootstrap.servers=kafka:9092
mp.messaging.incoming.stock.key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
mp.messaging.incoming.stock.value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
mp.messaging.incoming.stock.interceptor.classes=io.opentracing.contrib.kafka.TracingConsumerInterceptor

With this configuration our application will use a Kafka connector and will provide access to the messages using an incoming channel named stock. This channel will contain events from both topics stock-stockcomplete-events and stock-stockincomplete-events.

To get more details about the concepts of Connectors and Channels you may head over to the SmallRye Reactive Messaging Documentation

Have a look at the snippet below which is used do consume the messages

1
2
3
4


@Incoming("stock")
public CompletionStage<Void> onMessage(KafkaRecord<String, String> message) {
 // implementation
}

If you want to know more about reactive messaging in quarkus have a look at the Quarkus Reactive Messaging Guide .

Docs: 5.4 Preparing the Database

Mon, 01 Jan 0001 00:00:00 +0000

Since we had a brief look at the configuration we are now going to define the outboxevent table. We will also create an event history table consumedevent to ignore events which have already been processed (idempotency).

Outbox Event Table

The EventRouter used by the Debezium Connector expects the outboxevent table to have a certain layout.

In our application configuration application.properties we mostly stuck with the defaults. Therefore, our outboxevent table should look like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


$ admin=# \d outboxevent;
 Table "outboxevent"
 Column | Type | Collation | Nullable | Default
--------------------+-----------------------------+-----------+----------+---------
 id | uuid | | not null |
 aggregatetype | character varying(64) | | not null |
 aggregateid | character varying(64) | | not null |
 type | character varying(32) | | not null |
 timestamp | timestamp without time zone | | not null |
 tracingspancontext | character varying(255) | | |
 payload | text | | |

Indexes:
 "outboxevent_pkey" PRIMARY KEY, btree (id)

Task 5.4.1 - Create outbox table with Liquibase

Limited lab duration

This step is already implemented. For a better understanding we encourage you to have a quick read but you may also skip it.

Change the resources/db/changeLog.xml to include the specification for the outboxevent table. Create the table according the description above.

Task Hint

Your configuration for the outboxevent table should look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


 <createTable tableName="outboxevent">
 <column name="id" type="UUID">
 <constraints primaryKey="true" nullable="false"/>
 </column>
 <column name="aggregatetype" type="varchar(64)">
 <constraints nullable="false"/>
 </column>
 <column name="aggregateid" type="varchar(64)">
 <constraints nullable="false"/>
 </column>
 <column name="type" type="varchar(32)">
 <constraints nullable="false"/>
 </column>
 <column name="timestamp" type="timestamp">
 <constraints nullable="false"/>
 </column>
 <column name="tracingspancontext" type="varchar(255)">
 <constraints nullable="true"/>
 </column>
 <column name="payload" type="text">
 <constraints nullable="true"/>
 </column>
 </createTable>

full source changeLog.xml

Task 5.4.2 - Create table for event history

Limited lab duration

This step is already implemented. For a better understanding we encourage you to have a quick read but you may also skip it.

Since we want to track the events which were already processed we create a table consumedevent. In this table we are going to store the event id and a received timestamp. We can then use this to check if an incoming event has already been processed by just looking up the event id. This will provide us with idempotency for events which are delivered more than once.

As in the previous task, create a table with the following structure in the liquibase config changeLog.xml.

1
2
3
4
5
6
7
8
9


$ admin=# \d consumedevent;
 Table "consumedevent"
 Column | Type | Collation | Nullable | Default
----------+-----------------------------+-----------+----------+---------
 id | uuid | | not null |
 received | timestamp without time zone | | not null |

Indexes:
 "consumedevent_pkey" PRIMARY KEY, btree (id)

Task Hint

Your configuration for the consumedevent table should look like this:

1
2
3
4
5
6
7
8


 <createTable tableName="consumedevent">
 <column name="id" type="UUID">
 <constraints primaryKey="true" nullable="false"/>
 </column>
 <column name="received" type="timestamp">
 <constraints nullable="false"/>
 </column>
 </createTable>

full source changeLog.xml

Docs: 5.5 Define Events

Mon, 01 Jan 0001 00:00:00 +0000

In the previous section we have configured the tables for our event handling. Now it’s time to define our actual events for the application.

Event overview

We will build the following event flow. Keep in mind that messages are not directly written to Kafka.

Define Events

To define our events we have to implement ExportedEvent. This event implementation is responsible to fill up our outboxevent table. The definition of the values is as follows:

Field	Description
`id`	unique id. Can be used by consumers to detect duplicates. Should be generated when creating the event Object.
`aggregatetype`	the origin of a given event. It could be the microservice name or for example in case of domain-driven design the origin domain. For example “order” or “stock”.
`aggregateid`	contains the id affected by a given event. In case of the `aggregatetype` order this could be the order id. This id is used as the key for Kafka messages. Only messages with the same key are guaranteed to be processed in order as they are placed in the same partition of a kafka topic.
`type`	the actual type of the event. For example “OrderCreated” or “StockComplete”. We will use this information to trigger specific event handlers on the consumer side.
`payload`	in our example a JSON structure which contains the event body. For example the actual Order with Article id and Amount ordered.

Task 5.5.1 - Implement OrderCreated Event

Given the information above. Complete the ch.puzzle.mm.debezium.event.entity.OrderCreatedEvent event implementation.
- Set the required fields accordingly (AGGREGATE_TYPE and EVENT_TYPE)

For help, here’s an excerpt of the order outboxevent table. For simplicity the timestamp is omitted.

1
2
3
4
5
6
7


admin=# select id, aggregatetype, aggregateid, type, payload from outboxevent;
 id | aggregatetype | aggregateid | type | payload
-----------+---------------+-------------+----------------+---------------------------------------------------------
 bed8bd... | Order | 100000 | OrderCreated | {"orderId":100000,"items":[{"articleId":1,"amount":9}]}
 28e777... | Order | 100002 | OrderCreated | {"orderId":100002,"items":[{"articleId":1,"amount":3}]}
 e09acc... | Order | 100000 | OrderCancelled | {"orderId":100000,"items":[{"articleId":1,"amount":9}]}
 3709f2... | Order | 100004 | OrderCreated | {"orderId":100004,"items":[{"articleId":1,"amount":3}]}

Task Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60


public class OrderCreatedEvent implements ExportedEvent<String, JsonNode> {

 private static ObjectMapper mapper = new ObjectMapper();

 private static final String AGGREGATE_TYPE = "Order";
 private static final String EVENT_TYPE = "OrderCreated";

 private final UUID id;
 private final Long orderId;
 private final JsonNode jsonNode;
 private final Instant timestamp;

 public OrderCreatedEvent(Instant created, ShopOrder shopOrder) {
 this.id = UUID.randomUUID();
 this.orderId = shopOrder.id;
 this.timestamp = created;
 this.jsonNode = asJson(shopOrder);
 }

 @Override
 public String getAggregateId() {
 return String.valueOf(orderId);
 }

 @Override
 public String getAggregateType() {
 return AGGREGATE_TYPE;
 }

 @Override
 public String getType() {
 return EVENT_TYPE;
 }

 @Override
 public Instant getTimestamp() {
 return timestamp;
 }

 @Override
 public JsonNode getPayload() {
 return jsonNode;
 }

 public ObjectNode asJson(ShopOrder order) {
 ObjectNode asJson = mapper.createObjectNode()
 .put("orderId", order.id);

 ArrayNode items = asJson.putArray("items");

 for (ArticleOrder article : order.getArticleOrders()) {
 items.add(mapper.createObjectNode()
 .put("articleId", article.getArticleId())
 .put("amount", article.getAmount())
 );
 }

 return asJson;
 }
}

Docs: 5.6 Event Handling

Mon, 01 Jan 0001 00:00:00 +0000

In the previous section we defined our events. Now let’s have a look at the application logic to fire and consume these.

Event overview

We will build the following event flow. Keep in mind that messages are not directly written to Kafka.

Creating Orders

New orders are received by requests to our RESTful API /shop-orders. They are handled by the ch.puzzle.mm.debezium.order.boundary.ShopOrderResource.

The required POST body corresponds to the ShopOrderDTO class.

1
2
3
4
5
6
7
8


{
 "articleOrders" : [
 {
 "articleId" : 1,
 "amount" : 3
 }
 ]
}

Task 5.6.1 - Implement order creation and fire event

The actual work for creating an order is done by method createOrder in ch.puzzle.mm.debezium.order.control.ShopOrderService class.

1
2
3
4
5
6
7
8
9


public ShopOrder createOrder(ShopOrderDTO shopOrderDTO) {
 // TODO: implementation - create ArticleOrder list

 // TODO: implementation - create new shopOrder

 // TODO: fire OrderCreatedEvent

 return new ShopOrder();
}

Finish implementation of the createOrder Method:

Create list of ArticleOrder entities from article list in ShopOrderDto
Create a new ShopOrder
- status: Set to ShopOrderStatus.NEW
- articleOrders: Set created list of ArticleOrder
- persist the order
Fire the OrderCreatedEvent event
Return the newly created order

Hint list of ArticleOrder creation

1
2
3


 List<ArticleOrder> articleOrders = shopOrderDTO.articleOrders.stream()
 .map(s -> new ArticleOrder(s.articleId, s.amount))
 .collect(Collectors.toList());

Hint ShopOrder creation

1
2
3
4
5


// store order to shopOrder table
ShopOrder shopOrder = new ShopOrder();
shopOrder.setStatus(ShopOrderStatus.NEW);
shopOrder.setArticleOrders(articleOrders);
shopOrder.persist();

Hint event firing

Fire events with

1
2


// fire event (outbox table)
event.fire(new OrderCreatedEvent(Instant.now(), shopOrder));

Complete Task Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


@ApplicationScoped
public class ShopOrderService {

 @Inject
 Event<ExportedEvent<?, ?>> event;

 public ShopOrder createOrder(ShopOrderDTO shopOrderDTO) {
 List<ArticleOrder> articleOrders = shopOrderDTO.articleOrders.stream().map(s -> new ArticleOrder(s.articleId, s.amount)).collect(Collectors.toList());

 // store order to shopOrder table
 ShopOrder shopOrder = new ShopOrder();
 shopOrder.setStatus(ShopOrderStatus.NEW);
 shopOrder.setArticleOrders(articleOrders);
 shopOrder.persist();

 // fire event (outbox table)
 event.fire(new OrderCreatedEvent(Instant.now(), shopOrder));

 return shopOrder;
 }

 // ...
}

Cancelling Order

In this lab you can delete orders by POST to /shop-orders/{id}/status. For simplicity cancelling is only allowed if the order is in state completed (stock have been deducted).

Task 5.6.2 - Implement order cancellation and fire event

Limited lab duration

This step is already implemented. For a better understanding we encourage you to have a quick read but you may also skip it.

The actual work for cancelling an order is done by cancelOrder method in ch.puzzle.mm.debezium.order.control.ShopOrderService class.

Fetch the ShopOrder by orderId (already done)
Check Status (already done)
Set status of order to ShopOrderStatus.CANCELLED
Fire the OrderCancelledEvent event
Return the modified order

Complete Task Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


@ApplicationScoped
public class ShopOrderService {

 @Inject
 Event<ExportedEvent<?, ?>> event;

 // ...

 public ShopOrder cancelOrder(long orderId) {
 ShopOrder order = ShopOrder.getByIdOrThrow(orderId);
 if (order.getStatus().canCancel()) {
 order.setStatus(ShopOrderStatus.CANCELLED);
 event.fire(new OrderCancelledEvent(Instant.now(), order));
 return order;
 } else {
 throw new IllegalStateException("Cannot cancel Order " + orderId);
 }
 }
}

Consuming Stock Events

The following configuration defines the incoming stock channel containing the messages from the Kafka topics stock-stockcomplete-events and stock-stockincomplete-events

1

mp.messaging.incoming.stock.topics=stock-stockcomplete-events,stock-stockincomplete-events

Our event consumer class is the ch.puzzle.mm.debezium.event.boundary.KafkaEventConsumer.

Task 5.6.3 - Consuming Kafka messages

Consume incoming events in KafkaEventConsumer and delegate the processing to the StockEventHandler.

Finish implementation of onMessage in ch.puzzle.mm.debezium.event.boundary.KafkaEventConsumer

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


public CompletionStage<Void> onMessage(KafkaRecord<String, String> message) {
 return CompletableFuture.runAsync(() -> {
 try (final Scope span = tracer.buildSpan("handle-stock-message")
 .asChildOf(TracingKafkaUtils.extractSpanContext(message.getHeaders(), tracer))
 .startActive(true)) {

 // TODO: implementation - read id, eventType from headers

 // TODO: implementation - delegate message to StockEventHandler
 } catch (Exception e) {
 logger.error("Error while preparing articlestock", e);
 throw e;
 }
 }).thenRun(message::ack);
}

Annotate method onMessage as handler for the incoming stock channel with @Incoming(...).
Read the Kafka Headers id and eventType from the message using getHeadersAsString.
Inject the StockEventHandler and delegate processing to onStockEvent of StockEventHandler

Incoming definition Hint

Channels (incoming or outgoing) can be annotated on method level with @Incoming.

1
2


@Incoming("stock")
public CompletionStage<Void> onMessage(KafkaRecord<String, String> message) { }

Read Kafka Headers Hint

Use the provided method getHeaderAsString.

1
2


String eventId = getHeaderAsString(message, "id");
String eventType = getHeaderAsString(message, "eventType");

Calling StockEventHandler Hint

1
2
3
4
5
6


stockEventHandler.onStockEvent(
 UUID.fromString(eventId),
 eventType,
 message.getKey(),
 message.getPayload(),
 message.getTimestamp());

Complete Task Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


@Incoming("stock")
public CompletionStage<Void> onMessage(KafkaRecord<String, String> message) {
 return CompletableFuture.runAsync(() -> {
 try (final Scope span = tracer.buildSpan("handle-stock-message").asChildOf(TracingKafkaUtils.extractSpanContext(message.getHeaders(), tracer)).startActive(true)) {
 logger.debug("Kafka message with key = {} arrived", message.getKey());

 logHeaders(message);

 String eventId = getHeaderAsString(message, "id");
 String eventType = getHeaderAsString(message, "eventType");

 stockEventHandler.onStockEvent(
 UUID.fromString(eventId),
 eventType,
 message.getKey(),
 message.getPayload(),
 message.getTimestamp()
 );
 } catch (Exception e) {
 logger.error("Error while preparing articlestock", e);
 throw e;
 }
 }).thenRun(message::ack);
}

Task 5.6.4 - Creating an event log

Limited lab duration

This step is already implemented. For a better understanding we encourage you to have a quick read but you may also skip it.

The EventLog class uses the ConsumedEvent entity to keep track of processed events.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


@ApplicationScoped
public class EventLog {

 private static final Logger logger = LoggerFactory.getLogger(EventLog.class);

 public void processed(UUID eventId) {
 // TODO: implementation - store
 }

 public boolean alreadyProcessed(UUID eventId) {
 // TODO: implementation - check exists
 return false;
 }
}

Transaction management

Ensure that processed requires a transaction by using Transactional.TxType.MANDATORY
Ensure that alreadyProcessed requires a transaction by using Transactional.TxType.MANDATORY

Transaction management Hint

Adding the @Transactional annotation ensures that methods throw an exception if there isn’t an existing transaction context. Annotate both methods with:

1

@Transactional(value = Transactional.TxType.MANDATORY)

Remember that we use the Panache extension and that our entities provide methods to easily query database records. You can get more details about using the Active Record Pattern in Quarkus in the Simplified Hibernate ORM with Panache Guide .

Implement both EventLog methods.

Implement method processed to write a new ConsumedEvent with the eventId and the current timestamp to the database.
Implement method alreadyProcessed to check the database for having a record with the given eventId.

Complete EventLog Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


@Traced
@ApplicationScoped
public class EventLog {

 private static final Logger logger = LoggerFactory.getLogger(EventLog.class);

 @Transactional(value = Transactional.TxType.MANDATORY)
 public void processed(UUID eventId) {
 ConsumedEvent.persist(new ConsumedEvent(eventId, Instant.now()));
 }

 @Transactional(value = Transactional.TxType.MANDATORY)
 public boolean alreadyProcessed(UUID eventId) {
 logger.info("Looking for event with id {} in message log", eventId);
 return ConsumedEvent.findByIdOptional(eventId).isPresent();
 }
}

Task 5.6.5 - Processing stock events

Implement the event handling in StockEventHandler

Check if message is already processed by using the EventLog.
- If so, skip the message.
Depending on eventType call the corresponding method in ShopOrderService
- StockComplete: deserialize the event and call onStockCompleteEvent
- StockIncomplete: deserialize the event and call onStockIncompleteEvent
- If none of them: log warning about an unknown event.
Register the event as processed using the EventLog

StockEventHandler Hint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


public void onStockEvent(UUID eventId, String eventType, String key, String event, Instant ts) {
 if (eventLog.alreadyProcessed(eventId)) {
 logger.info("Event with id {} was already processed, ignore.", eventId);
 return;
 }

 logger.info("Received '{}' event {} - OrderId: {}, ts: '{}'", eventType, eventId, key, ts);
 if (eventType.equalsIgnoreCase("StockComplete")) {
 shopOrderService.onStockCompleteEvent(deserialize(event));
 } else if (eventType.equalsIgnoreCase("StockIncomplete")) {
 shopOrderService.onStockIncompleteEvent(deserialize(event));
 } else {
 logger.warn("Ignoring unknown event '{}'", eventType);
 }

 eventLog.processed(eventId);
}

Task 5.6.6 - Complete the order management

For the order to be managed completely we have to implement the methods onStockCompleteEvent and onStockIncompleteEvent in our ShopOrderService.

Implement the stock complete event

Find ShopOrder in database using the orderId from the event
Set status to ShopOrderStatus.COMPLETED

OnStockCompleteEvent Hint

1
2
3
4
5


public void onStockCompleteEvent(ShopOrderStockResponse stockComplete) {
 ShopOrder.findByIdOptional(stockComplete.orderId).ifPresent(o -> {
 ((ShopOrder)o).setStatus(ShopOrderStatus.COMPLETED);
 });
}

Implement the stock incomplete event

Find ShopOrder in database using the orderId from the event
Set status to ShopOrderStatus.STOCK_INCOMPLETE

OnStockIncompleteEvent Hint

1
2
3
4
5


public void onStockIncompleteEvent(ShopOrderStockResponse stockIncomplete) {
 ShopOrder.findByIdOptional(stockIncomplete.orderId).ifPresent(o -> {
 ((ShopOrder) o).setStatus(ShopOrderStatus.STOCK_INCOMPLETE);
 });
}

Docs: 5.7 Running the Lab

Mon, 01 Jan 0001 00:00:00 +0000

In the previous section we implemented the microservice. Let’s build and run it.

Building

The debezium-order microservice provides a multistage Dockerfile.multistage.jvm for building the java application and creating a docker container.

Task 5.7.1 - Build docker container

Docker Version

Change to base directory of this lab <PROJECT_ROOT>/code/debezium and run docker build.

1

docker build -f debezium-order/src/main/docker/Dockerfile.multistage.jvm -t microservices-lab-debezium-order:latest debezium-order

Running

Task 5.7.2 - Starting with Docker Compose

Enter a shell and change your directory to the <PROJECT_ROOT>/code/debezium/docker folder.

If you have any previous lab running stop these containers

1

docker-compose down --remove-orphans

Start the microservices with

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


docker-compose up -d
Creating debezium-grafana ... done
Creating debezium-order-db ... done
Creating debezium-prometheus ... done
Creating debezium-jaeger ... done
Creating debezium-alertmanager ... done
Creating debezium-stock-db ... done
Creating debezium-zookeeper ... done
Creating debezium-stock ... done
Creating debezium-order ... done
Creating debezium-kafka ... done
Creating debezium-connect ... done
Creating debezium-viewer ... done

Docs: 5.8 Using the Lab

Mon, 01 Jan 0001 00:00:00 +0000

We have finished our work on the environment and our order microservice. We are now ready to setup our debezium connectors.

Debezium Connectors

With starting the environment we have also started Kafka Connect. We need to configure our Debezium PostgreSQL Connector to listen for events on the order-db and stock-db database.

Task 5.8.2 - Configuring Debezium Connectors

Configuring the Debezium PostgreSQL Connector for the order-db database.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


curl --request POST \
 --url http://localhost:8083/connectors \
 --header 'Accept: application/json' \
 --header 'Content-Type: application/json' \
 --data '{
 "name": "order-events",
 "config": {
 "connector.class" : "io.debezium.connector.postgresql.PostgresConnector",
 "tasks.max" : "1",
 "database.hostname" : "order-db",
 "database.port" : "5432",
 "database.user" : "admin",
 "database.password" : "1234",
 "database.dbname" : "admin",
 "database.server.name" : "order-db",
 "schema.include.list" : "public",
 "table.include.list" : "public.outboxevent",
 "tombstones.on.delete" : "false",
 "transforms" : "outbox,topiccase",
 "transforms.outbox.type" : "io.debezium.transforms.outbox.EventRouter",
 "transforms.outbox.route.by.field" : "type",
 "transforms.outbox.route.topic.replacement" : "order-${routedByValue}-events",
 "transforms.outbox.table.field.event.timestamp" : "timestamp",
 "transforms.outbox.table.fields.additional.placement" : "type:header:eventType",
 "transforms.topiccase.type" : "ch.puzzle.mm.debezium.connect.smt.ChangeTopicCase",
 "transforms.topiccase.toCase" : "lower"
 }
}'

Configuring the Debezium PostgreSQL Connector for the stock-db database.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


curl --request POST \
 --url http://localhost:8083/connectors \
 --header 'Accept: application/json' \
 --header 'Content-Type: application/json' \
 --data '{
 "name": "stock-events",
 "config": {
 "connector.class" : "io.debezium.connector.postgresql.PostgresConnector",
 "tasks.max" : "1",
 "database.hostname" : "stock-db",
 "database.port" : "5432",
 "database.user" : "admin",
 "database.password" : "1234",
 "database.dbname" : "admin",
 "database.server.name" : "stock-db",
 "schema.include.list" : "public",
 "table.include.list" : "public.outboxevent",
 "tombstones.on.delete" : "false",
 "transforms" : "outbox,topiccase",
 "transforms.outbox.type" : "io.debezium.transforms.outbox.EventRouter",
 "transforms.outbox.route.by.field" : "type",
 "transforms.outbox.route.topic.replacement" : "stock-${routedByValue}-events",
 "transforms.outbox.table.field.event.timestamp" : "timestamp",
 "transforms.outbox.table.fields.additional.placement" : "type:header:eventType",
 "transforms.topiccase.type" : "ch.puzzle.mm.debezium.connect.smt.ChangeTopicCase",
 "transforms.topiccase.toCase" : "lower"
 }
}'

As you can see in the configuration of the connectors we specify our connection information and our tables with the table.include.list and our transforms. We use the EventRouter transformation which handles our outboxevent table.

Check if the connectors are created successfully

1
2
3
4
5
6
7
8


curl --request GET \
 --url http://127.0.0.1:8083/connectors \
 --header 'Accept: application/json'

[
 "order-events",
 "stock-events"
]

Sample Requests

We provide a requests/api.http file containing all requests needed for this lab. This file is compatible with IntelliJ IDEA and also includes the curl version for the requests.

Get article stock information

1

curl --request GET --url http://localhost:8081/article-stocks

Get order information

1

curl --request GET --url http://localhost:8080/shop-orders

Create new order

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


curl --request POST \
 --url http://localhost:8080/shop-orders \
 --header 'Content-Type: application/json' \
 --data '{
 "articleOrders" : [
 {
 "articleId" : 1,
 "amount" : 1
 }
 ]
}'

Task 5.8.3 - Using the application

There is a Viewer Microservice running which provides an overview of events in the Kafka Topics. It uses websockets and shows newly arriving events.

Feel free to test the following scenarios:

Order Article 1 with quantity 9 (e.g. orderId: 100000)
Order Article 1 with quantity 3

What do you expect? Was the second order rejected?

What happens if you proceed like this?

Cancel OrderId 100000
Order Article 1 with quantity 3

What do you expect? Was the second order rejected?

What happens if you proceed like this?

Delete the connectors (order-events and stock-events)
Order Article 1 with quantity 3
Order Article 1 with quantity 3
Order Article 1 with quantity 3
Recreate the connectors (order-events and stock-events)

Were there any errors when the connectors were down? Do the microservices catch up after recreating?

What does the Jaeger Tracing show?