Microservices Basics Training – 2. Mini-Monolith

Docs: 2.1 Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

Source Code

The source-code used for this lab is in the folder /code/monolith

Online Repository Link: https://github.com/acend/microservices-lab/tree/main/code/monolith .

Task 2.1.1 - Getting your workspace ready

We will only inspect files in this lab.

Open your IDE or favorite tool and switch to the referenced source folder.

Docs: 2.2 Docker Environment

Mon, 01 Jan 0001 00:00:00 +0000

Docker Compose

Our setup for the application is quite simple. It consists of the Quarkus service application and its database application-db. Further we have three more containers setup. We use jaeger as our tracing, prometheus to scrape the metrics and grafana to visualize them in dashboards.

All configuration for the docker environment is located in the folder docker. The folder docker/config is used to configure prometheus and grafana.

Host mapped container ports

Starting up the docker-compose will create a bridge network. We mapped most container ports to the docker host. This provides easy access for debugging like attaching the database to your IDE.

However, the most important ports of the containers are:

Container	Internal Port	Host Port	Purpose
`application`	8080	8080	Main application port
`application-db`	5432	5432	Database port
`jaeger`	16686	16686	Jaeger UI
`jaeger`	14268	14268	Port used to send traces from our Quarkus applications to Jaeger
`prometheus`	9090	9090	Prometheus service port (includes UI)
`grafana`	3000	3000	Grafana UI

Have a look at Jaeger Port Details if you need more information about the available Jaeger ports.

Task 2.2.1 - Inspecting the docker environment

Have a look at the /code/monolith/docker folder and get familiar with the docker environment specified in docker-compose.yaml

Docs: 2.3 Application Structure

Mon, 01 Jan 0001 00:00:00 +0000

Code Structure

The application in this lab is structured as follows:

monolith-application: Base directory for code or artifacts belonging to the application container
monolith-application/src/main/docker: Dockerfiles to build application and container
monolith-application/src/main/java/ch/puzzle/mm/monolith/: Location where most of the java application code resides
monolith-application/src/main/resources/db: contains the database migration scripts

The java code is structured by domains and the sub folders boundary, control and entity. Beside domains there are also helpers. For example the package monkey which contains a basic ChaosMonkey to inject errors.

Domains

In the application we have the following domains:

article: Static article management
order: Handles the management of the order and initiates the stock management
stock: Represents a stock management and handles the amount of items in stock

The most important classes in the application are:

order/boundary/ShopOrderResource.java: Provides the RESTful API endpoint for creating a new order
order/control/ShopOrderService.java: Handles the effective creation of the order and triggers the stock management
stock/control/ArticleStockService.java: Handles the effective management of the items in stock

Task 2.3.1 - Inspecting application structure

Have a look at the /code/monolith folder and get familiar with the application structure.

Docs: 2.4 Using the Lab

Mon, 01 Jan 0001 00:00:00 +0000

Starting the Lab

Open a shell and change your directory to the /code/monolith/docker folder.

Start the microservices with

1
2
3
4
5
6
7
8


$ docker-compose up -d
Creating network "docker_monolith" with driver "bridge"
Creating monolith-jaeger ... done
Creating monolith-grafana ... done
Creating monolith-alertmanager ... done
Creating monolith-application-db ... done
Creating monolith-prometheus ... done
Creating monolith-application ... done

Now your environment is up and running.

Using the Lab

Available Endpoints

Endpoint	Container	Purpose
http://localhost:8080/shop-orders	application	Return and create new orders
http://localhost:8080/article-stocks	application	List article stock count
http://localhost:8080/chaos-monkey	application	Lets you add a ChaosMonkey

Sample Requests

In the folder requests within the monolith lab folder there is an IntelliJ IDEA compatible HTTP Request file named api.http. You may run the requests directly from this file or build your own requests.

Get the article stock information

1

curl --request GET --url http://localhost:8080/article-stocks

Get order information

1

curl --request GET --url http://localhost:8080/shop-orders

Create new order

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


curl --request POST \
 --url http://localhost:8080/shop-orders \
 --header 'Content-Type: application/json' \
 --data '{
 "articleOrders" : [
 {
 "articleId" : 1,
 "amount" : 1
 }
 ]
}'

Task 2.4.1 - Testing scenario

Run the following queries:

Check the article-stock count
Create an order for one item of article 1
Check the orders and article-stock again

Answer the questions:

Is the article-stock count correct?
What happens if you are running out of stock?

ChaosMonkey

There is a basic implementation of a ChaosMonkey. It will let you inject errors to methods annotated with @ChaosMonkey. You may configure the ChaosMonkey using the provided RESTful API /chaos-monkey.

The following error types are available:

Error Type	Executed	Description
RateLimit	Before method	Rate limit in calls/sec. Default: `Long.MAX_VALUE`
Latency	After method	Issues a Thread.sleep() in ms. Default: 0
ErrorRate	After method	Lets you specify a percentage of Errors thrown (0 to 1). Default: 0
Exception	After method	Always triggers an exception if enabled. Default: `false`

Sample request for configuring the ChaosMonkey which will throw an exception:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


curl --request POST \
 --url http://127.0.0.1:8080/chaos-monkey \
 --header 'Content-Type: application/json' \
 --data '[
 {
 "clazzName": "ArticleStockService",
 "methodName": "orderArticles",
 "enabled": true,
 "errorRate": 0.0,
 "latencyMs": 0,
 "permitsPerSec": 9.223372036854776E18,
 "rateLimiterType": "BLOCK",
 "throwException": true
 }
]'

Task 2.4.2 - Injecting an error

Enable a ChaosMonkey for the method orderArticles in the ArticleStockService.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


curl --request POST \
 --url http://127.0.0.1:8080/chaos-monkey \
 --header 'Content-Type: application/json' \
 --data '[
 {
 "clazzName": "ArticleStockService",
 "methodName": "orderArticles",
 "enabled": true,
 "throwException": true
 }
]'

This will throw an InternalServerErrorException after execution of the method orderArticles.

Ask yourself what to expect if an exception is thrown?

Issue the request from the Task 2.4.1.

What did you observe? Is there any difference?
Is the data still consistent?
What’s the role of a transaction in this case?

Task Hint

Since the application throws an InternalServerErrorException the order is not created and the stock count remains the same.

This is due to the fact that the code runs in a single transaction. Throwing a RuntimeException from an active transaction triggers a rollback of the transaction.

Docs: 2.5 Metrics (optional)

Mon, 01 Jan 0001 00:00:00 +0000

Optional Section

The following content is optional. If you have time, feel free to proceed.

Metrics with Prometheus and Grafana

The involved components for our metrics to be working are:

MicroProfile Metrics: Expose the application metrics as a RESTful Endpoint
Prometheus: Scrape the exposed metrics periodically
Grafana: Visualize the scraped metrics

Dependencies

Our application includes the following dependency in the pom.xml.

GroupId	ArtifactId	Description	Detailed information
`io.quarkus`	`quarkus-smallrye-metrics`	Exposes various metrics and statistics about the application	MicroProfile Metrics

Including this dependency is enough for our application to expose the metrics at /metrics

Metric Details

The exposed metrics are using the OpenMetrics Format format. Metrics can be put in three different scopes.

Scopes

base: metrics that all MicroProfile vendors have to provide. e.g. JVM metrics
vendor: vendor specific metrics
application: application-specific metrics

Source: github.com/eclipse/microprofile-metrics

More Details: MicroProfile Metrics - Scope

We will only generate metrics for the scope application.

There are various type of metrics available. The most used types are:

Counters: Counters are a metric that is used to keep an incremental or a decremental count.
Gauges: represent metrics that are sampled to obtain their value.
Meters: Meters are used to track throughput.
Timers: Timers are used to aggregate timing duration, in nanoseconds, and provide duration and throughput statistics.

MicroProfile Metrics provides these types as annotations @Counted, @Timed, @Metered and @Gauge.

Task 2.5.1 - Inspecting metrics

Start the environment (if not already running) and point your browser to http://localhost:8080/metrics

What do you get?
Can you identify the different scopes?
Can you locate some application specific metrics?

Task Hint

You will get all metrics in the OpenMetrics TEXT Format. Each metric is on a new line.

The different scopes can be seen by having a look at the first part of a metric (e.g. vendor_).

Defining metrics from code

The following snipped is part of the RESTful API endpoint ShopOrderResource.

1
2
3
4
5
6
7
8
9


@POST
@Transactional
@Counted(name = "monolith_order_create_request", absolute = true, description = "number of orders requested", tags = {"application=monolith", "resource=ShopOrderResource"})
@Timed(name = "monolith_order_create_timer", absolute = true, description = "timer for processing a order creation", tags = {"application=monolith", "resource=ShopOrderResource"})
public Response createShopOrder(ShopOrderDTO shopOrderDTO) {
 ShopOrder shopOrder = shopOrderService.createOrder(shopOrderDTO);
 registerSuccessfulOrder();
 return Response.ok(shopOrder).build();
}

full source ShopOrderResource.java

This code defines the following metrics:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


application_monolith_order_create_request_total{application="monolith",resource="ShopOrderResource"} 28.0
application_monolith_order_create_timer_rate_per_second{application="monolith",resource="ShopOrderResource"} 0.21598823940031425
application_monolith_order_create_timer_one_min_rate_per_second{application="monolith",resource="ShopOrderResource"} 0.3083022515809183
application_monolith_order_create_timer_five_min_rate_per_second{application="monolith",resource="ShopOrderResource"} 0.08583728986138126
application_monolith_order_create_timer_fifteen_min_rate_per_second{application="monolith",resource="ShopOrderResource"} 0.030253491344307742
application_monolith_order_create_timer_min_seconds{application="monolith",resource="ShopOrderResource"} 0.004008576
application_monolith_order_create_timer_max_seconds{application="monolith",resource="ShopOrderResource"} 0.194742506
application_monolith_order_create_timer_mean_seconds{application="monolith",resource="ShopOrderResource"} 0.020320811804737986
application_monolith_order_create_timer_stddev_seconds{application="monolith",resource="ShopOrderResource"} 0.044306979654618196
application_monolith_order_create_timer_seconds_count{application="monolith",resource="ShopOrderResource"} 28.0
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.5"} 0.008546583
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.75"} 0.01223613
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.95"} 0.190428873
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.98"} 0.194742506
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.99"} 0.194742506
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.999"} 0.194742506

Task 2.5.2 - Interpret metrics

Try to interpret the metrics from the listing above

How many orders have been requested?
How long does the creation of an order take?
Were the requests made recently or some time ago?
Were there peaks or were the requests evenly distributed over time?

Task Hint

A total of 28 orders created
The count on rate_per_second, one_min_rate_per_second are higher than fifteen_min_rate_per_second which indicates that most requests are seconds/minutes ago.
There seem to be peaks as the rates for second, minute, five and fifteen are not evenly distributed.

The graph may look like this

Task 2.5.3 - Inspect grafana dashboards

Head over to the Grafana UI at http://localhost:3001 . Login with admin/admin and inspect the two available dashboards.

SmallRye MicroProfile Metrics: Generated Dashboard containing base and vendor metrics.
Monolith Dashboard: Custom Dashboard containing some basic metrics from the application.

Try to change an existing visualization

Open the Monolith Dashboard
Edit “Order Create Timer seconds” by clicking the title and selecting edit
- This graph contains multiple quantiles in the same graph. display only the 0.75 quantile.
- Apply a filter to the metric by adding {quantile="0.75"} to the metrics query. This applies a filter to a tag named quantile with value 0.75

The modified query is using the Prometheus Query Language (PromQL). Check the Querying Prometheus page for more details.

Task Hint

Prometheus scraping

We know how our application exposes metrics. We have also seen dashboards with simple visualizations. Now have a look at the Prometheus configuration docker/config/prometheus/prometheus.yaml to see how the scraping of these metrics is configured.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# my global config
global:
 scrape_interval: 15s
 evaluation_interval: 15s

# omitted

scrape_configs:
 - job_name: 'prometheus'

 static_configs:
 - targets: ['localhost:9090']

 - job_name: 'application'
 static_configs:
 - targets: ['application:8080']

full source prometheus.yaml

This defines that Prometheus scrapes our application at port 8080. Note that the metrics_path is /metrics per default. Prometheus will therefore query http://application:8080/metrics every 15 seconds to get new metrics.

For more information about the Prometheus configuration head over to the Prometheus configuration site.

Docs: 2.6 Application Tracing (optional)

Mon, 01 Jan 0001 00:00:00 +0000

Optional Section

The following content is optional. If you have time, feel free to proceed.

OpenTracing with Jaeger Tracing

We use OpenTracing for the distributed tracing. Our tracing backend is Jaeger.

Dependencies

Our application includes the following dependencies in the pom.xml.

GroupId	ArtifactId	Description	Detailed information
`io.quarkus`	`quarkus-smallrye-opentracing`	MicroProfile OpenTracing implementation	MicroProfile OpenTracing
`io.opentracing.contrib`	`opentracing-jdbc`	Provides traces for database queries.

Configuration

The following configurations is required in the application.properties to setup the tracing:

1
2
3
4


quarkus.jaeger.endpoint=http://jaeger:14268/api/traces
quarkus.jaeger.service-name=monolith-application
quarkus.jaeger.sampler-type=const
quarkus.jaeger.sampler-param=1

endpoint: defines where we will send our metrics
service-name: the name of the current service
sampler-type: const defines a constant sampling
sampler-param: percentage of request which are sampled. 1 defines that we want to sample all requests.

Further we have to use the tracing enabled database driver.

1
2
3
4


quarkus.datasource.db-kind=postgresql
quarkus.datasource.jdbc.url=jdbc:tracing:postgresql://application-db:5432/admin
quarkus.datasource.jdbc.driver=io.opentracing.contrib.jdbc.TracingDriver
quarkus.hibernate-orm.dialect=org.hibernate.dialect.PostgreSQLDialect

Last we made the Quarkus log format to include the traceId and spanId in the logs. This is not necessary but will help troubleshooting.

1

quarkus.log.console.format=%d{HH:mm:ss} %-5p traceId=%X{traceId}, spanId=%X{spanId}, sampled=%X{sampled} [%c{2.}] (%t) %s%e%n

Configuration

The configuration listed above may slightly differ from the one used in the solution repository. This is because some properties are overridden using environment variables in the docker-compose.yaml file.

full source application.properties

Application code

Using the configuration made is basically everything which is needed to get traces from our application. By default all requests to RESTful endpoints are traced. Using the TracingDriver provides spans for all database queries.

If you want to customize the tracing you can annotate any class or method with the @Traced annotation. You may also disable the default tracing with @Traced(false). Further you may inject the Tracer to your application code. Using this tracer you can create custom spans in business methods or add tags to span.

This snipped is from class ShopOrderService.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


@ApplicationScoped
public class ShopOrderService {
 // ...

 @Inject
 Tracer tracer;

 @Traced(operationName = "createOrder")
 public ShopOrder createOrder(ShopOrderDTO shopOrderDTO) {
 ShopOrder shopOrder = new ShopOrder();
 // ...
 shopOrder.persist();

 tracer.activeSpan().setTag("order.id", shopOrder.id);

 return shopOrder;
 }
}

This will create a span for the method createOrder and name it accordingly. Using the injected Tracer we tag the span with the just created order.id.

full source ShopOrderService.java

Task 2.6.1 - Inspect the application traces

Start up your environment if not already running and point your browser to the Jaeger UI at http://localhost:16686/

Issue requests as described in section 2.4

Do you find traces for created orders? You may list all traces or use the tag filter order.id=X
How do traces look like if the operationName is not set?
Do you see the tag order.id on the span?
What happens if you inject an exception as described in section 2.4?

Task Hint

Traces from application code with no explicit name are named by their class and method. If an exception is thrown, there will be a tag error=true on the span. The Jaeger UI will flag such spans with an exclamation mark.