2.5 Metrics (optional)

Metrics with Prometheus and Grafana

The involved components for our metrics to be working are:

• MicroProfile Metrics: Expose the application metrics as a RESTful Endpoint
• Prometheus: Scrape the exposed metrics periodically
• Grafana: Visualize the scraped metrics

Dependencies

Our application includes the following dependency in the pom.xml.

Including this dependency is enough for our application to expose the metrics at /metrics

Metric Details

The exposed metrics are using the OpenMetrics Format format. Metrics can be put in three different scopes.

We will only generate metrics for the scope application.

There are various type of metrics available. The most used types are:

• Counters: Counters are a metric that is used to keep an incremental or a decremental count.
• Gauges: represent metrics that are sampled to obtain their value.
• Meters: Meters are used to track throughput.
• Timers: Timers are used to aggregate timing duration, in nanoseconds, and provide duration and throughput statistics.

MicroProfile Metrics provides these types as annotations @Counted, @Timed, @Metered and @Gauge.

Task 2.5.1 - Inspecting metrics

Start the environment (if not already running) and point your browser to http://localhost:8080/metrics

• What do you get?
• Can you identify the different scopes?
• Can you locate some application specific metrics?

Defining metrics from code

The following snipped is part of the RESTful API endpoint ShopOrderResource.

1
2
3
4
5
6
7
8
9

@POST
@Transactional
@Counted(name = "monolith_order_create_request", absolute = true, description = "number of orders requested", tags = {"application=monolith", "resource=ShopOrderResource"})
@Timed(name = "monolith_order_create_timer", absolute = true, description = "timer for processing a order creation", tags = {"application=monolith", "resource=ShopOrderResource"})
public Response createShopOrder(ShopOrderDTO shopOrderDTO) {
    ShopOrder shopOrder = shopOrderService.createOrder(shopOrderDTO);
    registerSuccessfulOrder();
    return Response.ok(shopOrder).build();
}

full source ShopOrderResource.java

This code defines the following metrics:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

application_monolith_order_create_request_total{application="monolith",resource="ShopOrderResource"} 28.0
application_monolith_order_create_timer_rate_per_second{application="monolith",resource="ShopOrderResource"} 0.21598823940031425
application_monolith_order_create_timer_one_min_rate_per_second{application="monolith",resource="ShopOrderResource"} 0.3083022515809183
application_monolith_order_create_timer_five_min_rate_per_second{application="monolith",resource="ShopOrderResource"} 0.08583728986138126
application_monolith_order_create_timer_fifteen_min_rate_per_second{application="monolith",resource="ShopOrderResource"} 0.030253491344307742
application_monolith_order_create_timer_min_seconds{application="monolith",resource="ShopOrderResource"} 0.004008576
application_monolith_order_create_timer_max_seconds{application="monolith",resource="ShopOrderResource"} 0.194742506
application_monolith_order_create_timer_mean_seconds{application="monolith",resource="ShopOrderResource"} 0.020320811804737986
application_monolith_order_create_timer_stddev_seconds{application="monolith",resource="ShopOrderResource"} 0.044306979654618196
application_monolith_order_create_timer_seconds_count{application="monolith",resource="ShopOrderResource"} 28.0
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.5"} 0.008546583
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.75"} 0.01223613
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.95"} 0.190428873
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.98"} 0.194742506
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.99"} 0.194742506
application_monolith_order_create_timer_seconds{application="monolith",resource="ShopOrderResource",quantile="0.999"} 0.194742506

Task 2.5.2 - Interpret metrics

Try to interpret the metrics from the listing above

• How many orders have been requested?
• How long does the creation of an order take?
• Were the requests made recently or some time ago?
• Were there peaks or were the requests evenly distributed over time?

Task 2.5.3 - Inspect grafana dashboards

Head over to the Grafana UI at http://localhost:3001 . Login with admin/admin and inspect the two available dashboards.

• SmallRye MicroProfile Metrics: Generated Dashboard containing base and vendor metrics.
• Monolith Dashboard: Custom Dashboard containing some basic metrics from the application.

Try to change an existing visualization

• Open the Monolith Dashboard
• Edit “Order Create Timer seconds” by clicking the title and selecting edit
  • This graph contains multiple quantiles in the same graph. display only the 0.75 quantile.
  • Apply a filter to the metric by adding {quantile="0.75"} to the metrics query. This applies a filter to a tag named quantile with value 0.75

The modified query is using the Prometheus Query Language (PromQL). Check the Querying Prometheus page for more details.

Prometheus scraping

We know how our application exposes metrics. We have also seen dashboards with simple visualizations. Now have a look at the Prometheus configuration docker/config/prometheus/prometheus.yaml to see how the scraping of these metrics is configured.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# my global config
global:
  scrape_interval:     15s
  evaluation_interval: 15s

# omitted

scrape_configs:
  - job_name: 'prometheus'

    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'application'
    static_configs:
      - targets: ['application:8080']

full source prometheus.yaml

This defines that Prometheus scrapes our application at port 8080. Note that the metrics_path is /metrics per default. Prometheus will therefore query http://application:8080/metrics every 15 seconds to get new metrics.

For more information about the Prometheus configuration head over to the Prometheus configuration site.