3.6 Distributed Transactions (optional)

Understand how we can provide data consistency over multiple microservices.

Both our microservices use local transactions to store their data. There are several cases where data may get out of sync between order and stock microservice.

Problem

Think of the following scenario.

  • New order is requested using the RESTful API
    • Order microservice will create and store a new order
    • Order microservice will trigger the stock microservice to change the amount of in-stock items.
    • Stock microservice changes the stock count and returns successfully.
    • Order microservice fails for any reason

Monolith Split

Since the stock successfully applied the reservation, and the order microservice failed, this results in an inconsistent state.

MicroProfile LRA (Long Running Actions)

LRA Protocol Sequence - source: github.com/eclipse/microprofile-lra LRA Sequence

The MicroProfile LRA is built for microservices with RESTful communication. Each microservice participating in a LRA will have to provide a compensate action which will be invoked if the transaction is cancelled. The life cycle of LRAs can be managed by annotating the JAX-RS resources with the following annotations.

AnnotationDescription
@LRAControls the life cycle of an LRA.
@CompensateIndicates that the method should be invoked if the LRA is cancelled.
@CompleteIndicates that the method should be invoked if the LRA is closed.
@ForgetIndicates that the method may release any resources that were allocated for this LRA.
@LeaveIndicates that this class is no longer interested in this LRA.
@StatusWhen the annotated method is invoked it should report the status.
@AfterLRAWhen an LRA has reached a final state the annotated method is invoked.
Source: github.com/eclipse/microprofile-lra

See the basic sample below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

@Path("/")
@ApplicationScoped
public class SimpleLRAParticipant {
    @LRA(LRA.Type.REQUIRES_NEW)
    @Path("/cdi")
    @PUT
    public void doInTransaction(@HeaderParam(LRA_HTTP_CONTEXT_HEADER) URI lraId) {
        /*
         * Perform business actions in the context of the LRA identified by the
         * value in the injected JAX-RS header. This LRA was started just before
         * the method was entered (REQUIRES_NEW) and will be closed when the
         * method finishes at which point the completeWork method below will be
         * invoked.
         */
    }

    @Complete
    @Path("/complete")
    @PUT
    public Response completeWork(@HeaderParam(LRA_HTTP_CONTEXT_HEADER) URI lraId) {
        /*
         * Free up resources allocated in the context of the LRA identified by the
         * value in the injected JAX-RS header.
         *
         * Since there is no @Status method in this class, completeWork MUST be
         * idempotent and MUST return the status.
         */
         return Response.ok(ParticipantStatus.Completed.name()).build();
    }

    @Compensate
    @Path("/compensate")
    @PUT
    public Response compensateWork(@HeaderParam(LRA_HTTP_CONTEXT_HEADER) URI lraId) {
        /*
         * The LRA identified by the value in the injected JAX-RS header was
         * cancelled so the business logic should compensate for any actions
         * that have been performed while running in its context.
         *
         * Since there is no @Status method in this class, compensateWork MUST be
         * idempotent and MUST return the status
         */
         return Response.ok(ParticipantStatus.Compensated.name()).build();
    }
}

Source: github.com/eclipse/microprofile-lra

Handling Compensations

What does providing a compensation mean to our microservices?

Each operation like order creation is run in a LRA context. The LRA context is reflected by an associated lra Id. The lraId will be injected in the HTTP-Headers. The call for compensation will also contain this id. This means that we must be able to revert our database changes by using the lra id whenever a REST call is made to the method annotated with @Compensate.

Task 3.6.1 - How to track LRAs

Think of compensating an order, which changes are necessary to our entities that we can revert the change later on?

Task Hint

Lets assume the LRA with id lra-1 needs to be compensated.

To compensate the database changes we need to know:

  • Which order is created by lra-1
    • If the order is compensated we set a different order status
  • Which articles were reserved by lra-1
    • If the order is compensated we have to revert the stock deduction

For this to achieve a simple approach could be:

  • Store lra-1 which was used to create the order in the ShopOrder entity.
  • Track the ArticleOrder with the associated lra-1 in the stock microservice as a new table/entity articlestockchange

Database Changes

This leads to the following changes in our database schema for the order microservice.

1
2
3
4
5
6

<changeSet author="lra" id="2">
  <comment>Add LRA field to store associated lra-id</comment>
  <addColumn tableName="shoporder">
    <column name="lra" type="varchar(255)" />
  </addColumn>
</changeSet>

On the stock microservice we will add the following migration to create a new table articlestockchange.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

<createTable tableName="articlestockchange">
  <column name="id" type="bigint">
    <constraints primaryKey="true" nullable="false"/>
  </column>
  <column name="article_id" type="bigint">
    <constraints nullable="false" foreignKeyName="articlestockchange_article_fk" referencedColumnNames="id"/>
  </column>
  <column name="count" type="int"/>
  <column name="lra" type="varchar(255)"/>
</createTable>

LRA Coordinator

The LRA management will be made by an LRA coordinator. The Narayana Transaction Manager1, which is the default JTA Transaction manager in Quarkus and WildFly Application servers, provides an implementation of an LRA Coordinator.

We will use this coordinator for our environment. Therefore, the following changes are made in the docker/docker-compose.yaml:

Add LRA Coordinator (do not forget to add the volume)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

  lra-coordinator:
    image: quay.io/jbosstm/lra-coordinator:latest
    hostname: lra-coordinator
    container_name: rest-lra-coordinator
    volumes:
      - lra-data:/opt/jboss
    networks:
      - rest
    ports:
      - 8090:8080
    environment:
      - LOG_LEVEL=DEBUG

# ... omitted

volumes:
  # ....
  lra-data:

Dependencies

Our application includes the following dependency in the pom.xml.

GroupIdArtifactIdDescription
org.jboss.narayana.rtsnarayana-lraProvides MicroProfile LRA implementation

To fix the docker network issue and specify the needed system variables we add the following to the order container specification:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

  order:
    # ... omitted
    environment:
      - JAVA_TOOL_OPTIONS=-Dlra.http.host=lra-coordinator -Dlra.http.port=8080
      - APPLICATION_REWRITE_BASE_ENABLED=true
      - APPLICATION_REWRITE_BASE_HOST=order

  stock:
    # ... omitted
    environment:
      - JAVA_TOOL_OPTIONS=-Dlra.http.host=lra-coordinator -Dlra.http.port=8080
      - APPLICATION_REWRITE_BASE_ENABLED=true
      - APPLICATION_REWRITE_BASE_HOST=stock

Using LRAs

Now we are going to change our code to make use of LRAs. We have to annotate the JAX-RS resources with the specific LRA-Annotations.

Changes in Order Microservice

We have to make the following changes to our ch.puzzle.mm.rest.order.boundary.ShopOrderResource

Annotate the POST method createShopOrder and specify the HeaderParams to be used to extract the LRA ID. The extracted LRA ID will be stored in the database to be able to complete or compensate the LRA.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

@POST
@ChaosMonkey
@LRA(value = LRA.Type.REQUIRES_NEW,
        cancelOn = {
                Response.Status.INTERNAL_SERVER_ERROR // cancel on a 500 code
        },
        cancelOnFamily = {
                Response.Status.Family.CLIENT_ERROR // cancel on any 4xx code
        })
public Response createShopOrder(@HeaderParam(LRA.LRA_HTTP_CONTEXT_HEADER) String lraUrl,
                                ShopOrderDTO shopOrderDTO) {
        // extract lraId from headerstring
        String lraId = getLraId(lraUrl);

        // create ShopOrder locally
        ShopOrder shopOrder = shopOrderService.createOrder(shopOrderDTO, lraId);

        // call remote service
        articleStockService.orderArticles(shopOrderDTO.articleOrders);

        return Response.ok(shopOrder).build();
}

Implement @Complete method and set the COMPLETE status for this order. The LRA expects us to return the Status.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

@PUT
@Path("/complete")
@Complete
@Consumes(MediaType.TEXT_PLAIN)
@Transactional
public Response completeShopOrder(@HeaderParam(LRA.LRA_HTTP_CONTEXT_HEADER) String lraUrl) {
    // extract lraId from headerstring
    String lraId = getLraId(lraUrl);

    int updated = ShopOrder.setStatusByLra(ShopOrderStatus.COMPLETED, lraId);

    return Response.ok(ParticipantStatus.Completed.name()).build();
}

Implement @Compensate method and set the INCOMPLETE status. Also return the Status for the LRA.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

@PUT
@Path("/compensate")
@Compensate
@Consumes(MediaType.TEXT_PLAIN)
@Transactional
public Response compensateShopOrder(@HeaderParam(LRA.LRA_HTTP_CONTEXT_HEADER) String lraUrl) {
    // extract lraId from headerstring
    String lraId = getLraId(lraUrl);

    int updated = ShopOrder.setStatusByLra(ShopOrderStatus.INCOMPLETE, lraId);

    return Response.ok(ParticipantStatus.Compensated.name()).build();
}

Further you have to add a simple method to extract the LRA-id

1
2
3

String getLraId(String longRunningHeader) {
    return longRunningHeader.substring(longRunningHeader.lastIndexOf('/') + 1);
}

Complete Source of ShopOrderResource

We have also added method to the ShopOrder entity to change the ShopOrder by a given lraId. The entity also includes the lra field which we’ve been added to our database schema.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

@Entity
public class ShopOrder extends PanacheEntity {

    @NotNull
    @Column(unique=true, nullable = false)
    private String lra;

    // omitted

    public String getLra() {
        return lra;
    }

    public void setLra(String lra) {
        this.lra = lra;
    }

    public static int setStatusByLra(ShopOrderStatus status, String lra) {
        return update("status = :status where lra = :lra",
                Parameters.with("status", status).and("lra", lra));
    }
}

Complete Source of ShopOrder

Our ShopOrderService now takes an additional argument the lraId which is stored in the newly created field on the ShopOrder entity.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

@ApplicationScoped
public class ShopOrderService {

    @Transactional
    public ShopOrder createOrder(ShopOrderDTO dto, String lra) {
        // omitted

        // set lra id
        shopOrder.setLra(lra);

        // omitted
    }
}

Complete Source of ShopOrderService

At the end we switch our rest-client to the LRA-Enabled version for the stock service by changing the path to @Path("/lra/article-stocks").

Complete Source of ArticleStockService

Changes in Stock Microservice

We have to make the following changes to our ch.puzzle.mm.rest.stock.boundary.ArticleStockResource

Annotate the POST method createStockOrder and specify the HeaderParams to be used to extract the LRA ID. The extracted LRA ID will be stored in the database to be able to complete or compensate the LRA.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

@POST
@ChaosMonkey
@LRA(value = LRA.Type.MANDATORY, end = false)
public Response createStockOrder(@HeaderParam(LRA.LRA_HTTP_CONTEXT_HEADER) String lraUrl,
                                 List<ArticleOrderDTO> articles) {
    // extract lraId from headerstring
    String lraId = getLraId(lraUrl);

    // handle stock count
    articleStockService.orderArticles(articles, lraId);
}

Implement @Complete method to just return the LRA status.

1
2
3
4
5
6

@PUT
@Path("/complete")
@Complete
public Response completeShopOrder(@HeaderParam(LRA.LRA_HTTP_CONTEXT_HEADER) String lraUrl) {
    return Response.ok(ParticipantStatus.Completed.name()).build();
}

Implement @Compensate method which uses the stored LRA from the database and adds the reserved (decreased) amount of an article to the stock count.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

@PUT
@Path("/compensate")
@Compensate
public Response compensateShopOrder(@HeaderParam(LRA.LRA_HTTP_CONTEXT_HEADER) String lraUrl) {
    String lraId = getLraId(lraUrl);

    // find article stock change by LRA-ID and increment the stockcount.
    ArticleStockChange.findByLraId(lraId).forEach(asc -> {
        ArticleStock stock = ArticleStock.findByArticleId(asc.getArticle().id);
        int oldStockCount = stock.getCount();
        stock.setCount(stock.getCount() + asc.getCount());
        log.info("Compensating for Article {}: oldCount={}, release={}, newCount={}", stock.getArticle().id, oldStockCount, asc.getCount(), stock.getCount());
    });

    return Response.ok(ParticipantStatus.Compensated.name()).build();
}

Complete Source of ArticleStockLraResource

Changing to the LRA enabled version

To change to the LRA version, switch to the pre-built docker images.

Task 3.6.2 - Switching to the pre-built images

We provide a docker-compose.solution.yaml docker file which includes all changes necessary to use the LRA enabled version.

You may simply start the environment using this docker-compose file. Restart your docker environment with

1
2

docker-compose down --remove-orphans
docker-compose -f docker-compose.solution.yaml up -d

Source of docker-compose.solution.yaml

Task 3.6.3 - Rerun the tests from previous section

Issue the request from the Task 3.5.1.

  • What did you observe?
  • Is there any difference?
  • Is the data consistent between stock and order microservice?
  • What’s the role of a transaction in this case?
  • What’s the role of the LRA coordinator?

Use the chaos-monkey rest endpoint to inject an errors at the order microservice.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

curl --request POST \
  --url http://localhost:8080/chaos-monkey \
  --header 'Content-Type: application/json' \
  --data '[
    {
        "clazzName": "ShopOrderResource",
        "methodName": "createShopOrder",
        "enabled": true,
        "throwException": true
  }
]'
  • Rerun some tests with the monkey in place. What did happen?

You may delete the ChaosMonkey on the order microservice with

1
2
3

curl --request DELETE \
  --url http://127.0.0.1:8080/chaos-monkey/ShopOrderResource%23createShopOrder \
  --header 'Content-Type: application/json'

Now create a chaos-monkey at the stock microservice:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

curl --request POST \
  --url http://127.0.0.1:8081/chaos-monkey \
  --header 'Content-Type: application/json' \
  --data '[
  {
    "clazzName": "ArticleStockLraResource",
    "methodName": "createStockOrder",
    "enabled": true,
    "errorRate": 0,
    "latencyMs": 0,
    "permitsPerSec": 9.223372036854776E18,
    "rateLimiterType": "BLOCK",
    "throwException": true
  }
]'

Now rerun some tests with the monkey on the stock microservice.

  • Is your data still consistent?
Task Hint

Using LRAs the LRA coordinator tracks the state of the LRA. If an error is occurred it will notify every participant of the LRA about the error by triggering the method annotated with @Compensate. Since our microservices now track which entities have been created in which LRA contexts they are able to revert or compensate the changes.

You may see this clearly in the logs of the stock or order microservice.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

----------------------------
Create ArticleOrder: 0_ffffc0a86003_-1be006ed_5ff8d079_12
LRA Transaction Id: 0_ffffc0a86003_-1be006ed_5ff8d079_12
----------------------------
Article with id 1 processed. Stock oldCount=10, ordered=1, newCount=9

---------------------------
Compensate ArticleOrder
LRA Transaction Id: 0_ffffc0a86003_-1be006ed_5ff8d079_12
----------------------------
Compensating for Article 1: oldCount=9, release=1, newCount=10

  1. Narayana Transaction Manager: https://narayana.io  ↩︎

Previous
Next
Last modified February 12, 2024: fix debezium lab version (9e6013e)