AsyncAPI Logo
On this page

Adeo Group

Industry: RetailCustomers: 500MRevenue: 25.6B EURO turnover, including 768M EURO online.

Adeo owns different brands in retail industry focused on home improvement and DIY markets, like Leroy Merlin.

tl;dr just go and have a look atfull production-used AsyncAPI document

Adeo Group

Challenges

Cost Component Repository product, part of the ADEO tech products, is used to calculate and publish transfer prices between different internal locations globally. Different business units use different information systems. It is hard to learn how each business unit shares information about its systems, API and accuracy.

The initial solution was a developer portal with a list of all applications and reference to dedicated documentation. Some legacy systems had docs written in MS Excel.

There was a need for a standart way of describing event-driven architecture.

Solution

The API is now described with AsyncAPI. The AsyncAPI file, stored with the source code, generates HTML documentation in the same release pipeline for the product. Documentation is exposed internally as part of the product for other company units depending on the API.

Payloads are described with Avro schema. These schemas generate models and are referenced directly in AsyncAPI files thanks to the schemaFormat feature and $ref. This way, they make sure the code is aligned with the docs.

Shift to using AsyncAPI also enables the team to implement more use cases using AsyncAPI files.

Use Case

Document the API of the product, so its users know how it works and how to use it. AsyncAPI was selected as the standard that allows you to generate documentation from a machine-readable document that describes the API. The goal was to document API in a standardized way, so other internal products could follow to unify how APIs are documented across the company.

More Details

Languages: JavaFrameworks: SpringProtocols: Kafka

Testing strategy

For Kafka, e2e tests are done with Zerocode. Load tests are handled with JMeter with the kloadgen plugin that supports Kafka and Avro.

Approach to code generation

Java models generation. Avro schemas used as a source.

Architecture

The following enterprise integration patterns are applied:

  • Request/Reply Described with description field in AsyncAPI. Reply goes to dedicated reply channel. Example description of response channel: 
    1description: >
2This topic is used to REPLY Costing Requests and is targeted by the
3`REPLY_TOPIC` header.
  • Return Address Info that needs to be provided by the client so producer knows where to send a response. Information is sent in the message header with the REPLY_TOPIC property. The AsyncAPI file documents information as part of the Message Header object. Example of request message header with REPLY_TOPIC: 
    1headers:
2  type: object
3  required:
4    - REPLY_TOPIC
5  properties:
6    REPLY_TOPIC:
7      $ref: "#/components/schemas/ReplyTopic"
  • Correlation Identifier This pattern enables the identification of the request given to the sent response. The REQUEST_ID property is in the request message header. The CORRELATION_ID property is in the response message header. Both headers are described in the AsyncAPI Message Header object and referred to in the AsyncAPI correlationID property. This means that correlation identifier is represented by different property in the message header, depending if it is a request or reply. Example of request message header with REQUEST_ID: 
    1headers:
2  type: object
3  required:
4    - REQUEST_ID
5  properties:
6    REQUEST_ID:
7      $ref: "#/components/schemas/RequestId"
    Example of how correlationId points to REQUEST_ID: 
    1correlationId:
2  description: >
3    This correlation ID is used for message tracing and messages
4    correlation.
5    This correlation ID is generated at runtime based on the `REQUEST_ID`
6    and sent to the RESPONSE message.
7  location: $message.header#/REQUEST_ID
  • DeadLetter Channel Also known as Dead Letter Queue. In Kafka, it is just another channel where undelivered messages are sent. Not part of the AsyncAPI file, as API consumers will not listen to this channel. Consumers know what happens with wrong events.
  • Invalid Message Channel Invalid messages are routed to the dedicated channel for rejected requests but are not part of the AsyncAPI file, as API consumers will not listen to this channel. Consumers know what happens with wrong events.

Architecture Diagram

More Details about AsyncAPI

Version: 2.4.0Who maintains documents: DevelopersInternal users: trueExternal users: false

How AsyncAPI documents are stored

Git repository where source code is.

Where maintainers edit AsyncAPI documents

IntelliJ without any special plugins. Sometimes people use AsyncAPI Studio, but not regularly because of lack of support for references to local drive.

What extensions are used

Extensions are used to describe details about custom security:

1  x-sasl.jaas.config: >-
2    org.apache.kafka.common.security.plain.PlainLoginModule required
3    username="<CLUSTER_API_KEY>" password="<CLUSTER_API_SECRET>";
4  x-security.protocol: SASL_SSL
5  x-ssl.endpoint.identification.algorithm: https
6  x-sasl.mechanism: PLAIN

How documentation is generated

Documentation generated from AsyncAPI is hosted as part of the product on a dedicated endpoint using Spring controller. Publishing is part of the CI/CD pipeline for the product using GithubActions.

Related Maven configuration used to trigger docs generation with AsyncAPI Generator industry:

1 <profile>
2        <id>generate-asyncapi-doc</id>
3        <build>
4            <plugins>
5                <plugin>
6                    <groupId>com.github.eirslett</groupId>
7                    <artifactId>frontend-maven-plugin</artifactId>
8                    <!-- Use the latest released version:
9                    https://repo1.maven.org/maven2/com/github/eirslett/frontend-maven-plugin/ -->
10                    <version>${frontend-maven-plugin.version}</version>
11                    <configuration>
12                        <nodeVersion>v12.18.4</nodeVersion>
13                        <installDirectory>${node.installation.path}</installDirectory>
14                        <workingDirectory>${node.installation.path}</workingDirectory>
15                    </configuration>
16                    <executions>
17                        <execution>
18                            <id>install node and npm</id>
19                            <goals>
20                                <goal>install-node-and-npm</goal>
21                            </goals>
22                            <phase>generate-resources</phase>
23                        </execution>
24                        <execution>
25                            <id>install @asyncapi/generator globally</id>
26                            <goals>
27                                <goal>npm</goal>
28                            </goals>
29                            <configuration>
30                                <arguments>install @asyncapi/generator@${asyncapi.generator.version}</arguments>
31                            </configuration>
32                        </execution>
33                    </executions>
34                </plugin>
35                <plugin>
36                    <groupId>org.codehaus.mojo</groupId>
37                    <artifactId>exec-maven-plugin</artifactId>
38                    <version>1.6.0</version>
39
40                    <executions>
41                        <execution>
42                            <id>execute-generation</id>
43                            <goals>
44                                <goal>exec</goal>
45                            </goals>
46                            <phase>generate-resources</phase>
47
48                            <configuration>
49                                <!-- Access binary file in node_modules because it doesn't work on windows otherwise. -->
50                                <executable>${node.modules.installation.path}/${ag.binary.name}</executable>
51                                <commandlineArgs>
52                                    ${project.basedir}/src/docs/asyncapi/asyncapi.yaml @asyncapi/html-template@${asyncapi.htmltemplate.version} -p sidebarOrganization=byTags -p
53                                    version=${project.version} -o ${asyncapi.generation.dir}
54                                </commandlineArgs>
55                            </configuration>
56                        </execution>
57                    </executions>
58                </plugin>
59                <plugin>
60                    <groupId>org.apache.maven.plugins</groupId>
61                    <artifactId>maven-resources-plugin</artifactId>
62                    <executions>
63                        <execution>
64                            <id>copy-resources</id>
65                            <!-- here the phase you need -->
66                            <phase>generate-resources</phase>
67                            <goals>
68                                <goal>copy-resources</goal>
69                            </goals>
70                            <configuration>
71                                <outputDirectory>${asyncapi.generation.dir}/assets</outputDirectory>
72                                <resources>
73                                    <resource>
74                                        <directory>src/docs/asyncapi/assets</directory>
75                                        <filtering>true</filtering>
76                                    </resource>
77                                </resources>
78                            </configuration>
79                        </execution>
80                    </executions>
81                </plugin>
82            </plugins>
83        </build>
84    </profile>

Critical features of AsyncAPI related to documentation:

  • use of version parameter in the generator command to display the release version from the Maven pom
  • descriptions that support CommonMark (Markdown) as they allow to put detailed structured descriptions and screenshots inside generated docs
  • examples and validation information. In this case converted from Avro to JSON Schema to show it in documentation and have examples generated
  • Tags for tagging operations to categorize them to make it easier to navigate in documentation UI

What bindings are used

All Kafka bindings are used. Server, channel, operation and message bindings.

Example of server bindings:

1bindings:
2  kafka:
3    schema.registry.url: >-
4      https://schema-registry.prod.url/

Example of channel bindings:

1bindings:
2  kafka:
3    replicas: 3
4    partitions: 3
5    cleanup.policy: delete
6    retention.ms: 7 days

Example of operation bindings:

1bindings:
2  kafka:
3    groupId:
4      type: string
5      description: >
6        The groupId must be prefixed by your `svc` account, deliver by the
7        Adeo Kafka team.
8        This `svc` must have the write access to the topic.
9    value.subject.name.strategy:
10      type: string
11      description: >
12        We use the RecordNameStrategy to infer the messages schema.
13        Use
14        `value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy`
15        in your producer configuration.

Example of message bindings:

1bindings:
2  kafka:
3    key:
4      $ref: "https://asyncapi.com/resources/casestudies/adeo/CostingResponseKey.avsc"

What tools are used

Schemas

Spec: Avro 1.9

Storage strategy

Git repository where source code is. During release they are published to Confluent Schema Registry.

Schema Registry

Confluent Schema Registry.

Versioning of schemas

Versioning is based on git tags. The schema version pushed to Confluent Schema Registry matches the git tag version of the product. Every schema has a version information that matches with product tag version.

Example Avro schema with version information:

1{
2  "namespace": "com.adeo.casestudy.costingrequest",
3  "type": "record",
4  "name": "CostingRequestPayload",
5  "version": "1.1.0",
6  "fields": [ ... ]
7}

Validation of message schemas

Based on validation using Confluent Schema Registry.

Additional Resources

Watch this video presentation about AsyncAPI case study from Ludovic Dussart, Ineat & Antoine Delequeuche, Adeo.

Production-use AsyncAPI document

1asyncapi: 2.4.0
2info:
3  title: Adeo AsyncAPI Case Study
4  version: "%REPLACED_BY_MAVEN%"
5  description: >
6    This Adeo specification illustrates how ADEO uses AsyncAPI to document some of their exchanges.
7  contact:
8    name: AsyncAPI Community
9    email: case-study@asyncapi.com
10servers:
11  production:
12    url: "prod.url:9092"
13    protocol: kafka
14    description: Kafka PRODUCTION cluster
15    security:
16      - sasl-ssl: []
17    bindings:
18      kafka:
19        schema.registry.url: >-
20          https://schema-registry.prod.url/
21  staging:
22    url: "staging.url:9092"
23    protocol: kafka
24    description: Kafka STAGING cluster for `uat` and `preprod` environments
25    security:
26      - sasl-ssl: []
27    bindings:
28      kafka:
29        schema.registry.url: >-
30          https://schema-registry.staging.url/
31  dev:
32    url: "dev.url:9092"
33    protocol: kafka
34    description: Kafka DEV cluster for `dev` and `sit` environments
35    security:
36      - sasl-ssl: []
37    bindings:
38      kafka:
39        schema.registry.url: >-
40          https://schema-registry.dev.url/
41tags:
42  - name: costing
43    description: "Costing channels, used by Costing clients."
44channels:
45  "adeo-{env}-case-study-COSTING-REQUEST-{version}":
46    description: >
47      Use this topic to do a Costing Request to Costing product.
48      We use the
49      [**RecordNameStrategy**](https://docs.confluent.io/platform/current/schema-registry/serdes-develop/index.html#subject-name-strategy)
50      to infer the messages schema.
51      You have to define `value.subject.name.strategy` to
52      `io.confluent.kafka.serializers.subject.RecordNameStrategy` in your
53      producer to use the schema we manage.
54      The schema below illustrates how Costing Request messages are
55      handled.
56      ![](https://user-images.githubusercontent.com/5501911/188920831-689cec5f-8dc3-460b-8794-0b54ec8b0ac8.png)
57    parameters:
58      env:
59        $ref: "#/components/parameters/Env"
60      version:
61        $ref: "#/components/parameters/Version"
62    bindings:
63      kafka:
64        replicas: 3
65        partitions: 3
66        cleanup.policy: delete
67        retention.ms: 7 days
68    publish:
69      operationId: requestCosting
70      summary: |
71        [COSTING] Request one or more Costing calculation for any product
72      description: >
73        You can try a costing request using our [Conduktor producer
74        template](https://conduktor.url)
75      tags:
76        - name: costing
77      message:
78        oneOf:
79          - $ref: "#/components/messages/costingRequestV1"
80      bindings:
81        kafka:
82          groupId:
83            type: string
84            description: >
85              The groupId must be prefixed by your `svc` account, deliver by the
86              Adeo Kafka team.
87              This `svc` must have the write access to the topic.
88          value.subject.name.strategy:
89            type: string
90            description: >
91              We use the RecordNameStrategy to infer the messages schema.
92              Use
93              `value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy`
94              in your producer configuration.
95  "adeo-{env}-case-study-COSTING-RESPONSE-{version}":
96    description: >
97      This topic is used to REPLY Costing Requests and is targeted by the
98      `REPLY_TOPIC` header.
99      **You must grant PUBLISH access to our `svc-ccr-app` service account.**.
100      We use the
101      [**RecordNameStrategy**](https://docs.confluent.io/platform/current/schema-registry/serdes-develop/index.html#subject-name-strategy)
102      to infer the messages schema.
103      You have to define `key.subject.name.strategy` and
104      `value.subject.name.strategy` to
105      `io.confluent.kafka.serializers.subject.RecordNameStrategy` in your
106      consumer.
107      The schema below illustrates how Costing Response messages are
108      handled.
109           ![](https://user-images.githubusercontent.com/5501911/188920831-689cec5f-8dc3-460b-8794-0b54ec8b0ac8.png)
110    parameters:
111      env:
112        $ref: "#/components/parameters/Env"
113      version:
114        $ref: "#/components/parameters/Version"
115    bindings:
116      kafka:
117        groupId:
118          type: string
119          description: >
120            The groupId must be prefixed by your `svc` account, deliver by the
121            Adeo Kafka team.
122            This `svc` must have the read access to the topic.
123        key.subject.name.strategy:
124          type: string
125          description: >
126            We use the RecordNameStrategy to infer the messages schema.
127            Use
128            `key.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy`
129            in your consumer configuration.
130        value.subject.name.strategy:
131          type: string
132          description: >
133            We use the RecordNameStrategy to infer the messages schema.
134            Use
135            `value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy`
136            in your consumer configuration.
137    subscribe:
138      operationId: getCostingResponse
139      summary: >
140        [COSTING] Get the costing responses matching an initial Costing
141        Request.
142      tags:
143        - name: costing
144      message:
145        $ref: "#/components/messages/costingResponse"
146components:
147  correlationIds:
148    costingCorrelationId:
149      description: >
150        This correlation ID is used for message tracing and messages
151        correlation.
152        This correlation ID is generated at runtime based on the `REQUEST_ID`
153        and sent to the RESPONSE message.
154      location: $message.header#/REQUEST_ID
155  messages:
156    costingRequestV1:
157      name: CostingRequestV1
158      title: Costing Request V1
159      summary: Costing Request V1 inputs.
160      tags:
161        - name: costing
162      schemaFormat: application/vnd.apache.avro;version=1.9.0
163      correlationId:
164        $ref: "#/components/correlationIds/costingCorrelationId"
165      headers:
166        type: object
167        required:
168          - REQUESTER_ID
169          - REQUESTER_CODE
170          - REQUEST_ID
171          - REPLY_TOPIC
172        properties:
173          REQUEST_ID:
174            $ref: "#/components/schemas/RequestId"
175          REPLY_TOPIC:
176            $ref: "#/components/schemas/ReplyTopic"
177          REQUESTER_ID:
178            $ref: "#/components/schemas/RequesterId"
179          REQUESTER_CODE:
180            $ref: "#/components/schemas/RequesterCode"
181      payload:
182        $ref: "https://asyncapi.com/resources/casestudies/adeo/CostingRequestPayload.avsc"
183    costingResponse:
184      name: CostingResponse
185      title: Costing Response
186      summary: Costing Response ouputs.
187      tags:
188        - name: costing
189      description: >
190        Please refer to the `CostingResponseKey.avsc` schema, available on [our
191        github
192        project](https://github.url/).
193      schemaFormat: application/vnd.apache.avro;version=1.9.0
194      correlationId:
195        $ref: "#/components/correlationIds/costingCorrelationId"
196      headers:
197        type: object
198        properties:
199          CALCULATION_ID:
200            $ref: "#/components/schemas/MessageId"
201          CORRELATION_ID:
202            $ref: "#/components/schemas/CorrelationId"
203          REQUEST_TIMESTAMP:
204            type: string
205            format: date-time
206            description: Timestamp of the costing request
207          CALCULATION_TIMESTAMP:
208            type: string
209            format: date-time
210            description: Technical timestamp for the costing calculation
211      bindings:
212        kafka:
213          key:
214            $ref: "https://asyncapi.com/resources/casestudies/adeo/CostingResponseKey.avsc"
215      payload:
216        $ref: "https://asyncapi.com/resources/casestudies/adeo/CostingResponsePayload.avsc"
217  schemas:
218    RequesterId:
219      type: string
220      description: The Costing requester service account used to produce costing request.
221      example: svc-ecollect-app
222    RequesterCode:
223      type: string
224      description: >-
225        The Costing requester code (generally the BU Code). The requester code
226        is useful to get the dedicated context (tenant).
227      example: 1
228    MessageId:
229      type: string
230      format: uuid
231      description: A unique Message ID.
232      example: 1fa6ef40-8f47-40a8-8cf6-f8607d0066ef
233    RequestId:
234      type: string
235      format: uuid
236      description: >-
237        A unique Request ID needed to define a `CORRELATION_ID` for exchanges,
238        which will be sent back in the Costing Responses.
239      example: 1fa6ef40-8f47-40a8-8cf6-f8607d0066ef
240    CorrelationId:
241      type: string
242      format: uuid
243      description: >-
244        A unique Correlation ID defined from the `REQUEST_ID` or the
245        `MESSAGE_ID` provided in the Costing Request.
246      example: 1fa6ef40-8f47-40a8-8cf6-f8607d0066ef
247    BuCode:
248      type: string
249      description: The Business Unit code for which data are applicable.
250      example: 1
251    ReplyTopic:
252      type: string
253      description: >
254        The Kafka topic where to send the Costing Response. This is required for
255        the [Return Address EIP
256        pattern](https://www.enterpriseintegrationpatterns.com/patterns/messaging/ReturnAddress.html).
257        **You must grant WRITE access to our `svc-ccr-app` service account.**
258      example: adeo-case-study-COSTING-RESPONSE-V1
259    ErrorStep:
260      type: string
261      description: |
262        The woker that has thrown the error.
263      example: EXPOSE_RESULT
264    ErrorMessage:
265      type: string
266      description: |
267        The error message describing the error.
268      example: Error message
269    ErrorCode:
270      type: string
271      description: |
272        The error code.
273      example: CURRENCY_NOT_FOUND
274  parameters:
275    Env:
276      description: Adeo Kafka Environement for messages publications.
277      schema:
278        type: string
279        enum:
280          - dev
281          - sit
282          - uat1
283          - preprod
284          - prod
285    Version:
286      description: the topic version you want to use
287      schema:
288        type: string
289        example: V1
290        default: V1
291  securitySchemes:
292    sasl-ssl:
293      type: plain
294      x-sasl.jaas.config: >-
295        org.apache.kafka.common.security.plain.PlainLoginModule required
296        username="<CLUSTER_API_KEY>" password="<CLUSTER_API_SECRET>";
297      x-security.protocol: SASL_SSL
298      x-ssl.endpoint.identification.algorithm: https
299      x-sasl.mechanism: PLAIN
300      description: >
301        Use [SASL authentication with SSL
302        encryption](https://docs.confluent.io/platform/current/security/security_tutorial.html#configure-clients)
303        to connect to the ADEO Broker.