Implementing Distributed Tracing with Spring

Cloud Sleuth, Zipkin, AWS MQ - RabbitMQ

Author: thanhct
Email: caothanhthanh@gmail.com
Phone: 0983770244

References:
https://howtodoinjava.com/spring-cloud/spring-cloud-zipkin-sleuth-
tutorial/
https://www.baeldung.com/tracing-services-with-zipkin
https://dzone.com/articles/spring-cloud-amp-spring-
bootimplementing-zipkin-se
https://spring.io/blog/2016/02/15/distributed-tracing-with-spring-
cloud-sleuth-and-spring-cloud-zipkin

What is distributed tracing system?

Distributed Tracing is a troubleshooting and understanding

microservices. It’s useful when we need to track timming data for
latency problem and request go throuth some services. Here with
distributed tracing enable, we can measure which component took
how much time. It manages both the collection and lookup of the
data.

You can use in-memory, MySQL, Apache Cassandra or Elasticseach

to storage data and HTTP, Apache ActiveMQ, Apache Kafka, gRPC,
RabbitMQ and Scribe (Apache Thrift) as transport options.

In this case we use AWS MQ (ActiveMQ) as transport for our

Distributed System, Sleuh send log to ActiveMQ then ZIpkin will
consume it then store to elastich search.
AWS MQ Apache ActiveMQ
To create an ActiveMQ Broker please visit aws site to do that.
https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/
amazon-mq-getting-started.html

In our project create ActiveMQ as Active/Standby broker this Provides

high availability and automacti failover capability. Creates a single
broker instance in one Available Zone (AZ) and another standy broker
instance in a different AZ. Suitable for production workloads.
Input Broker Name, choose broker instance type, type the username
& password for web console access. Then hit Create to create your
ActiveMQ broker.

Creating the broker takes about 15 minutes.

When Aws MQ is created successfully. Aws MQ displays the Running
status and you can click on the Broker’s name you’ll see the
connection details

Now you can click on the AactiveMQ web console link to open the
ActiveMQ dasdboard
https://b-454903b5-4dc9-49d4-a8cf-f9669d8122e8-1.mq.ap-
southeast-2.amazonaws.com:8162

When you see the dashboard screen that mean your ActiveMQ broker
is created successfully.
If you can’t connect to you ActiveMQ dashboard please make sure
that Plublic Accessibility is turning on (yes)
And Secrurity Group already added ports to access from outside.

Next step we’ll install zipkin server.

To install zipkin you have some ways to do that. https://zipkin.io/
pages/quickstart.html

Zipkin Server
Start zipkin by Docker
docker run -d -p 9411:9411 -e
ACTIVEMQ_URL='failover:(ssl://
b-454903b5-4dc9-49d4-a8cf-f9669d8122e8-1.mq.ap-
southeast-2.amazonaws.com:61617,ssl://
b-454903b5-4dc9-49d4-a8cf-f9669d8122e8-2.mq.ap-
southeast-2.amazonaws.com:61617)' -e
ACTIVEMQ_USERNAME=thanhct -e
ACTIVEMQ_PASSWORD=xxxxx -e
ACTIVEMQ_CONCURRENCY=8 openzipkin/zipkin

Runing zipkin with Java jar file

curl -sSL https://zipkin.io/quickstart.sh | bash
-s
ACTIVEMQ_URL=ssl://b-454903b5-4dc9-49d4-a8cf-
f9669d8122e8-1.mq.ap-
southeast-2.amazonaws.com:61617
ACTIVEMQ_USERNAME=thanhct
ACTIVEMQ_PASSWORD=xxxxxxxx
ACTIVEMQ_CONCURRENCY=8 java -jar zipkin.jar

Running from Source Code

git clone https://github.com/openzipkin/zipkin
cd zipkin
# Build the server and also make its
dependencies
./mvnw -DskipTests --also-make -pl zipkin-server
clean install
# Run the server
ACTIVEMQ_URL=ssl://b-454903b5-4dc9-49d4-a8cf-
f9669d8122e8-1.mq.ap-
southeast-2.amazonaws.com:61617
ACTIVEMQ_USERNAME=thanhct
ACTIVEMQ_PASSWORD=xxxxxxxx
ACTIVEMQ_CONCURRENCY=8 java -jar ./zipkin-
server/target/zipkin-server-x.x.x-SNAPSHOT-
exec.jar

Running from docker compose

Clone you docker-compose from: http://
git.eplatform.vn/jienie/docker-zipkin
docker-compose -f docker-compose.yml -f docker-
compose-elasticsearch-activemq.yml up

If everything is ok now you are running the zipkin server and It’s
connected to ActiveMQ as a consumer.
Now move to browser then type http://localhost:9411 Zipkin’s
dashboard will appear.

Now you have to implement Spring boot, Spring cloud sleuth to

connect to ActiveMQ then send message from your microservices to
ActiveMQ to tracing you system

Intergrating Spring Cloud Sleuth

in this solution I tried to tracing our system as image below:

In this architechture. We’ll make an authentication and get token from

oAuth2 to get authorization for next request. All request will be going
to the ZUUL API GATEWAY (ZUUL PROXY) then request will be
forwading to the oAuth2 server from here oAuth2 request to master
common and buyer master common services by Feign to get user
info. How we can tracing that requests. Let’s see in the next step.

In all services you must add these dependancies in your POM.xml

Dependences
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency>
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-sender-activemq-client</artifactId>
</dependency>
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-reporter-spring-beans</artifactId>
</dependency>

<dependencyManagement>
<dependencies>
<!-- temporary until sleuth has next zipkin-reporter release -->
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-reporter-bom</artifactId>
<version>2.9.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

Now when you run your microservices you can see log as below:
All services you can see are different tracing log compare with after
you’re intergrated Spring Cloud Sleuth.
That’s [service-zuul-
gateway,d976e46e9c6e74df,d976e46e9c6e74df,true] It mean
you’re added sleuth successfully.
service-zuul-gateway: the name of the application that logged the
span.
d976e46e9c6e74df: the id of the latency graph that contains the
span.
d976e46e9c6e74df: the id of a specific operation that took place.
true: whether the log should be exported to zipkin or not.

Zuul gateway log

oAuth2 log

Master Common Service Log

Buyer Master Common Service Log.

Important thing:
Configure the percentage of spans exported using
spring.sleuth.sampler.probability: 1.0 (range: 0.1 -> 1.0)
In this solution we set it 1.0 it mean 100% spans will be exported.

Now, one more thing we have to do that’s pass a spring boot

configured AcitveMQConnectionFactory to its builder, and make sure
use the zipkin-reporter bpm to align versons.
@Bean
ActiveMQSenderFactoryBean senderFactory() {
ActiveMQSenderFactoryBean result = new
ActiveMQSenderFactoryBean();
result.setUrl(“xxxx”); // put ActiveMQ
openwire url you’ve created in last step
result.setPassword("admin"); // put ActiveMQ
username you’ve created in last step
result.setUsername("admin"); // put ActiveMQ
password url you’ve created in last step
return result;
}

this is the time you can run all your microservices(zuul, oAuth2,
eureka, master-common, buyer-master-common) to test your tracing
Use postman or CLI to call you microservices with code below:
curl -X POST \
http://localhost:8888/service-oauth/v1/api/
login/token \
-H 'Accept: application/json, text/plain, */*'
\
-H 'Authorization: basic
Qmh3LUVwcy1KaWVuaWU6c2VjcmV0' \
-H 'Content-Type: application/x-www-form-
urlencoded' \
-H 'Postman-Token: 5c70952f-
c336-4759-8164-82d858db8fb5' \
-H 'cache-control: no-cache' \
-d
'username=V2Applicant&password=test123&client_id
=Bhw-Eps-
Jienie&grant_type=password&company_group_code=00
01'

After API call succeed, we can see the latency statistics at zipkin UI
http://localhost:9411 at service name option choose all service
then click Find Traces button. Then swicth to Try Lens UI to have
more better look and feel. So now you can know where you can do
performance analysis by looking at tracing dat.
You can see overview of request and you can see timing, you can see
the details of request and how long does it take.