RTF Log Forwarding Approaches

Exported by Andrey Yunkin ©2018 MuleSoft, Inc.

RTF Log Forwarding Approaches
Introduction

1 Introduction
This article covers different approaches of log forwarding that can be used for Runtime Fabric (RTF)
solutions. As we know RTF is Mulesoft's runtime plane that can be hosted on customers environment,
either on other cloud based platforms (AWS, GS, Azure) or On-Prem.
RTF manages Mule runtimes in K8s managed docker containers called pods, all packaged with Gravity
(Kubernetes cluster and all its applications into a single file called a "Cluster Image")  for ease of
installation and portability. Common requirement is to find a way to forward the Mule application logs to
3rd party logging platforms like ELK or Splunk in order to maintain the logs between redeployments or in
the event of crashes.  In this article we consider following 3 approaches and the last one is the key area of
focus in this article:
1. Out of the box RTF log forwarding. 
2. Third party agent polling.
3.  Hybrid (Unified way of logging where customer is using both CLoudHub as well as RTF).

Exported by Andrey Yunkin - 2 - ©2018 MuleSoft, Inc.

RTF Log Forwarding Approaches
Expected Audience

2 Expected Audience
The article aims at the following audience:
• Solutions Architect
• API Developer/Senior Technical Consultant

Exported by Andrey Yunkin - 3 - ©2018 MuleSoft, Inc.

RTF Log Forwarding Approaches
Log forwarding approaches

3 Log forwarding approaches

• Introduction
• Expected Audience
• Log forwarding approaches
• Out of the box RTF log forwarding (Up to version 1.5)
• Out of the box RTF log forwarding ( from Version 1.6 + )
• As of Runtime Fabric version 1.6 we have following additional log forwarding
options also supported by RTF as per v1.6 Documentation:
• Third party Log Collection Agent
• Unified Hybrid (RTF + CloudHub)
• Solution:
• Performance and Load Test results:
• Application configuration for each runtime plane
• RTF
• CloudHub
• Conclusion
• Resources

3.1 Out of the box RTF log forwarding (Up to version 1.5)
RTF documentation provides out of the box log forwarding feature that is offered by Gravity. Gravity has a
concept of resources similar to K8s, and one of these resources is logforwarder which we can define on
Controller nodes in order to update the cluster configuration for log forwarding. 
Every Gravity Cluster is automatically set up to aggregate the logs from all running containers. By default,
the logs are kept inside the Cluster but they can be configured to be shipped to a remote log collector
such as a rsyslog server.
Following is an example of log-forwarder.yaml

kind: logforwarder
version: v2
metadata:
name: mani-rsys-udp-logforwarder
spec:
address: 10.2.2.1:514
protocol: udp

If you have an rsyslog server (can be easily installed on all linux hosts) then all you have to do is to run the
following command:
gravity resource create log-forwarder.yaml

Then you can check the result using following command:

gravity resource get logforwarder

In order to remove it you can use following command:

gravity resource rm <resource name>

example:

Exported by Andrey Yunkin - 4 - ©2018 MuleSoft, Inc.

RTF Log Forwarding Approaches
Log forwarding approaches

gravity resource rm logforwarder mani-rsys-udp-logforwarder

3.2 Out of the box RTF log forwarding ( from Version 1.6 + )

3.2.1 As of Runtime Fabric version 1.6 we have following additional log forwarding options also supported by RTF
as per v1.6 Documentation:

• Azure Log Analytics

• Elasticsearch
• Graylog Extended Log Format (GELF)
• Splunk

3.3 Third party Log Collection Agent

You can use Third party agents like Cloudwatch agents, Filebeat or Kinesis agent to collect and ship out
the logs from the following directory:

# log files to monitor

/var/lib/gravity/planet/log/containers/*.log
 
# gravity specific ops logs
/var/lib/gravity/site/**/*.log

or If you know the uudi of the container you can use following:

# prior to 1.4
/var/lib/gravity/planet/docker/containers/<uuid>/<uuid>-json.log
 
# RTF 1.5
/var/lib/gravity/planet/log/containers/<deployment>-<pattern>.log
 
# an 1.5 app example
# sysapi-prod-7587d88b9c-zrswj_8fcf073b-1235-4f09-
ae3d-7ae2d8e71515_app-49d791a9962b99692dd7fa7509573c33f71c974136030de199b9a9008705eba1.log

But you need to be aware that these logs are per container, thus you will need to aggregate them. 

 Log collection agent needs to be installed on all workers nodes as deployments are scheduled
across all workers (filters properly set to capture useful logs), and pods are by design transient
and short-lived.

Exported by Andrey Yunkin - 5 - ©2018 MuleSoft, Inc.

RTF Log Forwarding Approaches
Log forwarding approaches

3.4 Unified Hybrid (RTF + CloudHub)

The goal is to provide a unified standard solution that can be used to perform log forwarding for both
CloudHub deployments as well as RTF deployments.
Assumption is customer uses On-Prem or Cloud Based RTF runtime plane as well as CloudHub and the
objective is to forward all logs from both runtime planes to 3rd party logging platform like Splunk. In
addition, all logs must have the Application name in them, like JSON logger layout (this should be always
followed otherwise we can not differentiate the logs after loading them to 3rd party system).
Considerations:
• RTF Limitations and possible options to get the logs:
• Using custom log4j2 appenders in RTF with custom log appenders is not an option. 
• Polling the Logs from /var/lib/gravity/planet/docker/containers/*/*.log but its not
best option since the logs are container base and need 3rd party agent.
• Using Gravity logforwarder resource to RSysLog Server, we can not configure the
3rd party logging platform credentials for example Splunk needs username/
password/token. 
• For CloudHub the best option we have is to use custom log4j2 appenders by disabling
CloudHub logs, thus we can use log4j's socket appender. 
• RSysLog Server supports TCP or UDP protocols to send strings/logs on default port 514. It
has better performance with UDP due to TCP overhead, tradeoff is UDP doesn't guarantee
delivery (chance of packet loss over internet) but TCP does.  

3.4.1 Solution:
• Please refer to following architecture diagram. We will have 2 APIs acting as RSysLog
Server, one deployed in CloudHub and one deployed to RTF. Since both are native to
deployments, we won't have security concern for data at transport from runtimes to
RsysLog-Api. Then we will publish the logs to any message broker like Anypoint MQ/
Active MQ or etc. Next, we can have another API like System API that fetch the messages
from message broker and publish them to target logging platform like Splunk or ELK, this
API is where we can have all configurations and credentials required by 3rd party logging

Exported by Andrey Yunkin - 6 - ©2018 MuleSoft, Inc.

RTF Log Forwarding Approaches
Log forwarding approaches

platform(eg: Splunk needs username, password and token). 

• In RsysLog API, mule can act as a RSysLog server using Socket connector (Use version 1.1.6
due to known issues in earlier versions) configured for UDP for best performance and
throughput. Performance related considerations:
• In this case we are using Anypoint MQ, please note that every publish message in
Active MQ is an HTTP call which can be much slower than rate of incoming logs per
second. We can have 2 options Async Scope or VM.
• We better use reliability pattern here using VM for following reasons when using
Anypoint MQ, if using Active MQ which uses JMS is an option Async Scope can be
considered as well:
1. We won't lose messages and keep accepting incoming calls, in Async scope
under heavy concurrent load we may run out of threads.
2. We can asynchronously process all of them over time under the balanced
combination of high and low load. 

Exported by Andrey Yunkin - 7 - ©2018 MuleSoft, Inc.

RTF Log Forwarding Approaches
Log forwarding approaches

3. Large volume of concurrent incoming logs can bring down the 0.1 vCore
worker if we use async scope (each thread consumes 1MB in Java 8) when
publishing to Anypoint MQ.
• Dataweave becomes another bottleneck since it uses CPU intensive threads and
under high load it will limit the application, thus make sure to add
"mule.dwScript.processingType=BLOCKING" to application properties, Please refer
to support knowledge base. 
• I have increased the flow max concurrency to 10 and it performed very well under
the heavy load with minimum cpu and memory impact on the worker. 

• We should not use INO Loggers since they can impact the worker's performance
under the high load, only debug loggers can be used for troubleshooting. 

Exported by Andrey Yunkin - 8 - ©2018 MuleSoft, Inc.

RTF Log Forwarding Approaches
Log forwarding approaches

3.4.2 Performance and Load Test results:

In this test I have pushed 20,000 messages in 10 min using 50 threads (50
concurrent message) with payload size of 3kb. 

• JMeter results:

• CPU Stats during the test from Anypoint Monitoring:

• CPU credit and stats monitored from AWS Console:

Exported by Andrey Yunkin - 9 - ©2018 MuleSoft, Inc.

RTF Log Forwarding Approaches
Log forwarding approaches

• Number of messages arrived at Anypoint MQ are 19968, thus we have lost 32

messages which is 0.0016%. Remember that we are using UDP so message
loss is a factor, and in this test I pushed messages from On-Prem to RSysLog
API deployed to CloudHub which is the worst case scenario, while if you push
UDP to local network we will have far less packet loss. 

Exported by Andrey Yunkin - 10 - ©2018 MuleSoft, Inc.

RTF Log Forwarding Approaches
Log forwarding approaches

3.4.3 Application configuration for each runtime plane

RTF
Considerations:
• In RTF using log4j2 is not an option thus RTF will be configured to forward the logs
to our UDP API using following FT.(for detailed steps refer to Out of the box RTF log
forwarding section.)
• We can not send UDP via Public Load Balancers thus we must use mule-worker-
myapp.region.cloudhub.io to send requests directly to the application from outside
of Anypoint Platform or use mule-worker-internal-myapp.region.cloudhub.io from
within VPC.
• FT url format under address can not be used for load balancing
• if sending to custom port, in VPC firewall you need to allow UDP traffic for that port.

kind: logforwarder
version: v2
metadata:
name: RSysLog-Api
spec:
address: <ft-host1>:<ft-port1>, <ft-host2>:<ft-port2>
protocol: udp

CloudHub
For CloudHub deployed APIs we need to Disable CloudHub logs and add following appender to it. 

<Socket name="socketAppender" host="mule-worker-internal-rsyslog-api-v2.au-s1.cloudhub.io" port="8081"

protocol="UDP">
<JsonLayout properties="true" />
</Socket>

3.5 Conclusion
At this time, if we really want to have unified solution this approach works. Whether to use TCP or UDP is
something that needs to be decided during sessions with customers and tested and POCed.
If customers can tolerate very few log loss, it should be fine and you can achieve better performance using
UDP. Best practice is to have RSysLog API native to each runtime plane to have data security at transport
and far less packet loss due to UDP. 

Exported by Andrey Yunkin - 11 - ©2018 MuleSoft, Inc.

RTF Log Forwarding Approaches
Log forwarding approaches

3.6 Resources
My RSysLog API implementation
To performance tune your solution you can use my JMeter test case, you need to install JMeter Plugin
Manager first and next install UPDRequest plugin.
My log-forwarder.yaml for RTF deployed APIs. 
My log4j2.xml for CloudHub deployed APIs that push to CloudHUb logs as well as RSysLog API.

Exported by Andrey Yunkin - 12 - ©2018 MuleSoft, Inc.