MESSAGING QUEUES
Advantages of Message Oriented Middleware
SYSTEM INTEGRATION
May 22th, 2017
MESSAGE QUEUE
Disadvantages of Message Oriented
Middleware
The primary disadvantage of many broker based
message oriented middleware systems is that
they require an extra component in
the architecture, the message transfer agent
(message broker). As with any system, adding
another component can lead to reductions
in performance and reliability, and can also
make the system as a whole more difficult and
expensive to maintain.
Asynchronicity/Loose coupling between components: MOM technologies
use asynchronous message passing as opposed to request-response
architectures. In asynchronous systems, message queues provide temporary
storage when the destination program is busy or not connected. In addition,
most asynchronous MOM systems provide persistent storage to back up the
message queue. It also means that should the receiver application fail for any
reason, the senders can continue unaffected, as the messages they send will
simply accumulate in the message queue for later processing when the
receiver restarts.
Routing: Many message-oriented middleware implementations depend on
a message queue system. Some implementations permit routing logic to be
provided by the messaging layer itself, while others depend on client
applications to provide routing information or allow for a mix of both
paradigms
Criteria for evaluating
Durability: Whether messages can be written to disk, or
even committed to a DBMS for reliability in case of server
crashes.
Point-Point vs Publish-Subscribe: Whether the message
queue provider supports point-point or publish-subscribe
mode of delivery
Push vs Pull Model : Whether the consumers have to pull
the messages or the broker pushes the messages to the
consumers.
Message Acknowledgements: Whether messages
acknowledgement modes are supported between
consumers, brokers and producers.
 Criteria for evaluating
Delivery Policies: Whether a message should be delivered
at least once, or no more than once.
Purging Policies: Whether Queues or Messages support a
“time to live”.
Message Size/Format: What message sizes and types are
supported.
Message filtering/routing policies: Whether messages can
be filtered so that a consumer can see only messages
matching some criteria of interest and messages can be
routed to consumers based on defined policies.
DATABASE QUEUE
Criteria for evaluating
Message Ordering: Whether messages are delivered to
consumers in a FIFO ordering or not.
Batching Policies: Should messages be delivered
immediately, or should the system wait a bit and try to
deliver many messages at once.
Scalability: What order of throughput, latency and volume
of queues, messages is supported.
Availability: Does the queuing system support high
availability in case of failures of servers.
Operational and financial cost
Cross platform interoperability
EVALUATION
Durability: Messages are saved to the database always.
Point-Point vs Publish-Subscribe: Supports only Point-Point model.
Delivery Policies: Messages are guaranteed to be delivered as the
Message producer gets an acknowledgement. Messages are
consumed by only one consumer by using a status field in the table
to co-ordinate between multiple consumers.
Message Acknowledgements: Msg are acknowledged as soon as
they are stored in the persistent store
Purging Policies: Messages can be retained and can be purged
manually or through a database trigger based on some purging
policies.
Message Size/Format: Can store binary/text messages of any size
limited by the underlying disk space and datatype used for the
column. Even though there is no limitation on message size, its not
ideal to have messages of very large size.

Message filtering/routing Policies: Message filtering
has to be implemented in the MessageHandler
component of the consumer service. Does not support
routing of message to multiple services implicitly. Each
message will have to be sent to every other Service
that has to consume the message
Routing message to multiple consumers
Message Ordering: Consumers pick up messages from
the table based on the timestamp of message. This
does not result in strict FIFO due to multiple
consumers.
JMS QUEUES
EVALUATION
Batching Policies: Messages can be processed in batches
based on batching policies defined.
Scalability: Not very scalable. Would hit performance
issues due to lock contention on database read/write
operations and too much disk I/O.
Availability: Using a dynamic service discovery solution to
detect service instances being added or going down,
would achieve high availability.
Operational and Financial Cost: Does not incur any
financial cost except the initial effort to design the
solution; would incur minimal operational burden.
Cross Platform interoperability:
JMS QUEUES
In this model, there is no transformation required as both the Producer and
Consumer clients are using the same protocol to communicate. Although this
scenario seems to solve the cross-platform interoperability problem, there
are some inherent problems.
Both protocols may not support the same message body types.
Applications would be locked into one specific vendor solution (or in some
cases only a few vendor choices) due to the built-in message bridge.
Both protocols may not support same datatypes, custom properties and
header properties between the messaging clients.
These scenarios illustrate that JMS isnt’t the right solution for cross-platform
interoperability.
While vendors such as ActiveMQ, HornetQ, Apollo, SonicMQ etc provide
some level of cross platform interoperability, it is usually through proprietary
protocols, API’s and client libraries. Without some sort of messaging
standard, it is difficult to know what messaging system another company or
system is using and how to interface with it.
 AMQP Message Brokers Exchanges, Bindings and Queues

AMQP (Advanced Message Queuing Protocol) is a
standard binary wire level protocol that enables
conforming client applications to communicate with
conforming messaging middleware brokers. AMQP
allows cross platform services/systems between
different enterprises or within the enterprise to
easily exchange messages between each other
regardless of the message broker vendor and
platform.
The AMQP Model at a high level works like this:
messages are published to exchanges, which are
often compared to post offices or mailboxes.
Exchanges then distribute message copies
to queues using rules called bindings. Then AMQP
brokers either deliver messages to consumers
subscribed to queues, or consumers fetch/pull
messages from queues on demand. Various brokers
that have implemented the AMQP protocol like
RabbitMQ, StormMQ, Apache Qpid, Apache
ActiveMQ

Features of a RESTful Services AMQP PUSH

Every system uses resources. These resources

can be pictures, video files, Web pages, business
information, or anything that can be
represented in a computer-based system. The
purpose of a service is to provide a window to
its clients so that they can access these
resources. Service architects and developers
want this service to be easy to implement,
maintainable, extensible, and scalable.
 AMQP PULL
In the pull model, each consumer fetches messages using amqp basic.get method
(requests to basic.get are synchronized by the broker
The broker maintains information about the state of the queue and messages in
Erlang’s distributed database Mnesia
Consumer can send message to the same exchange to notify producer of status if
required. In case of an External Service, the external service can notify the producer
through a callback url.
AMQP EVALUATION
Purging Policies: Messages are removed from the queue once they are acknowledged.
Messages are stored in memory (also on disk for persistent messages) and retaining for long
periods of time would exhaust memory resulting in performance issues. Rejected messages
and negative acknowledged messages can be requeued or forwarded to dead letter exchange
which will route to appropriate queue for handling such messages.
Message Size/Format: Supports binary messages and message size is limited by the
amount of memory/disk space on the broker. When a queue gets too large exhausting
memory, messages published to the queue will be written to disk. Even though there is no
limitation on message size, its not ideal to have messages of very large size.
Message filtering/routing Policies: Supports a variety of filtering/routing techniques
through the different types of exchanges (direct, fanout, topic, headers etc). No other broker
solutions offer such a wide range of routing capabilities. AMQP supports more complex
bindings depending on the requirement of applications.
Message Ordering: Messages are delivered in FIFO order but message consumption may
not be in FIFO order due to multiple consumers; unless each queue is associated with an
exclusive consumer.
AMQP EVALUATION
Durability: Durability can be configured at exchange/queue/message level. Messages will
be persisted until they are acknowledged (either auto or explicit acknowledgement).
Point-Point vs Publish-Subscribe: Supports Point-Point model as well as publish-
subscribe through different exchange types (direct, fanout, topic etc).
Delivery Policies: Supports different message delivery policies
1. At-most once Delivery: When using auto-acknowledgement messages are delivered
at most once as a message is deleted as soon as it is delivered to the consumer
2. At-least once Delivery: When using explicit acknowledgement messages may be
delivered more than once if the broker does not receive an acknowledgement from the
consumer either due to network failure or consumer/broker crash. In such scenarios,
messages will re re-queued resulting in delivery again with a delivered flag set to true.
Consumers can also reject or nack messages which will result in re-queuing of messages for re-
delivery.
Message Acknowledgements: Supports message acknowledgements between broker-
>producer as well as consumer->broker.
AMQP EVALUATION
Batching Policies: This applies for both the push/pull model. Messages can be
pushed/fetched in batches by setting a Qos prefetch count either on the channel level or
consumer level. This value should be optimized based on the average round trip time (time
taken for broker to send a message and receive ack) and the processing time of the message
on the consumer so that the consumer can consume messages at the maximum possible rate.
Scalability: Highly Scalable in terms of throughput(million messages/sec in some cases;
varies with message size) and latency is in the order of tens or hundreds of millisecs. The
number of queues that can be supported would also be in the order of millions in a clustered
deployment limited only by the number of concurrent socket connections that the broker can
support.
Availability: Supports high availability by replicating entities(Exchanges and Bindings) to all
brokers in the cluster to recover from unexpected failures. Queues have to be mirrored to a
subset of brokers for making them highly available. Chooses Availability over Consistency in
the event of a network partition (AP model)
Operational and Financial Cost: Involves operational and financial cost to manage
servers and monitor deployments and health of the system.
 APACHE KAFKA
Kafka is a distributed, partitioned, replicated commit log service.
It provides the functionality of a messaging system, with a
unique design. In comparison to most messaging systems Kafka
has better throughput, built-in partitioning, replication, and
fault-tolerance which makes it a good solution for large scale
message processing applications
KAFKA
Scalability: Highly Scalable in terms of throughput(million writes/sec, order of
hundreds of MB/sec depending on message size) and latency is in the order of tens or
hundreds of millisecs. The number of topics that can be supported will be in the order
of a few thousands. This is limited due to the fact that the configuration information
of brokers, topic-partitions, consumer offsets, partition-owners is stored in zookeeper
znodes and zookeeper being a in-memory non-sharded data store, we will eventually
exhaust the memory.
Availability: Supports high availability by replicating partitions to other brokers in the
cluster based on a replication factor, rebalancing the partitions to other brokers in
case of broker crashes and by maintain ISR’s for each partition. Chooses Availability
over Consistency in the event of a network partition (AP model) in the latest stable
release; with future releases supporting a configuration for choosing consistency over
availability or availability over consistency.