A-TeamGroup

June 2009

Complex Event
Processing
Over the past couple of years, Complex Event Processing has
emerged as a hot technology for the financial markets, and its
flexibility has been leveraged in applications as diverse as market
data cleansing, to algorithmic trading, to compliance monitoring,
to risk management. CEP is a solution to many problems, which is
one reason why the emerging marketplace is growing, with many
vendor options to choose from.

But CEP is not a solution in itself, and so technologists need to

concern themselves with how to integrate it within a complete
application architecture, how it can best be leveraged, and how
to streamline development and deployment. And with multiple
vendor choices, choosing the right CEP approach for the job is an
important, and early, activity for firms looking to adopt it.

Expect CEP offerings to become more complete in terms of

functionality - especially in the event modelling area - and to
also exploit technologies that will boost their performance. And
expect CEP to become further embedded across the entire trade
processing and operations chain. The scenario of CEP becoming
an infrastructure component - just like a database or messaging
platform is now - is a near certainty. But as for when that will
happen is open to debate. The entry into the market by IT heavy
hitters like IBM and Oracle - and Microsoft is on its way - will likely
drive the infrastructure-level adoption of CEP.

Also expect the CEP marketplace to evolve commercially, as

offerings from different vendors become more full function.
We’ll see different business models, including cloud delivery,
partnerships, acquisitions and mergers. And open source will play
a bigger role. As will CEP itself, touching all aspects of the trading
enterprise.

Sponsored By:

An A-TEAMGROUP Publication www.a-teamgroup.com

Complex Event Processing - A-Team Group
Inside the Complex World
Of Event Processing
By Peter Harris, President of Americas & Editor-at-Large, A-Team Group
T
ruth be told, software engineers
in the financial markets have been
writing code to handle events since
the year dot. A price update from an ex-
change, a trading limit that gets broken,
a stock split. All are examples of events
that might cause some software process
to take place.
Much of that event processing code is
pretty generic. When the price of an NYSE
stock hits a certain limit and so causes an
event to be triggered, the process would
be pretty much the same if the stock was
a NASDAQ one, or one traded on an ECN.
Thus, the emergence of generic event
processing functionality and the birth of
a software segment that’s expected to
mushroom over the next few years.
That segment – known as event stream
processing to some and complex event
processing to others – was worth $99 mil-
lion in 2007, according to Aite Group, who
predict it will grow to $460 million in 2010.
Given the numerous applications that this
technology can be applied to, across all
asset classes, strong growth among ESP/
CEP vendors seems highly likely.
In recent years, a number of vendors
offering CEP packages have emerged,
creating a fairly fragmented marketplace
– and we’ve now begun to see some con-
solidation – such as the recent merger of
Aleri and Coral8. Commoditisation has
also begun to creep in, driven in part by
A-Team’s Special Report on Complex Event Processing is Presented By:
2 June 2009 • Issue 05
the emergence of credible open source
projects, such as Esper.
At the same time, we’ve seen new en-
trants, such as Oracle (which picked
up a CEP engine with its acquisition of
BEA Systems), IBM with its recently an-
nounced System S/InfoStreams offering,
and Sybase (which bought the source
code of the Coral8 CEP engine). And Mi-
crosoft is on its way too, with a project
codenamed Orinoco.
In-house CEP development also con-
tinues, in part because those firms with
the requisite resources still see merit in
proprietary coding, to give them an edge
for specific trading applications, over a ge-
neric package. But such applications are
limited, and CEP packages will surely find
widespread adoption further along the
trade processing chain.
CEP vendors have adopted different ap-
proaches in building their businesses. For
example, Vhayu has built a strong profes-
sional solutions business, designing and
building customised solutions for custom-
ers, and has expanded its Velocity product
in several directions, most recently lever-
aging hardware acceleration to store vast
amounts of market data.
Many vendors – both established and
upstarts – have built specific application
offerings on top of their core CEP en-
gines. Aleri has rolled out order manage-
ment and liquidity management offer-
ings. Progess/Apama has built products
for algorithmic trading and smart order
routing in both the equities and foreign
exchange markets. Sklyler Technology
has introduced solutions for market data
feed management, tick analysis, liquidity
discovery and order management, while
Truviso has turned its attention to FX al-
gorithmic trading in partnership with FX
ECN Hotspot FX.
Partnerships have also proved impor-
tant for the CEP players. As well as reselling
Vhayu’s software, Thomson Reuters also has
partnerships with StreamBase (to integrate
StreamBase’s Studio application designer
with its Reuters Market Data System) and
Skyler, to deliver its order book product to
Reuters Datafeed Direct customers.
And despite the competitive element,
Vhayu works with the likes of Apama and
StreamBase to provide a historical market
data framework to underpin CEP tools.
Co-opertition, partnerships and verti-
cal strategies are likely key elements in the
business futures for emerging CEP play-
ers, and for established vendors looking
to grow into new markets. Also a factor
is whether CEP will become a generic IT
component, like messaging middleware
or an application server. That scenario will
benefit the bigger players like IBM and
Oracle, and will probably fuel further con-
solidation. For sure, there are interesting,
if perhaps complex, times ahead of us!
An A-TEAMGROUP Publication

Aleri CEP Expands
Beyond the Trade
By Jeff Wootton, Vice-President Product Strategy, Aleri
E
arly adopters of complex event
processing (CEP) technology were
people building front office trad-
ing applications. It’s easy to see why:
high performance CEP implementations
are capable of handling very high mes-
sage rates with very low latency – exactly
what many trading applications require.
For the first time it allowed for rapid ap-
plication development and high speed
data analysis, two attributes that have
previously been mutually exclusive.
One aspect of CEP that early adop-
ters quickly put to use was the ability
to analyse high volume event streams
to watch for – and alert people to –
conditions that warrant immediate
attention – typically trading oppor-
tunities. In fact, some people think
of CEP as just a tool for monitoring
and alerting, but that’s just the tip of
the iceberg.
As so often happens with new tech-
nology, CEP has started to branch out
from the initial uses of the early adop-
ters. As users have become familiar with
the capabilities of CEP products on the
market, they have spotted other areas
that can benefit. Some of the other ways
that firms are applying CEP in the trad-
ing operation include:
• Market data enrichment: filter and
cleanse; normalise; compute value-
added fields; combine data from mul-
tiple sources
• Auto-quoting: continuously update
prices in step with the market
• Order book consolidation – liquid-
ity analysis: analyse liquidity in a frag-
mented market
• Smart order routing: take liquidity
analysis one step further and use the
insight to route orders
An A-TEAMGROUP Publication
Complex Event Processing - Aleri
• Order flow surveillance: moni-
tor order flow to gain insight or for
compliance
• Algorithmic trading: both quantita-
tive strategy trading as well as algo-
rithmic execution
• Algo monitoring: monitor the status/
progress and measure effectiveness of
trading algorithms
And now we’re seeing CEP spread
beyond trading. One of the most no-
ticeable is the growing number of
post-trade applications based on CEP
technology and in particular position
and risk monitoring.
With the current volatility in the
markets, firms are seeing a need
for more timely insight into their
positions and exposures
With the current volatility in the mar-
kets, firms are seeing a need for more
timely insight into their positions and ex-
posures. Getting an overnight report that
consolidates positions across systems
and values positions against end-of-day
prices just isn’t sufficient anymore. They
want to be able to see the current con-
solidated value of their positions, with re-
al-time mark-to-market. Along with this,
they want to be able to aggregate posi-
tions along multiple dimensions, looking
at exposure by counterparty, asset class,
trading group within the firm, industry
sector, currency, etc. Add to this the abil-
ity to track exposures against limits and
generate alerts when limits are reached
(or maybe approached).
Traditional relational database tech-
nology can’t handle the rate of change
within the data sets (as transactions are
June 2009 • Issue 05
logged and market prices change) and
the overhead of constantly re-comput-
ing net positions.
That’s where CEP comes in: the event-
driven architecture of a real-time CEP
platform can continuously re-value po-
sitions and update aggregate values
across all dimensions. Information can
be delivered to users on an event-driv-
en basis (e.g. alerts) as well as an on-de-
mand basis – allowing users to view and
analyze the current set of positions at
any time. Typically, it’s a combination of
both that gets deployed.
This is just one example of where CEP
is going beyond alerting and trading to
deliver what we at Aleri like to call Con-
tinuous IntelligenceTM - the ability to have
insight into what’s happening right now
in order respond faster and make better
decisions. Trading firms have been build-
ing real-time data analysis applications
for years in C++ and Java, but they are
expensive and time-consuming to build
and maintain.
Conventional data analysis tools like
databases, spreadsheets and reporting
tools make it easy, but can’t cope with
data that changes rapidly. CEP provides
a platform for quickly implementing
applications that can cope with rapidly
changing data, applying custom busi-
ness logic to extract insight an enable
immediate response.
What firms are now beginning to re-
alise is that CEP is more than just a tool
for algo trading and alerting. It’s a ver-
satile tool for extracting immediate
insight from fast moving event data.
Wherever there’s a need to understand
the impact of events as they happen, no
matter how fast they happen, CEP tech-
nology can help. It’s the foundation for
Continuous Intelligence.
3
Complex Event Processing - Apama
Capital Markets
Embrace CEP’s
Real-Time Control
By Dr John Bates, General Manager and Co-Founder, Progress Apama
C
omplex event processing (CEP) is
used in capital markets to monitor
fast-moving event streams, ana-
lyse events to identify business threats
and opportunities, and take appropriate
action in real-time. In algorithmic trad-
ing, automated strategies optimise price,
size, and more to achieve best execution.
Further, market data may be analysed for
alpha opportunities, before execution.
The tactical view is that CEP is focused on
this valuable, but narrowly-defined use
case; and trading firms need other tech-
nologies for adjacent or new applications.
The strategic view is that CEP represents a
fundamental paradigm shift – to monitor,
analyse, and act – on any business update
that can be treated as an event.
The potential for market abuse has
risen with increasing trade and market
data volumes, and growing market frag-
mentation. Markets must maintain a level
playing field to ensure that participants
are protected from unfair practices. And
participants need to proactively detect
and prevent abusive patterns from inter-
nal and external sources to avoid the reg-
ulators’ penalties. Typically, what should
be a preventative measure quickly falls
behind the increasingly large amounts of
real time data and analysis.
Leveraging CEP for surveillance allows
regulators, compliance, and risk officers
to create strategies to detect and prevent
fraudulent activity, where market par-
ticipants intentionally and illegally ben-
efit from market movements. Here, trade
order flows represent the event streams
to be monitored in real-time. Analysis in-
cludes trading pattern detection models
to identify any suspicious behavior. Ex-
ample abusive patterns include driving
4 June 2009 • Issue 05
the price of an investment or index up or
down to make a profit or avoid loss, known
as “ramping,” and “painting the tape,” a
series of transactions intended to give the
impression of activity or price movement.
Finally automated action may be taken -
to stop further trades or raise an alert to
an operator for further investigation.
Further, CEP driven market surveillance
applications let business users retain con-
trol. In an environment where individu-
al trading strategies are revised in three
months or less, the middle office can now
use the same event graphical modeling
tools available to algorithmic traders. By
quickly deploying new detection models,
compliance officers can react to the new
schemes faster than ever.
Middle-/back-office risk monitoring
technology generally does not keep pace
with trading. Most risk management tasks
are run end-of-day and cannot catch prob-
lems as they occur. As trading velocity in-
creases and as automation takes over, risk
managers increasingly face the challenge
of blending trading and risk management
without strangling the front office. High
frequency trading is complex, positions
are built in milliseconds, and exposure
calculations are resource intensive.
With the speed and flexibility provided
by a CEP platform, traders and risk man-
agers can now model and implement
their own proactive analytics to detect,
prevent and mitigate unacceptable trad-
ing risks with minimal impact on the ex-
ecution latency. Here again, streams
of trade order flow are the events to be
monitored, trade-by-trade in real time,
as are market prices and events derived
from relating the two, and other sources.
Analysis examples range from simple “fat
finger” checks to temporally driven P&L
and exposure calculations. Further, CEP
scalability enables firms to further analyse
risk cross-trader, cross-desk, and cross-as-
set to achieve an institution-wide view of
risk, P&L, and total exposure with the abil-
ity to do something about it in real time.
Actions resulting from violations include
alerts, automated position management
and auto-hedging instructions, and au-
tomated limit calculation and enforce-
ment. Here, a “Risk Firewall’ rejects trades
that breach limits before they are sent to
market. Further, real time views of a trad-
ing model’s actual P&L vs. potential P&L
provides opportunities to optimise trad-
ing operations against intraday trends
not visible from an end-of-day snapshot.
Not only does the event-based ap-
proach reduce risk, it enables traders to
pursue trading strategies that they oth-
erwise couldn’t without the safeguards.
The old saying in racing is “Never drive
faster than your guardian angel can fly.”
With risk management running at the
pace of trading, including the ability to
take immediate action, once conserva-
tive parameters may be set more ag-
gressively, increase trading volumes and
thereby profits.
A capital markets firm wins on several
fronts by finding other uses for its existing
CEP platform. Business users benefit from
retaining control, achieving quick imple-
mentations, and the flexibility to create
enhancements to applications vs. wait-
ing for traditional IT development lead
times. When budgets are strained, being
able to find new uses for existing technol-
ogy through CEP is a cost-effective way to
deploy applications and create a leading
edge at the same time.
An A-TEAMGROUP Publication
The Power of
Complex Event Processing
in Capital Markets

What trading platform...

I is proven at more than half of the world’s largest banks?
I delivers algorithmically-powered trading strategies for Equities, Derivatives, Foreign Exchange
and Fixed Income?
I is pioneering the use of real-time market surveillance by regulators, exchanges and trading firms?
I is providing algorithmic traders with the ability to integrate risk management into their
trading strategies?
I goes beyond algorithmic trading to deliver market aggregation, smart order routing and real-time
pricing applications for its clients?

Progress Apama ® ®

Bringing the power of Complex Event Processing to Capital Markets.

Visit www.progress.com/apama or contact us at

Europe, Middle East & Africa North & South America Asia & Australia
T +44 207 788 0137 T +1 800 979 2546 T + 61 2 9919 7100
E apamasolutions.emea@progress.com E apamasolutions.americas@progress.com E apamasolutions.asiapac@progress.com
Complex Event Processing - Roundtable

Roundtable:
CEP Today and
Dr John Bates, General Manager and Co-
Tomorrow...
Founder, Progress Apama

With some of the free CEP engines out

Why invest in a CEP pack-
there (Esper, Microsoft Orinoco), you can
age rather than simply writing
realise an immediate ROI by integrat-
specialised event processing
ing a third-party CEP engine with your
code?
application.
Adler: It really depends on the use
case and the complexity of the stream
Bates: One can, of course, write spe-
cialised code, but we see clear advan-
processing that the use case entails.
tages to leveraging a CEP product. The
For fairly simplistic cases where you are
main reasons to use a CEP platform are
performing simple filtering and calcula-
that you can use fit-for-purpose tooling
Daniel Chait, Managing Director, tions, you may find that writing custom
to build applications quickly, and that
Lab49 code in C# or Java is the simplest thing.
the underlying platform makes the ap-
Custom code also gives you complete
plication run as quickly as customer
control over the features and over the
code that could have taken 10-100 times
release cycle.
longer to develop.
However, CEP packages combine a lot
CEP allows the organisation to use
more than simple stream processing. For
technology that captures the collective
a certain class of applications, the use of
intellectual property of a vendor that
a streaming SQL dialect allows you to
is focused and committed to complex
perform complex streaming processing
event processing as their business. In
in few lines of code. CEP engines con-
our instance, the customer can lever-
tain various windowing constructs that
age Apama’s experience in addressing
allow you to perform calculations over a
issues that someone that is writing spe-
time series of data. Some come with rich
Jeff Wootton, Vice-President Product cialised event processing code may be
GUI-based development environments
Strategy, Aleri confronting for the first time. Arguably,
that allow a non-technical user to easily
we have already “been there, done that”
prototype a specific use case.
Marc Adler, in many instances, given our customer
In addition to the actual stream
Director and Chief Architect of Equities, experience and the sizable research and
processing engine, CEP packages also
Citigroup development investment in CEP that
come with a rich set of input and output
we make. That investment includes an
adapters that allow you to easily con-
engineering team of very bright people
nect to a variety of streaming data
whose sole focus is on addressing the
sources. For example, a CEP package
opportunities and challenges associat-
might contain adapters that connect to
ed with CEP.
various market data sources like Thom-
Secondly, by investment in a CEP
son Reuters, Bloomberg, and NYSE Tech-
product, you can focus your attention
nologies/Wombat. These adapters alone
on the elements of the deployment that
would take several weeks to write.
are unique to your organisation, rather
CEP engines are usually highly optimised
than sustaining an investment in the
for multi-threading and multi-core process-
infrastructure plumbing. Event-based
ing, something that would normally take a
applications are quite dynamic as the
bit of effort for most developers to write.

6 June 2009 • Issue 05 An A-TEAMGROUP Publication


circumstances that drive the events
change. With a CEP product, you can
delegate to the vendor/product many
of the challenges associated with keep-
ing pace with changes. Thus, the prod-
uct affords you the opportunity to write
event processing code as needed, to ad-
dress the unique elements of your appli-
cation and/or business.
Chait: CEP systems are most suited for
applications which consume and gen-
erate multiple streams of data, whose
data streams needs to be combined and
compared, and aggregated in a tempo-
ral fashion. For stream processing appli-
cations which only require filtering and
projection (map/select), CEP systems
are overkill as those operations can be
done on streaming data in contempo-
rary languages with little effort.
Some of the benefits of buying a pack-
age are the ability to take advantage of
normal benefits that come from buying
any major systems product – time to
market, prebuilt functionality and sup-
port. In addition, CEP vendor products
will bring with them many useful fea-
tures to aid in developer productiv-
ity (sophisticated debuggers, testing
harnesses, graphical editing tools, etc)
which typical in-house systems would
never build.
Wootton: For application developers,
a commercial CEP platform is a produc-
tivity tool. It lets you get an application
built and into production quicker, by
letting you focus on the business logic
and leaving all the data handling, scala-
bility, reliability, etc to the platform. The
result is the ability to do more, in less
time, with less development resources,
ultimately improving business agility.
For some firms, the benefit is in reduc-
ing the skill level required to build appli-
cations. You don’t have to be a network
programmer, skilled in building multi-
threaded event-driven software to build
an application on a CEP platform. It be-
comes accessible to the sort of user that
builds complex spreadsheets or that
builds database applications.
Our customers tell us that building
An A-TEAMGROUP Publication
Complex Event Processing - Roundtable
their applications on Aleri has cut their
time and effort in implementing a new
application by about 75% when com-
pared to simply writing the application
from scratch. This means that in many
cases the CEP license has paid for itself
the day the application goes live, just
from the up front cost savings – and yet
the benefits continue to accrue from
a lower ongoing cost to maintain the
application.
How does a system designer
determine when it would be
beneficial to leverage a CEP
package within an application
architecture?
Adler: It makes sense to consider a
CEP engine when you do not have the
time or the resources to code up the
entire ecosystem that is being offered
by the CEP vendors. When you buy a
CEP system, you get not only the actual
stream processing engine (which is
often optimised for multiple threads
and multiple CPUs), but you get a devel-
opment environment, input and output
adapters that provide connectivity to a
wide variety of data sources, visualisa-
tion tools, and a support staff behind
the scenes.
The complexity of the use case also
plays an important part in the decision
to use a CEP system. Most of the CEP sys-
tems come with a pattern matching fa-
cility that allow you to implement mul-
tiply-nested time-base queries that can
recognise the existence or non-existence
of a particular pattern. This is something
that might take a bit of time to imple-
ment efficiently if you were going to go
the route of custom coding.
For applications that have low-latency
requirements, it makes sense to do POCs
with a number of the CEP products, and
to compare it against custom code. If
your custom code performs magni-
tudes better than all of the CEP engines,
then it might make sense to stick with a
custom implementation. Also, you need
to ensure that the language that the CEP
engine is programmed in is expressive
and flexible enough for your needs. The
June 2009 • Issue 05
limitations of the streaming SQL para-
digm have been debated frequently in
the blogosphere. There is no excuse for
not doing a comprehensive evaluation
of all of the technologies available to
you for processing streams of data.
Bates: Though there are no hard and
fast rules, the simple rules might be (a)
is your data changing frequently; (b)
are your rules changing reasonably fre-
quently; (c) are your rules reasonably
complex; and (d) do you need to take
actions within a short time window to
maximise profit or reduce risk. Clearly,
the more sophisticated the logic and
the more dynamic the conditions, the
better the case for a CEP product like
Apama. Apama offers a proven, high
performance CEP engine, so organisa-
tions know that foundation is available
to them “out of the box.”
In addition, our customers are attract-
ed to the development tools that can ac-
celerate their deployments, so as the ap-
plications get more sophisticated and,
in many cases, more dynamic (changing
over time), the more attractive a product
like Apama is since the customer recog-
nises that CEP applications are not “one
and done.” Additionally, the connectiv-
ity that we offer is often a key value. In
capital markets, you need connectivity
to an array of liquidity and market data
sources, and we offer a range of pack-
aged adapters that our customers can
use, rather than having to write their
own.
Wootton: The first test I like to apply
is two look for the presence of two con-
ditions: data in motion and the need for
real-time analysis – in other words, to
understand and act on events as they
happen, rather than after the fact. It’s
the intersection of these two things
where CEP adds the most value. If you
are analysing static data sets, you gen-
erally don’t need CEP – you can use tra-
ditional database tools. And the same
is true if you are content to collect the
data an analyse it later, running batch
reports. But where you have streams of
event data that you need to understand
7
Complex Event Processing - Roundtable
and respond to as things happen – that’s
where CEP can help. It’s what we like to
call continuous intelligence.
Now there are places where CEP can
be useful in analysing static data sets –
particularly when looking for patterns
of events. And there are use cases where
CEP can add value in simply collecting
event data into a database – where
CEP can correlate, filter and transform
incoming event data such that what’s
stored in the database is optimised for
future analysis and reporting purposes.
But generally it comes down to the
need for continuous intelligence – where
you extract insight from live streams of
event data in order to respond faster
to changing conditions and to make
better decisions based on more timely
information.
Can a generic CEP package
really provide the real-time
performance required of fi-
nancial trading applications?
Adler: There was an effort under-
way in 2008 in the Stac Research A1
committee where the various CEP sys-
tems were going to be benchmarked.
There were going to be a number of
low-level benchmarks and there was
going to be a benchmark that was an
end-to-end trading app. Unfortunately,
this effort never materialised into any-
thing concrete. This would have given
us some concrete numbers surrounding
actual use cases.
Most people are mistrustful of the per-
formance that commercial CEP systems
can deliver. One algo trader I spoke to re-
cently told me that he would never use
a commercial CEP package because he
wanted to be able to tune every single
aspect in order to get the most speed
from it, and he did not want the guts of
his trading system to be a black box. As
an example, CEP systems that are written
in C# and Java have to aware of pauses
for garbage collection, especially when
dealing with a high-volume, low-latency
data. There is a lot of work done in the
guts of trading systems today in order to
minimise garbage collection pauses.
8 June 2009 • Issue 05
There can certainly been speed im-
provements if you marry a CEP system
with multi-core hardware. This ap-
proach is being taken with IBM in mar-
rying its System S with its Blue Gene
supercomputer. Speed improvements
can also happen at the messaging layer
using high-performance message infra-
structures like RTI, Solace and 29West.
CEP vendors need to write adapters for
these messaging layers. Integration with
caching products and grid computing
infrastructures is also important.
Bates: The term “generic” has some
pejorative connotations, suggesting a
vanilla capability that cannot address
extreme performance requirements
that might be available with a special-
ised offering or a custom developed
one. There are perhaps some CEP prod-
ucts that cannot perform at the required
throughput and latency that is required,
as they are not architected for it.
But clearly there are others that can
meet those requirements. We’ve proven
it through real experience. But the per-
formance requires some new physics
of software - not just any generic event
processing platform. Trading is not a
monolithic application; there are lots of
permutations and variations to trading
and a range of different requirements.
The validation of whether a product can
meet those requirements is having cus-
tomers that use the product. For Apama,
we would point to our customers and
their success with Apama. They are so-
phisticated trading organisations that
have chosen Apama and remain users.
Chait: CEP packages have been with
us for several years now, and I think that
industry experience has plainly shown
that today’s CEP engines are quite fast
enough for nearly all high-performance
applications. However, it still remains
challenging to effectively develop high-
performance applications on them;
these tools require specialised training
and understanding of their architec-
ture and design, or else risk having their
performance dragged down. For exam-
ple, an improperly implemented input
adapter can cause a bottleneck which
impacts the entire systems.
Wootton: It depends on the package,
of course, but a CEP implementation that
has been designed with a goal of provid-
ing high performance – in terms of both
high throughput and very low latency
– can deliver performance that comes
very close to a custom coded application.
I won’t deny that a programming team
with the right skills and enough time to
build highly optimised code will be able
to achieve higher performance with hard-
coded logic, but the marginal improve-
ment may not offset the addition time to
build and deploy and the dependence on
a highly skilled programming team.
What are the pros and cons
of a graphical event modeling
approach versus a lower level
Java or SQL-like language?
Adler: From what I have seen, devel-
oping with a pure GUI-based approach
is not satisfactory for anything but the
simplest of use cases. Sophisticated
use cases need the maximum amount
of flexibility and control. You will even-
tually need to dive into the underlying
programming language, and you may
need to develop your own custom user-
defined functions.
It is fairly easy to get up to speed with
the languages offered by the various
CEP products. Most of the vendors have
chosen a streaming SQL paradigm, so it is
relatively simple to find a database devel-
oper and bring them up to speed on a CEP
system. Most developers feel more com-
fortable in reading code that looking at a
bunch of connected icons on a palette.
That being said, there is a definite
need for an end-user GUI tool where
traders can formulate their own ad-hoc
queries and run them in the CEP engine
without having to learn the intricacies of
the underlying streaming SQL (or proce-
dural) language. But, then again, there is
a question of whether traders should be
allowed to interact with the CEP system,
as the wrong kind of query can bring
the CEP system to a crawl.
An A-TEAMGROUP Publication

Bates: I am not sure the distinctions are
between graphical modeling and Java/
SQL, or that the decisions are as binary
as the question implies. Apama supports
Java class libraries and our customers can
build CEP based applications using Java.
Some choose to do so. Apama also offers
an event processing language (EPL) that
is designed specifically for the require-
ments of CEP applications and recognis-
es the value of natively supporting the
event-oriented operations. Many of our
customers choose that option, since Java
has certain limitations that we address
within our EPL.
And Apama offers a graphical mode-
ling tool, as well. One of the advantages
of graphical tools is the ability to acceler-
ate the speed with which organisations
can develop and deploy CEP applications
– and evolve them over time. Graphical
tools can also provide a possible “lingua
franca” between business users and de-
velopers where it is much easier to mu-
tually understand the CEP logic and ex-
ecution flow of the applications.
The ideal situation is the ability to sup-
port multiple development constituen-
cies, with options for both. Even within
an organisation there are different types
of users, so approaching CEP with a
“one size fits all” development model is
likely to fail to address those different
constituencies.
Chait: The tendency of CEP vendors
to use a specialised development envi-
ronments and languages is a mistake.
For instance, Microsoft seems eager to
show via Orinocio, Live Labs Reactive
Framework, Reactive F#, and Axum how
multithreaded, stream based, and CEP
paradigms can be elegantly based on
existing language tools. As another ex-
ample, by adopting the industry stand-
ard Eclipse toolset, StreamBase greatly
reduced the learning curve and support
costs associated with their product.
Wootton: It’s really down to a combina-
tion of preference and skills. That’s why we
offer both – to appeal to a wider range of
users. A big advantage of visual authoring
is that it’s more approachable for non-pro-
An A-TEAMGROUP Publication
Complex Event Processing - Roundtable
grammers. With that said, many program-
mers also see value in a visual tool, in being
able to easily visualise the overall struc-
ture and flow of an application. Typically a
programmer will prefer language-based
authoring, however, primarily because
it’s more efficient (doesn’t require all the
mouse clicks) and they can more easily see
the logic being applied to the data. Bottom
line: it’s nice to be able to offer users both.
As CEP becomes more widely
used, is it likely to become a
ubiquitous middleware com-
ponent, like a messaging bus
or a database?
Adler: Definitely. You are going to see
this when Microsoft releases Orinoco.
The CEP engine will just be another com-
ponent in a comprehensive data strate-
gy. Developers will treat the CEP engine
and streaming LINQ queries in the same
way that they treat a database and the
SQL queries that they use to manipulate
the data.
Another example that I have seen is
with the wide use of Esper in custom-
built systems. Whenever developers
need to implement some kind of custom
filtering or window-based processing,
they are just as likely to use Esper as write
custom code. Esper is free, and it can
easily fit into a Java-based codebase.
This is why CEP vendors need to sup-
port a large number of adapters to other
pieces of commonly-used middleware.
A developer wants to take a look at the
entire middleware stack, and know that
the CEP engine can snap in at any point
in the stack. For example, many financial
institutions are using Kx Systems’ KDB+
to store tick and trade data. It should be
a breeze to hook up your CEP engine to
a subscription of data that is stored in
KDB.
Bates: We see CEP as achieving a pres-
ence in both, though would expect it to
become more prevalent within applica-
tion stacks. The real power of CEP is in the
ability to build event-driven applications
and that suggests that CEP will become
an element within both applications and
June 2009 • Issue 05
solutions - like algorithmic trading and
surveillance.
So it will be in infrastructure and ap-
plications platforms. That has been our
experience and why we have focused on
Solution Accelerators to assist our cus-
tomers in this area.
There are clearly applications for
CEP in conjunction with middleware.
Some example include monitoring
messaging events that are transmit-
ted over messaging infrastructures like
an ESB or monitoring processes con-
trolled by business process manage-
ment tools. Both might be considered
“middleware-centric” and we have cus-
tomers using Apama for those kinds of
applications.
Chait: Yes, in a sense I believe these
will become ubiquitous, in that they will
be widely adopted and various firms will
choose one preferred vendor platform
for all internal applications. However that
does not necessarily mean that they will
be provided as enterprise-wide services.
If you look at the state of relational data-
bases inside most banks today, that pro-
vides a clue; often individual groups or
teams are on their own to procure the rel-
evant resources such as servers, database
software, disk space, and the like, and I
don’t see why CEP will be any different.
Wootton: We certainly believe so.
Today people don’t really consider
writing their own message bus or da-
tabase when building an application
– why would you? It’s always cheaper
and faster when you can get what you
need off-the-shelf. As the usage of this
technology expands, the products will
continue to mature and familiarity will
become more common. We’ll get to the
point where it’s just one of the standard
building blocks used to address a certain
class of problems.
Where are standards playing
a role in CEP, or where are
they needed?
Adler: A common definition of an event
would be great for interoperability. “Spe-
9
Complex Event Processing - Roundtable
cialist systems” that calculate their own
alerts could feed these alerts into the
CEP engine for further correlation.
I think that all CEP vendors will even-
tually need to write “query providers”
to support the LINQ feature in C# (and
whatever the competitor to LINQ will be
in Java). This way, developers can embed
streaming queries within their C# or
Java applications, and the LINQ queries
will look the same, no matter which CEP
engine they are being directed to.
Bates: While it is a bit early in the game
for standards for CEP or Event Processing,
recognising that relevant standards may
emerge, it is important to both participate
in relevant working groups and support
standards in adjacent areas. For example,
Apama is a founding member of the Event
Processing Technical Society (EPTS) and is
actively involved with the Event Process-
ing community. Therefore there are a
number of key areas to consider.
For example, part of an understand-
ing of standards is to recognise that CEP
is usually deployed in support of appli-
cations in markets where standards are
already in place. To the extent that CEP
builds solutions in those markets, it is
the standards of those markets that are
important.
For example, in capital markets, a CEP-
based application has to support FIX as
a commonly used protocol for sending
order flow to trading counterparties and
destinations.. In fact, Apama provides
connectivity through FIX and other
adapters to a broad range of equities,
foreign exchange, futures and options,
and fixed income markets, market data
providers, and EMS/OMS/FIX gateways
Have Smarts to Show?
Promote your thought leadership via A-Team Group’s
supplement series.
Contact
Martyn Hodges,
martyn.hodges@a-teamgroup.com
10
via the existing protocols relevant to
those organisations, be those protocols
proprietary or standards-based.
This observation also raises the more
general issue of integration. From the
customer perspective it is important for
a CEP-based application to fit into an ex-
isting environment, rather than require
a re-design of a firm’s entire architecture
to enable a single application with CEP.
Therefore in addition to market-specif-
ic adapters as previously described, it
is important to adhere to a number of
infrastructure and development stan-
dards as well. In the case of Apama, for
interoperability this includes XML files,
JMS, ODBC and JDBC connectivity. For
development, Apama supports Java, C,
C++, and .NET for development. Though
supporting development standards as-
sists in leveraging existing code-base,
Apama’s native Event Processing Lan-
guage is better suited to the task of cre-
ating event-driven applications. CEP
language standards may eventually
emerge and become valuable. However,
to create a single CEP language standard
at this stage could restrict innovation,
which is essential to the shift towards
the event-driven paradigm.
Once a firm has the confidence that it
can integrate a CEP-enabled application
into its existing environment or relevant
ecosystem, the other remaining area of
concern from the customer perspective
is be able to quantify the expected and
realised benefits of using CEP.
Here, standards may be useful to the
extent of being able to benchmark per-
formance parameters of system applied
to a broad array of use cases as a result of
its flexibility. Apama is a member of the
June 2009 • Issue 05
Stac-A1 working group, participating in
the creation of specifications intended to
test event-processing platforms whose
workloads consist of taking inbound
events from one or more sources, apply-
ing specific algorithms to those events,
and generating outbound events.
Chait: An endpoint standard could be
useful. A name, URI, schema, transport,
required data rate for consumers, that
sort of thing. Also, a standard for ac-
cessing CEP data (similar to ODBC for
relational databases) would facilitate
adoption and application development
across a wider range of scenarios. How-
ever, at present the technical hurdles are
at least as significant as the market hur-
dles, so I don’t see this happening any-
time soon.
Wootton: To date, standards haven’t
played much of a role – as you’d expect
for a relatively new class of technology.
Over time, clearly standards will devel-
op. Where I think they are needed most
is at the level of event interchange. How
is information about an event handed
off from one system to another?
Even though a standard message bus
is typically used to get an event-mes-
sage from emitters to consumers, there
are no standards around the structure of
a event message.
This means that a lot of time gets
spent building adapters. Adapters don’t
add value – they just translate. So I see
real potential benefit in moving toward
standards on event structure, facilitat-
ing event interchange, and reducing
the effort we all put into adapters and
integration.
An A-TEAMGROUP Publication
 www.Electronic-Trading.com

A-TeamGroup
ElectronicTrading
Read the Latest Headlines & Get a FREE Issue Now!
Electronic-Trading.com

Electronic Trading is the only publication to focus exclusively on the electronic

trading marketplace. It offers insight into how technology is affecting the business
of liquidity: how sell-side firms are leveraging new, reliable and inexpensive
technologies to attract order flow, and how buy-side firms are turning new
capabilities into operationally superior business processes.

Electronic Trading will help you understand major firms’ activities in the areas of:

• Algorithmic trading models

• Matching systems for the buy- and sell-side and interdealer segments
• Trade order management systems
• Liquidity aggregators
• Exchange gateways, ECNs, ATSs
• Transaction messaging and protocols

View the latest Electronic Trading headlines, read the first paragraphs
and get a FREE issue at Electronic-Trading.com

To find out more, contact

info@a-teamgroup.com.
30-Day Free Trial Or call +44 (0)20 8090 2055

Risk-RegulationIT.com/free-trial
Or visit our website:
www.Electronic-Trading.com
Access all articles and
resources for free, no
credit card required
An A-TeamGroup publication
www.a-teamgroup.com

SPONSORS
 www.Risk-RegulationIT.com

A-TeamGroup
Risk & Regulation IT
MiFID Monitor Expands to
Become Risk & Regulation IT!
MiFID Monitor is now Risk & Regulation IT, with a broader focus on:

• Regulations and their impact on financial markets infrastructure, data, technology and systems

• Emerging regulations as they are discussed and brought into affect such as possible regulation
of hedge funds or ratings agencies

• Input from the global regulators themselves

• Risk management - data and technology-based approaches to managing market, credit,

operational, and liquidity risk, and the likely evolution to an enterprise risk management
framework

• Governance, risk and compliance systems and suppliers

• Impact of derivatives and alternatives on regulatory and risk systems

• Industry initiatives and best practices, and more.

To receive your 30-day free trial

or to sign up for a subscription
(£495 / $745), contact:
30-Day Free Trial ron@a-teamgroup.com.
Tel: +44 (0)1522 850 913
Risk-RegulationIT.com/free-trial
Or visit our website:
Access all articles and
www.Risk-RegulationIT.com
resources for free, no
credit card required
An A-TeamGroup publication
www.a-teamgroup.com

SPONSORS