Pragmatic Software

Development in q/kdb+

Ajay Rathore
My preferred tools for kdb+/q
development
Sublime Text3
Maven
Git
Jenkins
Docker
Sublime Text3

Light on the CPU, faster startup

Symbol indexing across projects, which allows you to
use Goto Definition and Goto Symbol to jump to related
code in other files. Pretty huge
Lots of open source plugins to enhance and customize
Kdb+ syntax highlighting sublime-q

Screenshot of my IDE below

Maven
Mavens primary goal is to allow a developer to comprehend the
complete state of a development effort in the shortest period of
time. In order to attain this goal there are several areas of concern
that Maven attempts to deal with:

Making the build process easy

Providing a uniform build system
Providing quality project information
Providing guidelines for best practices development
Allowing transparent migration to new features
Apart from the above benefits, one of the biggest
advantage of using maven is that it allows certain level of
modularity in big kdb+ projects. Using maven we can break
kdb+ libraries and services into smaller, manageable
artifacts and using nexus repositories we can publish and
share those artifacts across different teams.

Let me give a simple example.

Suppose we want to create a logging library which can be

shared across different kdb services without duplicating the
code.
Maven project layout will look something like below
Assembly.xml (This will create a zip archive maintaining the
dir structure relative to src/main/kdb)
pom.xml
When we run mvn install on above project, maven will create a zip
artifact logging-library-1.0-SNAPSHOT-dist.zip and push that into
nexus repository.
In order for other projects to use this logging library, they will need to
declare a maven dependency in their project pom.

<dependency>
<groupId>com.kdb.ajay</groupId>
<artifactId>logging-library</artifactId>
<version>1.0-SNAPSHOT</version>
<classifier>dist</classifier>
<type>zip</type>
</dependency>

Provided those other projects are also assembled as zip

distribution, maven dependency plugin will pull any dependencies
and package everything together in a single artifact while
maintaining the individual projects directory structure. Now for
using the functions in the logging library, other q files will need to
simply load the library using

\l code/logging.q

For deployment purpose, we can use maven-antrun-plugin to scp

the final zip archive to any target server and unzip it and initialize
the service.
Git
Git is probably the most popular distributed version control
system out there, it can be used as a server out of the box.
Dedicated Git server software helps, amongst other features,
to add access control, display the contents of a Git repository
via the web, and help managing multiple repositories.
For hosting the code, there are couple of options for free
web-based Git repository hosting service like github and
bitbucket. I prefer using bitbucket which allows private
repositories.
Git can be integrated with maven using <scm> tag, this allows
maven to automatically push any tagged artifacts when user
runs mvn release:prepare and release:perform.
Jenkins
Jenkins is an open source continuous integration server.
Basically Continuous Integration is the practice of running
your tests on a non-developer machine automatically every
time someone pushes new code into the git repository. It
gets tightly integrated with maven and git.
There are multiple ways of setting up jenkins git repository
monitoring, whenever new code is pushed to git either
master or feature branch, jenkins can download any
changes and build the branch and can deploy the artifact to
the dev or uat server. All of this can be automated and
speeds up develop-test-release cycles.
Docker
Docker is an open source technology and provides a
convenient way to ship any infrastructure code and
supporting libraries and tools along with the application
code. You can look into references section for more
resources on docker.

We can make use of couple of maven plugins for creating

docker images of our kdb services and push those into
docker registries via jenkins job.
Project structure will look something like

For building docker images and integrating into maven

lifecycle we can create a maven docker profile
To invoke the above docker profile we can simply call
mvn clean install -P docker

This will result in following sequence of events, first of all maven will
build project zip artifact and put it inside project build directory which
by default is project-root/target/
After that maven-resource plugin will run in the prepare-package
phase and will copy all files inside src/main/docker dir to target/docker
and project build artifact from target to target/docker dir, it will also
replace any property place holders via resource filtering. Now
everything we need to build docker image is inside target/docker dir
Next we invoke maven-antrun-plugin in package phase. It will perform
two tasks, first it sets up appropriate file permission on
target/docker/build.sh and then run that script to produce docker
image.
build.sh

The above script simply invokes docker build command inside

target/docker/ dir which has dockerfile with all the instructions for
building the kdb service image.
Dockerfile
Once the docker image is produced, we can simply launch the
container using container port mapping and docker volumes for
loading data from SAN or NAS.

docker run -v
/Users/ajay/deploy/kdb-tick/data/marketdata:/home/ajay/services/kdb-t
ick/data/marketdata --name marketdata -td -p 5000:5000 -p
5001:5001 -p 5002:5002 ajayrathore/kdb-tick-marketdata

The above container will launch a standard kdb tick triplet with
tickerplant on port 5000, realtime on port 5001 and hdb on port 5002.
Historic data is mounted from localhost using docker -v flag.
References

Maven

Git

Jenkins

Docker