Professional Documents
Culture Documents
Build Automation Report From IT Central Station 2015-07-04E5
Build Automation Report From IT Central Station 2015-07-04E5
Build Automation Report From IT Central Station 2015-07-04E5
2015
Disclaimer
IT Central Station Ltd. does not endorse or recommend any products or services. The views and opinions of reviewers
quoted in this document, IT Central Station websites, and IT Central Station materials (i.e. Content) do not reflect the
opinions of IT Central Station. We make no guarantees about the accuracy, currency, suitability, or quality of the content, and
we assume no responsibility for unintended, objectionable, inaccurate, misleading, or unlawful content made available by
Content authors or other third parties.
www.ITCentralStation.com
reports@ITCentralStation.com
+1 (646) 328-1944
This report is comprised of a comprehensive list of enterprise level Build Automation vendors. We have
also included several real user reviews posted on ITCentralStation.com. The reviewers of these
products have been validated as real users based on their LinkedIn profiles to ensure that they provide
reliable opinions and not those of product vendors.
IMPORTANT NOTICE
Did you find this whitepaper helpful? At IT Central Station, our philosophy is “Give to Get”. Our active
community and unbiased reviews are made possible by your participation and as such, we ask that you
share your expertise with us as well. Please email reviews@itcentralstation.com and one of our
community managers will be in touch with you shortly. You can choose to review anonymously or not
and your company name will not be included in the review.
If you found this report and/or the reviews on IT Central Station useful, we would greatly appreciate your
participation in giving back to our community.
Rating: 8.8
Jenkins Jenkins (14 reviews)
Rating: 8.38
JetBrains TeamCity (9 reviews)
Rating: 5.76
Atlassian Bamboo (7 reviews)
Rating: 8.0
CloudBees CloudBees (7 reviews)
Rating: 7.34
GNU GNU Make (3 reviews)
Rating: 7.5
Thoughtworks Thoughtworks Go (2 reviews)
Rating: 8.0
ZeroTurnaround JRebel (1 review)
Circle CI Circle CI
Codeship Codeship
XebiaLabs XL Release
Over 87,463 professionals have used IT Central Station research on enterprise tech. Here are the top
Build Automation vendors based on product reviews, ratings, and comparisons. All reviews and ratings
are from real users, validated by our triple authentication process.
TOP 5 TOP 5
Based on Followers Based on Rating
Overview: Jenkins is an award-winning application that monitors executions of repeated jobs, such as building a
software project or jobs run by cron.
Sample Airial, Clarus Financial Technology, cubetutor, Metawidget, mysocio, namma, silverpeas, Sokkva, So
Customers: Rave, tagzbox
Top Rational Build Forge vs. Jenkins Compared 34% of the time.
Comparisons:*
Bamboo vs. Jenkins Compared 18% of the time.
TeamCity vs. Jenkins Compared 9% of the time.
* Data is based on the aggregate profiles of IT Central Station Users researching this solution.
Stability Issues:
The default settings do not work properly on Mac OS X. You have to tweak JVM
parameters and allocate more heap memory as well as change other parameters to
have a stable Jenkins server.
Scalability Issues:
I haven't had to scale up yet. We have one build box which is running two agents on
it.
Customer Service:
As this is open source, there is no such thing as customer service, but there is a big
community to look for information and get answers.
As this is open source, there is no such thing as tech support, but there is a big
community to look for information and get answers.
Previous Solutions:
Initial Setup:
It was complex and additional knowledge about launch agents and daemons is
required. There are at least three, if not four, different ways to install and configure
Jenkins, which is not always good.
Implementation Team:
We implemented it in-house.
ROI:
Pricing:
The original setup. for us, was the cost of a new Mac Mini box which costs, from
$1,000 to $2,000 depending on the configuration.
Other Advice:
This is a very good, powerful and flexible product, but requires a lot of time
investment to get the maximum out of it.
Valuable Features:
The feature of this product that is most valuable to me is the robust community
support plugins.
Improvements to My Organization:
Jenkins has greatly improved the time it takes to deliver a software package to the
market.
Use of Solution:
Deployment Issues:
No issues encountered.
Stability Issues:
No issues encountered.
Scalability Issues:
No issues encountered.
Technical Support:
The community of people who use this open source tool are very helpful.
Previous Solutions:
Initial Setup:
Jenkins is very straightforward to install. Basically, it's a Java app that runs on an
app server.
ROI:
Pricing:
Alternate Solutions:
Other Advice:
Invest in time reading the support forums and newsgroups. Collaborate with other
professionals.
Overview: TeamCity is a Continuous Integration and Deployment server that provides out-of-the-box continuous unit
testing, code quality analysis, and early reporting on build problems. A simple installation process lets you
deploy TeamCity and start improving your release management practices in a matter of minutes. TeamCity
supports Java, .NET and Ruby development and integrates perfectly with major IDEs, version control
systems, and issue tracking systems.
* Data is based on the aggregate profiles of IT Central Station Users researching this solution.
It would also be nice to have a conditional build step: "I use it to increase
visibility and clarity of build
https://youtrack.jetbrains.com/issue/TW-17939 and deployment activities.
"
Having project templates
Use of Solution:
Deployment Issues:
Stability Issues:
As versions progress, TeamCity has greatly improved it's stability with it's
resourcing. Also following JetBrains' documentation about how to set up TeamCity,
distributes load in a more even fashion.
There have been times when making copies of a project has caused major stability
issues, especially when you have nested project structure based off of different
build configuration templates.
Customer Service:
Technical Support:
Previous Solutions:
Initial Setup:
Implementation Team:
We implemented it in-house.
Valuable Features:
Easy navigation
Useful to see how the build is progressing and how many tests are left to pass
before the build is green
Improvements to My Organization:
Improved the quality of builds by running unit tests as part of every build
Reduced the risks of regression defects by running automated tests as part of the
build process
Use of Solution:
Deployment Issues:
Only environment issues which caused failed deployments. Also, if people were
connected to the database on the backend, the build would fail.
Stability Issues:
It seemed a very stable tool with hardly any downtime in the six months I was using
it.
Not at all. In fact it was so stable we were able to add and create our own virtual
environments each time we needed to deploy. This meant that each user had their
own virtual environment, meaning nothing ever got overwritten.
Technical Support:
I had no need for it as we had DevOps engineers who fixed any issues.
Previous Solutions:
I used to use Jenkins in a previous job. I started using TeamCity when I changed
jobs as that was the tool that was being used, but I preferred TeamCity to Jenkins.
Initial Setup:
Implementation Team:
ROI:
We used to deploy multiple times a day too many different environments. Also, we
had offices in Australia who were using it when we were asleep! Effectively we
were using this tool 24 hours a day, seven days a week.
Other Advice:
Do it! Very easy to use and very stable. A must have tool for any teams using agile
methodologies.
* Data is based on the aggregate profiles of IT Central Station Users researching this solution.
Ease of use.
-Docker support
Deployment Issues:
Stability Issues:
Scalability Issues:
Yes, It needs to allow multiple plans to run on a agent(at same time). Dedicating
one whole agent isn't fair, I know we have to buy more agents, but in-terms buying
and maintaining more infrastructure isn't scalable either. .
Customer Service:
Technical Support:
Previous Solutions:
Initial Setup:
ROI:
Pricing:
We initially got 5 agents for around $2000 excluding infrastructure setup. Now we
are using docker containers to maximize the mileage on them.
Other Advice:
If you are looking for good integration with Atlassian products and then this is the
tool.
Valuable Features:
We are very fond of the flexibility it offers in terms of setting up our builds. We have
a wide variety of components we need to build which often require custom actions
or manipulations using in house technology. Bamboo allows us to set this up quite
easily by the flexible configuration features it includes.
Secondly we really like the integration of the build aspect and deploy aspect. One
of the recent major releases included this feature to link build and deploy projects
together allowing a deployment pipe-line to be setup completely in Bamboo. This
makes the status of deployments very visible and also allows for easy follow up and
even rollback in case a deployment causes too much havoc.
Improvements to My Organization:
The fact that all builds use a unified setup and infrastructure makes it much easier
to control and adapt the ever increasing number of builds.
We are lacking proper support for branched builds using multiple source
repositories. The current approach is quite clunky (or non-existent) but it seems
there is something in the pipe line for the end of the year according to a recent post
on the Bamboo Jira.
Use of Solution:
Deployment Issues:
Not at all, over all those years we only had a handful of issues and that's probably
because we always take each new update directly.
Stability Issues:
No, we never encountered any serious regression. For the few cases we
encountered bugs they were fixed in a reasonable time interval.
Scaling out a build system is not always easy, but thanks to the remote agent
technology we are able to scale out and add nodes in a transparent way.
Customer Service:
We use a 3d party for our license management, which makes it hard to judge this
but we never had direct interactions with Atlassian Customer support.
Technical Support:
Pretty good once you get trough the initial levels of the support team, it can take a
while before you are able to prove that there's a genuine issue.
Previous Solutions:
We came from Jenkins back in the day, I recall our most important reasons to
switch where the enterprise readiness of Bamboo and its integration in the
Atlassian stack (Jira and others).
Initial Setup:
Bamboo is very easy to deploy, all it takes is extracting the distribution and a JRE
to run it. This also goes for the remote agents which install by running a JAR which
already contains the configuration for communicating with its master.
Implementation Team:
We implemented in house.
ROI:
Pricing:
We started out using only a number of agents and moved to a bigger setup using
many remote agents over the years. The cost depends on the agents used and
amount of boxes deployed to run them on.
Alternate Solutions:
Plan before you start, Bamboo is 'only' that which automates. One should have a
decent design of how the build needs to work internally and have that (scripts,
servers, descriptors ...) in order before attempting to automate on a large scale.
Secondly, don't be afraid to change things to you application or pipeline to help the
automation to be more efficient - for example we replaced massive chunks of hard
SQL from the build scripts by a webservice to avoid dependencies to JDBC in our
builds.
Sample Acxiom, ARTstor, Choose Digital, Dominion Marine Media, Egraphs, Inc., Global Bank, jclouds, Movellas
Customers:
Top Heroku vs. CloudBees Compared 17% of the time.
Comparisons:*
Amazon AWS vs. CloudBees Compared 14% of the time.
Bamboo vs. CloudBees Compared 11% of the time.
* Data is based on the aggregate profiles of IT Central Station Users researching this solution.
Simple interface
Improvements to My Organization:
Yes. We were using Buildbot and started using Jenkins. It helped the team to get
visibility of the tests running and breaking builds. Jenkins has wall display setup so
that everybody can see build status including non-technical people. Using Jenkins
well doesn't make any difference unless it is used smartly.
Use of Solution:
Four years.
Deployment Issues:
Stability Issues:
Sometimes it randomly throws 'Error 500' while saving the settings of a job.
Scalability Issues:
No issues encountered.
Customer Service:
We were using Buildbot and Switched to Jenkins for its nice interface and visibility.
Initial Setup:
Initial setup of the Jenkins was not as easy as if you are setting master/slave
architecture. You need make sure that the master and all slaves are running OK as
well as to utilise slaves effectively.
ROI:
It's free and open-source so it's worth trying and you won't regret using it.
Alternate Solutions:
We looked at
Buildbot
Teamcity
Go
Bamboo
Other Advice:
Sample Lufax.com, Zopa, Manheim, Bresic Whitney, Kaplen, comparethemarket.com, Loof, iproperty.com,
Customers: Rosetta Stone
* Data is based on the aggregate profiles of IT Central Station Users researching this solution.
Use of Solution:
6 months.
Deployment Issues:
No major issues.
Stability Issues:
So far so good.
Scalability Issues:
Customer Service:
Good documentation but not much community support compare to other CI/ CD
tools
Previous Solutions:
Initial Setup:
Implementation Team:
In-house, and all our developers take part in keeping their GO server running.
ROI:
GO is open source, our initial cost is zero but it comes with a bit of learning curve
so if your delivery process isn't that complex, I think other tools might give you
better ROI.
Pricing:
Zero initial cost, and we host them on Azure, 8-9 VMs, at approximately $14 per
day.
Alternate Solutions:
Go was our winner when evaluated, along with Jenkins, TFS build, Atlassian
Bamboo and Team City.
Other Advice:
GO doesn't come with a lot of out of the box features like other tools, don't expect
that you can get it to work by clicking the NEXT button.
Valuable Features:
Improvements to My Organization:
Use of Solution:
6 months
Deployment Issues:
Stability Issues:
No stability issues.
Scalability Issues:
yet.
Customer Service:
Technical Support:
this.
Previous Solutions:
Initial Setup:
to do it.
Implementation Team:
In-house team.
ROI:
The product is free to use. In terms of value added it has more than achieved
everything we wanted from it and more.
Pricing:
Alternate Solutions:
Jenkins.
Other Advice:
Try out alternatives and pick the one which fits you the best.
* Data is based on the aggregate profiles of IT Central Station Users researching this solution.
Improvements to My Organization:
It gets the used time for redeploy in a development environment really fast and of
course that's a good point.
Development environment
Use of Solution:
3 years
This report features just a few of the Build Automation reviews on IT Central Station. If you would like to
read more about what real users are saying about the many Build Automation solutions on the market,
visit the Build Automation page at IT Central Station:
http://www.itcentralstation.com/category/build-automation
There are a number of ways you can participate in the IT Central Station community. Write a review,
read a comment, or just follow a product. Either way, we’ll be sure to let you know when people are
talking about the solutions you care about!
Give to Get
Did you find this whitepaper helpful? At IT Central Station, our philosophy is “Give to Get”. Our active
community and unbiased reviews are made possible by your participation and as such, we ask that you
share your expertise with us as well. Please expect to be contacted shortly by an IT Central Station
Community Manager. We will ask you for 10 minutes of your time to review a product that you use either
via a 10 minute phone interview or questionnaire. You can choose to review anonymously or not and
your company name will not be included in the review.
If you found this report and/or the reviews on IT Central Station useful, we would greatly appreciate your
participation in giving back to our community.