MongoDB X-Team

This doc: ​http://go/mongox

Team Mission
The MongoDB x-team will be the steward of the MongoDB platform for Foursquare. It will be the
ambassador to the rest of engineering on how to be good citizens of the system, and consult on
the best practices of how to use (or not use) the platform.

Rationale
MongoDB is part of the core foundation of Foursquare’s infrastructure. Most (all?) of the writes
from our mobile and web applications hit MongoDB, and most of our services need to be able to
read from MongoDB. As such, it is a highly visible point of failure whenever it experiences
problems. We actually leverage MongoDB in such a way right now that we cannot even
withstand a single poorly performing node.

While it is easy (and often times right) to point fingers at MongoDB, we also have been misusing
it in many ways for a long time. In order to ensure that the platform remains healthy and viable,
we need to have engineers from all over the organization understand how to best incorporate it
into their designs, and help others on all product teams use it appropriately. At a minimum,
teams that use MongoDB should have some level of expertise on the system within the team.

In short, it is no longer feasible for a single person to work on MongoDB part time and expect it
to sustain the rest of our infrastructure.

Why You Should Join

- the quality of our MongoDB backend really affects everyone's ability to do their job, all
the way from feature development to BizDev, so help us make everyone's lives better
and more productive by improving our most vital Foursquare system, which happens to
be one of the largest deployments of MongoDB in the world
- have a direct impact on improving our uptime (number of 9s)
- ensure that our data is accurate and available as we start to become more reliant on it
as a company
- if you’re on the dev side of things, come gain some insight on infra-y things, and see how
they fit into the overall tech stack
Potential Projects
https://app.asana.com/0/94649874561543/list

Some good starter projects highlighted from above asana project:

- get some appropriate monitoring in place -
https://app.asana.com/0/94649874561543/95975189005154
- dig into fixing some expensive queries -
https://app.asana.com/0/94649874561543/96071572852107
- consider sharding some large collections that currently sit on a single shard -
https://app.asana.com/0/94649874561543/95003154147819
- migrate non-user facing collections to logging cluster -
https://app.asana.com/0/94649874561543/95003154147831

Future:
- upgrade to 3.2? (this might be prod team’s responsibility)
- upgrade java driver to 3.X
- after 3.2 upgrade
- use document validation to prevent garbage documents from building up
- rename config servers to per cluster (mongo-configs-checkins-1 instead of
mongo-configs-1)
- make config servers into replica sets

Ongoing Tasks
- review new collection and index creation requests
- review new query patterns
- maintain and create tooling for MongoDB
- help teams monitor and alert on relevant metrics