CI/CD with OpenStack Cloud

In Viettel Networks
Cong Ha Minh
Chien Pham Tuong
Viettel Networks Engineer
Agenda
1. Why CI/CD and Cloud
2. Build CI/CD infrastructure with OpenStack
3. Automate staging environment provision/destroy with OpenStack
4. Autoscale Production Environment with Heat, Ansible

2
Why CI/CD & Cloud
Problems with previous development process
- Development Problem:
- Code integration nightmare, unexpected breaking old feature
- Repeated manual testing
- Incontrolable code style
- Operation Problem:
- Handy release process
- Fix physical infrastructure

3
Why CI/CD & Cloud
CI/CD and Cloud is the solution for these problems
- Automate dev & ops jobs with CI/CD.
- Unified dev, test and prod platform by Containerize.
- Use Cloud for computing resource management and autoscale.

4
CI/CD Environment

5
 Building CI/CD infrastructure
- Requirements:
- High Availability
- Scale on demand
- Flexible
- Our solutions:
- Keepalived + OpenStack volume
- OpenStack Cloud Plugin
- Build tool provided from Docker
image.

6
OpenStack Cloud Plugin
- Jenkins cloud extension point for OpenStack
- Scaling slaves by check queue length and available
executors stats

7
OpenStack Cloud Plugin
- When available executor stats exponential value below
MARGIN value, Jenkins scale additional slaves via
plugins
- Number of slaves spawned related with queue
length(workload) and number of executors per one slave

8
OpenStack Cloud Plugin

9
Continuous Integration Pipelines
- Push commit job: When feature is developing.

10
Continuous Integration Pipelines
- Merge request jobs: When feature is ready to merge.

11
Staging Environment Problems
In Merge Request Build, we need to create staging environment because:
- Some people don’t know how to write automate tests.
- Some complex and messy things is hard to write tests.
- Something need to manual test before merge (UI/UX)

But we need handle some problems:

- When and how create/destroy a staging environment ?
- Conflict when parallel builds in a merge request occurs.

12
Staging Environment Problems
To solve this problem, we combined Jenkins,
OpenStack and GitLab

- Use OpenStack Heat to ensure staging stack is created

before deploy build artifact.
- When merge request is accept/close, destroy its
staging stack by accept/close merge request pipeline.

13
Staging Environment Problems
To solve this problem, we combined
Jenkins, OpenStack and GitLab

- Use gitlab commit build status and gitlab merge

request commit list to keep track on list Jenkins
builds is running in a merge request

14
Staging Environment Problems
To solve this problem, we combined
Jenkins, OpenStack and GitLab

- When new merge request build start, we cancel

old builds by call to Jenkins API Endpoint, then
update build status

15
Continuous Delivery Pipeline

16
Results
Before:
● Manual test, build code
● Shared staging machine
● Deploy 2-4 times/month
● 2-3 days/release, manual deploy script
● Low success rate
● Manual resource management
After:
● Auto test, build by CI pipelines ~10 min
● Isolated staging environment ~20 min deploy
● Continuous Delivery
● Auto deploy, 40 min/release
● High success rate 95%
● Auto scale, 10-15 min response
17
Challenges
- Scaling time is high when compare with container orchestration
- Ansible Dynamic Inventory
- One time build Slave VM?
- Like Travis, Zuul
- Support Non-Containerize Build
- Clean Build Environment

18
Thank you
Question please