Installation Runbook - Milkbasket

April 14, 2023

PREPARED BY
devops@sourcefuse.com

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
 Installation Runbook - Milkbasket

Table of Contents

Section 1
Ansible- Dynamic Inventory

Section 2
Loki Set-up on EC2 Server
Promtail Agent Set-up on EC2 server

Section 3
Run NodeJS Application With Yarn using Systemctl service

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
Page 2 of 15
 Installation Runbook - Milkbasket

Ansible based on Dynamic Inventory

Pre-requisites:

● A server that has ansible installed in it.

● AWS Access Key & Secret Key (or) a role with full admin access attached to the ansible
server.

Follow the below steps for Ansible installation set-up:

Installing Ansible on Fedora or CentOS

On Fedora:

$ sudo dnf install dnf-plugins-core

$ sudo dnf install ansible-2.9.25
$ ansible --version

On CentOS:
$ sudo yum install epel-release
$ curl -O
https://releases.ansible.com/ansible/rpm/release/epel-7-x86_64/ansible-2.9.25-1.el7.ans.noarch
.rpm
$ sudo yum install python-paramiko python2-cryptography sshpass
$ sudo rpm -ivh ansible-2.9.25-1.el7.ans.noarch.rpm
$ ansible --version

Installing Ansible on Ubuntu

$ sudo apt-get update
$ sudo apt-get install python3-pip
$ sudo pip3 install ansible==2.9.25
$ ansible --version

Installing Ansible on ec2-amazon linux

$ sudo yum update -y
$ sudo amazon-linux-extras install epel
$ sudo yum install -y ansible

Note: For ec2-amazon linux machines, the amazon-linux-extras was removed for 2023 ami. We
are exploring this part, we will give a solution asap.

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
Page 3 of 15
 Installation Runbook - Milkbasket

Follow the below steps for the dynamic inventory setup.

Step 1: Install Ansible-EC2 plugin- Run the below command in order to install the Ansible-EC2
plugin

$ ansible-galaxy collection install amazon.aws

Step 2: Ensure you have python3 & pip3 installed in your Ansible server.

Python installation for centos, Redhat,

$ sudo yum install python3 -y

$ sudo yum –y install python3-pip

For Debian, Ubuntu,

$ sudo apt-get install python3 -y

$ sudo apt-get install python3-pip -

If you have used the Ansible ppa for installation, install pip using the following command.

$ sudo apt-get install python-boto3

Step 3: Create an inventory directory under /opt and cd into the directory.

$ sudo mkdir -p /opt/ansible/inventory

$ cd /opt/ansible/inventory

Step 4: Create a file named aws_ec2.yaml in the inventory directory.

$ sudo vi aws_ec2.yaml

Copy the following configuration to the file. If you are running an ansible server outside the AWS
environment, replace add your AWS access key and secret to the config file.

---
plugin: aws_ec2
aws_access_key: <YOUR-AWS-ACCESS-KEY-HERE>
aws_secret_key: <YOUR-AWS-SECRET-KEY-HERE>
keyed_groups:
- key: tags
prefix: tag

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
Page 4 of 15
 Installation Runbook - Milkbasket

If an ansible server is running inside the AWS environment, attach an ec2 instance role with the
required AWS ec2 permissions (please follow the step 5 else go to step 6).

Step 5: Create an IAM role

● In order to fetch the EC2 instance details, the ansible server must need an IAM role that
can allow interacting with EC2 instances.
● Attach AWS_EC2_FullAccess role to the ansible server

Step 6: Open /etc/ansible/ansible.cfg file.

$ sudo vi /etc/ansible/ansible.cfg

Find the [inventory] section and add the following line to enable the ec2 plugin.

[inventory]
enable_plugins = aws_ec2

Step 7: Run the below command in order to check the output

$ ansible-inventory -i /opt/ansible/inventory/aws_ec2.yaml --list

The above command returns the list of ec2 instances with all its parameters in JSON format.

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
Page 5 of 15
 Installation Runbook - Milkbasket

If you want to use the dynamic inventory as a default Ansible inventory, edit the
/etc/ansible/ansible.cfg file and search for inventory parameters under defaults. Change the
inventory parameter value as shown below.

inventory = /opt/ansible/inventory/aws_ec2.yaml

Now if you run the inventory list command without passing the inventory file, Ansible looks for
the default location and picks up the aws_ec2.yaml inventory file.

Step 8: Grouping of instances- Add the below code in the aws_ec2.yaml file

---
plugin: aws_ec2
aws_access_key: <YOUR-AWS-ACCESS-KEY-HERE>
aws_secret_key: <YOUR-AWS-SECRET-KEY-HERE>
keyed_groups:
- key: tags
prefix: tag
- prefix: instance_type
key: instance_type
- key: placement.region
prefix: aws_region

Run the below command in order to check the output for grouped instances.
$ ansible-inventory -i aws_ec2.yml --graph

Ansible command to run as a playbook:

$ ansible-playbook sample.yml -i aws_ec2.yml --private-key=<your-private-key>

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
Page 6 of 15
 Installation Runbook - Milkbasket

Configure LOKI on EC2 server ::

Step-1. Create loki user - sudo useradd -rs /bin/false loki

Step-2. Move to bin directory - cd /usr/local/bin
Step-3. Download and Install LOKI -

sudo curl -O -L "https://github.com/grafana/loki/releases/download/v2.8.0/loki-linux-amd64.zip"

(https://github.com/grafana/loki/releases/)
sudo unzip "loki-linux-amd64.zip"
sudo chmod a+x "loki-linux-amd64"

Step-4. Create configuration file for LOKI - sudo vi loki-config.yml

auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
instance_addr: <ip-address> (private ip address of the server)
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
Page 7 of 15
 Installation Runbook - Milkbasket

Step-5. Create data directory for loki and provide ownership to loki user - sudo mkdir /tmp/loki
&& sudo chown loki:loki /tmp/loki
Step-6. Create Systemctl Service for Loki - sudo vi /etc/systemd/system/loki.service
Description=Loki service
After=network.target
[Service]
Type=simple
User=loki
ExecStart=/usr/local/bin/loki-linux-amd64 -config.file /usr/local/bin/loki-config.yml
Restart=always
[Install]
WantedBy=multi-user.target

Step-7. Start the Loki service -

sudo systemctl daemon-reload
sudo systemctl start loki.service
sudo systemctl enable loki.service
sudo systemctl status loki.service

Step-8. Access the loki service on http://<ip-address>:3100/metrics

–--
Install LOKI using Ansible -

To install LOKI using ansible, you can refer to the repository -

https://bitbucket.org/milkbasket/terraform/src/master/ansible-playbook/

To run the ansible, please provide the key-pair file and run the following command :
ansible-playbook loki.yml -i inven-loki.txt

Config file and service file for loki is present -

https://bitbucket.org/milkbasket/terraform/src/master/ansible-playbook/loki/templates/

Configure Promtail Agent on EC2 server ::

Step-1. Create promtail user - sudo useradd -rs /bin/false promtail

Step-2. Move to bin directory - cd /usr/local/bin
Step-3. Download and Install Promtail -

sudo curl -O -L "https://github.com/grafana/loki/releases/download/v2.4.1/promtail-linux-amd64.zip"

sudo unzip "promtail-linux-amd64.zip"
sudo chmod a+x "promtail-linux-amd64"

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
Page 8 of 15
 Installation Runbook - Milkbasket

Step-4. Create configuration file for Promtail - sudo vi promtail-config.yml

Remark :: Please provide the ownership of the log file to promtail user to collect the logs and
Attach the IAM role with full EC2 permissions for sd_config (or provide access keys with full EC2
permissions)

server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: 'http://<loki-server-address>/loki/api/v1/push'
scrape_configs:
- job_name: journal
journal:
json: false
max_age: 12h
path: /var/log/journal
labels:
job: systemd-journal
relabel_configs:
- source_labels: ["__journal__systemd_unit"]
target_label: "unit"
- job_name: system
static_configs:
- targets: [<target-ip-address>]
labels:
job: varlog
__path__: /var/log/*.log
- job_name: 'aws'
ec2_sd_configs:
- region: "ap-south-1"
access_key: <access-key>
secret_key: <secret-access-key>
filters:
- name: instance-state-name
values:
- running
relabel_configs:
- source_labels: [__meta_ec2_tag_Name]

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
Page 9 of 15
 Installation Runbook - Milkbasket

target_label: name
action: replace
- source_labels: [__meta_ec2_tag_Account]
action: replace
target_label: Account
- source_labels: [__meta_ec2_instance_type]
target_label: instance_type
- action: replace
replacement: /var/log/secure
target_label: __path__
- source_labels: [__meta_ec2_instance_id]
target_label: instance_id
Step-5. Create Systemctl Service for Promtail - sudo vi /etc/systemd/system/promtail.service
[Unit]
Description=Promtail service
After=network.target
[Service]
Type=simple
User=promtail
ExecStart=/usr/local/bin/promtail-linux-amd64 -config.file
/usr/local/bin/promtail-config.yml
Restart=always
[Install]
WantedBy=multi-user.target

Step-7. Start the Promtail service -

sudo systemctl daemon-reload
sudo systemctl start promtail.service
sudo systemctl enable promtail.service
sudo systemctl status promtail.service

Step-8. Check the target of promtail agent on http://<ip-address>:9080

Grafana Integration With LOKI -

Step-1. Add LOKI data source in grafana.

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
Page 10 of 15
 Installation Runbook - Milkbasket

Step-2. Configure Loki Data source -

Provide the Loki server URL and test the connection.

Step-3. Explore the Labels -

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
Page 11 of 15
 Installation Runbook - Milkbasket

___

Steps to run the ansible playbook to install promtail on the host machines:

Prerequisite:

● A server that has ansible installed in it. (Please refer to the dynamic inventory set up
section).
● Finish the Initial set-up for dynamic inventory . (Please refer to the dynamic inventory set
up section).

–--
Install Promtail using Ansible -

To install Promtail using ansible, you can refer to the repository -

https://bitbucket.org/milkbasket/terraform/src/master/ansible-playbook/

Please update the log file which needs to be fetched in the task folder and according to the log
file, please update promtail-config.yml in the template folder. Also provide aws access keys for
ec2_sd_config in vars files with FullEC2 permissions.

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
Page 12 of 15
 Installation Runbook - Milkbasket

To run the ansible, please provide the key-pair file and run the following command :
ansible-playbook promtail.yml -i inven-prom.txt

Config file and service file for promtail is present -

https://bitbucket.org/milkbasket/terraform/src/master/ansible-playbook/promtail/templates/

Run NodeJS Application With Yarn using Systemctl service ::

To deploy a nodejs application with yarn as a no-shell user (app) on ec2-server (amazon linux)
with systemctl service, We have to follow these steps.

Step-1. Create No Shell User - sudo useradd -rs /bin/false app (app is a no-shell user)
Step-2. Install Nodejs and Yarn on EC2 server based on Linux AMi Version. (run these
command with user which have sudo privilege)

For 2022 AMI

sudo yum remove libuv -y
sudo yum install libuv --disableplugin=priorities
sudo yum -y install nodejs
sudo curl --silent --location https://dl.yarnpkg.com/rpm/yarn.repo | sudo tee
/etc/yum.repos.d/yarn.repo
sudo yum -y install yarn

For 2023 AMI

sudo yum -y install nodejs
sudo curl --silent --location https://dl.yarnpkg.com/rpm/yarn.repo | sudo tee
/etc/yum.repos.d/yarn.repo
sudo yum -y install yarn

Step-3. Make Directory for application - sudo mkdir /opt/myapp (/opt/myapp is app directory)
Step-4. Provide ownership of the application directory to the no shell user - sudo chown app:app
/opt/myapp
Step-5. Run yarn in the app directory - cd /opt/myapp/ && yarn install --production
Step-6. Create systemctl service - sudo vi /etc/systemd/system/myapp.service
[Unit]
Description=My Node.js Web Server

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
Page 13 of 15
 Installation Runbook - Milkbasket

After=network.target
[Service]
ExecStart=/usr/bin/yarn start
WorkingDirectory=/opt/myapp
User=app
Environment=PORT=3005 → (port where application will run)
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=nodejs_server
Restart=always
RestartSec=30s
[Install]
WantedBy=multi-user.target

Step-7. For Logging purpose install rsyslog - sudo yum -y install rsyslog
Step-8. Create Log file and provide ownership to no shell user - sudo touch /var/log/myapp.log
&& sudo chown app:app /var/log/myapp.log
Step-9. Reload the system daemon - sudo systemctl daemon-reload
Step-10. Start the node js service - sudo systemctl start myapp.service
Step-11. Create config for rsyslog file and paste the content - sudo vi /etc/rsyslog.d/myapp.conf

if $programname == 'nodejs_server' then /var/log/myapp.log

& stop

Step-12. Restart Syslog service - sudo systemctl restart rsyslog

Step-13. Check the status - sudo systemctl status myapp.service && sudo systemctl status
rsyslog
Step-14. Enable nodejs service so it will run when every time when machine will be boot - sudo
systemctl enable myapp.service
Step-15. Check logs - sudo tail -f /var/log/myapp.log
Step-16. To redirect nodejs service on Port 80, Install nginx - sudo yum install -y nodejs nginx
Step-17. Create nginx conf file - sudo vi /etc/nginx/conf.d/myapp.conf (replace proxy_pass url
with the url where nodejs app running)
server {
listen 80;
server_name example.com;
location / {
proxy_pass http://localhost:3005;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
Page 14 of 15
 Installation Runbook - Milkbasket

Step-18. Restart Nginx - sudo systemctl restart nginx && sudo systemctl enable nginx
Step-19. To automatically start nodejs systemctl service when code changes -
sudo vi /etc/systemd/system/watcher.service
[Unit]
Description=nodejs restarter
After=network.target
StartLimitIntervalSec=20
StartLimitBurst=5
[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl restart nodejs.service
[Install]
WantedBy=multi-user.target

AND

create path - sudo vi /etc/systemd/system/watcher.path

[Path]
PathModified=/opt/myapp/ → Application directory path
[Install]
WantedBy=multi-user.target

Step-20. Restart systemctl services -

sudo systemctl daemon-reload
sudo systemctl enable watcher.service && sudo systemctl enable watcher.path
sudo systemctl start watcher.service && sudo systemctl start watcher.path

Jenkins::

1. Jenkinsfile for infra creation using terraform -

https://bitbucket.org/milkbasket/terraform/src/master/JenkinsFile
2. Setup Jenkins Google Auth -
https://www.tothenew.com/blog/jenkins-google-authentication/

SourceFuse | 320 1st St N Suite 709 | Jacksonville Beach, FL 32250

1-800-578-FUSE | www.sourcefuse.com
Page 15 of 15