Day-to-day

- Make sure data is structured the right way (hai.nguyen1)

- Write to which dataset, any potential hit on cost
- Add people to groups
- Add permission to read BQ
- gcloud projects add-iam-policy-binding momovn-itc-shared --member
user:dung.chau@mservice.com.vn --role
organizations/368853197897/roles/BigQueryUserQueryOnly
- Maintenance Jenkins + Gerrit
- Clean up Prometheus PVC
- Menshen permission
- Create a new project, add log routing
- Similar to
https://console.cloud.google.com/logs/router?project=momovn-cornerstone-on-d
emand
- (BigQuery dataset)
- GCP reporting monthly
- Increase quotas for BQ if needed
- Permission for GKE
- gcloud projects add-iam-policy-binding momovn-prod --member
user:toan.mai1@mservice.com.vn --role
projects/momovn-prod/roles/gke_service_owner
- Bigtable
- Commitment review
- Quotas review
- Configurator permissions
- Pub/sub permissions
- VM permissions
-

Risks
- Miniapp sharing common data. A miniapp can get the user token and do something
fishy.
- Cloud team spirit

Notes
- Don’t bend to the request until you’re sure it’s absolutely necessary.

Data
Data on BigQuery (BQ) are organized into 2 main projects
- mmv2 (ID: project-5400504384186300846). This is the project that keeps data in its
original state. Access to data is limited since it has private information of users.
 - momovn-prod. This is the project that holds the copy of all data on mmv2, but with
sensitive information masked.
Each project is organized into datasets. Each dataset has multiple tables. Access can be
granted on the dataset level or table level.

There are 2 basic types of permissions for a dataset/table:

- READER: read only, cannot write.
- WRITER: can read and write. This permission is only granted for the following cases:
- Service accounts to generate data from a service, or copy it from a source.
- Ad-hoc analytics. There are designated datasets for this kind of activity. Ideally,
each BU/team has their own datasets for this kind of thing. For example,
BU_ECOM dataset is specifically for data analysts of BU e-commerce.

To grant permission for a person, a group of people or a service account, we can use the BQ
console or use a Python script in the codebase.
- Using BQ console: https://cloud.google.com/bigquery/docs/dataset-access-controls
- Using script:

python tools/scripts/add_bq_permission.py addpermission [--user=<email_id>]

[--group=<group_email>] --role=<role> --project=<project_id> --datasets=<list_of_datasets>
Where
- When giving permission to a user or a service account, use the param
--user=<email_id>
- When giving permission to a group of users, e.g. dabu-analytics@mservice.com.vn, use
the param --group=<group_email>
- Param `role` could be WRITER or READER
- Param `project` could be `momovn-prod` or `project-5400504384186300846`
- Param `datasets` is a list of datasets, separated by comma, e.g. APP_VARZ,SPIRAL
E.g.
● python tools/scripts/add_bq_permission.py addpermission
--group=dabu-analytics@mservice.com.vn --role=READER --project=momovn-prod
--datasets=APP_VARZ,SPIRAL
● python tools/scripts/add_bq_permission.py addpermission
--user=data-science-bq@momovn-prod.iam.gserviceaccount.com --role=WRITER
--project=project-5400504384186300846 --datasets=ENKI

Groups
Group is a convenient way to manage permissions. When a group gets certain permissions,
every person in the group will have those permissions as well. So when a new person joins, we
just need to assign her to the right group.
In general, each Business Unit (BU) has their own groups of data analysts. Some experienced
groups like Growth and MBI control their own groups, i.e., they can add people on their own. But
most other groups need a process to maintain control. The process is:
 - Anyone in need of using BQ for analytics must pass an exam organized by the MBI
team.
- When the MBI team confirms that the person passes the exam, she can be allowed to
use BQ.
- We will then add the person to the corresponding analytics group.

Groups are managed through Google Admin console: https://admin.google.com/ac/groups

Here are all the current analytics group

Group Department

bq-accountants@mservice.com.vn Phòng Kế toán làm nhiệm vụ đối soát

bu-data-data-analysts@mservice.com.vn BU Data (DABU)

bu-fi@mservice.com.vn BU Financial Service

bu-online-payment-analysts@mservice.com.vn BU Online

bu-opc-data-analysts@mservice.com.vn BU OPC (offline payment center)

bu-social-analysts@mservice.com.vn BU Social Payment

bu-utilities-analysts@mservice.com.vn BU Utilities

bubank@mservice.com.vn BU Bank

cloud-eng-itc@mservice.com.vn For engineers of ITC to view logs,

monitoring services & access to docker
registry

dabu-analytics@mservice.com.vn Data analysts of BU Data

mbi-data-analysts@mservice.com.vn Data analysts of MBI (MBI self- manage)

mbi-ci@mservice.com.vn Data analysts of MBI - Customer Insight

subteam (narrower permission than
mbi-data-analytics@)

data-analysts-restricted@mservice.com.vn Data analyst interns or new comers

data-engineers@mservice.com.vn Data engineers of Big Data team (only for

those passing probation)

data-engineers-intern@mservice.com.vn Data engineers of Big Data team (for

trainees, juniors, new comers)

data-ksnb@mservice.com.vn Kiểm soát nội bộ (a.k.a Internal Control).

 data-ksnb-limited@mservice.com.vn Kiểm soát nội bộ (a.k.a Internal Control).
For trainees or new comers

data-scientist-trainees@mservice.com.vn Data scientists of Big Data team (for

trainees and juniors)

data-scientists@mservice.com.vn Data scientists of Big Data team (only for

those passing probation)

growth-analysts@mservice.com.vn Data analysts of Growth team (Growth

self-manage)

itc-fi-eng@mservice.com.vn Engineers of team Finance-Insurance of

ITC (vu.nguyen)

internal-it@mservice.com.vn Engineers of team Internal IT (huy.lam)

itc-promotion-eng@mservice.com.vn Engineers of team Promotion (worked

with Kindle)

menshen-privileges@mservice.com.vn Special group for people having

permissions to read phone numbers

product-analysts@mservice.com.vn Analysts of Product team

risks-management@mservice.com.vn Product - Risk team

productopts-analysts@mservice.com.vn Product - Ops team

Menshen Group
Menshen group is a very special group. Only members of this group are allowed to read
BigQuery at the phone number level. This group is restricted, it’s only open for
● Some service accounts, this is granted case-by-case
● Some data analysts of MBI and Growth. Those people must get the approvals from the
highest managers of those groups. In the future, these permissions should be revoked.
● Some data scientists who work on building models. In the future, these permissions
should be revoked.
● Some data engineers who manage the Data Platform.

Cloud services
gcp-devops@mservice.com.vn
● Service account management
● Pub/sub admin
Gerrit
● Admins
○ huy.bui1@
○ hai.nguyen1@
● Gerrit is hosted on an EC2 instance in AWS.
● To ssh into the host
○ Download a pem file with your AWS account
○ Ask Gerrit admin to add your public key to the instance
○ If you have a separate pem file, use it; otherwise just ssh without it.
■ ssh -i <pem_file> ubuntu@gerrit.mservice.io
● There is 2 cron jobs running under sudo to:
○ Restart the server every day 4AM
○ Check every 5 mins to see if there are hanging connections, restart the server if
there is
● To see the cron jobs, run
○ sudo crontab -l
● To restart the server manually
○ sudo /ebs1/gerrit-3/code_review/bin/gerrit.sh restart

Jenkins
● To upgrade Jenkins with a new image
○ Edit gcloud/ci/jenkins/Dockerfile
○ Build and push new image
■ docker build -t asia.gcr.io/momovn-dev/jenkins-slave -f
gcloud/ci/jenkins/Dockerfile . && docker push
asia.gcr.io/momovn-dev/jenkins-slave
○ Note down the new SHA256
○ Update the SHA256 at gcloud/ci/jenkins/jenkins-slaves.yaml
■ Line `image: "asia.gcr.io/momovn-dev/jenkins-slave@sha256:xxx`
○ Switch K8s context to `ci-cluster` and apply the config
■ kubectl apply -f gcloud/ci/jenkins/jenkins-slaves.yaml
● There is Jenkins prod at jens.mservice.io to be considered as well
○ Edit gcloud/jenkins/slaves/Dockerfile
○ Build and push new image
■ docker build -t asia.gcr.io/momovn-prod/jenkins-slave -f
gcloud/jenkins/slaves/Dockerfile . && docker push
asia.gcr.io/momovn-prod/jenkins-slave
○ Update the new SHA256 at: https://jens.mservice.io/job/java-merge/configure
● How Jenkins & Gerrit work together
 ○ On Gerrit
■ There is a user with ID `Jenkins` on Gerrit. It belongs to this group
`Non-Interactive Users`
■ This Jenkins user can pull changes and post +1/-1 on verification
■ This Jenkins user has a password to connect to Gerrit
■ This Jenkins also has an assigned public key
○ On Jenkins
■ There is a plugin to connect to gerrit configured at
https://jenkins.mservice.io/gerrit-trigger/
■ For Jenkins to be triggered and pull code from Gerrit, it needs:
● Private key that matches with the public key
● Gerrit’s authorized key. This is contained in the file
/root/.ssh/known_hosts
■ Those are all mapped under a K8s secret named `ssh-key`. Check
gcloud/ci/jenkins/jenkins-slaves.yaml for the detailed mapping
■ This means: if Gerrit moves to a new host, we need to update the new
host’s signature and upload a secret key. This is an example of updating
the new secret
● kubectl create secret generic ssh-key
--from-file=<path/to/known_hosts>
--from-file=id_rsa.pub=<path/to/id_rsa.pub>
--from-file=id_rsa=<path/to/id_rsa> --dry-run -o yaml | k apply -f -

To clean up later
● Shut down the old Gerrit instance on AWS (ID: i-0e51911eeec345791, name: gerrit-2,
host: god.mservice.io) in about 6 months
●

To-do periodically
● Linkerd: renew certificates
○ https://linkerd.io/2.10/tasks/automatically-rotating-control-plane-tls-credentials/
● Review VMs commitments
● Review data access
○ Check BQ logs to see if there is any signs of downloading/processing data out of
the scope of someone’s work.
● Review quotas for all projects, increase quotas if necessary
○ https://console.cloud.google.com/iam-admin/quotas?project=momovn-prod
○ https://console.cloud.google.com/iam-admin/quotas?project=project-5400504384
186300846
To create a new project
● Add a new project to one of the following organization folders
○ Back Office: for Back Office to track their BQ usage
○ BUs: for BUs to track their BQ usage
○ ITC: for ITC to track the infras cost in general
○ Product: for Product to track their BQ usage
● Add Log Router to the project to log all queries running under that project
○ Go to https://console.cloud.google.com/logs/router
○ Select the new project
○ Create sink
○ Add a sink name
○ “Select sink service” -> BigQuery dataset
○ “Sink destination” ->
bigquery.googleapis.com/projects/momovn-prod/datasets/bq_audit_logs
○ “Choose logs to include in sink” -> resource.type="bigquery_resource"

Cert-manager
● Configs are kept in gcloud/cert-manager/
● Instructions are at gcloud/cert-manager/README.md

Miscellaneous scripts

Clean-up Prometheus PVC

● Check the alert to see which PVC is full, for example
`prometheus-prometheus-varz-db-prometheus-prometheus-varz-1`
● Delete the PVC
○ kubectl delete pvc -n monitoring
prometheus-prometheus-varz-db-prometheus-prometheus-varz-1
● Delete the corresponding pod
○ kubectl delete po -n monitoring prometheus-prometheus-varz-1
● Wait a couple of minutes, if the pod hangs, delete the pod again
○ kubectl delete po -n monitoring prometheus-prometheus-varz-1

Delete a node on K8s

● kubectl drain --force --ignore-daemonsets --delete-local-data --grace-period=150
<node_id> && kubectl delete node <node_id>
BigTable: Create new tables
● python3 ./tools/scripts/bigtable_create_tables.py createtable
--schema_file=<path/to/schema.yaml> --project=momovn-prod

BigTable: Add read permissions

● python3 tools/scripts/add_bigtable_permission.py bindpermission
--project=momovn-prod --user=<user_email> --instance=momovn-bigtable
--tables=<table_id> --role=roles/bigtable.viewer

Add a person to pull/push images from GCR

● On dev
○ gsutil iam ch user:<user_email>:roles/storage.objectAdmin
gs://asia.artifacts.momovn-dev.appspot.com/
● On prod
○ gsutil iam ch user:<user_email>:roles/storage.objectAdmin
gs://asia.artifacts.momovn-prod.appspot.com/

Add a person to deploy K8s

● On dev
○ gcloud projects add-iam-policy-binding momovn-dev --member
user:<user_email> --role projects/momovn-dev/roles/GkeDev
● On prod
○ gcloud projects add-iam-policy-binding momovn-prod --member
user:<user_email> --role projects/momovn-prod/roles/gke_service_owner

Ownerships
There are multiple groups
● Cloud Admins (cloud-admins@mservice.com.vn)
○ hai.nguyen1@
○ huy.bui1@
● Cloud DevOps (gcp-devops@mservice.com.vn)
○ an.vo@
○ phuc.tran2@
● Data Admins
○ long.pham2@
○ hai.nguyen1@
● GCP Admins (gcp-organization-admins@mservice.com.vn)
○ khoa.ngo@
 ○ hung.thai@

To request permissions for the following services, reach out the corresponding groups
● To pull images from GCR
○ Cloud Admins
○ Cloud DevOps
● Pub/sub
○ Cloud Admins
○ Cloud DevOps
● Add a new namespace on K8s
○ Cloud Admins
○ Cloud DevOps
● Add a new person to Gerrit
○ Cloud Admins
● Firestore/Configurator
○ Cloud Admins
● Increase quotas
○ GCP Admins
● BigTable
○ Cloud Admins
● VMs
○ Cloud DevOps
● Cloud Storage
○ Cloud Admins
● Create a new GCP project
○ GCP Admins
● Google Analytics
○ Data Admins