EKS, Thanos and S3

Overview
Thanos is an open source project that is capable of integrating with a Prometheus deployment, enabling a highly available
metrics system with long-term storage.

Thanos Components
Thanos Sidecar: Sidecar runs with every Prometheus instance. The sidecar uploads Prometheus data every two hours to
storage. It also serves real-time metrics that are not uploaded in bucket.

Thanos Store: Store serves metrics from long-term storage (S3)

Thanos Querier: Querier has a user interface similar to that of prometheus and it handles prometheus query API. Querier
queries Store and Sidecar to return the relevant metrics. Ig there are multiple Prometheus instances setup for HA, it can
also de-duplicate the metrics.

We can also install Thanos Compactor, which applied compaction procedure to Prometheus block data stored in an S3 bucket.
It is also responsible for downsampling of data.

Prerequisites
AWS Account with adequate permissions to operate IAM roles, IAM Policy, Amazon EKS and Amazon S3.

Running EKS cluster (1.13 or above)

Prometheus or Prometheus Operator

Helm 3.x

AWS CLI

eksctl version 0.22.0 or above

Confirm that all Thanos components are installed in the same Kubernetes namespace as Prometheus

Clone the Kubernetes manifests for Thanos Querier and store Deployment steps:

git clone -b release-0.12 https://github.com/thanos-io/kube-thanos.git

Thanos Compact Manifests

https://github.com/thanos-io/kube-thanos/tree/master/examples/all/manifests

Deployment Overview
Before beginning with Thanos deployment, we configure an S3 bucket to use as object storage and create IAM policy required
to access this bucket.

1. Enable Thanos Sidecar for Prometheus

2. Deploy Thanos Querier with the ability to talk to sidecar

3. Confirm that Thanos sidecar is able to upload Prometheus metrics to S3 bucket

4. Deploy Thanos Store to retrieve metrics data stored in long-term storage

5. Set up Thanos Compactor for data compaction and downsampling

EKS, Thanos and S3 1

 Configure S3 bucket and IAM Policy
1. To store metric data, create an S3 bucket in an AWS region local to the Prometheus environment.

2. Create an IAM policy to attach to the IAM role to give access to ServiceAccount used by Prometheus Pod.

{
"Version": "2012-10-17",
"Statement": {
"Sid": "BucketPolicy",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::thanos-metrics-s3storage/*",
"arn:aws:s3:::thanos-metrics-s3storage"
]

}
}

EKS Cluster Setup

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: thanosdemo
region: us-west-2
version: '1.16'
iam:
withOIDC: true
serviceAccounts:
- metadata:
name: prometheus-prometheus-oper-prometheus
namespace: monitoring
labels: {aws-usage: "application"}
attachPolicyARNs:
- "arn:aws:iam::454014481298:policy/thanos-metrics-s3storage-policy"
managedNodeGroups:
- name: ng0
minSize: 1
maxSize: 3
desiredCapacity: 2
ssh:
allow: true
publicKeyName: thanosdemo
labels: {role: mngworker}
iam:
withAddonPolicies:
imageBuilder: true
autoScaler: true
externalDNS: true
certManager: true
ebs: true
albIngress: true
xRay: true
cloudWatch: true
appMesh: true
cloudWatch:
clusterLogging:
enableTypes: ["*"]

eksctl create cluster -f eks-cluster-config.yaml

EKS, Thanos and S3 2

 verify the OIDC provider

aws eks describe-cluster --name thanosdemo -query "cluster.identity.oidc.issuer"

Add Chat repository

helm repo add stable https://kubernetes-charts.storage.googleapis.com/

Installing and Configuring Prometheus and Thanos

1. Get the prometheus-operator chart default configuration values

helm show values statble/prometheus-operator > values_default.yaml

2. The prometheus-operator chat creates the Kubernetes resources required to run Prometheus as part of the installation. We
must disable ServiceAccount creation for the Prometheus POD as ServiceAccount prometheus-prometheus-oper-
prometheus was created during the cluster install.

## Deploy a Prometheus instance

##
prometheus:
enabled: true
## Annotations for Prometheus
##
annotations: {}
## Service account for Prometheuses to use.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
##
serviceAccount:
create: false
name: "prometheus-prometheus-oper-prometheus"

3. Add Thanos Sidecar configuration after thanos

thanos:
baseImage: quay.io/thanos/thanos
version: v0.12.2
objectStorageConfig:
key: thanos-storage-config.yaml
name: thanos-storage-config

4. Configure ObjectStorageConfig witht the configuration file with the command thanos-storage-config.yaml :

type: s3
config:
bucket: thanos-metrics-s3storage #S3 bucket name
endpoint: s3.us-west-2.amazonaws.com #S3 Regional endpoint
encryptsse: true

5. Create Kubernetes secret:

kubectl -n monitoring create secret generic thanos-storage-config —from-file=thanos-storage-config.yaml=thanos-storage-config.yaml

EKS, Thanos and S3 3

 6. Install Thanos Sidecar with Prometheus POD:

helm install prometheus stable/prometheus-operator -f values_sa.yaml -n monitoring

7. Check the status of Prometheus POD and Thanos Sidecar:

kubectl get po -n monitoring -l app=prometheus

8. Check the status of Thanos Sidecar container in Prometheus POD:

kubectl describe pod prometheus-prometheus-prometheus-oper-prometheus-0 -n monitoring

Deploy Thanos Querier

Thanos Querier assists in retrieving metrics from all Prometheus instances. It can be used with Grafana because of its
compatibility with original PromQL and HTTP APIs.

1. Add metric store configuration as thanos-query-deployment.yaml under spec.spec.containers args query section.

--store=thanos-store.monitoring.svc.cluster.local:10901
--store=prometheus-operated.monitoring.svc.cluster.local:10901

EKS, Thanos and S3 4