Developing and Scaling a Microservice

dzone.com/articles/scaling-a-microservice

Vishwanath Eswarakrishnan

This article guides developers on key considerations for service

development and technologies to consider to build and scale a
microservice.

By

Vishwanath Eswarakrishnan
·
May. 01, 24 · Tutorial
Services, or servers, are software components or processes that execute operations on
specified inputs, producing either actions or data depending on their purpose. The party
making the request is the client, while the server manages the request process. Typically,
communication between client and server occurs over a network, utilizing protocols such as
HTTP for REST or gRPC. Services may include a User Interface (UI) or function solely as
backend processes. With this background, we can explore the steps and rationale behind
developing a scalable service.

1/9
NOTE: This article does not provide instructions on service or UI development, leaving you
the freedom to select the language or tech stack that suits your requirements. Instead, it
offers a comprehensive perspective on constructing and expanding a service, reflecting what
startups need to do in order to scale a service. Additionally, it's important to recognize that
while this approach offers valuable insights into computing concepts, it's not the sole method
for designing systems.

The Beginning: Version Control

Assuming clarity on the presence of a UI and the general purpose of the service, the initial
step prior to service development involves implementing a source control/version control
system to support the code. This typically entails utilizing tools like Git, Mercurial, or others to
back up the code and facilitate collaboration, especially as the number of contributors grows.
It's common for startups to begin with Git as their version control system, often leveraging
platforms like github.com for hosting Git repositories.

An essential element of version control is pull requests, facilitating peer reviews within your
team. This process enhances code quality by allowing multiple individuals to review and
approve proposed changes before integration. While I won't delve into specifics here, a quick
online search will provide ample information on the topic.

Developing the Service

Once version control is established, the next step involves setting up a repository and
initiating service development. This article adopts a language-agnostic approach, as delving
into specific languages and optimal tech stacks for every service function would be overly
detailed. For conciseness, let's focus on a service that executes functions based on inputs
and necessitates backend storage (while remaining neutral on the storage solution, which
will be discussed later).

As you commence service development, it's crucial to grasp how to run it locally on your
laptop or in any developer environment. One should consider this aspect carefully, as local
testing plays a pivotal role in efficient development. While crafting the service, ensure that
classes, functions, and other components are organized in a modular manner, into separate
files as necessary. This organizational approach promotes a structured repository and
facilitates comprehensive unit test coverage.

Unit tests represent a critical aspect of testing that developers should rigorously prioritize.
There should be no compromises in this regard! Countless incidents or production issues
could have been averted with the implementation of a few unit tests. Neglecting this practice
can potentially incur significant financial costs for a company.

2/9
I won't delve into the specifics of integrating the gRPC framework, REST packages, or any
other communication protocols. You'll have the freedom to explore and implement these as
you develop the service.

Once the service is executable and tested through unit tests and basic manual testing, the
next step is to explore how to make it "deployable."

Packaging the Service

Ensuring the service is "deployable" implies having a method to run the process in a more
manageable manner. Let's delve into this concept further. What exactly does this entail? Now
that we have a runnable process, who will initiate it initially? Moreover, where will it be
executed? Addressing these questions is crucial, and we'll now proceed to provide answers.

In my humble opinion, managing your own compute infrastructure might not be the best
approach. There are numerous intricacies involved in ensuring that your service is
accessible on the Internet. Opting for a cloud service provider (CSP) is a wiser choice, as
they handle much of the complexity behind the scenes. For our purposes, any available
cloud service provider will suffice.

Once a CSP is selected, the next consideration is how to manage the process. We aim to
avoid manual intervention every time the service crashes, especially without notification. The
solution lies in orchestrating our process through containerization. This involves creating a
container image for our process, essentially a filesystem containing all necessary
dependencies at the application layer. A "Dockerfile" is used to specify the steps for including
the process and dependencies in the container image. Upon completion of the Dockerfile,
the docker build cli can be used to generate an image with tags. This image is then stored
locally or pushed to a container registry, serving as a repository for container images that can
later be pulled onto a compute instance. With these steps outlined, the next question arises:
how does containerization orchestrate our process? This will be addressed in the following
section on executing a container.

Executing the Container

After building a container image, the subsequent step is its execution, which in turn initiates
the service we've developed. Various container runtimes, such as containerd, podman, and
others, are available to facilitate this process. In this context, we utilize the "docker" cli to
manage the container, which interacts with containerd in the background. Running a
container is straightforward: "docker run" executes the container and consequently, the
developed process. You may observe logs in the terminal (if not run as a daemon) or use
"docker logs" to inspect service logs if necessary. Additionally, options like "--restart" can be
included in the command to automatically restart the container (i.e., the process) in the event
of a crash, allowing for customization as required.

3/9
At this stage, we have our process encapsulated within a container, ready for
execution/orchestration as required. While this setup is suitable for local testing, our next
step involves exploring how to deploy this on a basic compute instance within our chosen
CSP.

Deploying the Container

Now that we have a container, it's advisable to publish it to a container registry. Numerous
container registries are available, managed by CSPs or docker itself. Once the container is
published, it becomes easily accessible from any CSP or platform. We can pull the image
and run it on a compute instance, such as a Virtual Machine (VM), allocated within the CSP.
Starting with this option is typically the most cost-effective and straightforward. While we
briefly touch on other forms of compute infrastructure later in this article, deploying on a VM
involves pulling a container image and running it, much like we did in our developer
environment.

Voila! Our service is deployed. However, ensuring accessibility to the world requires careful
consideration. While directly exposing the VM's IP to the external world may seem tempting,
it poses security risks. Implementing TLS for security is crucial. Instead, a better approach
involves using a reverse proxy to route requests to specific services. This ensures security
and facilitates the deployment of multiple services on the same VM.

To enable internet access to our service, we require a method for inbound traffic to reach our
VM. An effective solution involves installing a reverse proxy like Nginx directly on the VM.
This can be achieved by pulling the Nginx container image, typically labeled as "nginx:latest".
Before launching the container, it's necessary to configure Nginx settings such as servers,
locations, and additional configurations. Security measures like TLS can also be
implemented for enhanced protection.

Once the Nginx configuration is established, it can be exposed to the container through
volumes during container execution. This setup allows the reverse proxy to effectively route
incoming requests to the container running on the same VM, using a specified port. One
notable advantage is the ability to host multiple services within the VM, with routing efficiently
managed by the reverse proxy.

To finalize the setup, we must expose the VM's IP address and proxy port to the internet,
with TLS encryption supported by the reverse proxy. This configuration adjustment can
typically be configured through the CSP's settings.

NOTE: The examples of solutions provided below may reference GCP as the CSP. This is
solely for illustrative purposes and should not be interpreted as a recommendation. The
intention is solely to convey concepts effectively.

4/9
Consider the scenario where managing a single VM manually becomes laborious and lacks
scalability. To address this challenge, CSPs offer solutions akin to managed instance groups,
comprising multiple VMs configured identically. These groups often come with features like
startup scripts, which execute upon VM initialization. All the configurations discussed earlier
can be scripted into these startup scripts, simplifying the process of VM launch and
enhancing scalability. This setup proves beneficial when multiple VMs are required to handle
requests efficiently.

Now, the question arises: when dealing with multiple VMs, how do we decide where to route
requests? The solution is to employ a load balancer provided by the CSP. This load balancer
selects one VM from the pool to handle each request. Additionally, we can streamline the
process by implementing general load balancing. To remove individual reverse proxies, we
can utilize multiple instance groups for every service needed, accompanied by load
balancers for each. The general load balancer can expose its IP with TLS configuration and
route setup, ensuring that only service containers run on the VM. It's essential to ensure that
VM IPs and ports are accessible solely by the load balancer in the ingress path, a task
achievable through configurations provided by the CSP.

At this juncture, we have a load balancer securely managing requests, directing them to the
specific container service within a VM from a pool of VMs. This setup itself contributes to
scaling our service. To further enhance scalability and eliminate the need for continuous VM
operation, we can opt for an autoscaler policy. This policy dynamically scales the VM group
up or down based on parameters such as CPU, memory, or others provided by the CSP.

Now, let's delve into the concept of Infrastructure as Code (IaC), which holds significant
importance in efficiently managing CSP components that promote scale. Essentially, IaC
involves managing CSP infrastructure components through configuration files, interpreted by
an IaC tool (like Terraform) to manage CSP infrastructure accordingly. For more detailed
information, refer to the wiki.

Datastore
We've previously discussed scaling our service, but it's crucial to remember that there's
typically a requirement to maintain a state somewhere. This is where databases or
datastores play a pivotal role. From experience, handling this aspect can be quite tricky, and
I would once again advise against developing a custom solution. CSP solutions are ideally
suited for this purpose. CSPs generally handle the complexity associated with managing
databases, addressing concepts such as master-slave architecture, replica management,
synchronous-asynchronous replication, backups/restores, consistency, and other intricate
aspects more effectively. Managing a database can be challenging due to concerns about
data loss arising from improper configurations. Each CSP offers different database offerings,
and it's essential to consider the specific use cases the service deals with to choose the
appropriate offering. For instance, one may need to decide between using a relational

5/9
database offering versus a NoSQL offering. This article does not delve into these differences.
The database should be accessible from the VM group and serve as a central datastore for
all instances where the state is shared. It's worth noting that the database or datastore
should only be accessible within the VPC, and ideally, only from the VM group. This is crucial
to prevent exposing the ingress IP for the database, ensuring security and data integrity.

Queues
In service design, we often encounter scenarios where certain tasks need to be performed
asynchronously. This means that upon receiving a request, part of the processing can be
deferred to a later time without blocking the response to the client. One common approach is
to utilize databases as queues, where requests are ordered by some identifier. Alternatively,
CSP services such as Amazon SQS or GCP pub/sub can be employed for this purpose.
Messages published to the queue can then be retrieved for processing by a separate service
that listens to the queue. However, we won't delve into the specifics here.

Monitoring
In addition to the VM-level monitoring typically provided by the CSP, there may be a need for
more granular insights through service-level monitoring. For instance, one might require
latency metrics for database requests, metrics based on queue interactions, or metrics for
service CPU and memory utilization. These metrics should be collected and forwarded to a
monitoring solution such as Datadog, Prometheus, or others. These solutions are typically
backed by a time-series database (TSDB), allowing users to gain insights into the system's
state over a specific period of time. This monitoring setup also facilitates debugging certain
types of issues and can trigger alerts or alarms if configured to do so. Alternatively, you can
set up your own Prometheus deployment, as it is an open-source solution.

With the aforementioned concepts, it should be feasible to deploy a scalable service. This
level of scalability has proven sufficient for numerous startups that I have provided
consultation for. Moving forward, we'll explore the utilization of a "container orchestrator"
instead of deploying containers in VMs, as described earlier. In this article, we'll use
Kubernetes (k8s) as an example to illustrate this transition.

Container Orchestration: Enter Kubernetes (K8s)

Having implemented the aforementioned design, we can effectively manage numerous
requests to our service. Now, our objective is to achieve decoupling to further enhance
scalability. This decoupling is crucial because a bug in any service within a VM could lead to
the VM crashing, potentially causing the entire ecosystem to fail. Moreover, decoupled
services can be scaled independently. For instance, one service may have sufficient
scalability and effectively handle requests, while another may struggle with the load.
Consider the example of a shopping website where the catalog may receive significantly

6/9
more visits than the checkout page. Consequently, the scale of read requests may far
exceed that of checkouts. In such cases, deploying multiple service containers into
Kubernetes (K8s) as distinct services allows for independent scaling.

Before delving into specifics, it's worth noting that CSPs offer Kubernetes as a compute
platform option, which is essential for scaling to the next level.

Kubernetes (K8s)
We won't delve into the intricacies of Kubernetes controllers or other aspects in this article.
The information provided here will suffice to deploy a service on Kubernetes. Kubernetes
(K8s) serves as an abstraction over a cluster of nodes with storage and compute resources.
Depending on where the service is scheduled, the node provides the necessary compute
and storage capabilities.

Having container images is essential for deploying a service on Kubernetes (K8s).

Resources in K8s are represented by creating configurations, which can be in YAML or
JSON format, and they define specific K8s objects. These objects belong to a particular
"namespace" within the K8s cluster. The basic unit of compute within K8s is a "Pod," which
can run one or more containers. Therefore, a config for a pod can be created, and the
service can then be deployed onto a namespace using the K8s CLI, kubectl. Once the pod is
created, your service is essentially running, and you can monitor its state using kubectl with
the namespace as a parameter.

To deploy multiple pods, a "deployment" is required. Kubernetes (K8s) offers various

resources such as deployments, stateful sets, and daemon sets. The K8s documentation
provides sufficient explanations for these abstractions, we won't discuss each of them here.
A deployment is essentially a resource designed to deploy multiple pods of a similar kind.
This is achieved through the "replicas" option in the configuration, and you can also choose
an update strategy according to your requirements. Selecting the appropriate update
strategy is crucial to ensure there is no downtime during updates. Therefore, in our scenario,
we would utilize a deployment for our service that scales to multiple pods.

When employing a Deployment to oversee your application, Pods can be dynamically

generated and terminated. Consequently, the count and identities of operational and healthy
Pods may vary unpredictably. Kubernetes manages the creation and removal of Pods to
sustain the desired state of your cluster, treating Pods as transient resources with no
assured reliability or durability. Each Pod is assigned its own IP address, typically managed
by network plugins in Kubernetes. As a result, the set of Pods linked with a Deployment can
fluctuate over time, presenting a challenge for components within the cluster to consistently
locate and communicate with specific Pods. This challenge is mitigated by employing a
Service resource.

7/9
After establishing a service object, the subsequent topic of discussion is Ingress. Ingress is
responsible for routing to multiple services within the cluster. It facilitates the exposure of
HTTP, HTTPS, or even gRPC routes from outside the cluster to services within it. Traffic
routing is managed by rules specified on the Ingress resource, which is supported by a load
balancer operating in the background.

With all these components deployed, our service has attained a commendable level of
scalability. It's worth noting that the concepts discussed prior to entering the Kubernetes
realm are mirrored here in a way — we have load balancers, containers, and routes, albeit
implemented differently. Additionally, there are other objects such as Horizontal Pod
Autoscaler (HPA) for scaling pods based on memory/CPU utilization, and storage constructs
like Persistent volumes (PV) or Persistent Volume Claims (PVC), which we won't delve into
extensively. Feel free to explore these for a deeper understanding.

CI/CD
Lastly, I'd like to address an important aspect of enhancing developer efficiency: Continuous
Integration/Deployment (CI/CD). Continuous Integration (CI) involves running automated
tests (such as unit, end-to-end, or integration tests) on any developer pull request or check-in
to the version control system, typically before merging. This helps identify regressions and
bugs early in the development process. After merging, CI generates images and other
artifacts required for service deployment. Tools like Jenkins (Jenkins X), Tekton, Git actions
and others facilitate CI processes.

Continuous Deployment (CD) automates the deployment process, staging different

environments for deployment, such as development, staging, or production. Usually, the
development environment is deployed first, followed by running several end-to-end tests to
identify any issues. If everything functions correctly, CD proceeds to deploy to other
environments. All the aforementioned tools also support CD functionalities.

CI/CD tools significantly improve developer efficiency by reducing manual work. They are
essential to ensure developers don't spend hours on manual tasks. Additionally, during
manual deployments, it's crucial to ensure no one else is deploying to the same environment
simultaneously to avoid conflicts, a concern that can be addressed effectively by our CD
framework.

There are other aspects like dynamic config management and securely storing
secrets/passwords and logging system, though we won't delve into details, I would
encourage readers to look into the links provided.

Thank you for reading!

Version control Docker (software) Container Scalability Virtual Machine microservices

8/9
Opinions expressed by DZone contributors are their own.

9/9