5G Core Microservices and Containers ISSUE 1.7

5G Core Microservices and Containers
www.huawei.com
Copyright © Huawei Technologies Co., Ltd. All rights reserved.

Software Engineering Ideal: Building Software Systems Like
Building Houses (1/2)
+ =
Industry model
Service architecture (component) + Software
architecture = Software system
Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 3

Software Engineering Ideal: Building Software Systems Like
Building Houses (2/2)
5GC service
5GC software architecture?
architecture/component?
Microservice governance
Microservice
framework
5GC microservice deployment
Container-based deployment

Contents
1. 5GC Microservice Architecture
2. 5GC Microservice Governance Framework
3. 5GC Microservice Deployment (Container-based)

Microservice Concept
What is a microservice?
 Microservice is a software system architecture in which a complicated application is composed of multiple,

independent, micro services.
 These microservices communicate with each other through APIs that are irrelevant to languages.
 These microservices are small in size, loosely coupled, and focus on only a small task.
 Decomposed modular services facilitate system construction.
 Microservices are autonomous and complete, controlling all components, including UI, middleware,
access, and transactions.

Monolithic and Microservice Architectures
Monolithic architecture Microservice architecture
• Resource efficiency is maximized

Appropriate design for a
Resource
UI through microservice-based
efficienc
specific environment can
UI instantiation and scaling.
y
maximize efficiency. If the
• Excessive splitting increases the
Catalog environment changes, huge
basic overhead and cross-service
Service resource waste may occur.
Business Logic communication overhead.
Recommendation Customer
Data Access Layer Account Service
Service Service • Independent maintenance of
Maintenance
Development and maintenance microservices by a full-function
efficiency
become complicated with the team improves development and
increment of software volume. O&M efficiency.
Appropriate design can • Presenting too many details to
DB DB DB DB
simplify end users' operations. users increases the management
and maintenance costs.
Microservice definition
Microservice architecture characteristics Microservice architecture core
principles
Agilit
Too weak to support agile Excellent. Good decoupling
y
• Service autonomy, self-containing, • Function independence • Independent lifecycle
and self-management powered by further • Independent resource release. significantly improves agility.
• Independent service development decoupling of software logic. scaling
states, and separated platform and • Independent optional
Performance
language selection components Medium. Excessive splitting
• Independent service running states Excellent increases latency and deteriorates
and independent upgrades performance.
• Contractual interfaces between
services

5GC Microservice: UNC Microservice
UAM USM NRF NSSF
Term:
UAM: Unify Access Management
AM SM USM: Unify Session Management
APP Service Splitting principles:

USN AM
(Domain microservice, ADDR NRF NSSF Business perspective:
Policy
exclusive to NFs)
g. 1. Service value: Separate frequently changed service values, such as
rai n in policy and private customization capabilities, to shorten the TTM.

rt
SM
GBIM MCR CM
me
Policy Technical perspective:
s t o 2. Separation of distributed basic capabilities: Simplify the
cu
Link Service
d e for association between services and microservices (service mesh) to
NETM GTPC
sl i reduce complexity of full-mesh connections caused by the
h is
(Link microservice, NETM GTPC
et
splitting, which is the key to resolve the problem of microservices
shared by NFs)
t u s in a large quantity.
SBIM SBIM
n o
ANIM
Do 3. Domain-driven: Split the system by domain object (data and
nly. GXIM
(HTTP)
SBIM SBIM processing are independent) to determine the microservice context
o
u se
UPIM (HTTP) (HTTP) boundary, which is the core of microservice splitting.
t e rnal 4. Resource differentiation: Services that have different resource
fo r in requirements are split, improving resource efficiency.
eis 5. Scaling granularity: Services with different scaling granularities

Framework
(System microservice,his
slid UDR LI IPSEC CSDB CSLB FAC are split, improving resource efficiency.
shared globally)
T 6. Distinguish between process and atomic services: Protocol
function changes are separated. In the future, only a single atomic
Platform basic service set, such as SDR, CSR, etc. or process service needs to be replaced, eliminating E2E coupling
in the 3GPP process.
GO Large-granularity GO/C hybrid

Legend: microservice C microservice
microservices microservice

5GC Microservice: UNC Microservice
UAM USM NRF NSSF Term:

UAM: Unify Access Management
USM: Unify Session Management
AM SM NRF NSSF Splitting principles:

Business perspective:
APP Service 1. Service value: Separate frequently changed service values,
(Domain microservice, USN Policy CM Policy
ng . such as policy and private customization capabilities, to
in i shorten the TTM.
tr a
exclusive to NFs)
er
Technical perspective
tom
2. Separation of distributed basic capabilities: Simplify the
u s association between services and microservices (service mesh)
df or c to reduce complexity of full-mesh connections caused by the

Link Service
s e splitting, which is the key to resolve the problem of
(Link microservice, SBIM
ANIM
e i su SBIM
SBIM SBIM microservices in a large quantity.
(HTTP)
id (HTTP) (HTTP) 3. Domain-driven: Split the system by domain object (data and
s sl
shared by NFs)
T h i processing are independent) to determine the microservice

context boundary, which is the core of microservice splitting.
4. Resource differentiation: Services that have different resource
requirements are split, improving resource efficiency.
5. Scaling granularity: Services with different scaling
granularities are split, improving resource efficiency.
6. Distinguish between process and atomic services: Protocol
Framework CSDB CSLB function changes are separated. In the future, only a single
(System atomic or process service needs to be replaced, eliminating
microservice, E2E coupling in the 3GPP process.
shared globally)
Basic platform service set

5G Core Microservice Structure
GO/C microservices Large-granularity microservice
m er
o
t for main
usOM
Controller
Controller Active/standby f orc control Active/standby
Controller Executor for main control sli de OM for main
NLS for main control
e t hi s control
o t us
. Don
on ly ng.
l u se traini
r na
Executor r i nt e NLS
o
is f
Executor
sl i de N-way mode NLS N-way mode
T hi s
Executor Service execution logic NLS Service execution logic
Executor NLS
Note: The newly-developed 5GC microservices are all GO microservices. C Note: Large-granularity microservices directly reuse CE NLS, but container-
microservices are reconstructed from CE NLS. based reconstruction is performed during deployment.

5GC Microservice Structure
5GC microservices Solution Description

1. Controller and Executor nodes are involved in the
microservice architecture.
Controller
Controller Active/Standby 2. Controller nodes perform main control on microservices.
Controller Executor for main control
n g.
3. Executor nodes process service logics of microservices.
i
r t ra i n
4. Controller nodes work in active/standby mode, and
m e Executor nodes work in N-way mode.
us to
f orc
s e d
Executor
is u
l i de
Executor
s s N-way mode
Executor Thi Service execution logic
Executor

Contents

Microservice Governance Framework
NFVO OSS/BSS (U2020)

(VNF LCM)
Microservice governance
VNF
framework
App Micro-Service  Microservice registration and
Upd Conf CSDB discovery
Middleware
O&M UAM USM XXX
CHR ALM LI
 Inter-microservice
MCS1 MCS3 MCSa MCSc xxx xxx
communication
MCS2 ... MCSb ... xxx ...
VNFM Log Perf LB/IP  Load balancing of
(VNF LCM)
microservices
Framework
SCF MCSG SMSH HAF DCF
 Microservice reliability
Micro-Service
mechanism
 Microservice O&M
CaaS (FusionStage)  Middleware and support
components of microservices
VIM Cloud OS (FusionSphere OpenStack)

(FusionSphere
OpenStack
OM) Hardware (E9000, RH2288)

UNC System Architecture: Microservice Registration and
Discovery Mechanism
VNF
Service registration
Microservice A  Microservice A (pod 1, 2, ...) accesses and
registers with the Service Center.
POD A1 POD A2 …  Microservice A subscribes to
microservices B-n, and Service Center
returns the access list of microservices B-
Microservice A n to microservice A.
registration/deregistration Microservices can access each other using IP
address+port number. Dynamic service adjustment and self-
Service Center
Microservice B adaptation
registration/deregistration
Microservice B  If a new pod registers with the Service
Center, the Service Center automatically
POD B1 POD B2 … pushes the pod to associated
microservices.
Service access
 Microservice A can directly access
microservice B using the IP address+port
Microservice n number.
Service deregistration
 If a pod deregisters from the Service
... Center, the Service Center notifies
associated microservices of the
deregistration.

Transparent Communication for Microservice
Objective: inter-service decoupling
 Bind a service policy (topic/key type/key instance).
 Intra-service routing policy (key maps to token and root
› Non-coupling of the peer microservice type token topology)
.
age
› No awareness of the peer service topology Service A Service B
› g p
i ni n
No awareness of the peer token topology a
Service A Service B
controller ra controller
tSDR controller
er
Implementation (message routing based on the ust om
h ec
mt
receiver's policy) Executor pod Executor pod
Executor processfro
b
› Topic, key, and token are addressing objects for SDR c
ed
Executor process
communication. e l e t  Service A sends the

is1 d Executor
 Key mapping policy of Executor
 Topic is the external, functional interface (or sender's (service B) 6
d the receiver (service B)
y an
topic policy. d
onl
message type) provided by the microservice.
 Key is the label of a data object.
l u se 2 SDR agent SDR agent 5
n t e rna
 Tokens are decoupled from physical IP addresses,
fo r iand elastic
supporting fault-triggered migration 3 SDR exec SDR exec 4
scaling. Apps are unawareeofistokens for other MAC PAE SDR Payload
id
services. s sl
T hi The following uses the Executor message sent from service A to
[3] SDR executor routes the message.
• Locate the receiver's address based on the specified topic and
› The app's receiver delivers a receiving policy, service service B as an example:
key.
a-d: communication policy issuing process
binding policy, and intra-service routing policy to the • Determine the mapping about receiver's key and the root token.
1-6: message forwarding process
• Forward the root token to the destination Executor instance.
SDR. [1] The Executor sends a message. • Locate the mapping between the Executor instance and the
› The app's sender specifies the topic and key when • The topic and key are specified. destination address (for example, vFabric is TB/TP)
[2] The SDR agent routes the message. [4] The SDR executor on the receiver end transparently transmits the
sending a message. The SDR routes messages based on • If the key is the root token for affinity communication and located in message.
the receiver's policies. the current process, the message is routed within the process. [5] The DS actor in Executor routes the message to the actor.
• In other cases, the message is forwarded to the SDR executor. The exec DS actor routes the message to the actor.

Microservice Load Balancing
Basic principles of load balancing

SDR Microservice A  The system evenly allocates all tokens to each
Allocate tokens based
Controller 2 Synchronize token Controller on the capability of each Executor based on the number of Executor instances
distribution
executor instance and a
information to the 1 and a certain weight.
certain weight.
SDR through an  When a new service accesses, the SDR evenly
internal mechanism.
distributes services to the token based on the hash
budget to ensure load balance among Executor
Microservice A Microservice A Microservice A
instances.
Executor 1 Executor 2 Executor n
Basic principles of re-load-balancing
Token List Token List Token List
 During system scaling in/out, the number of
(1, 2, 3) (4, 5, 6, 7) (u, v, w,…)
Executor instances changes accordingly.
 The system adjusts token distribution to ensure that
tokens are evenly distributed to Executor instances

3 Distribute tokens evenly
to different Executors. based on the weight of each Executor.
 Token migration occurs during re-balancing.
SDR
Executor
Service flow

Configurations Decoupled from Microservices, Ensuring
Configuration Neutrality
The configuration function needs to be
EMS decomposed into the following multiple
microservices:
1. NBI microservices include NetConf,
Configuration Micro service Group RestConf, and Man-Machine-language
interfaces, which are used for interworking
with different configuration tools.
NetConf interface RestConf interface MML interface 2. The Rule Check function should be
decomposed into Module store and Rule
check framework. The Rule check
Configuration module Rule Check framework is steady, and variability of
modules is shielded by the Module store in
Module store which configuration modules are stored.
Database 3. Schema transformation and Data
distribution enable data to be published to
Schema transformation different App microservices in different
configuration files.
Data distribution 4. If an app microservice is added or
updated, the new configuration module
should be uploaded to the Module store
Configuration module upload
dynamically.
App micro service 1 App micro service 2 App micro service 3

Processing-Storage Separation of Microservices
Token,
Microservice framework X group ID Session DB (CSDB)
Process A and B here User microservice A

i n g.
instance 1 r ain A B Backup DB
er t
Service governance
Service
Service HA Mgmt.
governance B
Service Mesh Executor
(HAF)
st o m
Token assignment
or cu
e f
User microservice C
sl idinstance 2
Service governance
Token
Service Mgmt.
governance
Msg his
t D
Session DB instance 1
(DCF)
use A
Topic, Key n o t backup
D o User microservice E B A
ly. Service Mesh Executor instance 3
(token A)
o n
use
F
Service governance
n a l
Service governance
ServiceMesh (Policy
in ter B
or
Center) Link microservice Session DB instance 3
is f instance 1
s lide Service Mesh Executor
Th is
Service Governance Link microservice
(MCSG) instance 2
Link microservice
SCF instance 3

Processing-Storage Separation of Microservices
Token,
Microservice framework X group ID Session DB (CSDB)
Process A and B here User microservice A

instance 1 A B Backup DB
Service
Servicegovernance
governance B
Service HA Mgmt. Service Mesh Executor
g.
n in
Token assignment
a i
User microservice C
t r
Service governance m er instance 2
sto
Service governance Service Mesh Executor D Session DB instance 1
Token Mgmt. Msg
cu A
Topic, Key df or backup
u s e User microservice E B A
e is
(token A) instance 3 Session DB instance 1
s s lid Service Mesh Executor F
Th i B
Link microservice Session DB instance 3
instance 1
Link microservice
instance 2
Service governance Service Mesh Executor
Service governance
ServiceMesh
Link microservice
instance 3

Microservice Redundancy Mechanism
Token,
Platform-related microservices X group ID Session DB (CSDB)
1. Fault detection
User microservice A
g.
Service
Servicegovernance
Service HA Mgmt.
governance instance 1
r ai n in
er t
(HAF) Service Mesh Executor B
2. Fault notification
u sto m
rc
3. Redistribution notification
Service governance
Token
Service Mgmt.
governance
User microservice
d
C
e foA
sli D
instance 2
(DCF) Service Mesh Executor
th is A
se
t umicroservice
6. Session retrieval
4. Token A registration n oUser E
o B A
ly. D
instance 3 B Session DB instance 1
se on
Service Mesh Executor F
Service governance
Service governance
n a lu B
ServiceMesh (Policy
n te r Link microservice
or i
Center) Session DB instance 3
is f instance 1
lid e Service Mesh Executor
h iss
Service Governance T 5. Policy Link microservice
(MCSG) synchronization instance 2
Link microservice
SCF Msg instance 3
Topic, Key (token A)

Microservice redundancy mechanism
Token,
Platform-related microservices X group ID Session DB (CSDB)
1. Fault detection
User microservice A
Service governance
Service governance
Service HA Mgmt. instance 1
Service Mesh Executor B
2. Fault notification
3. Redistribution notification
i n g.
n A
tr a i
Service governance User microservice C
Service governance
Token Mgmt. instance 2
m er Session DB instance 1
u s to D
A
or c
6. Session retrieval
ed f User microservice E A
us
B
e i s instance 3 B Session DB instance 1
slid
Service Mesh Executor F
is
Th Link microservice
B
4. Token A registration
instance 1
Service governance
Service governance
ServiceMesh
Link microservice
instance 2
5. Policy synchronization Service Mesh Executor
Link microservice
Msg instance 3
Topic, Key (token A)

Microservice Orchestration Mechanism
1 Use algorithms to create slice templates. Then,

manually select a slice template. 2 View NFs in the template.
AMF: SMF:
UAM USM NRF
Latency: 5 ms Sub. Mgmt. UP Mgmt.
Slice UE Mgmt. PDN Mgmt.
Bandwidth: 15 Mbit/s NSSF UDM PCF
template Policy execution Policy execution
IoT feature Access Charging
UDG
IoT slice template
FMC feature
4 Complete the design and restore the new 3 Adjust microservices in the NF. Customized microservices:
template to the directory.
1) Replace normal access
AMF AMF
with FMC access.
AMF SMF UDM

SMF SMF 2) Delete policy execution in SMF.
UPF
IoT slice for tenant A PCF (deleted) 3) Delete the whole PCF.

Contents

Relationship Between Microservices and Containers
In essence, they are not directly related.

- The concept of microservice was proposed in the 1970s.
- Container technology was proposed in 2013, much later than microservices.
Microservice is a design mode of application software architecture, which advocates the principles of single responsibility,
service autonomy, lightweight communication, and clear interfaces. Based on these principles, containers can facilitate
microservice development, maintenance, and on-demand scaling.
(1) If containers are used as the infrastructure, microservices can be rapidly deployed and iterated according to the concept
of microservice.
(2) In this cloud computing era, "container as the infrastructure" attracts more attention than "VM as the infrastructure.“
(3) As the default container platform standard, Kubernetes (K8s) integrates the configuration center and registration center,
so there is no need to develop a new configuration center or registration center for the microservice architecture.
This is how microservices and containers are closely related. In most cases, they are mentioned in the same breath.

Container: Lightweight OS Virtualization
VM Vs. Container Docker: container engine
Apps Apps
Build Ship Run
Bins/Libs Base image (bins/libs)
Guest OS (kernel) Homogeneous OS with

container engine
Namespaces and control groups
• Image layering technology supports rapid software
Host OS with hypervisor development and deployment.
engine
• The centralized repository facilitates software
COTS hardware COTS hardware sharing and release.
Container is an OS kernel-based lightweight virtualization

The unified container engine and images make software
technology. Containers provide higher resource utilization and faster
deployment and sharing simple and efficient.
startup speed than VMs, but lower security isolation.

Overall Comparison Between Containers and VMs
Item Container VM
Design concept Application-oriented lightweight technology Resource-oriented system-level isolation
Implementation OS-level virtualization technology, providing an application running Device-level virtualization technology, providing a system
technology environment running environment
Using the hardware resource (I/O) directly Virtualized hardware resources, affecting the performance
Adapting to any CPU architecture, such as x86, ARM, and PPC, due to its Relying on hardware to facilitate high-performance virtualization.
Resource dependency high performance KVM has a complete ecosystem only on x86 servers.
Resource miniaturization for improving the resource efficiency N/A
Image release MB-level layered image 10 GB-level layered image
Providing bearers for microservices, implementing Build-Ship-Run N/A

Microservice ecosystem
Rich microservice ecosystem: third-party middleware, distributed
framework, tool system, and Docker Hub N/A
Development mode Supporting DevOps CI/CD N/A
Deployment in the millisecond level Deployment in about 5 minutes

Performance
Slightly better compute, network, and I/O virtualization N/A
Security Shared kernel space. Security isolation needs to be improved. Complete system isolation

Docker Is the Mainstream in the Container Ecosystem
Docker 2013.3 Rocket 2014.12 Garden 2013.10 LXD 2014.11
Community Commits ： 15340/493(total/month ） Commits ： 1280/91(total/month ） Commits ： 9322/1(total/month ） Commits ： 1079/(total/month ）

popularity Contributors ： 922 Contributors ： 59 Contributors ： 48 Contributors ： 21
Companies Docker and independent contributors ， such as Contribution to the community ： CoreOS and Contribution to the community ： Contribution to the community ：
IBM 、 Huawei 、 Red Hat 、 Google 、 and Endocode Vmware employees and Facebook Canonical
Microsoft(contributing a little) ranking the second(less than 5%)
Container App Container App Container App Container Syetem Container

type
Technical √Powerful and comprehensive functions with √Appc Spec standard, componentization,and √Similar to Docker,without the image √Similar to Docker,without the image
characteristics image packaging, container runtime, and image runtime repository repository
repository integrated √Deamon restart not affecting containers √Deamon restart not affecting √Deamon restart not affecting
√Deamon restart causing container restart √Not requiring the root permission containers containers
√Requiring the root permission √Distributed image repository √Requiring the root permission √Not requiring the root permission
√Centralized image repository Docker hub √Centralized image repostiory √No image repostiory
Popularity Prosperous ： There are 35,000Docker projects on The CoreOS component is used to build a Currently,the container is ued only Canonical’s Openstack version is
Github. container platform. with the CF and will be promted by integrated.
Vmware in the future.
Community Led by Docker Led by CoreOS Operated by the Cloud Foundry Led by Canonical
exposure Foundation
Development The current container implementation standards Multiple large companies support Rocket, and its Containers of Garden are not Containers of LCD may grow only in
analysis have become the foundation of the Docker container technology many develop. independently developed and depend the segmented OpenStack market.
Foundation. on Cloud Foundry.
Docker Google/Redhat VMware Ubuntu

1.Container formats and specifications are important.Multiple container formats and specifications coexist in the industry.Docker formats and specifications are the mainstream.
2.The OCI Foundation is founded,establishing the standards and community pattern for container exposure.

K8s Is the Mainstream in the Container Cluster Scheduling and
Application Orchestration Ecosystem

What Is K8s? (Concept - Structure - Process)
Kubernetes is an open-source system for automating deployment, scaling, and
management of containerized applications.
Cloud connector
 Troubleshooting  Multi-cloud synergy
 Auto scaling cloud

Kube-controller-
 Rolling upgrade Cloud-controller-
manager
manager
 Batch execution Core concepts:
 Operation
Pod, a container set that includes stateless and stateful pods. It is the
monitoring
minimum deployment and scheduling unit.
 Deployment and
Service, pod access proxy abstraction
Resource lifecycle startup
Kube-apiserver
kubelet kubelet kubelet
 Service
etcd Kube-proxy Kube-proxy Kube-proxy discovery
 Load
balancing
Kube-sched
Kubernetes Minions
Tips: The K8s ecosystem integrates
 Resource scheduling Plug-in mechanism  Network, storage, and acceleration resources (compute, storage, and
network) and is used in the service
(CNI/CNM/CSI) integration solutions running framework.

Huawei Container Cluster Scheduling and Application Orchestration
Solution – FusionStage (PaaS)
Application scheduling and resource management framework: Sets up K8s-based, enhanced, automatic lifecycle management, including application modeling, orchestration deployment,
resource scheduling, auto scaling, monitoring and self-healing.
Microservice running and governance framework: Provides a series of distributed/microservice governance capabilities, such as automatic registration, discovery, governance, isolation, and
invoking analysis, to shield the complexity of the distributed system.
Application development pipeline framework: Streamlines the CI/CD full-process automation from writing code submission to automatic compilation and packaging, continuous integration,
and automatic deployment.
Cloud middleware services: Refer to middleware services required for application cloudification and integrate traditional non-cloud middleware capabilities through service integration
management and control.
Management plane Data plane
PaaS cloud management system Legacy apps Virtual apps Cloud-based apps
Mixed
orchestration/d ERP e-Bank... CRM e-Commerce... Web Email…
eployment Service
Monitoring & integration/
Application
self-healing mgmt. Cloud middleware services
scheduling & Microservice running and
resource mgmt.
framework App resource governance framework
Auto scaling scheduling
Cross-cloud Service route Service discovery ELB Distributed cache service
adaptation
Code version Service governance External middleware

mgmt. Service registration Distributed message service
(isolation and fallbreak) integration
Application
Continuous development
pipeline IDE
integration Service monitoring (call Service definition
framework
Compiling and chain) management
packaging
IaaS
Some microservice components are open source.
The development pipeline is open source, which is
The FST 2.0 microservice framework provides POC
included in Huawei products and free for customers.
capabilities, and has been commercially used since 2018
Qualified suppliers provide customized services. Q1.

How to Introduce Containers to NFV Networks
Container-based VNF Container-based VNF Container-based VNF
Existing Existing
VNF VNF
Container platform Container platform Container platform
VM Bare metal VM Pure bare metal

IaaS IaaS
VM-based container NFVI Extended Bare-Metal Container Pure Bare-Metal Container

Infrastructure shared with existing
VNFs Yes Yes No
No. The container platform is integrated
Container platform decoupled from Yes. The NFVI shields hardware. with the NFVI, and the NFVI is coupled No. The container platform manages
hardware infrastructure with hardware. hardware infrastructure.
Yes. The NFVI provides multi-vendor
integration capabilities, and different No. Container platforms are still under quick development. Multi-vendor integration
Multi-vendor integration vendors can use their own container is difficult before container platforms are standardized.
platforms.
VMs are used to isolate containers. This
Security isolation of containers from enables security isolation between tenants Physical machines are used to isolate containers, implementing isolation between
multiple vendors more flexibly. tenants. This method is not as flexible as container isolation using VMs.
Performance Similar to VM performance Similar to physical machines
Container OS faults are within VMs, so
Reliability other VMs are not affected. Container OS faults are within bare-metal devices.
VMs can be used to implement advanced
Flexible resource management functions, such as live migration of Advanced functions, such as live migration of containers, are unavailable.
containers.
Use VM-based containers because bare-metal containers do not support multi-vendor integration before they are
standardized.
Impact of the VM + Container on the NFV Model
Network functions virtualization orchestrator

(NFVO):
Orchestrates NSs and VNF software
NFVO

The CaaS layer is added to packages.

(Manages NS  Manages NS life cycles.
the original NFV model and lifecycle)  Globally manages, authenticates, and
interfaces are added between authorizes NFVI resource requests.
 Manages policies on NS instances.
the CaaS layer and VNFM.
• Orchestrates, deploys, and
Virtualized network function manager (VNFM):
schedules containers.  Manages life cycles of VNF instances.
• Provides CT enhancement Virtualized Network Function

 Provides overall coordination between the NFVI and
capabilities for containers, such VNF (e.g. CloudIMS/CloudEPC) VNFM EMS.

 Functions as a VNF container resource management
as hugepage memory, shared (Manages VNF portal.
memory, DPDK, CPU pinning, lifecycle)  Manages life cycles of container-based VNFs, including
and isolation. instantiation, uninstallation, auto scaling, and transparent
CaaS transmission of upgrade requests.
• Supports container network  Monitors container alarms and KPIs.
capabilities, SR-IOV+DPDK,
and multiple network planes.
• Supports the IP SAN storage Cloud OS VIM Virtualized infrastructure manager
capability of VM-based NFVI (Hypervisor+Management Module) (VIM):
(Provisions
containers.  Controls and manages compute, storage,
virtualized and network resources.
Hardware resources)  Collects and reports infrastructure
(Server/Storage/Network) performance counters and events.
MANO
Management and
Orchestration

Basic Concepts Introduced by 5GC Deployment
App 1 App 2 App 3
App 1 App 2 App 3 VM Microservice
Bins/Libs Bins/Libs Bins/Libs
Bins/Libs Bins/Libs Bins/Libs

Pod Pod VM
Guest OS Guest OS Guest OS
Docker Engine Container 1 Container 1
Pod for controller
Hypervisor Operating System
Container 2 Container 2
Host Operating System Infrastructure VM
Infrastructure
Container 3 Container 3
Pod for executor
Virtual Machines
VM: Container: Relationships between VMs, containers, Relationships between microservices,

1. Each VM has an independent 1. Containers act as lightweight VMs. and pods: pods, and VMs:
guest OS, ensuring security They share the OS kernel. 1. Pod is a resource management concept 1. Microservice is a concept of logical
isolation. Containers are less isolated than defined in K8s and is not a running functions.
2. Hardware resources are VMs. entity. 2. The logical functions of microservices
virtualized, affecting the 2. No performance penalty for bare 2. Containers with a group of functions need to be carried by the VM or pod
performance. metal containers. form a pod, and are deployed by pod. entities.
3. Second-level instantiation, and 3. A pod is deployed on a VM.
agile deployment. 4. Containers within a pod cannot be
4. Multiple containers can run in a deployed on different VMs.
VM.

LCM Information Model for VNFs Running on VM-based Containers
Service model The items highlighted in yellow are the main objects managed by
the container-based VNF LCM.
NFVO
NS Software model
Resource model
1:N
1:N
N:1 1:1 VNFM
VNF VDU
N:1
1:N
N:1
1:1 1:N N:1
VNFC (Micro)Service Pod VM Host
1:N
CaaS
EMS Container

VM Specifications and Mapping with Microservices
VM Flavor Core vCPUs Memory (GB) Storage (GB) Anti-affinity Rule VM Label Key Pod
OMU=VM Mainly used by services, such as
4 8 32 470 Strong
HAFEXEC=MAVM,… VNFPOM and CSPOM
OMU
HAFEXEC=MAVM
4 8 32 470 Strong CSP services
OMU.G1=VM,…
64 PBU_C=VM
4 8 26 Strong LINK,SM…
(hostPath) HAFEXEC=MAVM,…
PBU_C=VM
4 8 26 64 Strong NETM,UPIM,…
HAFEXEC=MAVM,…
PBU_C=VM
4 8 26 64 Strong HAFETCD,VHA
PBU_C HAFEXEC=MAVM,…
PBU_C=VM
4 8 26 64 Strong SBIM,NRF,NSSF,GTP
HAFEXEC=MAVM,…
64 PBU_C=VM
4 8 26 Strong USN,AM
(hostPath) HAFEXEC=MAVM,…
PBU_C=VM
4 8 26 64 Strong CM,POLICY
HAFEXEC=MAVM,…
PBU_M=VM
PBU_M 4 8 34 50 Strong CSDB
HAFEXEC=MAVM,…
PBU_I=VM
PBU_I 4 8 24 50 Strong CSLB,IP
HAFEXEC=MAVM,…
Constraints: • Heterogeneous resource models are not supported for the same service or pod. For
• Each VDU is divided into one or more groups, and each group has a main label. example, an AM pod cannot use 4-core and 2-core resources at the same time.
• VMs become universal. Different VMs have different pod types and instance • If CSE-ETCD is deployed on IPUs, at least three IPUs are required.
quantity. • If hostPath is used, the number of instances must be the same as that of Label-VDU
• VNF instances cannot share pooled VDU resources. instances. The use of hostPath pods must be under restrict control.

Template Package Structure - VNFD + TOSCA
TemplatePackage Lecel1-Directory Lece2-Directory Lecel3-Directory Lecel4-Directory Lecel5-Directory File Description
Contains SHA values of each zip file in

Vercfg.xml
the same directory.
Vercfg.xml.cms Signature file of Vercfg.xml
Vercfg.xml.crl
VNFD-Types-Definitions.yaml Inherit original capabilities.

VNF_$VERSION definitions TOSCA-Normative-Types-
Definitions.yaml Define VNFD object types.
_Template.zip Inherit original capabilities.

blueprints *.yaml, eg VNFD-Template_Large_E9K.yaml
Define VMs.
Inherit original capabilities.

plans *.yaml, eg Large_E9K.yaml
Define input parameters.

scripts user-defined
Define script files

files user-defined
Define injected files.

Template Package Structure - VNFD + TOSCA
TemplatePackage Lecel1-Directory Lece2-Directory Lecel3-Directory Lecel4-Directory Lecel5-Directory File Description
AM AM.yaml
AM_Service AM service
…… ……
SM_Service SM SM.yaml SM service
…… ……
LINK_OM.yaml Public service

Common_Service LINK
LINK.yaml
…… ……
MME.yaml 2/3/4G service
MME
MME-SGSN_Service MME_OM.yaml
…… ……
CSLB_OM.yaml Middleware service
VNF_$VERSION TOSCA-5G-Core-
service_templates CSLB_IP.yaml
_Template.zip $VERSION.tar.gz
LBF
IP_MAIN.yaml
Middle_Service IP_CTRL.yaml
CSDB_OM.yaml
CSDB
CSDB.yaml
……. ……
OMLB OMLB.yaml OM service
OAM_Service
…… …….
SCFM.yaml Product piatform service
SCF
Service_Fabric SCFA.yaml
…… ……
Blueprint.yaml Main template entry

Structure of a TOSCA Package
node_templates:
app-example:
properties:
1. Image program of a container
# Define the number of app-example instances. 2. Deployment relationship between containers and VMs
instances: 2 3. Resource occupation configurations
annotations:
# Define the network plane required by the app-example instances.
- name: "network.alpha.kubernetes.io/network" # Mount /opt/host on the node
value: '[{"name": "base1","interface": "eth1"},{"name": "fab1","interface": "eth2"}]' to the /opt/container directory in the container.
resourceSelector: volumes:
# Define the deployment locations and affinity attributes of the app_example instances. The app- - name: example-dir
example instances will be deployed on VMs with the vm: omu label. mountPath: /opt/container
labels: # Define a volume.
- key: "vm" app-example-dir:
value: "omu" properties:
# Only one app-example instance can be deployed on each VM. name: example-dir
affinities: hostPath: /opt/host
antiself: true node_templates:
app-component: container-network-base:
properties: properties:
package: name: "base"
# Container image name # Define the type of the logical network.
image: swr ip:port/unc/app_component:1.0 type: "underlay_ipvlan"
# Define resources, including CPU, memory, and hugepage memory. providerNetworks: ["phy-network-Base"]
resource_spec: # Define the VLAN ID of the logical network plane.
requests: vlanID: 100
cpu: 12 # Define the network segment of the logical network plane.
memory: 74G ipam:
limits: subnet: "10.1.0.0/16"
cpu: 12 gateway: "10.1.0.254"
memory: 74G rangeStart: "10.1.0.1"
hugepage rangeEnd: "10.1.0.120"
- type: "1G"
number: 2

Container Images
VNFD template Specifies the information, such as the type, quantity, and network, about the VMs to be deployed.
Template
TOSCA template Specifies the information, such as the namespace, network, and image address, about the containers to be deployed.
Guest OS image package Euler OS image package for VMs.

image Image package of a group of container processes. The service software package for a VNF is split into several pod image
Pod image package packages, and these packages can be upgraded and maintained independently.
The pod image packages for a VNF may be combined as a software package for commercial release.
UNC product image package UNC template
Pod 1 software image Pod 2 software image From the internal technology
package package AOS template perspective, the software and
templates between pods are
Microservice
Microservice
Microservice
Microservice
FusionStage installation package
independent of each other and

process
process
can be delivered independently in
data
data
the development phase.
Pod 1 Pod 2
TOSCA TOSCA In the early phase of 5G version,
template template the A/B test and ongoing version
Docker basic image Docker basic image delivery processes are not ready,
and pod images are combined
into a product image package. All
the images use the same template.
The software package quantity
and operation mode are the same
Guest OS image package VNFD template as those for CloudEdge.

Package Management Procedure
User
Upload the VNFD template

1. When the VNFM northbound interface or portal receives the compression package()
uploaded template, the user can select the CaaS to which the Parse the VNFD()
template is to be synchronized. The CSM can connect to
multiple CaaSs. Store the VM template()
Upload the container
2. The CSM parses the VNFD and saves the VM template. template (API Call)
3. The CSM invokes the CaaS interface to upload the template.

Query the container template list()
4. The CSM invokes the CaaS interface to query the template list
and display the list to the user.
5. The CSM invokes the CaaS interface to create an application Create an application namespace()
namespace.
6. When the VNFM northbound or portal receives the uploaded
image, the user can select the CaaS to which the image is to be Upload the container image
synchronized. The CSM can connect to multiple CaaSs. compression package
(containing VNFs)() Create a namespace (optional)
7. The CSM invokes the CaaS interface to create an image
namespace and an image repository.
8. The CSM cyclically uploads all images in the image package to Create an image repository (optional)
the CaaS.
9. When the VNFM northbound or portal receives the uploaded
Upload the image()
image, the user can select the VIM to which the image is to be
synchronized. The CSM can connect to multiple VIMs.
10. The CSM invokes the VIM interface to synchronize the VM Query the image()
image to the VIM.
Upload the VM image()

Synchronize the VM image to the VIM()

VNF Instantiation Procedure
1. A user receives an NS instantiation request on

the NFVO portal.
2. After creating a network, the NFVO sends a
VNF creation request to the VNFM.
3. [Direct mode] The VNFM invokes the NFVO
interface to authorize VM resources.
Manage VMs()
4. The VNFM invokes the VIM interface to create
a VM and start the VM.
5. The VNFM invokes the CaaS interface to
manage the created VM and perform a series of
container creation actions, including CreateStack
and StartStack.
6. The NFVO invokes the VNFM interface to
query the VNF instantiation progress.

VNF Termination Procedure
1. A user initiates a request for terminating an NS

on the NFVO portal.
2. The NFVO invokes the VNFM to terminate the
VNF.
3. The VNFM invokes the CaaS interface to delete
the stack.
4. The VNFM periodically invokes the CaaS
interface to check whether the stack has been
deleted.
5. The VNFM invokes the CaaS interface to delete
the VM node.
6. The VNFM invokes the VIM interface to delete
the VM.
7. The NFVO invokes the VNFM interface to view
the VNF termination progress.

Auto Scale-out Procedure
Principles: Rapid container scale-out is an important

value point of containers. If more containers need to
VM scale-out
be added, ensure that required VM resources are
available.
[VM scale-out]
Method 1: Reserve VM resources during VNF
deployment.
Method 2: Manually perform VM scale-out on the
MANO in advance. For details, see the figure of VM
scale-out.
Method 3: The load is automatically detected based on a
policy. If the VM load exceeds the threshold, VM scale-
out is performed in advance. For details, see the figure
of VM scale-out.
Container scale-
[Container scale-out]
Method 1: A user performs container scale-out on the
VNFM. For details, see the figure of container scale-
out
out.
Method 2: The container load is automatically detected
based on a policy. If the container load exceeds the
threshold, the system starts to add more containers. For
details, see the figure of container scale-out.

Auto Scale-in Procedure
sd-VNF Scale-in (VM)
VNFM VIM CaaS

[VM scale-in]
User
1. A user initiates a VM scale-in operation on the VNFM.
2. The VNFM selects a low-load VM. ScaleIn(VM)
3. The VNFM invokes the CaaS interface to set the VM to The VNFM selects a VM with low load.()
VM scale-in
be unschedulable. The VNFM sets the VM to be unschedulable.()
4. The VNFM deletes the pod on the VM. After detecting a
Ack()
pod fault, the CaaS automatically migrates services and
rebuilds a pod. Delete Pods()
5. The VNFM invokes the VIM interface to delete the VM. Ack()
[Container scale-in]
Method 1: A user performs container scale-in on the VNFM. The CaaS reschedules Pods
Delete VMs() To other VM nodes.()
For details, see the figure of container scale-in.
6. A user initiates a container scale-in operation on the Ack()
VNFM.
7. The VNFM invokes the CaaS execution stack lifecycle
management interface to perform scale-in.
8. The VNFM invokes the CaaS query stack interface to
check whether scale-in has been complete.
Method 2: The container load is automatically detected based
on a policy. If the container load is lower than the threshold,
Container scale-
container scale-in is initiated. For details, see the figure of in
container scale-in. The procedure is the same as that described
in method 1.

5GC Container Network
Component 5GC 2.0 5GC 20.0
UNC
Base: IP VLAN
OMU VM Container network Base: IP VLAN
Fabric: SR-IOV
Fabric: OVS
NPS Pod GSC OMO HA Pod NLS main UNC: OVS/EVS
control pod VM network SR-IOV
Pod Pod UDG: SR-IOV
(multiple)
PaaS- Supporting both IP
Flat/VLAN
Agent VLAN and SR-IOV
Single/Multiple
IPAM Multiple vNICs Single vNIC
vNICs
Canal
Container
network <<PaaS_Mng>>
 Network planes
PaaS Management, OM, Base, Fabric, and External
VM  IP VLAN
network
The Linux traffic distribution policy is configured for pod ports so that
NIC 1 NIC 2 traffic received over vNICs is distributed to different pods based on the
configured policy. Therefore, only a small number of vNICs are required.
40GE 40GE 40GE 40GE
 Direct connection/SR-IOV
ILOGIC_BASE1 ILOGIC_BASE2 Each port of pod occupies one vNIC. When there are a large number of
IPHY1 IPHY2
pods, the vNICs will be insufficient.

Thank You
www.huawei.com

5G Core Microservices and Containers ISSUE 1.7

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

5G Core Microservices and Containers ISSUE 1.7

Uploaded by

Copyright:

Available Formats

5G Core Microservices and Containers

Copyright © Huawei Technologies Co., Ltd. All rights reserved.

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 3

5GC microservice deployment

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 4

1. 5GC Microservice Architecture

2. 5GC Microservice Governance Framework

3. 5GC Microservice Deployment (Container-based)

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 5

 Microservice is a software system architecture in which a complicated application is composed of multiple,

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 6

• Resource efficiency is maximized

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 7

APP Service Splitting principles:

rai n in policy and private customization capabilities, to shorten the TTM.

t e rnal 4. Resource differentiation: Services that have different resource

fo r in requirements are split, improving resource efficiency.

eis 5. Scaling granularity: Services with different scaling granularities

GO Large-granularity GO/C hybrid

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 8

UAM USM NRF NSSF Term:

AM SM NRF NSSF Splitting principles:

df or c to reduce complexity of full-mesh connections caused by the

T h i processing are independent) to determine the microservice

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 9

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 10

5GC microservices Solution Description

m e Executor nodes work in N-way mode.

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 11

1. 5GC Microservice Architecture

2. 5GC Microservice Governance Framework

3. 5GC Microservice Deployment (Container-based)

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 12

NFVO OSS/BSS (U2020)

VIM Cloud OS (FusionSphere OpenStack)

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 13

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 14

communication. e l e t  Service A sends the

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 15

Basic principles of load balancing

tokens are evenly distributed to Executor instances

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 16

App micro service 1 App micro service 2 App micro service 3

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 17

Process A and B here User microservice A

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 18

Process A and B here User microservice A

s s lid Service Mesh Executor F

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 19

lid e Service Mesh Executor

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 20

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 21

1 Use algorithms to create slice templates. Then,

AMF SMF UDM

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 22

1. 5GC Microservice Architecture

2. 5GC Microservice Governance Framework

3. 5GC Microservice Deployment (Container-based)

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 23

In essence, they are not directly related.

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 24

Guest OS (kernel) Homogeneous OS with

Container is an OS kernel-based lightweight virtualization

Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 25