Nova/Compute HA

1
Masakari
Mali Asemani
2022 Nov 23
Abramad Corporate
2 Masakari Module
 Provides VMs High Availability(VMHA),
 and rescues KVM-based VM from a failure.

 It covers:
 nova-compute host failure
 evacuate all the VMs from failure host according to th

 Provisioning process down

 restarts process,
 changes nova-compute service status to maintenance mode (use nova service-disable).

 VM process down
 restart vm (use nova stop API, and nova start API).
 Libvirt events will be also emitted by other failures.
3 Requirements

 Hosts need to be configured within zones/segments

 One or more clusters need to be exist

 pacemaker/pacemaker-remote/Consul

 Nova needs to use a shared storage.

 shared storage boundary could be a metric in making decision about the optimal host for fail-over
4 Main Components

 Masakari  Masakari Monitor

 Masakari-API  Provides monitoring tools
 Masakari-Engine
 Monitors Compute nodes and sends
 Masakari Python-Client notifications to Masakari using Masakari-
API
 a Python API (the masakariclient module)
 a command-line tool for masakari.
 Includes
 Host Monitor
 Instance Monitor
 Masakari Dashboard  Process Monitor
 the Horizon plugin  Inspectator Monitor
5 Main Components:
Masakari Monitor
 Host Monitor
 provides compute node HA by
automatically detecting compute nodes
failure via monitor driver.
 Driver Monitor: Consul
 Process Monitor
 automatically detecting the system-level
provides key process HA by automatically
detecting the process failure.
Note: If OpenStack service runs in container(pod),
processmonitor will not work as expected.
 Instance Monitor
 provides Virtual Machine HA by
automatically detecting VMs domain events
via libvirt.
 If it detects specific libvirt events, it
sends notifications to the masakari-api.
 Introspective instance Monitor
 provides Virtual Machine HA by
failure events via QEMU Guest Agent.
 If it detects VM heartbeat failure
events, it sends notifications to the
masakari-api.
6 Masakari Package Services
 Communications
 REST API
 The user-facing interface
 RPC
 Internal communications
 done via the oslo.messaging library
 an abstraction on top of message
queues.
 services
 Masakari-api :
 process REST requests
 Send them to the masakari-engin over RPC
 Typically involve
 database reads/writes,
 sending RPC messages to Masakari engine,
 generating responses to the REST calls.
 Masakari-engine
 will run on the same host where the Masakari
api is running,
 listen for RPC messages
 Processes the notifications received
from masakari-api
 execute the recovery workflow in async way.
7 Masakari Architecture

SQL database for data storage.

8 Evacuation

 When you evacuate the instance, Compute detects whether shared storage is
available on the target host.

Two different ways to evacuate instances from a failed compute node.

 nova evacuate command:
 to evacuate a single instance from a failed node.
 nova host-evacuate:
 to evacuate all instances of the host, in one shot.
9 Segments

 In Masakari the compute nodes are grouped into  Openstack zoned boundaries :
failover segments.
 System can be zoned from top to
down levels, into
 Segment concept enables you to

 Regions,
define several segments of hypervisor hosts
 based on the shared storage boundaries  Availability Zones
 or any other limitations may critical for selection of  and Host Aggregates (or Cells).
the failover host.
 and assign the failover host/hosts for each of them.  Zones can be managed by the nova
scheduler.
 In the event of a failure  Operator may want to use other types
 guests are moved onto other nodes within the same of boundaries such as rack layout and
segment. powering.
 recovery method of a segment determines the
compute node to house the evacuated guests
10 Recovery Methods
(using Nova Evacuate API)

 Auto  auto_priority
 the guests are relocated to any of the available  chain the previous methods together.
nodes in the same segment
 evacuate all the VMs with no destination node for
 At First: attempts to move the guest using
nova scheduler the auto method
 Dis-Adv:  if that fails: tries the reserved_host method
 no guarantee for resources availability

  rh_priority
reserved_host
 compute hosts are allocated as reserved  chain the previous methods together.
 which allows an operator to guarantee there is  At First: attempts to move the guest using
sufficient capacity available for any guests in need the reserved_host method
of evacuation
 if that fails it tries the auto method.
 evacuate all the VMs with reserved hosts as the
destination nodes for nova scheduler
11
How Does it Work?
12 How does it work as a whole?
13 How does it work as a whole?
Recovery Request
Segments API
Host API
Notifications API

b le
isa

No
ic eD Ms

v
v V
Ser

aE
te Start

va
m pu an d

cu
Co Sto
p

ate
o v a
N

Process failure
Notification

Instance failure
Notification
Host failure Notification Pacemaker-remote Pacemaker-remote
14 Host Monitor

 Provides compute node HA

 by automatically detecting compute nodes failure
 via monitor driver.

 Monitor Driver:

 Pacemaker and Pacemaker Remote

 Consul
15 Monitor Driver: Pacemaker

 The compute node’s status is depending on

 the heartbeat between the compute node and the cluster.

 Once the node lost the heartbeat,

 masakari-hostmonitor in other nodes will detect the failure
 and send notifications to masakari-api.
16 Monitor Driver: Pacemaker
17 Monitor Drivers: Consul
 If the nodes in the cloud have multiple interfaces, consul could handle it.
 Interfaces: management network, tenant network or storage network
 Consul agents are required to install into all nodes,
 which make up multiple consul clusters.
 The compute node’s status is depending on
 assembly of multiple interfaces connectivity status,
 which are retrieved from multiple consul clusters.
 Trigger host failure recovery according to defined HA strategy.

Management tenant storage actions Management tenant storage actions

Up up up No Act Up up down recovery

Up down up No Act Up down down recovery

Down up up No Act Down up down recovery

down down up No Act down down down recovery

18 Monitor Drivers: Consul
19 Instance Monitor
 provides Virtual Machine HA by automatically detecting VMs domain events via libvirt.
 If it detects specific libvirt events, it sends notifications to the masakari-api.

 Set libvirt connection uri in the config file

 It runs libvirt event loop in a background thread.

 Invoking libvirt.virEventRegisterDefaultImpl()
 register libvirt’s default event loop implementation.
 Invoking libvirt.virEventRunDefaultImpl()
 will perform one iteration of the libvirt default event loop.
 Invoking conn.domainEventRegisterAny()
 will register event callbacks against libvirt connection instances.
 The callbacks registered will be triggered from the execution context of libvirt.virEventRunDefaultImpl(),
 which will send notifications to the masakari-api.
 It will reconnect to libvirt and reprocess if disconnected.
20 Introspective instance Monitor
 provides Virtual Machine HA
 by automatically detecting the system-level failure events
 via QEMU Guest Agent.

 If it detects VM heartbeat failure events,

 it sends notifications to the masakari-api.

 libvirt and QEMU Guest Agent are used as the underlying protocol for messaging to and from VM.
 The host-side qemu-agent sockets are used to determine whether VMs are configured with QEMU Guest
Agent.
 qemu-guest-ping is used as the monitoring heartbeat.
 For the future release, we can pass through arbitrary guest agent commands to check the health of the
applications inside a VM.
21 Process Monitor

 Provides key process HA  Sample of process definitions :

 by automatically detecting the process failure.

 Processes to be monitored should be pre-configured in

process_list.yaml file

 If masakari-processmonitor detects one process failure,

 it will try to restart it firstly.
 After several retries failed, it sends notification to
masakari-api

 Note:
 If OpenStack service runs in container(pod),
processmonitor will not work as expected.
 Any Question?
Thanks All :)

22