Professional Documents
Culture Documents
Kexec
Kexec
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 1 / 34
A Tale in Three Parts
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 2 / 34
A Tale in Three Parts
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 2 / 34
A Tale in Three Parts
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 2 / 34
Part I
Crash-Dump Analysis
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 3 / 34
Crash-Dump Analysis
User-Space Core-Dump
When an application crashes it is useful to analyse the state of the
programme at the time of the crash
Operating systems provide a facility to dump the memory core of
applications
The core can be analysed using tools such as gdb
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 4 / 34
Crash-Dump Analysis
User-Space Core-Dump
When an application crashes it is useful to analyse the state of the
programme at the time of the crash
Operating systems provide a facility to dump the memory core of
applications
The core can be analysed using tools such as gdb
Operating System Crash-Dump
When an operating system crashes it is also useful to analyse the state
of the system at the time of the crash
The difficulty arises because it is the operating system that would
naturally collect the memory dump, however it has crashed
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 4 / 34
Who is Interested In It?
Enterprise Users
Want deep analysis of crashes
Are willing to go to third parties for analysis
But they want to use vendor-supplied kernels
So a standardised solution is needed
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 5 / 34
Who is Interested In It?
Enterprise Users
Want deep analysis of crashes
Are willing to go to third parties for analysis
But they want to use vendor-supplied kernels
So a standardised solution is needed
Kernel Developers
If end-users could collect crash dumps, it could be useful for kernel
developers to isolate bugs.
The facility would need to be provided by distributions.
Crash-Dump size may be a problem when transferring between
end-users and kernel-developers
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 5 / 34
Some Crash-Dump Solutions for Linux
Full Disclosure: Mini Kernel Dump was developed in part by my employer, VA Linux Systems Japan K.K.
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 6 / 34
Part II
Kexec
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 7 / 34
Kexec
Kexec is a feature of the Linux kernel that allows a new kernel to run
in place of the running kernel
Kernel kexec
Second
Kernel
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 8 / 34
Motivation for Kexec
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 9 / 34
Performing Kexec
The new kernel is loaded into the running kernel from user-space
using kexec-tool
# kexec -l /boot/vmlinuz
The new kernel can then be kexeced using the same tool
# kexec -e
Kexec is a method of warm or soft-booting machines
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 10 / 34
Kexec Flow
1. Kexec Load
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 11 / 34
Kexec Flow
1. Kexec Load
2. Kexec Execute
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 12 / 34
Kdump
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 13 / 34
Kexec vs Kdump
Kexec
Uses (all) system memory as any booting kernel would
The second kernel runs when triggered by kexec-tool
Kdump
Runs the crash kernel in a reserved area of memory,
so as to leave the panicked kernels memory untouched
for forensic analysis
The crash kernel runs when the running kernel panics
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 14 / 34
Kdump Flow
1. Kdump Load
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 15 / 34
Kdump Flow
1. Kdump Load
2. Kdump Execute
Panic!
Kernel Purgatory Crash
Kernel Execute
Kernel
Segments Boot Param
Kernel Image Kernel Image
Initramfs / Initrd Initramfs / Initrd
Point of Command Line Command Line
No Return Purgatory
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 16 / 34
Triggering Kdump
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 17 / 34
Triggering Kdump
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 17 / 34
VMCore
Visible as /proc/vmcore
from within a crash kernel run by kdump
Includes the entire physical memory,
as visible to the panicked kernel
crash
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 18 / 34
Core Files
Memory
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 19 / 34
Core Files
Memory
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 19 / 34
Core Files
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 19 / 34
Core Files
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 19 / 34
Core Files
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 19 / 34
VMCore Files
Preamble
Kexecs vmcore is also ELF format
Crash Notes
Physical
Memory
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 20 / 34
VMCore Files
Preamble
Kexecs vmcore is also ELF format
Crash Notes
Preamble
Physical
Memory
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 20 / 34
VMCore Files
Preamble
Kexecs vmcore is also ELF format
Crash Notes
Preamble
Then per-CPU crash notes
Physical
Memory
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 20 / 34
VMCore Files
Preamble
Kexecs vmcore is also ELF format
Crash Notes
Preamble
Then per-CPU crash notes
Physical
And then the entire physical memory of the Memory
crashed machine
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 20 / 34
Using Kexec for Crash-Dump Analysis
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 21 / 34
Using Kexec for Crash-Dump Analysis
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 21 / 34
Using Kexec for Crash-Dump Analysis
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 21 / 34
Part III
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 22 / 34
Motivation for Porting Kexec to Xen
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 23 / 34
Motivation for Porting Kexec to Xen
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 23 / 34
Motivation for Porting Kexec to Xen
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 23 / 34
Motivation for Porting Kexec to Xen
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 23 / 34
Motivation for Porting Kexec to Xen
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 23 / 34
Implementation in Brief
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 24 / 34
Implementation in Brief
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 24 / 34
Implementation in Brief
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 24 / 34
Xen Kexec Flow
1. Kexec Load
* Hypervisor Image is only needed if kexecing into Xen, it should be omitted when kexecing into Linux
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 25 / 34
Kexec Flow
1. Kexec Load
2. Kexec Execute * Hypervisor Image is only needed if kexecing into Xen, it should be omitted when kexecing into Linux
Domain 0 Linux
kexectool Execute Hypervsior Purgatory or
Kernel
Segments Boot Param Xen
Hypervisor Image* Hypervisor Image*
Kernel Image Kernel Image
Point of Initramfs / Initrd Initramfs / Initrd
Command Line Command Line
No Return Purgatory
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 26 / 34
Xen Kdump Flow
1. Kdump Load
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 27 / 34
Xen Kdump Flow
1. Kdump Load
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 28 / 34
Xen Kdump Flow
1. Kdump Load
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 29 / 34
VMCore Files for Xen
* Full Disclosure: Both Kazuo Moriwaka and Itsuro Oda are colleagues of mine
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 30 / 34
Implementation Problems: Constants
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 31 / 34
Implementation Problems: Constants
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 31 / 34
Implementation Problems: Constants
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 31 / 34
Implementation Problems: Constants
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 31 / 34
Implementation Problems: Constants
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 31 / 34
Implementation Problems: Crash Notes
Problem
Per-cpu memory reserved at kernel boot time
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 32 / 34
Implementation Problems: Crash Notes
Problem
Per-cpu memory reserved at kernel boot time
Crash note mapping for VMcore is set up at kexec-load time
/sys/devices/system/cpu/cpu*/crash notes
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 32 / 34
Implementation Problems: Crash Notes
Problem
Per-cpu memory reserved at kernel boot time
Crash note mapping for VMcore is set up at kexec-load time
/sys/devices/system/cpu/cpu*/crash notes
Written to at crash-time for each cpu
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 32 / 34
Implementation Problems: Crash Notes
Problem
Per-cpu memory reserved at kernel boot time
Crash note mapping for VMcore is set up at kexec-load time
/sys/devices/system/cpu/cpu*/crash notes
Written to at crash-time for each cpu
When using xen, the domain U kernels see virtual cpus
May only be a subset of physical cpus
Physical to virtual mapping may change at any time
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 32 / 34
Implementation Problems: Crash Notes
Problem
Per-cpu memory reserved at kernel boot time
Crash note mapping for VMcore is set up at kexec-load time
/sys/devices/system/cpu/cpu*/crash notes
Written to at crash-time for each cpu
When using xen, the domain U kernels see virtual cpus
May only be a subset of physical cpus
Physical to virtual mapping may change at any time
Solution
Leave crash notes as-is in the domain 0 kernel
The virtual cpus at the time of kexec-load will be visible in the VMCore
The virtual cpus at the time of crash will be saved, if the domain 0
kernel crashes
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 32 / 34
Implementation Problems: Crash Notes
Problem
Per-cpu memory reserved at kernel boot time
Crash note mapping for VMcore is set up at kexec-load time
/sys/devices/system/cpu/cpu*/crash notes
Written to at crash-time for each cpu
When using xen, the domain U kernels see virtual cpus
May only be a subset of physical cpus
Physical to virtual mapping may change at any time
Solution
Leave crash notes as-is in the domain 0 kernel
The virtual cpus at the time of kexec-load will be visible in the VMCore
The virtual cpus at the time of crash will be saved, if the domain 0
kernel crashes
A separate area is reserved by the hypervisor to save per-cpu
crash-note-like information
Saved regardless of if it is the hypervisor or domain 0 kernel that crash
Visible as ELF sections in the VMCore
But a customised tool is needed anyway
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 32 / 34
Implementation Problems: Crash Notes
Problem
Per-cpu memory reserved at kernel boot time
Crash note mapping for VMcore is set up at kexec-load time
/sys/devices/system/cpu/cpu*/crash notes
Written to at crash-time for each cpu
When using xen, the domain U kernels see virtual cpus
May only be a subset of physical cpus
Physical to virtual mapping may change at any time
Solution
Leave crash notes as-is in the domain 0 kernel
The virtual cpus at the time of kexec-load will be visible in the VMCore
The virtual cpus at the time of crash will be saved, if the domain 0
kernel crashes
A separate area is reserved by the hypervisor to save per-cpu
crash-note-like information
Saved regardless of if it is the hypervisor or domain 0 kernel that crash
Visible as ELF sections in the VMCore
But a customised tool is needed anyway
N.B: This is probably broken if cpus are hotpluged, regardless of Xen
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 32 / 34
Status of the Xen Port
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 33 / 34
Questions?
Horms (VA Linux Systems Japan K.K.) Kexec 18th January 2007 34 / 34