Demystifying systemd

A Practical Guide
Ben Breard, RHCA
Solutions Architect, Red Hat
Lennart Poettering
Senior Engineer, Red Hat

Agenda
systemd 101
Customizing Units
Resource Management
Conerting init scri!ts
"he #ournal

systemd
Controls $units% rather than #ust daemons
Handles de!endency &et'een units(
"rac)s !rocesses 'ith serice in*ormation
Serices are o'ned &y a cgrou!(
Sim!le to con*igure $S+As% &ased on CPU, Memory, and ,-
Pro!erly )ill daemons
Minimal &oot times
.e&ugga&ility / no early &oot messages are lost
Easy to learn and &ac)'ards com!ati&le

RESISTANCE IS FUTILE!

RESISTANCE IS FUTILE!
systemd !
units, systemctl, trou&leshooting

systemd Units
serice(serice
soc)et(soc)et
deice(deice
mount(mount
automount(automount
s'a!(s'a!
target(target
!ath(!ath
timer(timer
sna!shot(sna!shot
slice(slice
sco!e(sco!e

systemd Units " #tt$d%ser&i'e
0Unit1
.escri!tion2"he A!ache H""P Serer
A*ter2remote3*s(target nss3loo)u!(target
0Serice1
"y!e2noti*y
Enironment4ile25etc5syscon*ig5htt!d
E6ecStart25usr5s&in5htt!d 7-P",-8S 3.4-REGR-U8.
E6ecReload25usr5s&in5htt!d 7-P",-8S 3) grace*ul
E6ecSto!25usr5s&in5htt!d 7-P",-8S 3) grace*ul3sto!
9illSignal2S,GC-8"
Priate"m!2true
0,nstall1
:anted;y2multi3user(target <Comments 'ere remoed *or reada&ility

(anaging Ser&i'es " Unit fi)es
,nit
,nit scri!ts= 5etc5init(d > called *rom 5etc5rc<
systemd
Maintainer *iles= 5usr5li&5systemd5system
Administrator *iles= 5etc5systemd5system
8on3!ersistent, runtime data= 5run5systemd
Note* unit *iles under 5etc 'ill ta)e !recedence oer 5usr

(anaging Ser&i'es " Start+Sto$
,nit
serice htt!d ?start,sto!,restart,reload@
systemd
systemctl ?start,sto!,restart,reload@ htt!d(serice

(anaging Ser&i'es " Start+Sto$
,nit
serice htt!d ?start,sto!,restart,reload@
systemd
systemctl ?start,sto!,restart,reload@ htt!d(serice

(anaging Ser&i'es " Start+Sto$
Glo& units 'hen needed(
systemctl restart htt!d mariad&
systemctl ena&le htt!d mariad& nt!d lmAsensors 0etc1
,* a unit ty!e isnBt s!eci*ied, (serice is assumed(
systemctl start htt!d 22 systemctl start htt!d(serice
Shell com!letion is highly recommended
,nstall &ash3com!letion
Add &ash3com!letion to minimal )ic)starts
Connect to remote hosts oer SSH using $3H%

(anaging Ser&i'es " Stat,s
,nit
serice htt!d status
systemd
systemctl status htt!d(serice
"i!= 3l 'onBt truncate the out!ut

(anaging Ser&i'es " Stat,s

(anaging Ser&i'es " Stat,s
"hatBs a little more hel!*ul than=


(anaging Ser&i'es " Stat,s
+ist loaded serices=
systemctl 3t serice
+ist installed serices=
systemctl list3unit3*iles 3t serice Csimilar to ch)con*ig 33listD
Eie' state=
systemctl 33state *ailed

(anaging Ser&i'es " Ena-)e+Disa-)e
,nit
ch)con*ig htt!d ?on,o**@
systemd
systemctl ?ena&le, disa&le, mas), unmas)@ htt!d(serice

Targets .. R,n)e&e)s
$Runleels% are e6!osed as target units
"arget names are more releant=
multi3user(target s( runleelF
gra!hical(target s( runleelG
Eie' the de*ault target ia= Hsystemctl get3de*aultH
Set the de*ault target ia= Hsystemctl set3de*ault 0target1H
Change at run3time ia= Hsystemctl isolate 0target1H
Note* 5etc5initta& is no longer used(

Tro,-)es#ooting
A!!end systemd(unit20target1 to the )ernel
Rescue mode= single, s, S, or 1
Emergency Csimilar to init25&in5&ashD= 3& or emergency
,nteractie &oot a!!end= systemd(con*irmAs!a'n21
Ena&le de&ugging a!!end=
de&ug
de&ug systemd(logAtarget2)msg logA&u*Alen21M
de&ug systemd(logAtarget2console console2ttyS0
htt!=55*reedes)to!(org5'i)i5So*t'are5systemd5.e&ugging5

Tro,-)es#ooting
Early &oot shell on ttyI
systemctl ena&le de&ug3shell(serice
EJuialent to= Hln 3s 5usr5li&5systemd5system5de&ug3shell(serice K
5etc5systemd5system5sysinit(target('ants5H
systemctl list3#o&s
systemd3analyze
Use B&lameB, B!lotB, or Bcritical3chainB *or more details
rc(local is su!!orted, &ut no longer runs last
chmod L6 5etc5rc(d5rc(local
C,stomi/ing Units
dro!3ins

0#at1s a&ai)a-)e22
+ist a unitBs !ro!erties=
systemctl sho' 33all htt!d
Muery a single !ro!erty=
systemctl sho' 3! Restart htt!d
Restart2no
Hel!*ul man *iles= systemd(e6ec and systemd(serice
Restart, 8ice, CPUA**inity, --MScoreAd#ust, +imit8-4,+E, etc
.isclaimer= #ust &ecause you 'an con*igure something doesnBt mean you s#o,)dN

Dro$3ins
1D m)dir 5etc5systemd5system50name(ty!e(d15<(con*
OD im 5etc5systemd5system5htt!d(serice(d5G03htt!d(con*
0Serice1 Remem&er the BSB is ca!italized
Restart2al'ays
CPUA**inity20 1 O F
--MScoreAd#ust231000
FD systemctl daemon3reload
Changes 'ill &e a!!lied on to! o* maintainer unit *iles(

Dro$3ins

Dro$3ins
Hsystemd3deltaH is your *riend(
Sim!le to use 'ith con*iguration tools li)e Satellite, Pu!!et, etc(
Sim!ly delete the dro!3in to reert to de*aults(
.onBt *orget Hsystemctl daemon3reloadH 'hen modi*ying units(
Reso,r'e (anagement
slices, sco!es, serices

Contro) 4ro,$s (ade Sim$)e
Resource Management 'ith cgrou!s can reduce a!!lication or EM
contention and im!roe through!ut and !redicta&ility

Easi)y t#e -est RHEL 5 feat,re t#at no one ,ses

Contain 6o,r A$$)i'ations and 4,arantee Ser&i'e
Con*igure ho' a!!lications,
containers, and EMs 'ill &ehae 'hen
resources are under contention(
"riial to )ee! !oorly 'ritten
a!!lications *rom stom!ing on your
system(
My daughter 'as not harmed during the ma)ing o* this !resentation

S)i'es, S'o$es, Ser&i'es
S)i'e / Unit ty!e *or creating the cgrou! hierarchy *or resource
management(
S'o$e / -rganizational unit that grou!s a sericesB 'or)er
!rocesses(
Ser&i'e / Process or grou! o* !rocesses controlled &y systemd

Understanding t#e Hierar'#y
35
systemd im!lements a standard, single3root hierarchy under
5sys5*s5cgrou!5systemd

Understanding t#e Hierar'#y
35
machine(slice system(slice
user(slice
Each slice gets eJual CPU time on the scheduler(

Understanding t#e Hierar'#y
35
machine(slice system(slice
user(slice
CPUShares210OP
Each slice gets eJual CPU time on the scheduler(
CPUShares210OP
CPUShares210OP

Understanding t#e Hierar'#y
35
machine(slice system(slice
user(slice
user31000(slice
session3F(sco!e
sshd= user
&ash
user31001(slice

Understanding t#e Hierar'#y
35
machine(slice system(slice
user(slice
user31000(slice
session3F(sco!e
sshd= user
&ash
user31001(slice
CPUShares210OP

Understanding t#e Hierar'#y
35
machine(slice system(slice
user(slice
user31000(slice
session3F(sco!e
sshd= user
&ash
tomcat(serice
sshd(serice
mariad&(serice
htt!d(serice
user31001(slice

Understanding t#e Hierar'#y
35
machine(slice system(slice
user(slice
user31000(slice
session3F(sco!e
sshd= user
&ash
tomcat(serice
sshd(serice
mariad&(serice
htt!d(serice
user31001(slice
CPUShares210OP

Understanding t#e Hierar'#y
35
machine(slice system(slice
user(slice
user31000(slice
session3F(sco!e
sshd= user
&ash
tomcat(serice
sshd(serice
mariad&(serice
htt!d(serice
user31001(slice
m1(sco!e
5usr5&in5Jemu
mO(sco!e
5usr5&in5Jemu

Reso,r'e (anagement " systemd3'g)s

Reso,r'e (anagement " systemd3'gto$

Reso,r'e (anagement " Config,ration
Ena&le the desired controllerCsD / CPU, Memory, ;loc),-
Con*igure cgrou! attri&utes=
systemctl set3!ro!erty 33runtime htt!d(serice CPUShares2O0PQ
.ro! $33runtime% to !ersist=
systemctl set3!ro!erty htt!d(serice CPUShares2O0PQ
-r !lace in the unit *ile=
0Serice1
CPUShares2O0PQ
htt!=550!ointer(de5&log5!ro#ects5resources(html

Reso,r'e (anagement 3 CPU
CPUAccounting21 to ena&le
CPUShares / de*ault is 10OP(
,ncrease to assign more CPU to a serice
e(g( CPUShares21R00
htt!s=55'''()ernel(org5doc5.ocumentation5scheduler5sched3design3C4S(t6t

Reso,r'e (anagement 3 (emory
MemoryAccounting21 to ena&le
Memory+imit2
Use 9, M, G, " su**i6es
Memory+imit21G
htt!s=55'''()ernel(org5doc5.ocumentation5cgrou!s5memory(t6t

Reso,r'e (anagement 3 B)7I8
;loc),-Accounting21
;loc),-:eight2 assigns an ,- 'eight to a s!eci*ic serice
CreJuires C4MD
Similar to CPU shares
.e*ault is 1000
Range 10 / 1000
Can &e de*ined !er deice Cor mount !ointD
;loc),-Read;and'idth > ;loc),-:rite;and'idth
;loc),-:rite;and'ith25ar5log GM
htt!s=55'''()ernel(org5doc5.ocumentation5cgrou!s5&l)io3controller(t6t
Con&erting Init S'ri$ts
Sou can do itN ,tBs easyN

Remem&er 'hat init scri!ts loo) li)eT

+et'+init%d+#tt$d
( 5etc5rc(d5init(d5*unctions
i* 0 3* 5etc5syscon*ig5htt!d 1U then
( 5etc5syscon*ig5htt!d
*i
H""P.A+A8G27?H""P.A+A8G3VCV@
,8,"+-GAARGS2VV
a!achectl25usr5s&in5a!achectl
htt!d27?H""P.35usr5s&in5htt!d@
!rog2htt!d
!id*ile27?P,.4,+E35ar5run5htt!d5htt!d(!id@
loc)*ile27?+-C94,+E35ar5loc)5su&sys5htt!d@
RE"EA+20
S"-PA",ME-U"27?S"-PA",ME-U"310@
startCD ?
echo 3n 7VStarting 7!rog= V
+A8G27H""P.A+A8G daemon 33!id*ile27?!id*ile@ 7htt!d 7-P",-8S
RE"EA+27T
echo
0 7RE"EA+ 2 0 1 >> touch 7?loc)*ile@
return 7RE"EA+
@
sto!CD ?
echo 3n 7VSto!!ing 7!rog= V
)ill!roc 3! 7?!id*ile@ 3d 7?S"-PA",ME-U"@ 7htt!d
RE"EA+27T
echo
0 7RE"EA+ 2 0 1 >> rm 3* 7?loc)*ile@ 7?!id*ile@
@
4rom RHE+ R(PU comments remoed

Init " #tt$d 'ontin,ed
reloadCD ?
echo 3n 7VReloading 7!rog= V
i* N +A8G27H""P.A+A8G 7htt!d 7-P",-8S 3t W>5de5nullU then
RE"EA+2R
echo 7Vnot reloading due to con*iguration synta6 errorV
*ailure 7Vnot reloading 7htt!d due to con*iguration synta6 errorV
else
+S;21 )ill!roc 3! 7?!id*ile@ 7htt!d 3HUP
RE"EA+27T
i* 0 7RE"EA+ 3eJ X 1U then
*ailure 7Vhtt!d shutdo'nV
*i
*i
echo
@
case V71V in
startD
start
UU
sto!D
sto!
UU
statusD
status 3! 7?!id*ile@ 7htt!d
RE"EA+27T
UU

Init " #tt$d 'ontin,ed
restartD
sto!
start
UU
condrestartYtry3restartD
i* status 3! 7?!id*ile@ 7htt!d W>5de5nullU then
sto!
start
*i
UU
*orce3reloadYreloadD
reload
UU
grace*ulYhel!Ycon*igtestY*ullstatusD
7a!achectl 7Z
RE"EA+27T
UU
<D
echo 7VUsage= 7!rog ?startYsto!YrestartYcondrestartYtry3restartY*orce3reloadYreloadYstatusY*ullstatusYgrace*ulYhel!Ycon*igtest@V
RE"EA+2O
esac
e6it 7RE"EA+


Systemd " #tt$d%ser&i'e
0Unit1
.escri!tion2"he A!ache H""P Serer
A*ter2remote3*s(target nss3loo)u!(target
0Serice1
"y!e2noti*y
Enironment4ile25etc5syscon*ig5htt!d
E6ecStart25usr5s&in5htt!d 7-P",-8S 3.4-REGR-U8.
E6ecReload25usr5s&in5htt!d 7-P",-8S 3) grace*ul
E6ecSto!25usr5s&in5htt!d 7-P",-8S 3) grace*ul3sto!
9illSignal2S,GC-8"
Priate"m!2true
0,nstall1
:anted;y2multi3user(target <Comments 'ere remoed *or reada&ility

To -e ')ear
Systemd maintains II[ &ac)'ards com!ati&ility 'ith +S;
com!ati&le initscri!ts and the e6ce!tions are 'ell documented(
:hile 'e do encourage eeryone to conert legacy scri!ts to
serice unit *iles, itBs not a reJuirement(
,ncom!ati&ilities are listed here=
htt!=55'''(*reedes)to!(org5'i)i5So*t'are5systemd5,ncom!ati&ilities5
Conerting SysE ,nit Scri!ts=
htt!=550!ointer(de5&log5!ro#ects5systemd3*or3admins3F(html

Unit fi)e )ayo,t " C,stom a$$)i'ation e9am$)e
0Unit1
.escri!tion2.escri&e the daemon
0Serice1
E6ecStart25usr5s&in50mya!!1 3.
"y!e2*or)ing
P,.4ile25ar5run5mya!!(!id
0,nstall1
:anted;y2multi3user(target

EAP E9am$)e
:Unit;
.escri!tion2\;oss Enter!rise A!!lication Plat*orm
:Ser&i'e;
User2#&oss3as
Enironment2\;-SSAUSER2#&oss3as
Enironment2\;-SSAH-ME25usr5local5EAP3R(1(15#&oss3ea!3R(1
Enironment2\;-SSAC-8S-+EA+-G25ar5log5#&ossas5console(log
E6ecStart25usr5local5EAP3R(1(15#&oss3ea!3R(15&in5standalone(sh
P,.4ile25run5#&oss3as5#&oss3as3standalone(!id
Syslog,denti*ier2#&oss3as
+imit8-4,+E210ORPO
Slice2#&oss(slice
:Insta));
:anted;y2multi3user(target
8ote= ,* you donBt de*ine $"y!e2% it 'ill &e $sim!le% &y de*ault

EAP E9am$)e

EAP E9am$)e

Unit fi)e )ayo,t " Test yo,r ,nit fi)e
Co!y the unit *ile
c! 0mya!!1(serice 5etc5systemd5system5
Alert systemd o* the changes=
systemctl daemon3reload
Start serice
systemctl start 0mya!!1(serice
Eie' status
systemctl status 0mya!!1(serice
htt!=550!ointer(de5&log5!ro#ects5systemd3*or3admins3F(html
T#e <o,rna)
#ournalctl 101

<o,rna)
,nde6ed
4ormatted
Errors in red
:arnings in &old
Security
Relia&ility
,ntelligently rotated
htt!=550!ointer(de5&log5!ro#ects5#ournalctl(html

<o,rna)
.oes not re!lace rsyslog in RHE+ X
rsyslog is ena&led &y de*ault
"he #ournal is not !ersistent &y de*ault(
Ena&le !ersistence= Hm)dir 5ar5log5#ournalH
Stored in )ey3alue !airs
#ournalctl 0ta&1 0ta&1
Man X systemd(#ournal3*ields
Collects eent metadata along 'ith the message
Sim!le to *ilter
,nterleae units, &inaries, etc(

<o,rna) " =o,rna)'t)

Using t#e <o,rna)
"ail the #ournal= H#ournalctl 3*H
Sho' ] num&er o* lines= H#ournalctl 3n G0H
Eie' *rom &oot= H#ournalctl 3&H
4ilter &y !riority= H#ournalctl 3! 0leel1H
0 emerg
1 alert
O crit
F err
P 'arning
G notice
R de&ug

Using t#e <o,rna)
-ther use*ul *ilters=
3r reerse order
3u 0unit1
&inary e(g( 5usr5s&in5dnsmasJ 0additional &inaries1
33since2yesterday or SSSS3MM3.. CHH=MM=SSD
33until2SSSS3MM3..
Eie' entire #ournal
#ournalctl 3o er&ose Cuse*ul *or gre!D

Systemd Reso,r'es
RHE+ X documentation=
htt!s=55access(redhat(com5site5documentation5RedAHatAEnter!riseA+inu65
Systemd !ro#ect !age=
htt!=55'''(*reedes)to!(org5'i)i5So*t'are5systemd5
+ennart PoetteringBs systemd &log entries= Cread them allD
htt!=550!ointer(de5&log5!ro#ects5systemd3*or3admins31(html
Red Hat System Administration ,, > ,,, CRH1FP5RHOGPD
htt!=55redhat(com5training5
Systemd 4AM
"i!s > "ric)s

$MuestionsT%
THAN> 68U