ZCS6 Day 2 Slides

6.
DAY 2
Zabbix 6.0 Certified Specialist ● Day 2 © 2022 by Zabbix. All rights reserved
COPYRIGHT NOTICE
Rules
It is prohibited to make any video and/or audio recordings during the whole period of this course.
This course is intended only for the officially enrolled student. Subject to the Copyright Notice below, the student is not allowed to
share his credentials for attending this course, to allow others to join and take part, or otherwise make use of these Materials.
Copyright notice
© Zabbix, 2022. All rights reserved.
Unless otherwise indicated, Zabbix owns the copyright and other intellectual property rights in the text, graphics, information,
designs, data, verbal/audio/video presentations and files, comments, drawings, exam questions and exam answers, and other
training content, lab manuals and practical tasks, and training courses themselves (further – Materials).
The Materials are protected by watermarks, copyright statements, and other means. It is prohibited to remove any of watermarks
and copyright statements, or in any other way to amend or change the content or appearance of the Materials.
Any unauthorized reprint, publication, reproduction, sharing, or use of the Materials is prohibited. No part of the Materials may be
reproduced, transmitted, or published in any form or by any means, electronic or mechanical, including photocopying, recording,
or by any information storage and retrieval system without the express signed written permission from Zabbix.
All course Materials made available to the student during the course of the training may be used solely by the student enrolled in
the relevant course for personal and educational purposes only. Materials provided to the student should be treated as
confidential information shared with the student only for the purpose of the student performing Zabbix Certified training.
The student acknowledges that damages alone would not be an adequate remedy for the breach of this copyright and the student
shall be entitled to the granting of equitable relief concerning any threatened or actual breach of any of the provisions of this
Copyright notice.
Zabbix 6.0 Certified Specialist ● Day 2 © 2022 by Zabbix. All rights reserved Theory 2
AGENDA
Zabbix agent Zabbix agent
User parameters
passive checks active checks
Human-readable
Update intervals Preprocessing
data
Triggers Problems Trigger functions
Zabbix 6.0 Certified Specialist ● Day 2 © 2022 by Zabbix. All rights reserved 3
Zabbix agent passive checks
Zabbix 6.0 Certified Specialist ● Day 2 © 2022 by Zabbix. All rights reserved 15 minutes 4
PASSIVE CHECKS
A passive check is a simple data request:
Zabbix server or proxy asks for a single metric
Zabbix agent gets the request and sends back the result
system.cpu.load = ?
0.57
Data from Zabbix passive agent is collected by the poller process:

Start more pollers on the server or proxies if you have a lot of passive checks
### Option: StartPollers

# Number of pre-forked instances of pollers.
# Range: 0-1000
StartPollers=20
FRONTEND SETTINGS
Settings for connecting to a passive agent are defined in Zabbix frontend:
Add one or multiple interfaces with the type "Agent" on a host
Specify IP address or DNS name of the target host
Specify proxy server if the host will be monitored through a proxy

Host name does not matter at all for passive checks
The host availability icon will become green if at least one passive check will succeed
SERVER PARAMETER
Zabbix agent needs to be configured to accept requests from Zabbix server:
Server parameter must match Zabbix server or proxy address
Multiple IP addresses, IP address ranges or DNS names can be specified
Specifying 0.0.0.0/0 will accept connection from any address (very insecure setting!)
### Option: Server

# List of comma delimited IP addresses, or DNS names of Zabbix servers and Zabbix proxies.
# Incoming connections will be accepted only from the hosts listed here.
Server=192.168.0.5
Agent will reject connection if the address of Zabbix server or proxy will not match
system.cpu.load = ?
Error: agent dropped connection

because of access permissions.
IP=192.168.0.5 Server=127.0.0.1
AGENT PORT AND PROCESSES
Zabbix passive agent listens to port 10050 by default:
Different port can be specified using ListenPort parameter:
### Option: ListenPort

# Agent will listen on this port for connections from the server.
# Range: 1024-32767
ListenPort=10050
Multiple passive agent processes can be started at once:

Each process can answer only one request from a poller at a time
More processes may be required if some of the checks are very slow
Setting StartAgents parameter to 0 will disable the passive mode for Zabbix agent
### Option: StartAgents
# Number of pre-forked instances of zabbix_agentd that process passive checks.
# If set to 0, disables passive checks and the agent will not listen on any TCP port.
## Range: 0-100
StartAgents=3
TIMEOUTS
Zabbix server/proxy and agent timeouts both have effect on passive checks:
Zabbix server timeout is the connection timeout
Zabbix agent timeout is the service timeout
Timeouts can be adjusted in the server/proxy and agent configuration files

Increasing timeouts unreasonably may degrade monitoring performance
Pollers can become much busier waiting for unresponsive checks with large timeouts
I will wait only I have only 3 seconds

4 seconds to collect metric
for an answer and send it back
system.sw.packages = ?
Timeout=4s Timeout=3s
ZABBIX GET UTILITY
Command line utility zabbix_get may be used to connect to a passive agent:
Can be used to troubleshoot agent availability or item keys
Can be used in custom scripts to collect the data
Utility is standalone and can be installed on any Linux/Windows workstation
Additional parameters can be displayed using zabbix_get --help command
The agent must accept connections from the machine executing zabbix_get
# zabbix_get -s prod-server.example.com -k agent.version
6.0.0
agent.version = ?
6.0.0
PRACTICAL SETUP
1) Create a new host
Host name: Training-VM-XX
Groups: Training/Servers
Interface: Type: Zabbix agent DNS: student-xx.zabbix.training
2) Create an item on the Training-VM-XX host with 10 seconds update interval:
CPU load
3) Create two items on the Training-VM-XX host with 1 minute update interval:
System uptime
Free memory size
4) Create an item on the Training-VM-XX host with one hour update interval
Total memory size
5) Force the data collection by pressing "Execute now"
Make sure that the items receive data
6) Install zabbix_get utility from official Zabbix repository
List installed Zabbix packages using zabbix_get utility
Zabbix 6.0 Certified Specialist ● Day 2 © 2022 by Zabbix. All rights reserved 20 minutes Practical task No: 8 11
Zabbix agent active checks
ACTIVE CHECKS
In active mode, Zabbix agent initializes connection to Zabbix server or proxy:
Active agent requests configuration by sending its Hostname
Server responds with configuration (item keys and update intervals)
Agent collects metrics using received configuration and sends them back at regular intervals
hostname=Prod server
configuration = {....}
system.cpu.load = 0.75
net.if.in[eth0] = 143535
Data from active agents is accepted by the trapper process on port 10051:
Start more trappers on server or proxies if you have a lot of active agents
### Option: StartTrappers
# Number of pre-forked instances of trappers.
# Range: 0-1000
StartTrappers=5
FRONTEND SETTINGS
Settings for an active agent in Zabbix frontend:
Only Host name field is used to accept incoming requests from active agents
Visible name is not used!
Only this field is used
None of the interfaces (even if defined) are used for active checks
Active agent does not have any interface in the item configuration
Passive agent Active agent
ACTIVE AGENT CONFIGURATION
The two most important parts in active agent configuration are:
Hostname The hostname must match hostname in Zabbix frontend
ServerActive IP address of Zabbix server or proxy
### Option: Hostname
# List of comma delimited unique, case sensitive hostnames.
# Required for active checks and must match hostnames as configured on the server.#
Hostname=Production server
### Option: ServerActive

# List of comma delimited IP / DNS pairs of Zabbix servers and Zabbix proxies
ServerActive=192.168.0.5
Hello 192.168.0.5, Hello "Production

my hostname is server", here is your
"Production server" configuration!
ServerActive=192.168.0.5 IP=192.168.0.5
SPECIFY THE HOSTNAME
There are three ways how to specify a hostname:
Hostname just write a hostname into the configuration file
If multiple hostnames are specified, the agent will send values to all hostnames defined here
### Option: Hostname
# List of comma delimited unique, case sensitive hostnames.
# Required for active checks and must match hostnames as configured on the server.#
HostnameItem dynamically creates hostname from the agent key

Used only if Hostname is undefined
### Option: HostnameItem
# Item used for generating Hostname if it is undefined. Ignored if Hostname is defined.
HostnameItem=system.run[hostname.sh]
Comment out both options and use system hostname from the OS instead
The value of system.hostname item is used (without a domain name)
#Hostname=Production server
#HostnameItem=system.hostname
SERVER ACTIVE ADDRESSES
ServerActive specifies the addresses of Zabbix servers
Usually just one Zabbix server address is used
Hostname=Production server IP=192.168.0.5

If more than one address is specified, the same Zabbix agent will report data to all Zabbix
servers listed here
ServerActive=10.10.10.5,192.168.0.5
Zabbix Server 1 Hostname=Production server Zabbix Server 2

IP=10.10.10.5 ServerActive=192.168.0.5,10.10.10.5 IP=192.168.0.5
ACTIVE CHECK PROCESSING
Active checks are processed by Zabbix agent and values are sent back:
Agent retrieves a list of items from Zabbix server for independent processing:
On agent start
Every 120 seconds when running (by default)
### Option: RefreshActiveChecks

# How often list of active checks is refreshed, in seconds.
## Range: 60-3600
RefreshActiveChecks=120
Collected values are sent to Zabbix server or proxy in bulk:

Timestamps of data collection time using the host clock are preserved when sending data
Inaccurate host clock may cause false alarms
Metrics are sent to Zabbix server using JSON format
Zabbix Agent 2 is compressing the data
Zabbix agent in active mode collects all metrics immediately after start / restart
MEMORY BUFFER
Zabbix agent in active mode is storing collected values in a memory buffer:
SQLite can be used as a buffer for Zabbix Agent 2 (optionally)
Collected values are sent in bulk to Zabbix server or proxy:

Every BufferSend seconds (5 by default)
#### Option: BufferSend
# Do not keep data longer than N seconds in buffer.
# Range: 1-3600
BufferSend=5
If the buffer specified by BufferSize is full in less than BufferSend seconds

### Option: BufferSize
# Maximum number of values in a memory buffer. The agent will send
# all collected data to Zabbix Server or Proxy if the buffer is full.
# Range: 2-65535
BufferSize=100
AVAILABILITY ICON
Active checks do not have any availability icon:
Active agent does not use or require the agent interface!
No interface is defined
The availability icon will be gray if an interface is defined but not used by any passive item
Interface is defined but not used
PRACTICAL SETUP
1) Configure Zabbix agent for active checks
2) Full Clone the host "Training-VM-XX"
Host name: student-XX (training VMs hostname)
Visible name: Training-VM-XX active checks
3) Change the type of all items on this host to Zabbix agent (active)
4) Remove Zabbix agent host interface
5) Change the Hostname in Zabbix agent configuration file and restart agent
6) Make sure that the items receive data
Zabbix agent
troubleshooting
TYPICAL MISTAKES
Wrong Hostname defined in the agent configuration is a typical problem:
Metrics are collected, but sent to a different host configured in Zabbix frontend
Technically, the monitoring is working properly as configured
Default Hostname on new agent installations is "Zabbix server"!
Thank you, I will store

Hello, here are some
the metrics into
data, my hostname is
"Development server"
"Development server"
host
Production server Zabbix Server

Hostname=Development server
TYPICAL MISTAKES
The host is not defined on Zabbix server at all:
Visible name field is not used for identification, only the Host name
Both Zabbix agent and server will complain about non existing hosts in the log files:
cannot send list of active checks to "192.168.1.5": host [Production server] not found
Hello, my hostname is
Sorry, I do not have
"Production server" and
any configuration for
I need monitoring
"Production server"
configuration
Configuration request
Production server Zabbix Server

TYPICAL MISTAKES
The host is monitored by proxy, but Zabbix server address is used in
ServerActive:
When moving hosts between proxies, do not forget to change ServerActive
Multiple ServerActive entries are possible, but they will create a lot of errors in logs
Hello 10.10.10.1,
please send me Sorry, I am
monitoring not monitoring
configuration "Production server"
Configuration request
Production server IP=10.10.10.1

IP=10.10.10.5
TYPICAL MISTAKES
The same Zabbix server is specified multiple times in ServerActive
It is not possible to specify the same IP or DNS name twice, but:
IP and DNS may match the same host
Multiple DNS names may match the same host
Zabbix agent will think these are separate Zabbix instances
Zabbix will send the same configuration multiple times
Agent will send back metrics to each instance, which will result in duplicated values
Production server Zabbix server

IP=10.10.10.1
ServerActive=10.10.10.1,zabbix.example.com
DNS=zabbix.example.com
Values are
duplicated
TIMESTAMP OF COLLECTED VALUE
Local UNIXTIME is used to timestamp all collected values:
Passive checks are timestamped by Zabbix server or proxy
agent.version = ?
Timestamp Value
6.0.0
1642318203 6.0.0
local time=2022.01.16 09:30:03 local time=2022.01.16 09:29:45

UNIXTIME = 1642318203 UNIXTIME = 1642318185
Active checks are always timestamped by Zabbix agent
agent.version = 6.0.0 Timestamp Value

timestamp = 1642318185
1642318185 6.0.0
local time=2022.01.16 09:30:03 local time=2022.01.16 09:29:45

UNIXTIME = 1642318203 UNIXTIME = 1642318185
User parameters
USER PARAMETERS
UserParameter allows to run a check that does not come predefined with Zabbix:
Shell commands
Custom scripts
All commands are executed from the OS user which is used to run agent:
The user must have enough permissions to execute the command or script
The return value of the command is the standard output together with the standard error
UserParameter must be configured for every agent where they will be used:
Directly in zabbix_agentd.conf file
Included from zabbix_agentd.d directory (recommended)
### Option: UserParameter
# User-defined parameter to monitor. There can be several user-defined parameters.
#
UserParameter=mykey,echo 1
DEFINING USER PARAMETERS
More than one UserParameter can be defined per agent:
Multiple include files can be specified with different sets of parameters
All keys per agent must be unique or Zabbix agent will exit with the error message:
ERROR: cannot add user parameter "mysql.qps,mysqladmin status : key "mysql.qps" already exists
Working directory for commands can be specified using UserParameterDir:

UserParameter commands can be specified using the relative ./ prefix
### Option: UserParameterDir
# When executing UserParameter commands the agent will change the working directory to the one
# specified in the UserParameterDir option.
UserParameterDir=/home/zabbix/workingdir
Example: a script is located at /home/zabbix/workdir/myscript.sh

UserParameter=mykey,./myscript.sh
RELOAD LIST OF USER PARAMETERS
Zabbix agent will be able to use the new item key only after it is loaded into
configuration
To reload the list of user parameters two methods are available:

Restart Zabbix agent to re-read the entire configuration file:
# systemctl restart zabbix-agent
Use Zabbix agent runtime command userparameter_reload:

Works for both Agent and Agent2
Only user parameters are affected, other configuration changes are ignored
Not supported for zabbix_agentd on OpenBSD, NetBSD and Windows
# zabbix_agentd -R userparameter_reload
zabbix_agentd [1651700]: command sent successfully
USER PARAMETER SYNTAX
A user parameter has the following syntax: UserParameter=key,command
The key will be used by the item, any unique key name can be entered
The command will be executed when the key is requested
UserParameter can be simple or flexible:

Simple: UserParameter=mysql.qps,mysqladmin status | cut -f9 -d":"
Flexible: UserParameter=mysql.ping[*],mysqladmin -u$1 -p$2 ping | grep -c alive
UNSAFE USER PARAMETERS
Certain symbols can not be passed as arguments by default:
\'"`*?[]{}~$!&;()<>|#@
Additionally, newline characters are not allowed
Can beUnsafeUserParameters
### Option: allowed by setting UnsafeUserParameters=1 in zabbix_agentd.conf file
# Allow all characters to be passed in arguments to user-defined parameters.
# The following characters and newline characters are not allowed:
# \ ' " ` * ? [ ] { } ~ $ ! & ; ( ) < > | # @
# Range: 0-1
UnsafeUserParameters=1
SLOW USER PARAMETERS
Slow UserParameters can delay other metric collection:
They will respect timeouts defined in the server and agent configuration
When used as a passive check:
Pollers will become busy waiting for new values, more pollers may be needed
More passive agent processes may be needed, because some will be utilized for long time
I need to wait poller busy 20 seconds

for answer
from agent
When used as an active check:

C-based active agent has only one data collector process - slow checks will delay other metrics
Agent 2 is not affected by this
Other metrics can
wait until I
20 seconds finish this one
data missing
PRACTICAL SETUP
1) Add a simple user parameter:
Name: MySQL Uptime (simple)
Key: mysql.uptime.s
Command line: mysqladmin -uzabbix -pP455w0RD status 2>/dev/null | awk '{print $2}'
2) Restart Zabbix agent
3) Add a new item to Training-VM-XX host to monitor this user parameter
4) Add a flexible user parameter:
Name: MySQL Uptime (flexible)
Key: mysql.uptime.f[*]
Command line: mysqladmin -u$1 -p$2 status 2>/dev/null | awk '{print $$2}'
5) Add the user parameter to Zabbix agent without restarting it
6) Add a new item to Training-VM-XX host to monitor this user parameter
Use macros to specify the username
Use secret macro to specify the password
7) Make sure both items are receiving data
Human-readable data
UNITS
Unit symbols are used for a human-readable representation of data:
Can be used with numerical data only (unsigned / float)
If units are defined, K/M/G/T/P/E/Z/Y prefix fill be used
Stored in database Units Displayed in frontend

5000 Hz 5 KHz
5341623 bits 5.34 Mbits
3765986812 W 3.76 GW
Special processing is used to display B, Bps, unixtime, uptime, s units:

1024 B 1 KB
125 uptime 00:02:05
1630316392 unixtime 2021.08.30 09:39:52
614 s 10m 14s
UNIT CONVERSION OPTIONS
Any unit can be prevented from being converted by using a ! prefix
5000 Hz 5 KHz
5000 !Hz 5000 Hz
614 s 10m 14s
614 !s 614s
1500 ms 1.5 Kms
1500 !ms 1500 ms
Try to store base units (bytes, seconds, etc.) to avoid strange results in the
frontend:
Data can be converted using preprocessing steps if needed (discussed later)
3200 MB 3.2 KMB
3200 !MB 3200 MB
3435972078 B 3.2 GB
VALUE MAPS
Value maps contain the mapping between raw values and string representations:
Used both by Zabbix frontend and notifications
Value maps are defined on the Host or Template level
Available value map can be specified in the item configuration form
Can be used with items having type of information:
Numeric (unsigned)
Numeric (float)
Character
VALUE MAP INHERITANCE
Value maps do not have inheritance from templates:
They can be defined on both template or host level
Linking a template to the host does not make the host inherit the value maps:
A local item on a host can use only the value map from the host
A templated item on a host will use the value map from the template
TEMPLATE HOST
Value map X Value Map Y
Item A Template: Item A
Item B Template: Item B

This item may
Item C use Value Map Y
only
! It is not possible to create a template with a value map and use it for host items!
APPLYING VALUE MAPS
Usually, multiple mappings are defined per value map:
Mapping is applied according to the order of mapping rules
If multiple mappings overlap, the first match will be applied
It is possible to reorder mappings by dragging
Supported mapping types:

equals - equal values will be mapped
is greater than or equals - equal or greater values will be mapped
is less than or equals - equal or smaller values will be mapped
in range - values in range will be mapped. Multiple ranges are supported (e.g., 1-10, 20 -25)
regexp - values corresponding to the regular expression will be mapped
default - all outstanding values will be mapped, other than these with specific mappings
VALUE MAP OVERLAP
Multiple value mappings can overlap:
Mapping is applied according to the order of mapping rules (first match)
It is possible to reorder mappings by dragging
"220" will be mapped as "Normal" "220" will be mapped as "Perfect"
The last mapping will be

Mappings are reordered
ignored because "Normal"
properly
range is matched first
PRACTICAL SETUP
1) Add corresponding units to the items on Training-VM-XX host
System uptime s
Total memory size B
Free memory size B
MySQL uptime (use s or uptime units)
2) Create a new value map for Training resources host

Name: Web performance
Mappings:
0 for service down
0 - 100 for fast response time
> 100 for slow response time
3) Improve "Service Web performance" item

Assign the value map " Web performance"
Add seconds as item unit
4) Add value maps to ICMP ping status and Service NTP availability items
5) Check the results
Update intervals
UPDATE INTERVAL
Retrieve a new value for this item every N seconds:
Minimum update interval is 1 second
Maximum allowed update interval is 1 day
Time suffixes are supported, e.g., 30s, 1m, 2h, 1d
A user macro can be used in this field
Defaults to seconds if a suffix is not used (600 = 10m)
Can be set to 0 (never checked) if a custom interval exists with a non-zero value
Update interval cannot be set for the following item types:

Zabbix trapper
SNMP trapper
Dependent items
UPDATE INTERVAL
Update interval specifies the time between item checks:
All checks are distributed equally over time
This design is implemented to avoid data collection spikes
Two items have the same update interval 10 seconds:

Incoming traffic is monitored every 04, 14, 24, 34, 44, 54 second
Outgoing traffic is monitored every 00, 10, 20, 30, 40, 50 second
The next item will have different timings
10 seconds 10 seconds
time
Incoming traffic 08:17:24 08:17:34 08:17:44 08:17:54
Outgoing traffic 08:17:30 08:17:40 08:17:50 08:18:00
FLEXIBLE INTERVALS
Flexible intervals allow to redefine the default update interval for specific time
periods:
Interval the update interval for the specified time period
Period the time period when the flexible interval is active
Multiple flexible intervals can be defined:

Outside the flexible intervals the default update interval is used.
If multiple intervals overlap, the smallest interval value is used for the overlapping period
If the flexible interval is set to '0', the item is not polled during the flexible interval
! Zabbix server time zone is used to specify flexible intervals
SCHEDULED INTERVALS
Scheduling intervals are used to additionally check items at specific times:
Independent checking schedule, which is executed in parallel to "Update interval" checks
If update interval is set to 0, only scheduled checks will be executed
Zabbix server time zone is used
Scheduled
check
10 seconds
time
06:59:34 06:59:44 06:59:54 07:00:04
07:00:00
SCHEDULED INTERVALS
A scheduling interval is defined as:
md<filter>wd<filter>h<filter>m<filter>s<filter>
md month days 1-31
wd week days 1-7
h hours 0-23
m minutes 0-59
s seconds 0-59
<filter> is defined as: [<from>[-<to>]][/<step>]
Examples:
wd1-5h9 - every Monday till Friday at 9:00
h9m/30;h10 - execute at 9:00, 9:30, 10:00
h9-10m10-40/30 - execute at 9:10, 9:40, 10:10, 10:40
md1wd1h9m30 - every 1st day of each month at 9:30 if it is Monday
! Massive use of scheduled items can cause big spikes on monitored environment!
CUSTOM INTERVALS
Custom intervals can be used:
For all passive checks
As active checks only for Zabbix agent 2
Update interval, flexible intervals and scheduled checks can be combined:

Update interval 0 is supported only in the combination with Custom intervals
Flexible intervals will override update interval
Scheduled checks will be executed in addition to update/flexible intervals
EXECUTE NOW
Pressing Execute now will perform an extra item check on demand:
The actual data collection may be slightly delayed (a few seconds) after pressing the button
The value will be collected and processed as usual
Will work locally or through active/passive proxies
Can be executed for multiple items at once
Active items (Zabbix agent active, trapper, etc.) cannot be forcibly executed
Status of the execution request will be reported (This does not mean successful execution!)
Execute
now
time
06:20:00 06:30:00 06:40:00 06:50:00
06:45:03
PRACTICAL SETUP
1) Define flexible update intervals for the CPU load item
Nighttime (00:00 - 06:00) once per minute
Weekends (Saturday, Sunday) once per 5 minutes
2) Create a new item to monitor swapped in pages with scheduled interval
Once per hour exactly at the beginning of the hour
Preprocessing
PREPROCESSING
Preprocessing allows to define transformation rules for the received item values:
One or several transformations are possible
Transformations are executed in the order in which they are defined
Preprocessing can be performed by:

Zabbix server
Zabbix proxies (if a host is monitored by proxy)
Monitored Collected Preprocessing Transformed Database

device data data
PREPROCESSING STEPS
Preprocessing steps are defined in a separate tab of the item configuration form:
The number of steps is displayed in the tab
The order of execution can be changed by dragging and dropping
Type of information for the item is also displayed:

It is possible to change the type of information without leaving the Preprocessing tab
CUSTOM ON FAIL
If a preprocessing step fails multiple scenarios are possible:
Error handling can be introduced using "custom on fail" scenarios
Otherwise, the item will become unsupported, and no more steps will be processed
Custom on fail offers three options:

Discard value The value is discarded, and no more steps are processed
Set value to The value will be changed to the predefined value
Set error to The item will become not supported with a custom error
! If value is discarded, next preprocessing steps are ignored!
PREPOCESSING INTERNALS
Every collected value goes through preprocessing queue:
Data are passed to preprocessing manager and placed into preprocessing queue
Preprocessing manager assigns tasks from the preprocessing queue to workers
Preprocessing workers are performing the preprocessing steps
### Option: StartPreprocessors

# Number of pre-forked instances of preprocessing workers.
# The preprocessing manager process is automatically started when preprocessor worker is started.
# Range: 1-1000
StartPreprocessors=10
Preprocessing steps
CUSTOM MULTIPLIER
Multiply the value by the specified integer or floating-point value:
User macro can be used to specify the multiplier
Scientific notation is supported (e.g. 1e+70)
Division is also possible if multiplier < 1 is used:
24 * 0.125 = 3
Notes for Numeric (unsigned) item type:

Incoming values with a fractional part will be trimmed before multiplying
The result will also be trimmed before saving
numeric (unsigned) 2.5 * 2.9 = 2 * 2.9 = 5.8 = 5

numeric (float) 2.5 * 2.9 = 7.25
SIMPLE CHANGE
Simple change calculates the difference between the current value and the
previous value:
Formula: value - previous value 135 - 80 = 55
Speed per second calculates the result based on the two values and time:
value − previous value 135 − 80 55

Formula: = = 5.5
time − previous time 1600329600 − 1600329590 10
A few important notes:

These steps can only be used for a constantly growing value
Zabbix stores nothing if the current value is smaller than the previous value
Only one change operation per item is allowed
SPEED PER SECOND
The following Zabbix agent keys report back metrics as counters:
net.if.in, net.if.out, net.if.total
system.cpu.intr, system.cpu.switches
system.swap.in, system.swap.out
Many other data collectors (e.g. SNMP) can also report data as counters
Without change per second With change per second
preprocessing step preprocessing step
REGULAR EXPRESSIONS
Regular expression captures the matched text into a numbered group:
Zabbix supports PCRE (Perl Compatible Regular Expressions) format
Both PCRE and PCRE2 libraries are supported starting from Zabbix 6.0
Regular expression supports extraction of maximum 10 captured groups

An \N (where N=1…9) escape sequence is replaced with the Nth matched group
\0 escape sequence is replaced with the matched text
Source data Regular Expression
total used free

Mem: 1827 840 75 Mem:\s+(\d+)\s+(\d+)\s+(\d+)
Capturing
groups \0 = Mem: 1827 840 75 \1 = 1827 \2 = 840 \3 = 75
REGULAR EXPRESSIONS
In Zabbix frontend, two fields must be specified:
Pattern - regular expression
Output - capturing group Regex pattern Capturing group
Result can be
tested from
the frontend
DISCARD UNCHANGED
Discard unchanged
Time Values Discard unchanged Explanation
with heartbeat 30s
00:00 0 0 0
00:05 0 No value because same as previous
00:10 0
00:15 1 1 1 Received different value
00:20 1
00:25 1
00:30 1
00:35 1
00:45 0
00:50 0
00:55 0
30s
01:00 0
01:05 0
01:10 0 0 Value written because of heartbeat 30s
01:15 0
01:25 1
01:35 0
01:40 0
30s
01:45 0
01:50 0
01:55 0
02:00 0 0 Value written because of heartbeat 30s
02:05 0
CHECK FOR NOT SUPPORTED VALUE
Check if there was an error in collecting item value:
Normally that would lead to the item turning unsupported
Modify that behaviour by specifying the "Custom on fail" error-handling options
This step is always executed as the first preprocessing step
Timeout 0 0
Check for not Custom

supported value multiplier 8
PREPROCESSING COURSES
Preprocessing steps are covered in other Zabbix courses:
Zabbix 6.0 Certified Professional course:
XML Xpath
JSONPath
Prometheus patterns
Conversion to JSON
A specialized one-day Preprocessing course:

How preprocessing works
All preprocessing steps in much more details
i https://www.zabbix.com/training?language=english#training_courses
PRACTICAL SETUP
1) Create two new items on Training-VM-XX host to monitor network traffic:
Incoming network traffic
Outgoing network traffic
2) Assign preprocessing steps to the items:
Monitor speed per second
Convert bytes to bits
3) Check that items are reporting correct data
4) Update "swapped in pages" item to report pages per hour
Not supported items
NOT SUPPORTED ITEMS
Items may become "Not supported" under multiple scenarios:
On data collection time:
Wrong item key or parameters
Timeout, communication error
Wrong credentials or permission problems
Data collection errors can be corrected by "check for not supported value" preprocessing step
In preprocessing steps
Failed regular expression, XML Xpath or JSONPath, incorrect Prometheus pattern
Arithmetical operations with character data
If a step fails, no more steps will be processed
Use "custom on fail" options to override errors
When writing into database

Wrong value type (writing text into numerical field, for example)
NOT SUPPORTED ERROR CORRECTION
Data collection Preprocessing Writing into DB
Error can be fixed with Error can be fixed with Error cannot be fixed
"Check for not supported "Custom on fail" option
item" preprocessing step
Unsupported item key Preprocessing step failed Cannot write char to integer
Timeout
Wrong password
NOT SUPPORTED ITEMS
Not supported items may degrade Zabbix performance:
Not supported items will be checked on every item update interval
Data collectors will be busy trying to collect nonworking items
Errors like timeouts may keep data collectors busy for significant time
Not supported items can be troubleshooted using Zabbix frontend:

By clicking on the error message in the items part of host configuration form
By using "Test item" form to test the data collection and preprocessing steps
Not supported items may be rechecked by pressing Execute now

Do not forget to update configuration cache before testing results
An item will become supported immediately if a correct value will be reported
Item testing
ITEM TESTING
Zabbix can test any passive item before creating it:
Item key and other required parameters must be entered correctly
Testing is performed by Zabbix server/proxy or agent same as in a real monitoring setup
A test can be executed before creating the item
Only testing of the item configuration will be performed, the result is not stored
Received value is not saved into the database
Problem detection is not performed with this value
Test request
Result
ITEM TESTING
Testing can be performed by pressing Test button on the Item configuration
form
The button will be grayed out for item types that cannot be tested (active checks, icmp
ping)
Test values can be entered manually or received from the host
Get value will only get a raw value
Get value and test will emulate writing the value to the database
Get value from host Enter value manually
PREPROCESSING TESTING
Preprocessing steps also can be tested:
Using values received from the host or entered manually (will be preserved)
For some preprocessing steps, two values are required
Tests are performed by Zabbix server and will match actual results
Results for each step
Final result
Click on error
for details
Triggers
TRIGGER DEFINITION
Triggers are logical expressions that "evaluate" data gathered by items:
Trigger expressions allow to define a threshold of what state of data is "not acceptable"
avg(/production server/system.cpu.load,5m)>2
If the incoming data surpass the acceptable state, a trigger changes status to PROBLEM
Trigger status Description

OK A normal trigger state
PROBLEM Problem is detected by evaluating data
The expression may include a combination of multiple functions and multiple thresholds
Most trigger functions are evaluated based on history data
Some trigger functions for long-term analytics use trend data (trendavg(), trendcount(), etc.)
TRIGGER EXPRESSIONS
Trigger expressions have new syntax in Zabbix 6.0:
Expression uses a function that is applied to the item with some parameters
The function result is compared to the threshold, using an operator and a constant.
function(/host/key,<additional parameters>)<operator><constant>
function ( /host /key , parameters ) operator constant

last ( /Zabbix server /system.uptime ) < 10m
min ( /Linux by Zabbix agent /system.cpu.load[all] , 5m ) >= {$CPU.LOAD}
avg ( /Production server /vm.memory.utilization , {$PERIOD} ) > 1G
The referenced item key must exist on the host or template

The referenced item must be in a supported state for the trigger to be calculated
nodata() function is the exception here
CREATING TRIGGER
Triggers are created directly on a host or on a template (recommended):
Go to Configuration -> Hosts / Templates
Click on Triggers for selected Host / Template
Press Create trigger or click on an existing trigger name to update the configuration
Two fields are *mandatory:

Trigger name
Trigger expression
Their combination must be unique per host
Two tabs with additional properties:

Tags for trigger-level tags
Dependencies for trigger dependencies
TRIGGER NAME
Trigger name:
Trigger name is used to generate name of the Problem
If trigger name is changed, it will not affect already created problems
Not recommended to include host name or latest values in the trigger name, because:
Host name is already displayed in the Host field (sortable)
Latest values are already displayed in the Operational data field
Examples:
/etc/passwd has been changed
Getting closer to process limit (over 80% used)
Zabbix agent is not available (for 3m)
Host Trigger name Latest values
MACROS IN TRIGGER NAME
Using macros in the trigger name can create dynamic problem names:
Supported macros:
{HOST.HOST}, {HOST.NAME}
{HOST.DNS}, {HOST.IP}, {HOST.CONN}, {HOST.PORT}
{ITEM.VALUE}, {ITEM.LASTVALUE}, {ITEM.LOG*}
User macros {$MACRO}
Examples
Zabbix agent is not available for {$AGENT.TIMEOUT}
{HOST.NAME} has been restarted (uptime < {$MIN.HOST.UPTIME})
Duplicated
host name
EVENT NAME
Event name can redefine the problem name generated by a trigger:
By default, the event name is the same as the trigger name
Can be used to build detailed problem names containing aggregated data
The same set of macros is supported as in the trigger name, plus:
{TIME} Current time in hh:mm:ss format
{?EXPRESSION} Expression macros
Event name
Problem name is redefined
EXPRESSION MACROS
Expression macros have the following syntax
{?expression}
Expression macro can have an expression containing:

Any of the supported functions (last, min, max, avg, etc.)
Any item key which exists on some existing host or template by using /host/item syntax
{?avg(/Production server/system.cpu.load,1h)}
The current host can be omitted (//key,parameter)

If multiple hosts are used in the trigger expression, item values from the first host will be used
{?avg(//system.cpu.load,1h)}
User macros are not supported inside expression macros
MACRO FUNCTIONS
The output of the macro can be modified using macro function:
fmtnum (<digits>)
digits - the number of digits after decimal point
This function can be applied to following macros:

Built-in macros {ITEM.VALUE}, {ITEM.LASTVALUE}
Expression macros {?MACRO}
The macro function syntax: {{?MACRO}.fmtnum(#)}

Macro syntax Macro value Output
{{?avg(//vm.memory.size[free],1h)}.fmtnum(2)} 24.3413523 24.34
{{ITEM.LASTVALUE}.fmtnum(0)} 8.245678 8
i https://www.zabbix.com/documentation/6.0/en/manual/config/macros/macro_functions
OPERATIONAL DATA
Operational data is used to display live values on the Problems page:
Macros in trigger name will resolve to values at the time of problem creation
Macros in the operational data will display the very latest information dynamically
The same set of macros is supported as in trigger name
If undefined, will display {ITEM.LASTVALUE} from all items used in expressions
Fixed value Actual data
Multiple values
DISPLAY OPERATIONAL DATA
Operational data must be enabled on the Problems page:
None - do not display at all
Separately - display in a separate column
With problem name - separated by parentheses
Operational data
DISPLAYING ITEM VALUE
Macros can be used to display item value:
{ITEM.VALUE} the at-the-time-of-event value
{ITEM.LASTVALUE} the most recent value
1000
900
{ITEM.LASTVALUE}
800
{ITEM.VALUE}
700
600
500
400
300
200
100
0
9:00:00 9:00:10 9:00:20 9:00:30 9:00:40 9:00:50 9:01:00 9:01:10 9:01:20 9:01:30 9:01:40 9:01:50 9:02:00
Interface eth0: Bits in last(/host/net.if.in[eth0])>500M
INDEX IN MACRO NAMES
Index can be used to specify first, second, etc. item in a trigger configuration
{ITEM.VALUE<1-9>} the at-the-time-of-event value of the Nth item in the trigger expression
{ITEM.LASTVALUE<1-9>} the most recent value of the Nth item in the trigger expression
1000
900 {ITEM.VALUE1}
{ITEM.LASTVALUE1}
800
700
600
500
400
300
{ITEM.VALUE2}
{ITEM.LASTVALUE2}
200
100
0
9:00:00 9:00:10 9:00:20 9:00:30 9:00:40 9:00:50 9:01:00 9:01:10 9:01:20 9:01:30 9:01:40 9:01:50 9:02:00
Interface eth0: Bits in Interface eth0: Bits out last(/host/net.if.in[eth0])>500M or last(/host/net.if.out[eth0])>500M
TRIGGER SEVERITIES
One of the six severities can be assigned to a trigger:
Default trigger severity names and colors can be changed in

Administration > General > Trigger displaying options
Trigger severities can be used for:

Visual representation of triggers and problems (different colors)
Filtering based on severities (in Problems, Maps, Dashboards, etc.)
Different audio sounds in global alarms
Different user media used for different severities, for example:
SMS - only problems with High and Disaster severities
Email – All problems
Limiting actions by conditions against trigger severities
RESOLVING PROBLEMS
Problems may be resolved using different scenarios:
By using the trigger expression only
Trigger expression = FALSE
By specifying additional recovery expression
Trigger expression = FALSE and recovery expression = TRUE
By using manual close of the problem
Manual close option must be enabled in the trigger configuration
A few important notes about manually closing problems:

Manual close must be allowed in the trigger configuration
If a problem is manually closed, it may be detected again on the next trigger evaluation
Manual close is useful for problems which cannot be automatically resolved
! Recovery expression will not have any effect if trigger expression is still TRUE!
USING RECOVERY EXPRESSION
Recovery expression may reduce trigger flapping:
Once a problem is detected, a different expression is used for recovery
Both Trigger and Recovery expressions are used for OK event calculation
3.5
2.5
2
Trigger
expression
1.5
Recovery
1
expression
0.5
0
9:00:00 9:00:10 9:00:20 9:00:30 9:00:40 9:00:50 9:01:00 9:01:10 9:01:20 9:01:30 9:01:40 9:01:50 9:02:00
CPU Load last(/host/system.cpu.load)>2 last(/host/system.cpu.load)<=1
RECOVERY EXPRESSION EXAMPLE
Trigger
expression
Recovery
expression
MANUAL CLOSE
Manual close will close any existing problem immediately:
The trigger will be recalculated on the next item update
A new problem will be opened if the trigger expression is still true
Manual close
09:00:45
PROBLEM PROBLEM
6
5.5
5
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0
9:00:00 9:00:10 9:00:20 9:00:30 9:00:40 9:00:50 9:01:00 9:01:10 9:01:20 9:01:30 9:01:40
CPU Load last(/host/system.cpu.load)>2
PRACTICAL SETUP
1) On the host "Training-VM-XX":
Create a user macro for high CPU load (>1.5)
Create a trigger to detect high CPU load with a Warning severity
2) Use "cat /dev/urandom | md5sum" command
3) Wait for problem to be detected
Show operational data for the problem
Create a user macro for CPU recovery (<0.75)
Add a recovery expression to the trigger
5) Stop the "cat /dev/urandom | md5sum" by pressing CTRL+C
6) Watch Problems page for Operational data values
7) Wait for problem to be closed
Problems
PROBLEM DETECTION
Problem detection in Zabbix is based on:
Data collection at defined intervals using items
Evaluation of collected data using triggers
Problems are raised based on the result of a trigger expression:

PROBLEM Trigger expression is TRUE
OK Trigger expression is FALSE
Trigger status is recalculated:

When any of the items included in the trigger receive new data
Additionally, every 30 seconds if time-based functions are used
EVENTS
Each time the trigger changes its state, a trigger event is generated:
The event contains details of the trigger state change
The date and time when the change has happened
The new state of the trigger
Two types of events are created by triggers - Problem and Recovery

When a trigger detects the problem, it creates a new problem event
When the trigger detects that the problem is gone, it creates a recovery event
Problem
PROBLEM EVENT
Recovery
OK EVENT
PROBLEMS
Problems are generated based on the events:
Problem event will create a new problem
Recovery event will resolve the problem
Problem duration is the time between problem and recovery events:

PROBLEM RECOVERY
6.00
5.00
4.00
3.00
Trigger 2.00
CPU load > 2
1.00
0.00
09:00:00 09:00:10 09:00:20 09:00:30 09:00:40 09:00:50 09:01:00
Problem duration = 00:00:50

MULTIPLE EVENT GENERATION
Zabbix supports two event generation modes:
Single only one event is generated, when a trigger goes into the Problem state
Only one event for problem is created
Multiple a new event is generated upon every evaluation of the trigger

Used for log files, SNMP traps and in other specific use cases
A new event for every received value is created (wrong use in this example)
! Do not use Multiple problem generation mode without a specific reason!

PROBLEMS FILTER
Problems are displayed on the Monitoring > Problems page
The top section of the page contains a huge set of different filters
Filters can be saved as additional tabs and reused later
Tabs can have custom time periods
Number of active problems can be displayed on the tab
Number of problems
PROBLEMS PAGE
The main part of the Problems page contains the list of current problems:
The representation of problems depends on a global filter:
Recent problems open problems and problems resolved in the last 5 minutes are displayed
Problems only open problems are displayed
History custom history period for open and resolved problems can be specified
PROBLEM UPDATE
Status of a problem can be changed by:
Adding a text message
Changing problem severity
Acknowledging
Manually closing (if allowed)
All problem updates are displayed in frontend:

Comments have been made
Problem severity changed
Actions have been taken
PROBLEM DISPLAYING OPTIONS
Problem displaying options can be adjusted:
Go to the Administration > General > Trigger displaying options:
Redefine event status colors
Change problem displaying options
Change trigger severity names and colors
PRACTICAL SETUP
1) Use "cat /dev/urandom | md5sum" command to generate a problem
2) Acknowledge the problem with a message and change problem severity
3) Allow manual close in the trigger configuration
4) Close the problem using manual close
5) Watch how the problem is detected again soon
Trigger functions
FUNCTIONS
Zabbix supports almost 100 different functions:
Aggregate functions avg, count, min, max, sum, kurtosis etc.
Bitwise functions bitand, bitlshift, bitnot, bitor, bitrshift, bitxor
Date and time functions date, dayofmonth, dayofweek, now, time
History functions last, first, change, logseverity, monoinc, nodata, etc.
Trend-based functions trendavg, trendcount, trendmax, trendmin, trendsum
Mathematical functions abs, cos, sin, tan, ceil, floor, degrees, e, exp, log, rand, etc.
Operator functions in, between
Prediction functions forecast, timeleft
String functions ascii, bitlength, char, concat, find, left, length, trim, mid etc.
Foreach functions avg_foreach, last_foreach, sum_foreach (calculated items only)
Advanced functions and absolute time shift periods are discussed in ZCP 6.0
FUNCTION PARAMETERS
Most of the functions require one or multiple parameters:
/host/key is a common mandatory first parameter for history functions
last(/prod/system.cpu.load)
Other parameters are placed after the /host/key separated by a comma

If an evaluation period or range is required, it always goes as a second parameter
min(/prod/vm.memory.size[free],1h)
More than one parameter may be required for some functions
count(/prod/log[/var/log/myApp.log],10m,"like","error" )
! /host/key and evaluation period or range parameters must never be quoted
TIME PERIODS
Most of numeric functions accept time or number of values as a parameter:
Seconds will be used if no time suffix is specified
Time suffixes may be used to specify time units (10s, 5m, 1h, etc.)
If preceded by a hashtag #, the parameter indicates the number of values (#5, #10, etc.)
sum(/host/key,10m) Sum of values in the last 10 minutes
sum(/host/key,10s) Sum of values in the last 10 seconds
sum(/host/key,10) Sum of values in the last 10 seconds
sum(/host/key,#10) Sum of the last 10 values
FUNCTION PARAMETERS
Example:
Item update interval: 5s
Item collected every minute at 00, 05, 10, 15 ,etc.
max(/host/key,1m)
min(/host/key,#10)
avg(/host/key,10)
time
20 25 30 35 40 45 50 55 00 05 10 15 20 25 30 35 40 45
09:05:00 09:05:45
This value This value This value

is not included is included is not included
in the calculation in the calculation in the calculation
MATHEMATICAL OPERATIONS
Mathematical operations can be applied to the trigger functions:
To the result of trigger function
avg(/host/net.if.in[eth0,bytes],10m) * 8 > 10M result is multiplied by 8
Between results of trigger functions
min(/host/system.cpu.load,5m) / last(/host/system.cpu.num) > 1.5 Load per CPU
To the results of multiple trigger functions

last(/host/proc.num) / last(/host/kernel.maxproc) * 100 > 80 80 percent calculated
Mathematical operations must be added after the function
Correct Correct Wrong
min(/host/key,#5) min(/host/key,#5)*10 min(/host/key,#5*10)
FUNCTION PARAMETERS
Aggregate functions will accept other expressions as function parameters:
function(function_a(),function_b(),...)
Function beginning with host and item key will use time period as parameter
min(/host/key1,1h) smallest value from 1 hour of

historical data
Function beginning with other expressions will use them as parameters
min(avg(/host/key1,1h),min(/host/key2,#5)*10) smallest value from the result of

other expressions
function1 function2
OPERATORS
The following operators are supported for triggers:
Unary minus - (change the sign of an operand)
Mathematical operations ( +, -, *, / )
Compare (<, <=, >, >=, =, <>)
Logical operators (and, or, not)
Case-sensitive and must be in lowercase
Must be surrounded by spaces or parentheses.
Notes:
Most operators expect numerical variables
Operators = or <> can be used to compare strings
last(/host/vfs.file.cksum[/etc/passwd],#1) <> last(/host/vfs.file.cksum[/etc/passwd],#2)

last(/host1/system.hw.macaddr[eth0,short],#1) = last(/host2/system.hw.macaddr[eth0,short],#1)
FUNCTION COMPARE
When building trigger expressions, it is possible to compare function against:
Fixed value
last(/prod/agent.version) <> '6.0.0'
User macro
min(/prod/system.cpu.load,5m) > {$CPU.LOAD}
Result of another trigger function
last(/node1/hw.macaddr) = last(/node2/hw.macaddr)
Result of calculation
last(/node1/system.cpu.num) > last(/node2/system.cpu.num) * 1.5
RELATIVE TIMESHIFT
An optional time shift is supported in the time parameter:
This parameter allows to reference the data from a period of time in the past
Time shift starts with now (the current time), followed by:
+N<time unit> to add N time units
-N<time unit> to subtract N time units
avg(/host/key,4h:now-1d)
Complex expressions using multiple time units in the calculation are supported
avg(/host/key,6h:now-1d+8h)
6h:now-16h
4h:now-1d 6h:now-1d+8h 4h
time
22 23 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 1 2 3 4 5 6
Yesterday Today Now
2021-10-20 2021-10-21 2021-10-21 06:00
i https://www.zabbix.com/documentation/6.0/en/manual/config/triggers/expression
LAST FUNCTION
The last() function returns the last received value:
The time period is not supported
last(/host/key)
Using hashtag # will denote the Nth previous value
last(/host/key,#1) Last value
last(/host/key,#3) 3rd previous value
Time shift parameters are supported

last(/host/key,#1:now-1h) Value received one hour ago
last(/host/key)
last(/host/key,#4) last(/host/key,#3) last(/host/key,#2) last(/host/key,#1)
time
FUNCTION LAST()
Function last() is very sensitive:
Every received value which exceeds the threshold will generate a new alert
This can lead to "trigger flapping"
4.5
3.5
2.5
1.5
0.5
0
9:00:00 9:00:10 9:00:20 9:00:30 9:00:40 9:00:50 9:01:00 9:01:10 9:01:20 9:01:30 9:01:40 9:01:50 9:02:00
CPU Load last(/host/system.cpu.load)>2
FUNCTION MIN()
Function min() is a simple way to reduce false problem detection:
Minimum value must drop below the threshold for some time period to detect a problem
The problem will resolve immediately
30 seconds PROBLEM
5
4.5
3.5
2.5 Problem resolves

immediately
2
1.5
0.5
0
9:00:00 9:00:10 9:00:20 9:00:30 9:00:40 9:00:50 9:01:00 9:01:10 9:01:20 9:01:30 9:01:40 9:01:50 9:02:00
CPU Load min(/host/system.cpu.load,30s)>2
FUNCTION MAX()
Function max() may be used to detect availability issues:
Multiple availability checks in a row must fail to detect a problem
30 seconds PROBLEM
1.5
0.5
0
9:00:00 9:00:10 9:00:20 9:00:30 9:00:40 9:00:50 9:01:00 9:01:10 9:01:20 9:01:30 9:01:40 9:01:50 9:02:00
ICMP ping max(/host/icmpping,30s)=0
FUNCTION COUNT()
Function count() is another option:
More than one anomaly will be required to detect a problem
1 minute PROBLEM
3.5
2.5
1.5
0.5
0
9:00:00 9:00:10 9:00:20 9:00:30 9:00:40 9:00:50 9:01:00 9:01:10 9:01:20 9:01:30 9:01:40 9:01:50 9:02:00
CPU Load count(/host/system.cpu.load,1m,"gt",2)>=3
MONOTONIC FUNCTIONS
Monotonic functions can be used to monitor queues or disk space:
monoinc() detects monotonic increase of values collected
monodec() detects monotonic decrease of values collected
60 seconds PROBLEM
1100
1000
900
800
700
600
500
400
300
200
9:00:00 9:00:10 9:00:20 9:00:30 9:00:40 9:00:50 9:01:00 9:01:10 9:01:20 9:01:30 9:01:40 9:01:50 9:02:00
Free disk space on / monodec(/host/vfs.fs.size[/,free],60s,strict)=1
PRACTICAL SETUP
Replace last() function for CPU load triggers with a minimum for 1 minute
2) Use "cat /dev/urandom | md5sum" command to test triggers
3) Create a new trigger to compare memory usage:
Compare average free memory for current hour with average free memory for previous hour
Generate alert if the free memory has decreased by more than 25%
Time-based functions
TIME-BASED FUNCTIONS
All time-based functions are recalculated every 30 seconds:
If both time-based and non-time-based functions are used in an expression, it is
recalculated when a new value is received and additionally every 30 seconds
Recalculation schedule is distributed evenly between all time-based functions
Example trigger is recalculated:
every 10 seconds based on the item update interval
additionally, every 30 seconds because time-based function is used in the expression
30 seconds
2
1.5
0.5
0
9:00:00 9:00:10 9:00:20 9:00:30 9:00:40 9:00:50 9:01:00 9:01:10 9:01:20 9:01:30 9:01:40 9:01:50 9:02:00
min(/host/system.cpu.load,30s) > 2 and dayofweek() > 5
DATE AND TIME FUNCTIONS
All date and time functions are time-based:
date current date in YYYYMMDD format
time current time in HHMMSS format
dayofweek current day of week 1-7 (Mon - 1, Sun - 7)
dayofmonth current day of month 1-31
now current time in HHMMSS format
These functions can be used to specify time periods from trigger calculation
Detect problems only on weekends
min(/host/system.cpu.load,30s) > 2 and dayofweek() > 5
Ignore scheduled backups between 01:00:00 and 03:00:00
avg(/host/system.cpu.util[,iowait],5m) > 5 and (time() < 010000 or time() > 030000)
! All date and time functions use Zabbix server time zone!
NODATA() FUNCTION
Time-based history function checks for no data received:
nodata(/host/key,time period,<mode>)
Time period should not be less than 30 seconds (nodata(0) is not allowed)
Returns:
1 - if no data received during the defined period of time
0 - otherwise
The 'nodata' triggers monitored by proxy are, by default, sensitive to proxy availability
They will not fire if the data is expected from a proxy, which is currently offline
"strict" mode will ignore proxy availability
nodata()
NODATA EXAMPLE
Function nodata() can be used to detect:
Zabbix agent availability (agent.ping item never returns 0)
The problem is
Changes in the log files monitored by Zabbix resolved
immediately when
Data received (or not received) on the regular intervals the data arrives
2 minutes of no data received

2
0
9:01:15 9:02:15 9:03:15 9:04:15 9:05:15 9:06:15 9:07:15
Zabbix agent ping nodata(/host/agent.ping,2m)=1
NOTES
Zabbix server time zone is used to calculate time-based functions
User time zone settings may differ from Zabbix server time zone
Time-based functions in triggers with multiple event generation mode will create
a new problem every 30 seconds
The trigger will be evaluated every 30 seconds even if there isn't any new data received
Nodata() function may return false positives if:

There are time differences between Zabbix server, proxy and agent
Discard unchanged preprocessing steps are used
History is not saved for the item
Nodata() function is evaluated for "not supported" items also
PRACTICAL SETUP
1) On the "Training-VM-XX active checks" host
Create Zabbix agent ping item with the 10 second update interval
Create a trigger to check agent ping last value
2) Stop Zabbix agent on your virtual machine
3) Wait 1 minute to test if the trigger detects a problem
4) On the Training-VM-XX active checks host:
Replace the last() trigger function with no data received for 1 minute
5) Wait 1 minute to test if the trigger detects a problem
6) Start Zabbix agent on your virtual machine
TRIGGER DEPENDENCIES
TRIGGER DEPENDENCIES
Dependencies can be defined between multiple triggers:
Problems will be suppressed if the trigger they depend on is in the PROBLEM state
Zabbix does not support dependencies between hosts directly
Dependencies between triggers can be defined:

On the same host:
Problem level (different severities)
Same host dependencies
Different hosts:
Network devices Low disk space
Applications
Other resources
Critically low disk space
Different host dependencies
Server Switch Router Out of disk space
TRIGGERS - DEPENDENCIES
It is possible to create complex architecture with multi-level dependency:
Multiple levels:
Server A > Switch 1 > Router Server A Server B Server C Server D
Multiple dependencies:
Switch 1 > Router
Switch 1 Switch 2
Switch 2 > Router
Router
If, for example, a router is down, and dependencies are defined:

Problems generated by the dependent triggers will be suppressed and hidden
Zabbix will not execute actions for the dependent trigger
The dependent trigger will be re-evaluated and will change its state only after the parent
trigger returns to the OK state and new metrics are received.
DEPENDENT TRIGGER EVALUATION
Dependent trigger will only be re-evaluated when:
Parent trigger has changed its state to "OK"
A new value is received for item used in dependent trigger expression
Triggers are evaluated independently of their dependencies:

It is possible that problem with a dependent trigger will be detected first
In this scenario, the dependent trigger will fire as usual
It will become suppressed later when problem with a parent trigger will be detected
PRACTICAL SETUP
Create user macro for very high CPU load ( >2.5) with a "High" severity
Create a trigger to detect very high CPU load
2) Create a dependency between triggers for high and very high CPU load
3) Use "yes > /dev/null" command to test this setup
QUESTIONS?
Zabbix 6.0 Certified Specialist ● Day 2 © 2022 by Zabbix. All rights reserved 134
Time for a break :)
Zabbix 6.0 Certified Specialist ● Day 2 © 2022 by Zabbix. All rights reserved

ZCS6 Day 2 Slides

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ZCS6 Day 2 Slides

Uploaded by

Copyright:

Available Formats

6.

Triggers Problems Trigger functions

Data from Zabbix passive agent is collected by the poller process:

### Option: StartPollers

Specify proxy server if the host will be monitored through a proxy

### Option: Server

Error: agent dropped connection

### Option: ListenPort

Multiple passive agent processes can be started at once:

Timeouts can be adjusted in the server/proxy and agent configuration files

I will wait only I have only 3 seconds

Only this field is used

Passive agent Active agent

### Option: ServerActive

Hello 192.168.0.5, Hello "Production

HostnameItem dynamically creates hostname from the agent key

Hostname=Production server IP=192.168.0.5

Zabbix Server 1 Hostname=Production server Zabbix Server 2

### Option: RefreshActiveChecks

Collected values are sent to Zabbix server or proxy in bulk:

Collected values are sent in bulk to Zabbix server or proxy:

If the buffer specified by BufferSize is full in less than BufferSend seconds

Thank you, I will store

Production server Zabbix Server

Production server Zabbix Server

Production server IP=10.10.10.1

Production server Zabbix server

local time=2022.01.16 09:30:03 local time=2022.01.16 09:29:45

Active checks are always timestamped by Zabbix agent

agent.version = 6.0.0 Timestamp Value

local time=2022.01.16 09:30:03 local time=2022.01.16 09:29:45

Working directory for commands can be specified using UserParameterDir:

Example: a script is located at /home/zabbix/workdir/myscript.sh

To reload the list of user parameters two methods are available:

# systemctl restart zabbix-agent

Use Zabbix agent runtime command userparameter_reload:

UserParameter can be simple or flexible:

Flexible: UserParameter=mysql.ping[*],mysqladmin -u$1 -p$2 ping | grep -c alive

I need to wait poller busy 20 seconds

When used as an active check:

Stored in database Units Displayed in frontend

Special processing is used to display B, Bps, unixtime, uptime, s units:

Value map X Value Map Y

Item A Template: Item A

Item B Template: Item B

Supported mapping types:

"220" will be mapped as "Normal" "220" will be mapped as "Perfect"

The last mapping will be

2) Create a new value map for Training resources host

3) Improve "Service Web performance" item

Update interval cannot be set for the following item types:

Two items have the same update interval 10 seconds:

Multiple flexible intervals can be defined:

! Zabbix server time zone is used to specify flexible intervals

Update interval, flexible intervals and scheduled checks can be combined:

Preprocessing can be performed by:

Monitored Collected Preprocessing Transformed Database

Type of information for the item is also displayed:

Custom on fail offers three options:

! If value is discarded, next preprocessing steps are ignored!

### Option: StartPreprocessors

Division is also possible if multiplier < 1 is used:

Notes for Numeric (unsigned) item type:

avg(/host/net.if.in[eth0,bytes],10m) * 8 > 10M result is multiplied by 8

min(/host/key,#5) min(/host/key,#5)10 min(/host/key,#510)

min(/host/key1,1h) smallest value from 1 hour of

min(avg(/host/key1,1h),min(/host/key2,#5)*10) smallest value from the result of