Table of Contents

• Overview of NetBackup Processes

– What are the various NetBackup processes
– What do they do?
– How do they work together?
• Monitoring/ Logging
– Tips/tricks for legacy logging, Vxlog gathering using Vxlogview, etc.
– Log types and associated processes
– Logging levels. How to turn up and down logging levels and why.
– Logging locations
– Gatherng up logs. Logging assistant.
– Cleaning up logs
• Reading and Understanding Logs
– What is a Process Identified (PID) and how does NetBackup use it?
– How to know where to look
– Quickly locating errors
– Understanding job details
• Other tools of note
– NBSU
Process Overview
The Processes

• Netbackup is not a program but rather a collection of processes that work together
• Process name prefixes – why are they different?
– Bp_ = legacy process (bp comes from Backup plus the original product). Examples:
• bprd= request daemon.
• bpdmn = Database Manager.
– Nb_ = newer processes. Multithreaded. ( made their debut in 6.x) Always running. Examples:
• Nbrb = NetBackup Resource Broker. Allocates and tracks resources.
• Nbproxy = NetBackup Proxy used to talk to legacy processes.
• As a service
– Some processes run as a service. They are started when NetBackup starts and stay running.
– Bprd, bpcd, bpdbm, nbpem, nbrb, and nbemm are exampls of these.
• Per job processes
– Bpbkar = Backup Archive and Restore. A misnomer since it does not handle restores.
– Tar.. The oddball. A legaxy process without the bp prefix.
– These processes are called per job, with start, perform their job, and exit. A good examples is bpcd (Communication
daemon).
How Process Share Information

• Through the network

– This is where the majority of the communication happens. Even Locally on the
same host.
– Several processes manage the communication
• Bpcd, vnetd, nbproxy, and pbx.
• Command line parameters.
– Bpbksrv –r 1814400 –ru root –dt 68122 –to 7200
– 05:21:16.065 [374.1] <2> bpbrm spawn_child: /usr/openv/netbackup/bin/bptpm
bptm –r
– Event Files
• DD AIR import files.
– A file is placed that another process will look for and process.
Process Flow Chart
• Master
– Starts jobs and manages them NetBackup
– Manages the catalog and resources Master Server
• Media
– Moves the data to storage

• Storage Server
– MSDP, Advanced Disk, OST Plugin for Dxi, or Data Domain.

• Client Media
– Sends the Data Server

• API
– NDMP, Vmware and Hyper-V

• Upgrading
– Netbackup should always be updated from the top down. Opscenter, Master Server, Media Server, Client.
– They do not all need to be done at the same time Client
– A master can work with mixed media server versions and mixed client versions with some limitations and exceptions.
– Opscenter must ALWAYS be the highest level or at least match the master server.
Master Server UNIX Client

Configuration Backup policy NetBackup User

File Database Command Line
Database management Interface

bpdbm nbproxy bprd

nbproxy bpbackup or
nbpem
bpcd EMM Database bparchive

nbjm
nbemm nbrb

Master or Media Server

bpcd
bptm bptm
nbrmms (child)** bpcd Catalog info
(parent)
Ba
ck bpkar
e

up
ag

mount i
Shared mag
im

e Tape or Optical Backup image

p

Memory bpbrm
u
ck
Ba

Disk Volume ltid* Client Disk

mount

6
Sample Process Flow

• Backup Flow at the process level

– NBPEM checks for resources, max jobs per policy, and sends a start request to NBJM
– NBJM contacts BPJOBD and add the job to the jobDB. The job shows queued
– NBJM requests needed resources from NBRB
– NBRB contacts NBEMM to create allocations and lock the resources
– NBRM confirms NBEMM has locked the resources and notifies NBJM
– NBJM contact BPJOBD to confirm resources and the job goes active
– NBJM contacts BPDBM using NBPROXY to create the image DB record.
– NBJM uses BPCOMPATB to pass arguments to BPBRM on the media server.
– BPBRM contacts BPBKAR on the client and sends the file list.
– BPBRM contacts BPTM with the allocation ID from NBEMM.
– BPTM contacs LTID, tldcd, avrd to mount tapes if needed and spawns a child for tape
backups.

7
Sample Process Flow II

• Backup Flow at the process level (continued)

– BPKAR sends the data to BPTM child which puts in info buffers.
– BPTM parents pull the data from buffers and puts it to storage.
– Once a fragment has been moved, BPTM will contact BPBRM up update the image DB.
– Once all the data is backed up, BPBKAR will send the image summary to BPBRM and
then close connection to BPBRM and BPTM.
– BPBRM sends the image summary to BPDBM on the master and notifies BPJOBD that the
data has been backed up.
– BPBRM waits for BPTM to confirm exiting and asks BPDBM to validate the image is
complete.
– NBJM tells BPJOBD to mark the job complete in activity monitor and de allocate resources
used by notifying NBRB.
– NBJM notifies NBPEM that the job is complete and update the next due time for the policy.

8
 Capturing and Logging

9
Logging Types

• Legacy logging.
– bp_ processes.
– These are single threaded which means each launch has a new PID.
• Robust logging
– An enhancement to legacy logging which allows for better log management including breaking
logs into smaller logs, recycling logs to save space, and intelligent pruning options.
• Vxlogging
– Nb_ processes use these logs.
– Logging is still stored in the same logging locations and uses the legacy log folder structure.
However the logs stored in a different format requiring vxlogview to “query” the logs and extract
the details.
– Far more manageable.
– Allows for gathering specific details that previously required a lot of time.
• PID parsing, timframe, etc.

10
Logging levels for Legacy Processes

• Legacy process logging

– Increasing levels from the master server.
• Done by expanding host properties and selecting the host type and the hosts.
– Increasing levels from the client.
• Open the BAR gui as administrator, file -> NetBackup Client Procerties -> Troubleshooting tab.
• If the loggin is at minimum click the down arrow to go to the highest level.
– Bp.conf
• On Non-Windows systems the levels can be changed by adding VERBOSE = <level> to the bp.conf file located in
/usr/openv/netbackup.
– VERBOSE = 5
– There are specific process logging options acailable as well.
• BPRD_VERBOSE = 5
• <process name>_VERBOSE = 5 usually works.
• To put the changes into effect run /usr/openv/netbackup/bin/bprdreq –rereadconfig
– This will tell NetBackup to read in the bp.conf but only per job processes will log at the new level since they will start and see the
change in loggin level.
– Process checks for its loggin folder at startup and will log there if found. If the folder is created after starting that process will not log.
Process like bpbkar will log the next time it starts.
– Some processes need to be restarted for logging levels to change. Remember early we discussed services. Those processes that are
always running need to be restarted for logging changes up or down. They often lock their active log file and prevent deletion of it or
the folder.

11
Logging Levels for nb processes

• Robust Logging
– Enable within the GUI. Options are all controlled in either the java console or the BAR GUI.
• Vxlogging
– Can be configured by the command line or in the GUI. Some processes such as NBEMM,
NBPROXY, and PBX have to be configured through the command line using vxlogcfg.
– Take effect right away as long as the folder exists.
– Use vxlogview to retrieve there logs.
• /NetBackup/bin vxlogview –p 51216 –o 117 –t 01:30:00 >c:\temp\NBJM-117.txt NBJM log
• -p is the program ID. NetBackup, OpsCenter, BMR, Storage Foundation, etc.
• -o is the originator ID. This is the process that create the message.
• https://www.veritas.com/content/support/en_US/doc/86063237-127664549-0/v40600850-
127664549is
a current list of OIDs (Originator IDs)

12
Gathering up logs

• One by One (manually)

– Navigate to the folder and collect the log.
– Naming format
– Robust Logging.
• Vxlogging
– Vxlogview and switches.
• Vxlogview –t 00:30:00 will retrieve all log entries for the last 30 minutes for all processes.
• Logging Assistant from the Master server Java console
– The logging assistant is an excellent tool built into NetBackup that will set up loggin based on selections chosen during the New Logging Record wizard.
After a failure using the logging wizard will bundle up all the logs across all the servers involved and create a zip file for review. This is an excellent all in one
place to gather logs.
• Nbcplogs
– NBCPLogs is a command line tools used to collect logs from other hosts.
– Windows: <install folder>\netbackup\bin\support
– Non Windows: /usr/openv/netbackup/bin/support
– Examples
• Run .\nbcplogs.exe –s mm/dd-hh:mm –e mm/dd-hh:mm <destination_directory>
* There will be a question confirming enough space is available in the destination directory once the utility has calculated the space requirements.
** Run .\nbcplogs –help for other available options if needed.
• /usr/openv/netbackup/bin/support/nbcplogs –s dd/mm/yyyy –e dd/mm/yyyy –l puredisk, nbstserv,bptm,bpdm /log/.LOGROOT/ `hostname`_`date +%m
%d%y`
* Answer y if there is enough space in /log to hold the collection. See nbcplogs –help if other options are needed to limit the collection date range.

13
Cleaning Up

• Cleaning up
– Vxlogs
• Netbackup\bin
• Vxlogmgr –F tells NetBackup to purge all vxlogs except those currently in
use
– Legacy logs
• Use the GUI or edit bp.conf to lower the logging level
• Delete the folder. Some processes will log the log and the folder cannot be
removed. Either bring down services or wait for the process to end
depending on the process. Bpbkar for example is per job.

14
Reading Logs and Understanding them

15
Sample Job Details

16
Knowing Where to Look

• When tracking down an issue it is important to know where the breakdown is happening.
– A backup failure status code 71 means not all of the files were backup up. This is typically a client side issue.
– The Master told the media and client what was to be backed up but the client did not present all the data.
– Reviewing the job details from the admin console to identify the errors followed by reviewing the client logs.
– The job details in activity monitor provides direction on what to look for.
• Job ID is passed process to process. It is often a great way to find on what to look for.
• Job details in activity monitor will usually show the pid of a process.
– Pulling out all lines with a specific PID will show the entire “life” of the PID for per job processes.
• Timestamp are another useful way to look through a log
• Process Exit Status.
– 22:41:39.980 [11544.9748] <16> bpcd exit_server: FTL – BPCD EXIT STATUS 46
• Line Code.
– 22:41:39.730 [11544.9748] <16> bpcd valid_server: MEDIA1 is not a media server either
• Other useful information.
– Job Overview tab.
• File list
• Policy Name, Start Time, Elapsed, End Time.
• Schedule name, Type, and Retentions.
• Client name and Master name.
– Details Tab.
• Attempt number
• Job PID
• Media Server and Storage Unit
• Files and KB Written.

17
Logs at a Glance

• Log File Structure

– <timestamp> <PID> <severity> <message>
– Timestamp usually in 24 hour format.
– PID is the unique numveric value assigned to a process at startup. It cannot be
changed.
• Windows format is two number separated by . [3743.1]
• Non Windows format is one number [3741]
– Severity is the level of importance of the message. Typically the higher the
numver the more severe.
• <2> informational
• <16> something to note but may only be informational. Can also se an error.
• <32> is typically a severe error.

18
Logs at a Glance II

• Interesting Notes

19
Sample Job Details

20
 Other Tools: NBSU and DataCollect

21
NBSU

• NBSU stands for NetBackup Security Utility.

• It's a tool provided by Veritas NetBackup, a backup and recovery software suite, that helps manage security configurations
within NetBackup environments.
• NBSU allows administrators to set up security policies, manage authentication, configure encryption, and perform other
security-related tasks to ensure the protection of data in transit and at rest within the NetBackup infrastructure.
• Gather a large amount of system configuration and logs
• System and Event Viewer logs and message file
• Network configuration and network statistics
• NetBackup versions and file information
• Hardware versions and file information
• Hardware specs of the system
• History of NetBackup Installation
• Specific NetBackup configuration
• Tuning and touch files used by NetBackup

22
NBSU II

• When troubleshooting issues Engineers often will request a NBSU and for NetBackup appliances and
updated DataCollect.
• These help support engineers by providing a in depth view of the configuration and design of the
environment as it relates to NetBackup. This data provides a “jump start” on the troubleshooting
process.
• Application specific logs. These logs are typically internal to the application and contain “the other side”
of the issue. Often times there are two views of the issue. And example would be a client loses
connection to a media server. The client says the media server wnt away.. The media server thins the
client went away.
– MS SQL Enterprise Manager Logs.
– Exchange Server logs from Event Viewer.
– SharePoint specific logs from withing SharePoint Admin Portal.
– NDMPD logs from NDMP filers.
– Oracle internal logs.
– Active Directory Logging.

23
How to run the NBSU

• NBSU is supported on almost every NetBackup install type available including master,
media, and clients
• Is included in the installation
• Typically is just run from the command line using nbsu with no switches. It will typically
detect the installation type
• Location
– /usr/openv/netbackup/bin/support for Non windows
– <install folder>\netbackup\bin\support form windows
– On NetBackup Appliances it can be found under support in the CLISH
• Non Intrusive
– It will take anywhere from 2 minutes to 15 minutes to run and does not interfere with
NetBackup.
24
What is DataCollect

• DataCollect refers to a feature or utility designed to gather diagnostic

information, logs, and other relevant data from the NetBackup Appliance
environment. This collected data can then be used for troubleshooting,
performance analysis, and maintenance purposes.

25
What data to collect?

• DataCollect typically gathers a wide range of information including:

– System logs: Logs related to system events, errors, warnings, and other relevant information.
– Configuration information: Details about the configuration of the NetBackup Appliance including
hardware setup, software versions, and network settings.
– Performance metrics: Information on system performance, resource utilization, and any bottlenecks
that may be affecting performance.
– Backup and restore logs: Logs related to backup and restore operations performed by NetBackup on
the appliance.
– Storage information: Details about storage usage, capacity, and any issues related to storage
devices.
– Network information: Information about network connectivity, bandwidth usage, and any network-
related issues.
– The collected data can help administrators and support personnel diagnose and resolve issues more
effectively, optimize performance, and ensure the smooth operation of the NetBackup Appliance
environment. It's an essential tool for maintaining the health and reliability of backup and recovery
systems.

26