3-UCCX DB Replication Video

UCCX DB Replication
Konstantin Vaksin
TECHNICAL CONSULTING ENGINEER
• Databases overview
• Platform Replication
Agenda • UCCX Replication
• Databases summary
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
Databases overview
Database
Instance Datastore DBspace Size Contents

Platform DB cuic_data cuic_db_dbs 100 MB CUIC tables
Platform DB db_phx_config db_phx_config_dbs 150 MB Finesse tables
UCCX DB db_cra db_cra_dbs 500 MB Config tables
UCCX DB db_cra db_hist_dbs 32 GB * Historical tables
Prompts, scripts,
db_cra_repository_dbs
UCCX DB db_cra_repository 40 MB * grammars,
db_cra_repository2_dbs
documents
Platform DB ccm ccm_dbs DRF/DRS data
UCCX Solution Databases
CCX Application Configuration Datastore
Historical Datastore UCCX DB

CCX DB
Repository Datastore
IBM Informix
Platform DB
Finesse CUIC
CM DB
UCOS
CentOS Linux
Platform Database Instance
• Appears in services list as “A Cisco DB”
• CLI: run sql <your command here>
• ccm DB holds cluster and platform info, configuration
• Table processnode contains server names, table ‘applicationuser’ contains app user login
• cuic_data DB holds CUIC application data

• Contains report configurations, temporary data sets, and information synchronized from UCCX
• db_phx_config DB holds Finesse config

• No connect permissions via CLI, remote support account (TAC) required
Query processnode Query cuic_data:
table:
Cisco Unified CCX Database
Stores all the data to generate historical
and real-time reports
Historical
Datastore (HDS)
db_cra
Stores all the configuration data from CCX Configuration
Admin such as Applications, CSQs, etc
Datastore (CDS)
Stores scripts, prompt files, custom
documents and grammars Repository
Datastore (RDS)
db_cra_repository
• CLI: run uccx sql db_cra <your command here>
• CLI: run uccx sql db_cra_repository <your command here>
Use “run uccx sql db_cra”
Platform Replication
Before VOS 6.0
Hub and spoke topology.
The publisher establishes a connection to every server in

the cluster and the subscribers establish a connection the
local database and the publisher only. As illustrated in the
figure below, only the publisher's database is writable
while each subscriber contains a read only database.
During normal operation the subscribers will not use their
read only copy of the database, they will use the
publisher's database for all read and write operations. In
the event the publisher goes down or becomes
inaccessible the subscribers will use their local copy of the
database. Since the subscriber's database is read only and
the publisher's database is inaccessible, no changes are
permitted to the database during the failover period.
After VOS 6.0
Fully meshed topology
The publisher and each subscriber connect logically

to every server in the cluster and each server can
update all servers (including the publisher) on user
facing features such as call forward all. The full list
of user facing features is located on the following
slide. This change in topology overcomes previous
limitations in replication architecture, as changes
can now be made to local subscriber databases for
user facing faetures even while the publisher is
inaccessible. Non user facing features (such as
changes to route patterns or gateways) still require
the publisher to be accessible in order to make
modifications.
How it works? 1. IPSEC communication
Publisher Subscriber
TCP Port 8500
Cluster Manager communication
How it works? 1. IPSEC communication
• show open ports regexp 8500
• file view activelog
platform/log/clustermgrxxxx.log
What is happening after?
TCP Port 8002
TCP Port 1500
DB communication & intracluster

communication
But we will check all of that from the Dbreplication files
What can be the issue?
• Checks entries of the

hostnames of the servers
• Checks NTP status
• Check DNS resolutions
What can be the issue?
admin:show network cluster
192.168.33.53 finesse12s.cc.lab finesse12s Subscriber finesse
DBSub authenticated using TCP since Sun Oct 13 10:23:22 2019
192.168.33.52 finesse12p.cc.lab finesse12p Publisher DBPub
authenticated
Server Table (processnode) Entries

----------------------------------
finesse12p
finesse12s
admin:
We can check name of the subscribers in the DB

Same command as “run sql select * from processnode”
utils dbreplication runtimestate
admin:utils dbreplication runtimestate
Server Time: Fri Oct 18 14:12:46 CEST 2019
Cluster Replication State: Awaiting subscribers for replication setup request...
DB Version: ccm12_0_1_10000_17
Repltimeout set to: 300s

PROCESS option set to: 1
Cluster Detailed View from finesse12p (2 Servers):
PING DB/RPC/ REPL. Replication REPLICATION SETUP

SERVER-NAME IP ADDRESS (msec) DbMon? QUEUE Group ID (RTMT) & Details
----------- ---------- ------ ------- ----- ----------- ------------------
finesse12p 192.168.33.52 0.066 Y/Y/Y -- (-) (-) Replication Not Setup
finesse12s 192.168.33.53 0.279 Y/Y/Y -- (-) (-) DB-diff
Possible states?
Value Meaning Description
It’s the setting up process of Replication. CLI’s reset, rebuild, autorecovery might
0 Initialization State
have caused it.
It indicates that set up is still in progress. But huge delay in changing into the state of
1 The Number of replicates is incorrect
2 might implicate that something is wrong.
Logical connections are established and the tables are matched with the other
2 Replication is good
servers on the cluster.
Logical connections are present but not sure whether the tables match across the
cluster. Check dbmon logs.
3 Mismatched tables/Out of Sync
Probable reasons would be – a node might be down.
-update/data from one node would not have been passed onto other.
No active logical connection present. Absence of any replication. Network issue,

4 Setup Failed/Dropped
Intra-cluster communication failure etc might have caused it. Check ccm logs.
How can we reset replication?
Step 1. Stop replication!!!!! In earlier versions, you need to go to each subscriber. Now
you can use command from publisher:
• utils dbreplication stop all
Step 2. If you had to correct DB entries, you need to drop admindb

• utils dbreplication dropadmindb
Step 3. Reset replication from publisher

• utils dbreplication reset all
Step4. Monitor Replication
Replication Monitor
Cluster Replication State: PUB SETUP Started at 2019-10-18-15-29

Setup Progress: 1 node(s) added to the replication network
Setup Errors: No errors

Cluster Detailed View from fin-a (2 Servers):

----------- ---------- ------ ------- ----- ----------- ------------------
fin-a 192.168.33.22 0.030 Y/Y/Y 0 (g_2) (0) Setup Completed
fin-b 192.168.33.23 0.304 Y/Y/Y -- (-) (-) Not Requested
admin:file list activelog cm/trace/dbl/ date details

18 Oct,2019 15:30:52 141,718 20191018_152958_fin-a_g_2_ccm12_0_1_10000_17_dbl_repl_cdr_define.log
dir count = 3, file count = 14
Replication Monitor
Cluster Replication State: BROADCAST SYNC Started on 1 server(s) at: 2019-10-18-15-33

Use CLI to see detail: 'file view activelog cm/trace/dbl/20191018_153329_dbl_repl_output_Broadcast.log'


----------- ---------- ------ ------- ----- ----------- ------------------
fin-b 192.168.33.23 0.379 Y/Y/Y 0 (g_3) (0) Syncing...

18 Oct,2019 15:33:10 3,762 20191018_153239_fin-b_g_3_ccm12_0_1_10000_17_dbl_repl_cdr_define.log
Replication Monitor
admin:file view activelog cm/trace/dbl/20191018_153239_fin-b_g_3_ccm12_0_1_10000_17_dbl_repl_cdr_define.log
[10/18/2019 15:32:39] dbname passed [ccm12_0_1_10000_17] local-dbname [ccm12_0_1_10000_17]

[10/18/2019 15:32:39]
[10/18/2019 15:32:39] -------------------------------------- Inside deleteQuiescent --------------------------
…
[10/18/2019 15:33:09] ====================================== locateFailedServer End

======================================
[10/18/2019 15:33:09]
[10/18/2019 15:33:09] Executing cdr err -a
[10/18/2019 15:33:09] Executing su - informix -c "source /usr/local/cm/db/informix/local/ids.env; cdr err -a
2>&1 > /usr/local/cm/db/cdr_err.out"
[10/18/2019 15:33:10] size of cdr_err.out is [64]
[10/18/2019 15:33:10]
[10/18/2019 15:33:10] Server g_3_ccm12_0_1_10000_17 Defined Successfully
Replication Monitor
18 Oct,2019 15:35:09 380,574 20191018_153356_dbl_repl_cdr_Broadcast_ccmdb.log
18 Oct,2019 15:35:12 14,568 20191018_153510_dbl_repl_cdr_Broadcast_phxdb.log
file view activelog cm/trace/dbl/20191018_153356_dbl_repl_cdr_Broadcast_ccmdb.log
Oct 18 2019 15:34:05 ------ Table scan for ccmdbtemplate_g_2_ccm12_0_1_10000_17_1_147_typedberrors start --

------
Node Rows Extra Missing Mismatch Processed

---------------- --------- --------- --------- --------- ---------
g_2_ccm12_0_1_10000_17 1668 0 0 0 0
g_3_ccm12_0_1_10000_17 1668 0 0 0 0
Oct 18 2019 15:34:05 ------ Table scan for ccmdbtemplate_g_2_ccm12_0_1_10000_17_1_147_typedberrors end ---
------
Oct 18 2019 15:34:05 ------ Table scan for

ccmdbtemplate_g_2_ccm12_0_1_10000_17_1_352_typeroutingdatabasecachetimer start --------
Node Rows Extra Missing Mismatch Processed

---------------- --------- --------- --------- --------- ---------
g_2_ccm12_0_1_10000_17 97 0 0 0 0
g_3_ccm12_0_1_10000_17 97 0 0 0 0
Oct 18 2019 15:34:06 ------ Table scan for

ccmdbtemplate_g_2_ccm12_0_1_10000_17_1_352_typeroutingdatabasecachetimer end ---------
Replication completed
Cluster Replication State: BROADCAST SYNC ended at: 2019-10-18-15-35

Sync Result: SYNC COMPLETED on 745 tables out of 745
Sync Status: All Tables are in sync
Use CLI to see detail: 'file view activelog cm/trace/dbl/20191018_153329_dbl_repl_output_Broadcast.log'


----------- ---------- ------ ------- ----- ----------- ------------------
fin-b 192.168.33.23 0.315 Y/Y/Y 0 (g_3) (2) Setup Completed
UCCX Replication
CCX Database Replication
• View from CCX Admin or CLI
• Connected / Active state shows healthy replication
• Dropped / Timed Out status indicates replication issue
Tools-> Datastore Control Center-> Replication

Servers
admin:show uccx dbreplication servers

This operation may take a few minutes to complete. Please wait...
SERVER ID STATE STATUS QUEUE CONNECTION CHANGED

-----------------------------------------------------------------------
g_uc115ccxpri_uccx 1 Active Local 0
g_uc115ccxsec_uccx 2 Active Connected 0 May 9 18:07:55
CLI Command: Command successful.
show uccx dbreplication servers
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential admin: 25
Database commands at your finger tips!
utils uccx database dbserver integrity

Checks the integrity of the database server disk structures and displays results. It
also checks the DB config integrity and does a fix if integrity is broken.
When to use?
• CSQ/Application page doesn’t list any CSQs/Applications
• Newly created skills don’t appear or the entire Skills page is empty
Database commands at your finger tips!
utils uccx database dbreplication <tag>
• setup: will setup the UCCX DB Replication between the nodes
• repair: This command repairs mismatched data between cluster
nodes; it does not repair replication setup
• reset: does 3 steps
1. Remove Database replication
2. Setup Database replication
3. Initiate a data repair process
utils uccx database forcedatasync
This command gets the data from the other node in the cluster; effectively
overwriting the data on this node.
When to use?
• A single node in the cluster is corrupted and all efforts to restore database has
failed
• It puts a strain on the links between the nodes based on database side, so be
mindful of the same
Database Repair - Checklist
CCX Serviceability - UI Driven Command Line Interface Driven
Reset Replication Reset Replication
Enable/Disable CDS/HDS forcedatasync
Synchronize Data dbserver integrity
Database Replication Reset
• Replication can be broken if Subscriber is unavailable for too long and send queues buffer is exceeded
• Typically 3-4 days (*can vary with load)
Alert Raised!
• Alert will be raised DBReplicationStopped
Tools-> Datastore Control Center-> Replication

Servers
Issuing a Reset Replication causes the following to occur:

1. Remove database replication
utils uccx dbreplication teardown
2. Setup database replication
utils uccx dbreplication setup
3. Perform data repair process
utils uccx dbreplication repair all
Database Replication Reset
UI Driven CLI Driven
DB Master DB Standby
1. Teardown Replication.
CDS 2. Setup Replication. CDS
3. Merge Data
HDS HDS
RDS RDS
Disable/Enable CDS/HDS
Tools -> Datastore Control Center ->

Replication Servers
Node 1 Disable CDS/HDS Node 2

1. Tear down HDS replication.
CDS 2. Deactivate CDS on node 2. CDS
HDS HDS
Enable CDS/HDS
RDS RDS
1. Setup HDS replication.
2. Merge HDS data.
3. Copy node 1 CDS to node 2.
4. Activate node 2 CDS.
Synchronize Data - GUI
Tools -> Datastore Control Center ->
Datastores
CDS •Merges a single datastore. CDS
•CDS not eligible (no ER).

HDS HDS
RDS RDS
Force Database Sync – CLI Command
Local DB Remote DB
CDS
Overwrite Target = CDS
Local!
HDS HDS
RDS RDS
Cluster Reboot + Replication

Reset required!
Database Integrity
utils uccx database dbserver integrity
• Uses informix “oncheck” to check database for inconsistencies
• System Catalog Tables
• Chunk and extents
• Root reserved pages
• Output logged. CLI: file view activelog /uccx/cli/DbServerIntegrity_xxxx.txt
• Repairs NULL data entries in Config Tables

• Output logged. CLI: file view activelog /uccx/cli/DbConfigIntegrity_xxxx.txt
Reporting Review - Datasources
Publisher CUIC uses

Subscriber’s db_cra
Subscriber’s CUIC
Also uses Subscriber’s
db_cra
Database Issues – Further Analysis
• From Real-Time Monitoring Tool (RTMT) collect logs for Cisco Unified CCX Database
DB
online.uccx.log file for
issues related to database
service
DB REPLICATION
uccx_repl_output_util.log
file for issues related to
database replication
If you need to dig a bit deeper with database replication

issues you can collect the uccx_repl_output_util.log file
for further analysis.
Reporting Review – User Accounts
• uccxhruser - ”Historical Reporting User”
• CUIC, Wallboards, 3rd Party Reporting Software, Workforce Management
uccxwallboard
Wallboard User limited to
RtCSQsSummary and
RtICDStatistics tables
uccxworkforce: Workforce User used by Calabrio WFO.

Large queries on historical tables.
uccxhruser : Historical Reporting User for

CUIC co-resident and custom report access to
all historical tables.
Reporting and Database Performance
• Review the configured CUIC reports and their frequency
• Check the historical tables size
Reporting and Database Performance
• If tables have millions of rows, review purge config:
Default of 90 months can be reduced

Specify the Run Time for the purge to run
Databases summary
Platform vs UCCX Replication
• Platform stored in different database instance
• Platform replication commands start with “utils dbreplication *”
• UCCX replication commands start with “utils uccx dbreplication *”
• Platform replication status also shown in RTMT
• CUIC and Finesse use platform database instance
UCCX Replication
Maintaining database consistency across both nodes
Attempt configuration
change on Node 1
Call completes Replication of

Write to Node1 historical data
configuration DB
Call data written

Write to Node2 into historical
Configuration DB tables on Node1
After a few
2-phase Commit change seconds, data is
commit pushed to Node2
for config
changes
If the database is down for any reason, the call data is written
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
into flat files that can be used to restore into the database
44
later.
DB Replication States – basic check
utils dbreplication runtimestate – Provides the status of Platform DB

replication.
CUIC and Finesse share the platform DB.
utils uccx dbreplication status – Provides the status of the UCCX DB

replication.
All the CCX historical, configuration and repository data is stored on this DB.
CLI: utils network connectivity <hostname of secondary>
CLI: utils ntp status
CLI: utils diagnose test

3-UCCX DB Replication Video

Uploaded by

Copyright:

Available Formats

You might also like

3-UCCX DB Replication Video

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

3-UCCX DB Replication Video

Uploaded by

Copyright:

Available Formats

UCCX DB Replication

Instance Datastore DBspace Size Contents

CCX Application Configuration Datastore

Historical Datastore UCCX DB

• cuic_data DB holds CUIC application data

• db_phx_config DB holds Finesse config

The publisher establishes a connection to every server in

The publisher and each subscriber connect logically

TCP Port 8500

Cluster Manager communication

DB communication & intracluster

But we will check all of that from the Dbreplication files

• Checks entries of the

Server Table (processnode) Entries

We can check name of the subscribers in the DB

Server Time: Fri Oct 18 14:12:46 CEST 2019

Cluster Replication State: Awaiting subscribers for replication setup request...

Repltimeout set to: 300s

Cluster Detailed View from finesse12p (2 Servers):

PING DB/RPC/ REPL. Replication REPLICATION SETUP

No active logical connection present. Absence of any replication. Network issue,

Step 2. If you had to correct DB entries, you need to drop admindb

Step 3. Reset replication from publisher

Step4. Monitor Replication

Server Time: Fri Oct 18 15:31:43 CEST 2019

Cluster Replication State: PUB SETUP Started at 2019-10-18-15-29

Repltimeout set to: 300s

Cluster Detailed View from fin-a (2 Servers):

PING DB/RPC/ REPL. Replication REPLICATION SETUP

admin:file list activelog cm/trace/dbl/ date details

Server Time: Fri Oct 18 15:33:53 CEST 2019

Cluster Replication State: BROADCAST SYNC Started on 1 server(s) at: 2019-10-18-15-33

Repltimeout set to: 300s

Cluster Detailed View from fin-a (2 Servers):

PING DB/RPC/ REPL. Replication REPLICATION SETUP

admin:file list activelog cm/trace/dbl/ date details

[10/18/2019 15:32:39] dbname passed [ccm12_0_1_10000_17] local-dbname [ccm12_0_1_10000_17]

[10/18/2019 15:33:09] ====================================== locateFailedServer End

file view activelog cm/trace/dbl/20191018_153356_dbl_repl_cdr_Broadcast_ccmdb.log

Oct 18 2019 15:34:05 ------ Table scan for ccmdbtemplate_g_2_ccm12_0_1_10000_17_1_147_typedberrors start --

Node Rows Extra Missing Mismatch Processed

Oct 18 2019 15:34:05 ------ Table scan for

Node Rows Extra Missing Mismatch Processed

Oct 18 2019 15:34:06 ------ Table scan for

Server Time: Fri Oct 18 15:43:36 CEST 2019

Cluster Replication State: BROADCAST SYNC ended at: 2019-10-18-15-35

Repltimeout set to: 300s

Cluster Detailed View from fin-a (2 Servers):

PING DB/RPC/ REPL. Replication REPLICATION SETUP

Tools-> Datastore Control Center-> Replication

admin:show uccx dbreplication servers

SERVER ID STATE STATUS QUEUE CONNECTION CHANGED

utils uccx database dbserver integrity

CCX Serviceability - UI Driven Command Line Interface Driven

Reset Replication Reset Replication

Enable/Disable CDS/HDS forcedatasync

Synchronize Data dbserver integrity

Tools-> Datastore Control Center-> Replication