IBM Software Group

Troubleshooting on WebSphere Voice Response 4.2 on AIX

Presented by Yuchin Fu yuchinfu@us.ibm.com

WebSphere Support Technical Exchange

IBM Software Group

Agenda
Objectives Common problems / solutions WVR traces useful PD tools Java / VXML specifics References / Resources Q&A

WebSphere Support Technical Exchange

IBM Software Group

Session Objectives
Resolve / avoid some common problems Identify possible problem areas Know how to use common PD tools Know how to run different WVR traces Locate more problem solving information Gather useful data for PMR to work with IBM Support

WebSphere Support Technical Exchange

IBM Software Group

Product components
WVR is built up with the components: base runtime Java / VXML application environment State Table application environment Telephony interfaces

WebSphere Support Technical Exchange

IBM Software Group

Installation FAQ / common issues

No adapter / trunk available after installation  AIX 32-bit kernel for systems with voice adapters What is the supported environment  AIX 5.2 with ML 2 or later or AIX 5.3 with ML2 or later  Java 1.4.1 is the pre-req also support JDK 1.4.2 or 1.5  Communications Server (sna) 6.1 or 6.3  DTXA / DTTA / DTEA placement details available at http://www306.ibm.com/software/pervasive/products/voice/universalaccess_2_125.shtml

WebSphere Support Technical Exchange

IBM Software Group

Migration common issues / solutions

saveDT / restoreDT or DTdatabase m  Follow the steps in Installation (GC34-6380)  Install the latest WVR PTF (AIX 5.3 requires WVR 4.2.0.181 +)  reboot after all installation before DTdatabase -m  Check correct hostname in db2nodes.cfg  Check TechNote (1235594 ) for DB2 LIBPATH  Do NOT backup & restore /home/dtdb23in  WVR user (dtuser) and dtdb23in in same group (staff)  hostname defined in /etc/hosts collect /tmp/DTdatabase.out.[pid]

WebSphere Support Technical Exchange

IBM Software Group

Cloning checklist to avoid problems

(mksysb) can be used to build additional WVR systems. After loading mksysb image and change the hostname - check /etc/hosts for correct hostname / IP address  run WVR fsupdate script  check $DTJ_DIR/default.cff on hostname and IP address  check /home/dtdb23in/sqllib/db2nodes.cfg  sna and 3270 session configuration need fresh setup  Check TechNote (1228218) for LUM configuration reset  set adapater ownership: dt_setowner q[adapter ID] to query current setup dt_setowner s[adapter ID] for DTXA dt_setowner x s[adapter ID] for DTTA

WebSphere Support Technical Exchange

IBM Software Group

Start with the basics

identify the date / time of the events Check WVR System Monitor check the logs from $OAM_LOG_PATH
errorlog DTstatus.out

check the logs from $DTJ_HOME/native/aix for DTJxxxx

[VR node].out [app node].out dtjflog o [output] log.[n].log

AIX error report: errpt -a

Developing Java Application (GC34-6377) Problem Determination (GC34-6382)

WebSphere Support Technical Exchange

IBM Software Group

Start with the basics (cont)

If VRBE related, query system status with
dtjqhost dtjqnode dtjqapps

check Javacore / Javadump in native/aix if JVM node disappeared

WebSphere Support Technical Exchange

IBM Software Group

WVR base runtime trace

$ trace a 1 L [trace file size] T [buffer size] $ trcstop 1 $ print_trace > [ASCII trace file]

setup and limitations

 AIX trace for the whole WVR system  pre-defined trace file and trace buffer  events could be wrapped out on busy system in short time frame

Information provided
 Most of the WVR background proc events (DBHEALTH, DBIM, etc)  Signal events on voice channels (ring, answer, transfer, call terminate)  State table application actions and returned edges Pub:Problem Determination (GC34-6382)

WebSphere Support Technical Exchange

10

IBM Software Group

WVR base runtime trace (cont)

Would be helpful for issues like
    call events from CAS protocol (answer, transfer) application flow from state table applications Calls answered with wrong applications events related to the specific error_id in WVR errorlog

Use with the DTalarm daemon

 update $SYS_DIR/dtalarmd.update with trace stop command like alarm_id <ID> command sleep 5; trcstop 1  refresh DTalarm definition $ DTalarm u  start trace $ trace a 1 L 8000000 T 1000000  run print_trace after the error_id event pub: Managing and Monitoring the System (GC34-6384)

WebSphere Support Technical Exchange

11

IBM Software Group

WVR base runtime trace (cont)

Verify trace is valid for the event
Check the top lines in the trace output -Fri Sep 1 16:07:02 2006 date/time when trace is turned on System: AIX 5.2 Node: beech1 system OS and hostname information Machine: 000C751F4C00 Internet Address: 092A7926 9.42.121.38

Verify if the specific WVR error_id was logged

$ grep ERROR_ENROLL <ASCII trace file>

Collect dtProblem output soon after the trace

WebSphere Support Technical Exchange

12

IBM Software Group

WVR VRBE trace

set up and limitations
 setup from WVR VRBE trace daemon  binary trace files $DTJ_HOME/dtj_logs/wvrtrace.*  trace option reset after stop / start VRBE environment  Trace events can be wrapped out quickly on busy system  NO WVR base runtime events  NO signal interface events

Would be helpful for issues like

 unexpected flow or event from VRBE VXML applications  DTJXXXX errors in <node>.out  problems to start or stop VRBE environment  Other issues from the Java / VXML environment

WebSphere Support Technical Exchange

13

IBM Software Group

WVR VRBE trace (cont)

How to turn on VRBE trace
 Stop VRBE environment: dtjstop; dtjshost exit; trcserv -k  Edit $DTJ_HOME/dtj.ini to have trace.option=9  Edit $DTJ_HOME/native/aix/trcserv.ini to have trace.mode=3  Start VRBE environment again: dtjshost, dtjstart

How to collect trace data

 Collect dtbeProblem output immediately after the event  tar up all files under dtj_logs as backup

turn off the VRBE trace

 trace events will be logged while trace.mode=3 is set  remove the trace.option and trace.mode to avoid extra system load  restart VRBE environment after above changes

WebSphere Support Technical Exchange

14

IBM Software Group

Combined WVR runtime and VRBE trace

set up and limitations
 redirect VRBE trace into WVR runtime trace file  trace file can be wrapped out event sooner with combined events  may need to start runtime trace with larger trace file / buffer  stop the trace from command line or from DTalarm daemon

Would be helpful for issues like

 call flow issues involve state table and Java / VXML  Issues on the interface of VRBE and base runtime  Call flow issues involve Java / VXML and signalling

WebSphere Support Technical Exchange

15

IBM Software Group

Combined WVR runtime and VRBE trace (cont)

How to turn on the combined trace
 stop VRBE environment: dtjstop; dtjshost exit; trcserv -k  edit $DTJ_HOME/dtj.ini to have trace.option=9  edit $DTJ_HOME/native/aix/trcserv.ini to have trace.mode=5  start VRBE environment again: dtjshost, dtjstart  start runtime trace with larger trace file size and buffer size $ trace a -1 L 32000000 T 1000000

Turn off trace AS SOON AS POSSIBLE after the particular event

$ trcstop -1 $ print_trace | dtjtrcfmt > [output file] $ grep DTBeans [output file] VRBE trace events

WebSphere Support Technical Exchange

16

IBM Software Group

Some issues in VRBE environment

Java OOM exception in [node].out file
 Each node defined in default.cff is one JVM  Each node can have its own JavaCommand= line  A node with large number of instances, may need larger heap size  Setup in default.cff for the node with. For example --Xms32M Xmx448M will have the initial heap size of 32MB with the maximun of 448MB  To debug memory usage, add the option in JavaCommand line -verbosegc

VRBE environment fails to start

 All JVM communication is TCP/IP based  Non-responding DNS can cause dtjshost dtjstart failure

WebSphere Support Technical Exchange

17

IBM Software Group

Some issues in VRBE environment (cont)

All application instances are in use
 Instance could stay in use even after caller hang up  Application should set timeout check for all backend query

Calls are answered, but caller hear silence

 Play audio from local system before any backend query

WebSphere Support Technical Exchange

18

IBM Software Group

Must-Gather
dtProblem output dtbeProblem output if system has VRBE environment WVR traces http://www-1.ibm.com/support/docview.wss?uid=swg21240832

WebSphere Support Technical Exchange

19

IBM Software Group

Find out more help information

WVR product page:
http://www-306.ibm.com/software/pervasive/voice_response_aix/

TechNote search under Support page:

http://www-306.ibm.com/software/pervasive/voice_response_aix/support/

ALL WVR publication are available from this link:

http://www-128.ibm.com/developerworks/websphere/zones/voice/vresponse_aix.html

Latest WVR PTF information:

http://www-306.ibm.com/software/pervasive/products/voice/universalaccess_2_124.shtml

WebSphere Support Technical Exchange

20

IBM Software Group

Additional WebSphere Product Resources

Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: www.ibm.com/developerworks/websphere/community/ Learn about other upcoming webcasts, conferences and events: www.ibm.com/software/websphere/events_1.html Join the Global WebSphere User Group Community: www.websphere.org Access key product show-me demos and tutorials by visiting IBM Education Assistant: ibm.com/software/info/education/assistant Learn about the Electronic Service Request (ESR) tool for submitting problems electronically: www.ibm.com/software/support/viewlet/probsub/ESR_Overview_viewlet_swf .html Sign up to receive weekly technical My support emails: www.ibm.com/software/support/einfo.html

WebSphere Support Technical Exchange

21

IBM Software Group

Questions and Answers

WebSphere Support Technical Exchange

22