ssrnoig ‘SAP Startup Troubleshooting Gude for Netweaver Application Server- Applicaton Server Infrastructure - SCN Wiki Community WIKI SAP Community Were, Gast Lon 7 ‘Search he Conran ‘Applicaton Sever infastctue / SAP Glont Server Technology 1 SAP Slr Sonice SAP Startup Troubleshooting Guide for Netweaver Application Server & I Purpose “This document was designed to help SAP customers to troubleshoot andresolve the maarityf SAP system startup problems own their own, minimizing the business impacts and costs attached toa standard support ticket lifecycle. As ime goes by, new tops and startup error scenarios willbe added to this guide, Stay tuned! NEWS from December 2016: if SAP is running on AIX 7.1 0r7.2 and you are unable to start SAP after updating the AIK ead the SAP note 2402207, [1 Pretiminary Analysis ‘his initial checklist will approach mandatory settings that have proven to be root cause of startup issues in several occasions. With tisin mind, please *DONOT" skip any phases. 111. IS DATABASE UP AND RUNNING? “hiss absolute prerequisite for SAP SYSTEM to start. In order to test this, operating system command fSirans-@_ can be executed {as SIDadm user). ‘Any return code different than 0 means that DB cannot be reached by DIALOG work processes, and so SAP system wil nt initiate, The following return indicates that database is up, running and reachable by SAP work processes: Kone (release 742 = 18.11.14 = 20:14:09). ‘© Theres NO POINT in moving the analysis further while the return codes different than zero. ‘= Incase the return codes different than zero, please involve your DBA/Network/OS teams. 1.2. 1S SAP STARTUP SERVICE CORRECTLY DEFINED? UNIX: File /usr/sap/sapservices" must contain the correct service definitions. D_LIBRARY_PAT=/ 1D>/exe:$LD_LIBRARY_PAT >Jexe/sapst: pf=/ust/sap//S¥s/profile/_<: >_shostnane> WINDOWS: 1 Run"services msc Look forthe relevant service SAPeSID>_; ‘Check the commandline assigned tot : \uer\sap\\\oxe\sapstartery pf=: \uer\sap\\SYS\profite\ 1.3, START PROFILES: hips sn. sap. comikidsplay/SUSAP-*Startup+Troubleshooting+Guidefor+NatweavertApplicalon+Server nassrnoig ‘SAP Startup Troubleshooting Gude for Netweaver Application Server- Applicaton Server Infrastructure - SCN Wiki Inorder to correctly start the required processes such as message server, enqueue server and the disp work, the SAP startup agent (sapstartsrv} needs tofind a correct scenario for that. This scenario consists on alist of "Start Program_ USER: Itis mandatory that SIDadm user should be able to log.on at operating system level WITHOUT ANY" other prompts suchas: Company policies; User policies; Next scheduled downtimesy/System news; Longin as another user and SUdoing to SIDadm; ‘This can cause several problems throughout different scenarios, bt mainly on upgrade procedures. Further details on SAP note 1201712. 1901712-UNIK sapstarsry: Determining the environment (15:18 SAP STARTUP AGENT (SAPSTARTSRV) RUNNING? “There mustbe ONLY ONE sapsartsrv process runing for eachinstance. Examples foreach platform below: Examples ‘As anexample, we willuse a centralized system with the following 03 instances running on the same host (ASCS, PAS and DI) The following output should eturnfor the operating system evel command: >/S¥S/protite sap//5¥S/prot ‘© Profile pattern names returned withthe command can be such as “START_ chostname>"Inolder releases. ‘© There can be another sapstartsrv process (as below!) but it will lay no partsin the startup ofthe instance, hips: sn. sap. comikideplay/SUSAP- Startup+Troubleshooting+Guidefor+NtweavertApplicalon+Senver anassrnoig ‘SAP Startup Troubleshooting Gude for Netweaver Application Server- Applicaton Server Infrastructure - SCN Wiki “© When sapstartsr service snot running you could run manually the above command for the desired instance «© Ifthe service doesnt start and nothing it's written n “sapstartsrvog" it might happen the binaries are damaged inthe local exe folder. In this case has tobe replaced WINDOWS: ‘You shoud frst configure task manager output soit splays the command line related tothe selected process: Run task manager — Switch totab "Processes" —- Menu "view" — “columns” ~ PID and ‘COMMAND LINE} “Then using task manager tab "Processes look forthe instance sapstartsrv process with similar : \use\ sap D>\\exe\sapstartary pf-: \uer\gap\\SYS\proCite\ 1.6, 1S THERE ANY SAP PROGESS ALREADY RUNNING? Alter trying to start the instance (ether through the SAP MMC, the "startsap" command or using "sapcontrol. function Star’), no error is returned, However, the SAP processes (Dispatcher -ABAP: jstart- Java) do not start No trace files updated ether. ‘Verity whether any process is already in GREEN" or "YELLOW' status (eg, the IGS) 2.10.20) seas swork, Dispateh Stopped yy iit inane eiciadm AL> wees All processes must have the "GRAY" status. Otherwise, the SAP Startup Agent (sapstartsrv) will ignore! the start command. Stop the instance ist, soll processes are stopped. Then, start the instance again. 1. PERMISSIONS: UNIX: "TIMP" DIRECTORY: “tmp directory’s permissions must match the following. The final‘ for indicates sticky bit: dewxewnewt 22 © 927637 -Web service authentication in sapstartsrv as of Release 7.00 ‘SAPSTARTSRV.LOG: ‘This ile located on "fusr5ap///work/"should be ovmed by user. For specific cases when owed by root, sapstartsrv (running as user) will not be able to access the fle and thus the system wil fall to start Following entry is seen in ar/messages" open trace file sap: log ea) SAPUXUSERCHEGK FILE ‘The permissions for this ile should laok like this: ewer-x 1 128: Apr 06 2015 sapuxuserchk hips: sn. sap. comikidsplay/SUSAP-*Startup+Troubleshooting+Guidefor+NatweavertApplicalon+Server anassrnoig ‘SAP Startup Troubleshooting Gude for Netweaver Application Server- Applicaton Server Infrastructure - SCN Wiki ‘There's one SAPUXUSERCHECK file per instance /ust/sap// /exe) and the permissions and ownership should look alike forall files. The following command can help tracking the ies 1.8: IP / HOSTNAME RESOLUTION: LOCALHOST startup commands from sapcontrol will be using "localhost" implicitly (hen no '-host” option is explicitly informed) and fr such the loopback IP 127.00.1" should be resolved fort. What isthe output for ‘niping-v-H localhost"? The loopback IP should be the only ene returned. Ifnot, please involve your network team. 1147638 Wrong localhost name resolution setting, HOSTNAME ‘What IPs resolved fo the local hostname (output of OS level command niping-v-H )? There sa known error described in SAP note 1054467 where the loopback IP 127.0.0.1is wrongly resolved for it 11054467 -Local host name refers to loopback address l 2. IDENTIFYING AFFECTED STAGE ‘This troubleshooting (starting from here) ‘considers TRUE* that all pints from the initial checklist are done. Please keep in mind that all of above steps ‘were taken from real incidents and they were in several of those cases, the required fixthe forthe ssue resolution. The very fist question that needs to be answered is: which component/process has failed? Looking atthe image below, it possible to find out the sequence is which the logs willbe written _and therefore where the specific component information can be found, ‘Trace eof the startup procedures — Located in Inome/adm directory Standard trace fle ofthe message server Time: ‘Standard trace fles ofthe aispatcher ‘Standard trace fles of SAP workprocesses “The main approachis to begin forthe latest updated files, based on this image's timeline. As an example, incase the work processes traces (dev_wit#) are not updated after the latest failed start attempt, then the dispatcher trace (dev disp} needs tobe verified and soon. [| s.oisparcHer (dev dsp): 1 Dispatcher is usually the frst and most important trace file analyzed for startup issues Most ofthe times the reason forthe startup failure willbehere. htpssWik sn. sap. comikideplay/SUSAP- Startup+Troubleshooting+Guidefor-NatweavertApplicalon+Server anassrnoig ‘SAP Startup Troubleshooting Gude for Netweaver Application Server- Applicaton Server Infrastructure - SCN Wiki Inthe cispatcher trace, wel ntally look for critica error situations, which directly contributed to the startup fll. fsuch straightforward occurrences, ‘cannot be found, then we ook for any situation which would, indirectly, lead to the same situation. As an example, another component has crashed, ‘therefore dispatcher has to shut itself down, Example of error where't's possible to see the cause right away: s#+20G QoK—> DpMantcach, macon { ) (dpMessageser 2652) s#t Dp SAGAL EEROR => Dpésaetacht local Rostnane ‘server name>" is cesolved to loopback address (cf. SAP note 1054467 for details) ~ SES"DISPAECHER EMESGENCY SHUTDOWN *** Example of dispatcher startup error caused by the crash of all work processes. The very next step inthis case is to check the “dev_we" traces and find out whats killing the disp work processes and, ultimately what isthe cause ofthe instance fallure: +s ERROR => Dplidloeadip: WE4 (pid 1898) died (severit Dptracewascatue: child (pide1898) exited with exit cose 2 wth "Dp PAPAL EAROR => DpWPCheck: no nore work processes S++ DISPATCHER EMERGENCY SHUTDOWN **" 1 status-65280) [dpxcep.c 1739] ‘** IMPORTANT: > Whenever suspicious about a dispatcher crash andthe following entries are found: Please keep in mind that a manual shut down happened. This can be proved by the "signal 2" entry and also by the (normal)sign next tothe shut down sign, ‘:: MOSTCOMMON DISPATCHER CRASH SITUATIONS: Inthis ist we bring the most common causes for a "dispatcher crash, along with its resolution: LTSLEPORT_JNUSE": ‘These error entries are preventing the dispatcher from initialize and they are not caused by SAP but network inconsistencies. tthe time dispatchers trying to star and bind its default port 22itfinds that there is alteady a process binding this port, thus, it aborts lagging below entries: 6 hie) WitbindSocet: bird (0812: WEALACEES: Persson dened) {nlmn.cop 3788) < erable. (Gi from jue /ctyy 1) st ee.b.a00) [atedcop 30) SSPexnon=s oocaeninite iautscten teens) [pedisore 203 hips sn. sap. comikideplay/SUSAP- Startup+Troubleshooting+Guidefor+NetweavertApplicalon+Server enassrnoig ‘SAP Startup Troubleshooting Gude for Netweaver Application Server- Applicaton Server Infrastructure - SCN Wiki RESOLUTION: ‘You must involve your network team and find out what process is listening on part 32instance,number> and ensure thatthe dispatcher ports free at the timeltis starting up. Asan example, in UNIX you can try the OS command "netstat -an |grep 32" when searching for a process Tstening on that port. |LPLATFORM PATCHNO...BUTMY PATCHNO IS: Following errors sen for scenarios where the path release of dispatcher and message server isnot the same. This can happen for couple of reasons but ‘the major number of occurrences's related to issues on SAPCPE wiich failed to copy newest fils from the GLOBAL EXE folder tothe INSTANCE EXE folder, resulting in mixed releases. se un) Sie anal RESOLUTION: ‘You must ensure that ll instances are running wit the same kernel release. Asa potential fx, youcan copy manually the files fromthe global "EXE" {older tothe instance "EXE" folder. |ILBAD MEMORY PARAMETRIZATION: an o> sia. AacreneSnaed: exsinathare 0) return 2 fesusiad.c 2484) eso © botainite atone (0) epmeiep oR 22 ERROR © pterEnLt: opemintt (2) (ot » SES. Vale ange 12 2. 46 (enoutez.c 181 RESOLUTION: hips: sn. sap. comikidsplay/SUSAP- Statup+Troubleshooting+Guidefor+NatweavertApplicalon+Senver enassrnoig ‘SAP Startup Toubleshooting Guide for Netweaver Application Server Application Server Infastwture - SCN Wik “Thereare.a few interconnected memory parameters that must reach each other requisites. Mainly following SAP notes have tobe consultee! #1998299 -IBM :SHM2_stnit invalid argument ES/SHM,MAX_SHARED.SEGS 1#1014010- EG memory upto 32 GB (em/global_area. MB up to 32000) ‘And mainly those two rules shouldbe reviewed: MSLEADDR NAVAIL: “This erroris commonly seen when “localhost” cannot be resolved 25°127.0.0.1" sige gure» nstnndsoceat: ied (68: Can't assign requested sates) nten.cpp 239 (Gi Soon jen; as be me yzeziw300) [neetegp "3239, RESOLUTION: isresolved as loopback IP "127.0.0.1" and vice-versa ‘See PRELIMINARY CHECKLIST above, topic [1.7] and ensure tha NO MORE WORK PROCESSES: Dispatcher is shutting itself down because there are no more work processes avallabe for processing of requests low the initialization entries and soon the ERROR entries with the Inorder to pinpoint the cause, work processes traces have tobe analyzed. Just root cause willbe seen. RESOLUTION: Whenever this errors the root cause fora dispatcher crash, it means that in ths very same trace you'llse lots of entries related to the crash of every single work process “These entries lok tke this hips: sn. sap. comikidsplay/SUSAP- Statup+Troubleshooting+Guidefor+NatweavertApplicalon+Senver massrnoig ‘SAP Startup Toubleshooting Guide for Netweaver Application Server Application Server Infastcture - SCN Wiki ERROR => DpHalDeadWp: WO (pid 21104) died (severty-0, status~65280)[doxxwp.c 1727] 1104) exited with exit code 255 DpTraceWpstatus: child (pi ‘We will then search in the dispatcher trace forthe very first occurrence of work process that crashed with entries similar tothe one above. “There we ee that the work process 0 (zero) died. Then we need to change the Investigation to the work proces trace. Inthiscaseitwill be the ‘dev_w0" located a the work directory ofthe instance. CCheck the 'usr/sap///work/dev. wil" trace to find out why work processes are being killed, Most common cause are problems to reach/connect to database, (FAILTO SPOT TOPIC [1.1] FROM PRELIMINARY CHECKLIST) Vi DISPATCHER NOT ALLOWED TO CONNECT TO MESSAGE SERVER: ihecane Sa ispnssenytnarnal) Lépedtsne ete! harden server rseh0 e680 (ror) RESOLUTION: “SAP note # 1285956 - Not allowed to connect to message server ‘Vis GATEWAY COULD NOTBESTARTED": Straightforward. Dispatcher shut down as the gateway reader could not be started. 28 => gateny (pid 2908) ded (omndsp.€ 1872 RESOLUTION: Gateway trace file "dev_rd should be checked forthe rootcause. Avery common cause isthe unavailability of the gateway’s default port '38cinstance NR» during instance startup. IF this isthe case, the following entries are sen in "dev." hips: sn. sap. comikideplay/SUSAP-+Startup+Troubleshooting+Guidefor-NatweavertApplicalon+Sorver anassrnorg (SL EPOnT_twst/2a6; Ta; 51; #8. 02330) Tain cp 37381 In this case the instance number was 10, hence "0.0.00:3310" Ifyou face the same situation, you have to involve your network/OS teams and find out what process has already bound the port. IIL Es2iResCreateShm: shmgetfalled: “Tis issue is mainly seen on SAP systems based on HP-UX servers. > csniestremesm: emgetfailee(otze274R581248, Cc OF kre 5D GSptscresterties: tavtesernte“esles [esnse 2) 5S Ehtecrete: tamescrenctites salt [erences 307 Dh apenimacs esanescrente (1) iptdtee ra Een © boreraits openiie Cs) f@uniee 34), sreter spanax, (22: Enalsd argurert) (estou 21) RESOLUTION: ‘You must correctly st the OSkernel parameters as per below SAP note: #172787 -SAP on HP-UX: OS kernel parameter recommendations Below SAP nates can provide additional nfo as well: #180251 -HP-UX 64 Bit Kernel, Startup schlaegt felt #1677388 - System cannot start due to Es2IResCreate -shimgetfalled- Fle exsts error ‘4 MULTICASTERROR: ‘SAP Startup Troubleshooting Gude for Netweaver Application Server- Applicaton Server Infrastructure - SCN Wiki ‘This erroris present only as of kernel 740. The SAP Kernel release 740 uses a multicast address at the network 224.0.00/24 to dispatch tasks tothe work processes. ‘When required settings related to multicast are notin place the logged: (Gis MAsL/i. 14; D6; THPROT.IP<70-A.PEMOERSAIP) [asl cop 954 Ba 0.0120 incartoce Qube (ober.cpp 9572] Ets © fagine: Mogi otneoap failed (reoi8) epguecee 353) hips sn. sap, comikidsplay/SUSAP- Startup+Troubleshooting+Guidefor-NatweavertApplicalon+Senver tance willnot start andin the “dev cisp.new* trace file, the following entries are onassrnoig ‘SAP Startup Toubleshooting Guide for Netweaver Application Server Application Server Infastcture - SCN Wiki RESOLUTION: -> For customers with kerne! 740, the following notes guide you on how to set up multicast ina correct way: #1982555 - Dispatcher startup issue with SAP Kernel 740, #1951675 -1P multicast configuration for dispatcher wakeup mechanism -> For customers with kernel equal othigher than 741 PL.47, SAP does reccommend the usage of 2 new implementation based on events, This features enabled with the following parameter (which only exists as of kernel 741 PL 47} ralsp/queue/use_events for lspatching=on ~> This parameters default values ON starting from SAP kernel 742. XCAPPLICATION SERVER'S SHARED MEMORY CORRUPTION: ean => 22 sono 344909, (8-1 614), ) (22 Tals anger) eanune 577) i Ext opresiotaiocae: Stig feouttoola.¢ 633) SS ester 2 Cz TMeecussea,(eravena).2) Gas Soulia argarent) [eenn.e 577) RESOLUTION: ‘As these entries are related toa crash other than filed startup attempts, the resolution would be to start the instance once again, “The easiest way to reproduce this scenario (and most common cause ofthe crash sby removing the instance’ IPC keys on runtime wich, by obvious reasons, “SHOULD NOT BE DONE* #1641668. Various semop errors in workprocess traces XL WRONG EG (EXTENDED GLOBAL MEMORY) CONFIGURATION: si extn = cgnnts request sopene 2520 (48532736208 bytes) ton Large Ma Hie = 2ST yer [epuce 244) Ets © esennseeentnse (3) epeasen.© 0737) on 2 fomernits apenidt (5) eonréep.e 248731 RESOLUTION: ‘The instance will fail to start andlog the above entries in case the EG configuration is not maintained as per below SAP notes: hips: sn. sap. comikidsplay/SUSAP- Startup+Troubleshooting+Guidefor-NatweavertApplicalon+Sorver sonssrnoig ‘SAP Startup Troubleshooting Gude for Netweaver Application Server- Applicaton Server Infrastructure - SCN Wiki #705365 - Extended Global Memory (EG) increased from2.GB te 8 GB #1014010 EG memory up to 32 GB (em/global area MB up to32000) © Hint £By following this logic, any dispatcher related probem can be handled, even for different logged errors. © tint ‘There are SAP documentation for most of the possible cases. Research on them. SARA) 04. SAPCONTROL'S INSTANCE MANAGEMENT COMMANDS ‘SAPSTARTSRV (through SAPCONTROL interface) offers a variety of web methods that canbe used to monitor/manage the instances ofthe system. “They are specially useful for handling startup error scenarios. The "must-know" ones are listed below and als he general pattern for executing them: Getsysteminstancezist oR hostname, instancelie, hetprort, festservor001, 0, 50023, Eesteeeverdol, 1, 501 testserverd01, 2, 50213, 5 spsvort, startPriority, feat $0014, 3, ABAD GATENAY|TCMAN| 105, GRE: ) MESSAGESERVER| E4QUE, GREEN 3) ABAP GATEWAY| TOMAN| FoS, GREED 5, dispstatus intgol ~ne -function 41: "GetSystemlnstancelist’ “This one returns the lst of active instances forthe system, along with component list and the current status. Getsysteminstanceiist on hostname, instanceNr, hetpPore, hetpsPort, starteriority, feat testserverd01, 0, 50013, 50014, 3, ASAP GATENAT| ICMAN| IGS, GRE EestserverD0l, 1, 50113, 50114, 1, MeSsAGESARVEn| EnQUE, GREEN testserverd01, 2, 50213, 50214, 3, ASAP GATENAY| ICMAN| ToS, GRR! 5, dispstatus EXPECTED OUTPUT: ‘One entry per instance ofthe system, including ASCS, SCS INSTANCES; [GREEN values forall entries on column “aispstatus. Forallinstances running onthe same host (column hostname), there must be a differente value on “instanceNr, 42;"GetProcessList”: htps:Wik sn. sap. comikideplay/SUSAP- Startup+Troubleshooting+Guidefor-NatweavertApplicalon+Server nnassrnoig ‘SAP Startup Troubleshooting Gude for Netweaver Application Server- Applicaton Server Infrastructure - SCN Wiki .GetProcessList web method wil bring the component ist currently in lace on any of the instances seen in te first command. It ill omen handy as soon as the web method "GetSysteminstanceList” returns one instance reporting as YELLOW. This will happen when at least one component ispresenting some kind of prober, so now itis posible to check that affected process with more details: GetProcessList ok name, description, dispstatus,textstatus,starttime,elapsedtime, pid disp+work, Dispatcher, GREEN, Running, 2015 03 23 23:51:95, 268:27:26, 25549 legswd. mt, IGS Watchdog. GREEN, Running, 201503 23 22:51:25, 263:27:26, 25550 ‘evrd, Gateway, GREEN, Running, 2015 03 23 28:51:26, 2632725, 25590 ‘cman, ICM, GREEN, Running, 2015 0323 23:51:36, 263:27:25, 25591 43: "GetVersioninfo": Great for kernel consistency checks, this output will return the current kernel release + patch for each and every ofits components within the instance informed as argumentin the command ine Excluding the DB Iibraries all the remaining components of the instance must always share the same kernel and patch release on aconsistent scenario, us sop sabre sapstartsre, 72, pater 2, chong (ev 26 834, 38:45:48), Une nasi /ar/eanaso(oveousoeexe/aepocork, aa, pateh 2, cangeist asian, MKS cengeiniity Level, oped (Ww 36 Dole, sD:4ses), iinuaaay Soke 32 26-1093 Seti (ov 35 tue, so:4sei), iroussa, Send Le 2848 iy Level, ope (Nor 36 034, 15-44), leno, 201830 Je ves abgu (un 33 2s, 2-250), Linu, 014 2 8.37 iy Level 8, apt (iv 26 22, 918543}, noe, 20 12 26 3 dota, bein tin pe (hor 26 za, ast) EXPECTED OUTPUT: All entries must share the same kernel release and patch. In this case 742 patch 28 (withthe exception of the ones starting with ab) © Hint ‘There are hundreds of web methods available forthe most diverse cases. They canbe listed by typing just “sapcontrof* command as S|Dadmuser. tps: sn. sap. comikidsplay/SUSAP-*Startup+Troubleshooting+Guidefor-NatweavertApplicalon+Senver venassrnorg ‘SAP Startup Toubleshooting Guide for Netweaver Application Server Application Server Infastture - SCN Wiki © tint ‘These instance management commands can help to quickly find out version and specitic component details S| concERNING THIS DOCUMENT. “The original concept ofthis documents that be never finished. Instead, constantly improved, based on the incoming feedback. For this reason, you are invited t share whether or not the information present here helped you to solve a startup problem and, even more important, incase it did't. In ‘that case you are weleame to share your own scenara and inform what was the slution applied, constraints, specifics Ex] COMING SOON. ** STARTUP issues faced during uperade procedure. hips: sn. sap. comikidsplay/SUSAP- Statup+Troubleshooting+Guidefor+NatweavertApplicalon+Senver sana