Diagnostics Agent Troubleshooting

SAP Solu Solu tion Mana Manager ger 7.0 EH EHP1, P1, 7.1 7.10 0

End-to-End Diagnostics Dia iag gnost stic ics s Age gen nt Tr Tro ouble lesh sho oot otin ing g Gu Guid ide e Version 3.0, August 2013

END-TO-END DIAGNOSTICS DIAGNOSTICS AGENT TROUBLESHOOTING

Type Style Example Text

Represents Represents Words or characters quoted from the screen. These include field names, screen titles, pushbuttons labels, menu names, menu paths, and menu options. Cross-references to other documentation.

Example text

Emphasized words or phrases in body text, graphic titles, and table titles.

EXAMPLE TEXT

Technical names of system objects. These include report names, program names, transaction codes, table names, and key concepts of a programming language when they are surrounded by body text, for example, SELECT and INCLUDE.

Exampl e t ext

Output on the screen. This includes file and directory names and their paths, messages, names of variables and parameters, source text, and names of installation, upgrade and database tools.

Example text

Exact user entry. These are words or characters that you enter in the system exactly as they appear in the documentation.

Variable user entry. Angle brackets indicate that you replace these words and characters with appropriate entries to make entries in the system.

EXAMPLE TEXT TEXT

Keys on the keyboard, for example, F2 or ENTER.

Ic o n

Mean i n g Caution Example Note Recommendation Syntax

END-TO-END DIAGNOSTICS DIAGNOSTICS AGENT TROUBLESHOOTING

Contents 1

Document Status .......... .............. ......... .......... ......... ......... .......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... .......... ......... ......... ....... .. 5 1.1 New New Document Loc ation .......... .............. ......... .......... ......... ......... .......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... ....... .. 5 1.2 Loc ation of Interm ediate Document .......... ............... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... .......... ......... ......... ....... .. 5 1.3 Old Documen t Loc ation ....... ............ ......... ......... .......... ......... ......... .......... ......... ......... .......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... ....... .. 5

2

Pre-requisit es ......... .............. .......... ......... ......... ......... ......... .......... ......... ......... .......... ......... ......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... ......... ......... .......... ......... .... 6

2.1 JDK comp atibi lit y ........... ............... ......... ......... ......... .......... ......... ......... .......... ......... ......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... ......... ......... .......... ..... 6 2.2 Managed Managed System .......... ............... ......... ......... ......... ......... .......... ......... ......... .......... ......... ......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... ......... ......... ....... .. 6 2.3 SLD p rod uct ion syst em ....... ............ ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... .......... ......... ......... .......... ......... ......... ....... .. 7 3 Agent ins tallatio n str ategy ......... .............. ......... ......... ......... ......... ......... ......... .......... ......... ......... ......... ......... .......... ......... ......... ........ ........ ......... ......... ......... ......... .......... ........ ... 10 3.1 3.2

Managin Managin g sys tem scalin g ...... .......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ........ ........ ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ........ ... 10 Managed Managed System envir onment ....... ........... ......... ......... ......... ......... ......... ......... ......... ......... ......... ........ ........ ......... ......... ......... ......... ......... ......... .......... ......... ....... ... 10

3.3 Managed Managed Systems HA/Failover envir onment ........ ............. .......... ......... ......... ......... ........ ......... .......... ......... ......... ......... ......... ......... ......... ......... .... 10 4 Agent ins taller............. taller.................. .......... ......... ......... ......... ......... .......... ......... ......... .......... ......... ......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... ......... ......... ........ ... 11 4.1 SAP Host Agent ........... ............... ......... .......... ......... ......... ......... ......... .......... ......... ......... .......... ......... ......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... ....... 11 4.1.1 E2E RCA RCA configur configuration ation for for Solutio Solution n Manager Manager 7.0x 7.0x or or higher ................................... .................................................. ............... 11 4.1.2 E2E RCA RCA config configura uration tion for Soluti Solution on Manag Manager er 7.1x 7.1x only................. only.................................. .................................. ........................ ....... 11 4.2 4.3 4.4 5

OS co nfig uratio n chec ks ........ ............. .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ....... 12 JVM parameters ......... .............. ......... ......... .......... ......... ......... ......... ......... ......... ......... .......... ......... ......... ......... ......... ........ ........ .......... ......... ......... ......... ......... ......... ......... ......... .... 12 Startu p scr ipts ......... .............. .......... ......... ......... .......... ......... ......... ......... ......... .......... ......... ......... .......... ......... ......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... .... 13

Agent con necti vity .......... ............... ......... ......... ......... ......... ......... ......... .......... ......... ......... ......... ......... .......... ......... ......... ......... ......... ......... ......... .......... ......... ......... ......... ......... ......... .... 17 17 5.1 Managin Managin g sys tem ......... ............. ......... .......... ......... ......... ......... ......... ......... ......... .......... ......... ......... ......... ......... ......... ........ ........ ......... .......... ......... ......... ......... ......... .......... ....... .. 17 5.2 Managed Managed syst em .......... .............. ......... ......... ......... .......... ......... ......... ......... ......... .......... ......... ......... ......... ......... ......... ........ ........ ......... .......... ......... ......... ......... ......... ......... ....... ... 19 Connection configuration for managed systems .................................................................................. 22 5.3 5.4

6

Self-Management Self-Management .......... .............. ......... ......... ......... .......... ......... ......... ......... ......... ......... ......... .......... .......... ......... ......... ......... ......... ......... ......... .......... ......... ......... ......... ......... ..... 22 SAP ro uter ......... .............. ......... ......... .......... ......... ......... .......... ......... ......... ......... ......... .......... ......... ......... .......... ......... ......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ....... 23

Agent Mainten Mainten ance ........ ............. ......... ......... .......... ......... ......... ......... ......... ......... ......... .......... ......... ......... ......... ......... ........ ........ .......... ......... ......... ......... ......... ......... ......... .......... ....... .. 24 6.1 Config urati on............ on................. .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ....... .. 24 6.1.1 Change Change Path of the JDK.................. JDK.................................... ................................... ............................... ................................ .................................... .................... .. 24 6.1.2 Change Change JVM Paramete Parameters rs .................................. .................................................... ................................ ................................ .................................... .................. 24 6.1.3 Replacin Replacing g JDKs JDKs 1.4.2 with SAP JVM JVM 4 ................................. ................................................... ................................. ............................... ................ 24 6.1.4 Verifying Verifying Logical Logical hostname hostname ..................................... ................................................... ................................ ................................. ............................... ................ 25 6.1.5 Usage Usage of smdsetup smdsetup script script.................................... ..................................................... ................................ ................................ ................................... .................. 25 6.1.6 Change Change Agent log level level .................................... ................................................... ................................. ................................ ................................. ....................... .... 28

6.2 Managin Managin g J2EE user ......... ............. ......... ......... ......... ......... ......... .......... ......... ......... ......... ......... ......... ......... .......... ......... ........ ........ ......... ......... ......... .......... ......... ......... ....... .. 29 7 Trackin g Agent issu es .......... ............... ......... ......... ......... ......... .......... ......... ......... ......... ......... ......... ......... .......... ......... ......... .......... ......... ......... ......... ......... .......... ......... ......... ....... .. 30 7.1 OSS message pre-requisi tes ...... ........... .......... ......... ......... ......... ......... ......... ......... ......... ......... ........ ........ ......... ......... ......... ......... ......... ......... ......... ......... ......... ...... .. 30 7.2 7.3 7.4 8

Agent crash invest igatio n........... n............... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... .......... ......... ......... ........ ........ ......... ......... ......... ......... ....... .. 30 Thread dump .......... ............... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... .......... ......... ......... .......... ......... ......... .......... ......... ......... ..... 32 Act ivate P4 log s ......... .............. .......... ......... ......... ......... ......... .......... ......... ......... ......... ......... ......... ......... .......... ......... ........ ........ ......... ......... ......... ......... ......... .......... ......... ........ .... 33

Scalabilit y........... y................ ......... ......... ......... ......... ......... ......... .......... ......... ......... ......... ......... ......... ......... .......... ......... ......... ......... ......... .......... ......... ......... ......... ......... ......... ......... .......... ......... .... 34

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

8.1 9 10

4

Soluti on Manager Manager J 2EE stack settin gs .......... ............... .......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ........ ... 34

Telnet SMD SMD ser ver co mmands ......... .............. ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... .......... .......... ......... ......... ......... ......... ......... ......... ......... ......... ....... 35 Full Qualif ied Name Name resolu tio n iss ue ........ ............ ......... .......... ......... ......... .......... .......... ......... ......... .......... ......... ......... .......... ......... ......... .......... ......... ......... ........ ... 36

©SAP AG 2013


1

Document Status 1.1 1. 1

New Ne w Docum ent Loc ation

The new location for the troubleshooting content originally covered by this document is on the SDN WIKI



Here http://wiki.sdn.sap.com/wiki/display/SMSETUP/Diagnostics+Agent+Troubleshooting

(also accessible via this “tiny” link http://wiki.sdn.sap.com/wiki/x/aQSAF http://wiki.sdn.sap.com/wiki/x/aQSAF))

1.2 1. 2

Locatio n of Intermediate Document

While we are reviewing the content of this document, and migrate it to the SDN WIKI, this intermediate version of this document will be available as an attachment to SAP Note 1833501, 1833501, as DiagnosticsAgentTroubleshooting.pdf.

1.3 1. 3

Old Document Loc ation

The old location of t his document is on the SMP (both this SPM location and the document on SMP are deprecated/obsolete) 

https://service.sap.com/~sapdownload/002007974700000409092009E/DiagAgent_TroubleShooting.pdf

©SAP AG 2013

5


2

6

Pre-requisites 2.1 2. 1

JDK comp atibilit y

The following table describes the li st of different JDKs supported by t he Diagnostics Agent for each O S platform: OS

Di ag n o st i c s A g en t 7.00 / 7.01

Server 2003/2008

Windows

5.3 64bit on Power 4, 5 6.x 64bit on Power 4, 5 11.31 on IA64 64bit 11.11, PA-RISC 64bit

AIX

HP

Solaris

Linux

2

10 on SPARC 64bit / x86_64 IA32 32bit / IA64 64bit x86_64 64bit Power 64bit SLES9 or higer on zSeries 64bit

Diagnostics Dia gnostics Agent 7.11

Diagnostics Dia gnostics Agent 7.20

Diagnostics Ag ent 7.30 SP2

SAPJVM5

SAPJVM6

SAPJVM6

Note 716604 Note 716927 Note 716927 Note 716926 Note 716926 Note 716604 Note 716604 Note 861215 Note 810008 Note 746299

I5/OS

V5R3, V5R4 64bit

Note 717376

Z/OS

1.6, 1.7, 1.8, 1.9 64bit

-

-

-

The Diagnostics Agent installation packages 7.1x, 7.20 and 7.30 include the SAPJVM and the SAP Kernel DVD. Therefore, it is not necessary to install separately the SAPJVM before installing one of those Agents. The above listed SAP notes could be helpful to determine the adequate JDK versionbut versionbut do not apply apply JVM paramete parameters rs p roposed in these SAP SAP notes fo r the Diagnostics Agent. (They are only intended to SAP J2EE Engines).

2.2

Managed Sys System tem

The following list describes the SAP systems for which SAPInst installs a Diagnostics Agent at the end of the SAP Instance (central or dialog) installation. Sy st em SAP NW 7.0/BS 2005(ERP/CRM/SCM/SRM 5.0 AB AP 5.0 SR3) based on AS AS ABAP

SMD SA Pi n st r el ease -

SAP NW 7.0/BS 2005(ERP/CRM/SCM/SRM 5.0 Java/Java + ABA P 5.0 SR3) based on AS AS Java/Java SAP Solution Solution Manager Manager 7.0 (SR4)

SMD SAPinst 7.0 (SR3)

SAP NW 7.01 SR1/BS 2008 SR1(ERP 6.00 EHP4) AB AP EHP4) based on AS ABAP SAP NW 7.01 SR1/BS 2008 SR1(ERP 6.00 EHP4) Java/Java + ABA P EHP4) based on AS Java/Java

SMD SAPinst 7.01 SR1

2

Read SAP note 171356 before to install the agent

©SAP AG 2013


7

-

SAP NW 7.1 (SAP PI/CE) SAP NW 7.11 (SAP PI/CE/Mobile/AS ABAP/AS Java/Web Dispatcher/Workplace )

SMD SAPinst 7.11

SAP NW 7.2 (SAP CE/AS ABAP/AS Java/Web Dispatcher/Workplace)

SMD SAPinst 7.2/7.30

SAP NW 7.3 ( AS AS ABAP/AS ABAP/AS Java/Web Java/Web Dispatcher/Wo Dispatcher/Workplace rkplace)

SMD SAPinst 7.2/7.30

IMPORTANT: In any case refer to the SAP note 1365123 1365123 and and its attached document.

2.3 2. 3

SLD pro duc tion syst em

The SLD CIM Model version must be 1.5.26 or higher to higher to be able to register a Diagnostics Agent. Agent. How to d ete etermine rmine if I need need to registe registerr my age agent nt i n SLD or not? Refer to the SAP note 1365123 and its attached document  How to d ete etermine rmine which version of CIM Model Model is i nstalled? Go in SLD with url http://sldhostname:port/sld  Logon with J2EE user with role SAP_SLD_ADMINISTR SAP_SLD_ADMINISTRATOR ATOR  Click on link ‘About’, ‘About’, the CIM Model version is displayed in a window popup.  How to upgrade th e CIM CIM Model? Refer to SAP Note 669669 669669 / / Updating the SAP Component Repository in the SLD 

©SAP AG 2013


8

How to check i f the Diagnostics Diagnostics Agent is k nown in a gi ven SLD SLD in order to connect it to Solution Manager? Manager? Go to the SLD with URL http://sldhostname:port/sld  Logon with J2EE user with role SAP_SLD_ADMINISTR SAP_SLD_ADMINISTRATOR ATOR  Go to the Administration section  For SLDs based on NW 7.10 and and higher, click on the link ‘Maintenance’ / ‘CIM Instances’  For SLDs SLDs based on earlier NW NW releases, releases, click click on the link ‘Content Maintenance’  Select the subset: ‘All with Instances’ and Class: ‘SAP_DiagnosticsAgentInstance’  N.B.: No Diagnostics Agent is currently registered, in case the class ‘Diagnostics Agent Instance’ does not appear in drop down list. In column column “Name” of of table, table, check check if the desired Diagnostics Agent is present.  Or since Solution Manager 7.1 use rather the step “Connect Diagnostics Agent” in transaction solman_setup, scenario “System Preparation” This step displays for the select ed SLD the list of known Diagnostics Agents Remind that in case the landscape includes several SLDs, having an SLD bridge forwarding in p lace, lace, the above above approach approach wil l ONLY allow validating that a specific Diagnostics Diagnostics Agent is registered in one of those SLDs. But it does NOT necessarily mean that this Agent is directly regist ered in the precise SLD, where the look-up is b een een perfo rmed, like explained above. This means, means, to be able to connect Diagnostic Diagnostic s Agents using the “ Connect Diagnosti Diagnosti cs Agent Ag ent s” featur feat ure e in solm so lm an_setup an_s etup trans tr ans act io n / Syst Sy st em Preparat Pr eparat i on or in Managed System Sys tem Configuration / Assign Diagnostics Agent, it is im portant to select the SLD SLD in which t he Diagnosti Diagnosti cs Agent is directly registered. registered. This direct registration t arget arget SLD is decided and configured in most cases at Diagnostic Diagnostic s Agent installation ti me. N.B.: N.B.: Remind to have a look at the runt ime.properties ime.properties file of the Diagnostic Diagnostic s Agent, in order to find o ut in whi ch SLD the Agent is directly registered registered in.

Which J 2E 2EE E Group is assigned to the SLD user user u sed by t he Diagnostics Diagnostics Agent? The J2EE SAP_SLD_DATA_SUPPLIER group or the security role DataSupplierLD should be assigned to the  user used by the Diagnostics Agent. What is the SLD registration registration freque frequency ncy of Dia Diagnostic gnostic s Agent? Every 12 hours, the Diagnostics agent sends sends an SLD request to update the agent agent model information.  What is the frequency frequency on which t he Diagnostics Diagnostics Agent ch ecks within it s “ associa associated” ted” SLD for Solution Manager connectivity data updates? If the Diagnostics agent agent is registered in an SLD and not yet connected to a Solution Manager Manager System, it will will  check every 20 minutes if an association has been created in that SLD, using the “Connect Diagnostics Agents” feature in solman_setup transaction / System Preparation or in Managed System Configuration / Assign Diagnostics Agent. If the Diagnostics agent agent registered in an SLD and currently connected to a Solution Manager Manager System, System, it is  going to check every hour if a new / modified association has been created in that SLD.

©SAP AG 2013


9

Why is th e Diagnostics Diagnostics Agent not visible in th e Agent Agent Admin istration UI? Verify the below listed items, in case the Diagnostics Agent is not connecting to the Solution Manager system  after you used the “Connect Diagnostics Agents” feature, via solman_setup transaction / scenario: “Managed System Configuration” / Step: “Assign Diagnostics Agent”, or scenario: “System Preparation”. N.B.: Make sure to also search the Agent entry in the Agent Administration UI, using the filter function on the “Host name” column (visible when the “More details” button is enabled). Also, neither use the “Server name” (as it might be not assigned yet), nor the “Fully Qualified Hostname” (as hostname resolution issues will for example make the UI display the physical host) as f ilter criteria. You can access the Agent Administration UI in general via a hyperlink displayed in the above mentioned solman_setup steps.

o

Has the adequate SLD been selected when when performing the “Connect Diagnostics Agents” operation? operation? See also above Q&A item. enough? See See also above Q&A item. o Did you wait long enough? Agent still able to interact with its “associated” “associated” SLD? o Is the Diagnostics Agent Check for SLD CIM or SLD credentials errors in the Diagnostics Agent SMDSystem.?.log log files Manager connectivity connectivity data change change in the SLD? SLD? o Is the Diagnostics Agent detecting a Solution Manager Check for other kind of issues in the Diagnostics Agent SMDSystem.?.log log files. Alternatively simply connect the Diagnostics Agent to the Solution Manager system with the script ‘smdsetup’ (see part 6.1.5 Usage of smdsetup script) instead script) instead of using indirectly the SLD to do so.

Note that in case the foll owi ng message appears appears i n form er Agent Candid Candid ates UI : “2010 2010.. 03. 03. 17 14: 14: 50: 50: 26, 26, Manag anagii ng Syst Syst em ' . Syst emHome. ' i s r egi egi st er ed i n t he SLD but but t he CI CI M decl arati on i s not not val val i d, t he ag agent ent s cann cannot be att ach ached cor cor r ect ect l y. Check SLD SLD Manag anagii ng syst syst em decl arat i on. on. ( det ai l : CI M_ERR _ERR_FAI _FAI LED LED: No such such i nst ance: ance: sl d/ act i ve: ve: SAP_J SAP_J 2EE 2EEEngi Engi neCl eCl ust er. Cr eat eat i onC onCl assNam assName="SAP_J "SAP_J 2EEEng Engi neC neCl ust er ", Name=" . Syst emHome. " ) “

This error simpl y reports th at Solution Manager Manager instance is not foun d in SLD with the mentioned hostname BUT it does not prevent the agent attachment process to succeed. The UI will ask for the hostn ame and and p ort o f Soluti on Manager Manager during the attachment attachment sequence. sequence. This error message is sol ved in 7.01 SP7 P1 or hi gher.

©SAP AG 2013


3

10

Agent installation strateg egy y 3.1 3. 1

Managin Ma nagin g sys tem sc scaling aling

Use the SAP Solution Manager Sizing Toolkit to Toolkit to check if is necessary to configure your solution manager system as cluster environment. If it is necessary we strongly recommend using additional Dialog instances instead of adding Java server nodes to already existing Instances (like 1 dispatcher / 2 or more server node). Because the dispatcher is the critical component to connect the Agents to Solution Manager, it is more interesting to have many dispatchers (and therefore instances) for failover/load balancing capabilities. Last but not least also consider the currently known limitation documented in SAP Note 1480419 1480419 / Limitations in Alert Al ert in g and Moni tori to ring ng

3.2 3. 2

Managed Ma naged Syst em envir onm ent

To determine the Agent version and the amount of Agents to install (on Managed System hosts) refer to SAP Note 1365123 The following check list must be v alidated before installing a new Diagnostics agent on a managed host: Check Check whether a Diagnostics Diagnostics Agent is already already installed: Verify if the managed system has been installed  with a SAPInst version greater than NW04S SR3 (see part 2.2 part 2.2 Managed System) System) or check on the file system whether a folder like /usr/sap/DAA or /usr/sap/SMD already exists. In both cases an agent is already installed on the system. If the Diagnostics Agent(s) is/are already installed on the Managed system host(s), check if the Agent(s) is/are present in Agent Administration UI. If not, double check whether the Agent(s) is/are known in the SLD where the managed system is registered. In this last case connect the Agent to the Solution Manager System using the “Connect Diagnostics Agents” feature (or the Agent candidate Management with Solution Manager 7.0 EHP1). Alternatively simply connect the Diagnostics Agent to the Solution Manager system with the script ‘smdsetup’ (see part 6.1.5 Usage of smdsetup script) instead script) instead of using indirectly the SLD to do so.



In case no Diagnostics Agent is installed, first find out the logical hostname or physical/virtual hostname of of the managed system. Install the Diagnostics Agent following the rules defined in the document of SAPN SAP Note 1365123, 1365123 , on

3.3 3. 3

Managed Ma naged Syst ems HA/Fa HA/Failo ilo ver envir onm ent

With Solution Manager 7.1 SP05 and higher refer to the following article to put in place the so called Agents On-the-fly: Diagnostics+Agent+and+HA+support.. Diagnostics+Agent+and+HA+support In the context of earlier Solution Manager Releases, it is recommended to install the Diagnostics Agent on shared/cluster file system. The goal is to move the Agent each time together with the Managed System (only for Application Server Instances, not ASCS/SCS), but adding the “Agent stop, file system un-mounting, file system remounting and re-start” in the cluster fail-over scripts. In case the Managed instance and Agent are installed on different file system partitions, it is important when the Managed SAP instance and the Diagnostics Agent are moved to another cluster node to mount the Agent Instanc e fil e partit ion before the SAP Instance partiti on, to allo allo w the Intros cope Byte Code Code Adapter to start . This is due to the fact that the Wily Setup wizard makes the SAP java instance classpath depending on some jar files contained in the Diagnostics agent folder, especially if the Introscope Byte Code Adapter has been setup with Solution Manager (prior to Solution Manager 7.1 SP05). See also Diagnostics+Agent+and+HA+support for further details.

©SAP AG 2013


4

11

Agent installer 4.1 4. 1

SAP Hos Hostt Agen t Startup commands 

The SAPHOSTAGENT SAPHOSTAGENT is located in /usr/sap/hostctrl/exe /usr/sap/hostctrl/exe (on UNIX) On Windows it is usually located in “C:\Progam “C:\Progam Files\SAP\hostctrl\exe” In general you need to work with with the root or Administrator user to execute the SAPHOSTAGENT SAPHOSTAGENT commands

 



4.1.1

Actt io n Ac

Comman Com mand d

restart SA SAPHOSTAGENT

saphostexec -r -restart

Check its status

saphostexec -status

Check its v ersion

saphostexec –v ersion

On Windows it is usually usually installed as a service, meaning meaning that it can be started started or stopped stopped via Windows Services

E2E E2 E RCA RCA config uration for Solutio n Manager Manager 7.0x 7.0x or higher

To allow the Diagnostics Agent to retrieve CPU, Memory and disk usage via the SAP Host Agent, it is necessary to check the SAP Host Agent profile is configured as follow: L oc oc at at i on : " C: C: \ Pr o gr gr am F i l es \ SAP\ hos t c t r l \ exe\ hos t _ pr pr of i l e" or hos t _ pr pr o f i l e”

“ / us r / s ap ap/ hos t c t r l / exe/

Property to enable the different webservice port type:

SAPH PHost Cont ont r ol SA SAPO POscol SAPCCM ser ser vi ce/ ce/ por t t ypes = SA SAPCCMS SAPCont r ol

4.1.2

E2E E2 E RCA RCA config uration for Solutio n Manager Manager 7.1x 7.1x onl y

For some E2E RCA applications like LogViewer it is necessary to update the SAP Host Agent profile to add the OS user of Diagnostics Agent as user authorized to call specific protected methods via dedicated authentication. Therefore add or update the property “service/admin_users” in the SAP Host Agent profile "C:\Program Files\SAP\hostctrl\exe\host_profile" or “/usr/sap/hostctrl/exe/ host_profile” to list all all Diagnostics Agent OS users that are running on that Physical or Virtual host: For Wi ndows: s er vi ce/ admi n_us er s = SER> SER> ex: s er vi ce/ admi n_user s= MY_DO Y_DOMAI N\ SAPSer SAPSer vi ceDA ceDAA MY_D MY_DO OMAI N\ SAPSe SAPSerr vi ceDAB ceDAB For UNI X: s er vi ce/ admi n_us er s = SER> SER> ex: ser vi ce/ adm admi n_user s= daaad daaadm m dabad abadm m

©SAP AG 2013


4.2 4. 2

12

OS conf igur ation checks

After the installation of the Diagnostics Agent under UNIX platform, check the following OS parameters of the Diagnostics Agent OS user (ex: daaadm) 

Check the Diagnostics agent OS user limit : agent OS user user ‘su ‘su – smdadm’  Logon with the agent shell with the command ‘sh’  Open an sh shell  Execute the command ‘ulimit -a’ following recommended values  Compare the values with the following L i mi t time(seconds) file(blocks) data(kbytes) stack(kbytes) memory(Kbytes) coredump(blocks) nofiles(descriptors)

Rec o m m en d ed v al u e unlimited unlimited unlimited 4194304 unlimited 4108901 8092

If the user’s limit parameters for the OS Diagnostics Agent user are not correct, someOutOfMemory some OutOfMemory exceptions can for example occur during the extraction activities performed by the Diagnostics Agent. Also check the note 1437105 1437105.. Once the limits are modified also remember to restart the Diagnostics Agent system processes (sapstartsrv) so that the changes are taken into account. For AIX system see note 323816 323816.. If during a zip extractor job (datacollect), an access denied error occurs (could be found in the Diagnostics Agent log ‘SMDApplication.?.log’), double check if the folder / usr / sap/ / J XX/ SMDAgent gent / t emp is writable for the sapsys group. Furthermore the file system of the Managed system must be readable by t he Diagnostics agent. Logon with an agent OS user (su – smdadm), go in the managed system folder (ex: / us r / s ap ap/ SI D/ SYS/ pr o f i l e) , and open one profil e with the command ‘more __’. __’. If the “more” command is not working, the access rights are not correct for the Diagnostics agents. In that case, check with your system Administrator to adapt the file system rights. Further details could be found in SAP note 1163751 1163751..

4.3

JVM param eters

In this section, you will find the recommended JVM parameters for each JDK vendor 1.4 of older Diagnostics Agents 7.00/7.01: J DK J VM Par am et er s SA P No t e -DP4ClassLoad=P4Connection -Xj9 -Xgcpolicy:gencon -Xmn50m -Xmx256m 1093831 IBM JDK -Xms256m SUN JDK JDK 32 bit s -DP4ClassLoad=P4Connection -Xmx128m -Xms128m SUN JDK JDK 64 bit s -DP4ClassLoad=P4Connection –Xmx256m –Xms256m n.a HP JDK 32 bits -DP4ClassLoad=P4Connection -Xmx128m -Xms128m HP JDK 64 bits -DP4ClassLoad=P4Connection –Xmx256m –Xms256m

©SAP AG 2013


13

With agent 7.1x .1x or higher higher an an SAP JVM is used to execute ex ecute the Diagnostics Agent, the recommended JVM parameters are: J DK J VM Par am et er s SAP JVM4/5/6 32 32 bit s -DP4ClassLoad=P4Connection -Xmx128m -Xms128m -XX:MaxPermSize=64m SAP JVM4/5/6 64 64 bit s -DP4ClassLoad=P4Connection –Xmx256m –Xms256m -XX:MaxPermSize=128m -XX:MaxPermSize=128m Don’t put the java parameter verbose:gc in the Diagnostics Agents JVM parameters, because this parameter will steadily increase the size of the file work/std_SMDAgent.out

4.4 4. 4

Start Sta rt up scri pts

Information about Diagnostics Agents Agents 7.11 or higher higher With Diagnostics Agents 7.11, 7.20 & 7.30 the startup system is compliant to the SAP startup mechanism. It means that the former commands smdstart.sh and smdstop.sh are no longer available. 

Under Windows the SAP MMC MMC allows to start or stop the Diagnostics Diagnostics Agent. The Windows service of the Diagnostics Agent is configured t o start automatically during the Windows startup.

The green status in SAP MMC only indicates that the Diagnostics Agent process is running. It does however NOT mean that the Agent is operational / connected to the Solution Manager system. Only the Solution Manager Agent Administration UI allows to confirm that the Diagnostics Agent is successfully connected to the Solution Manager system and operational. (In the status is gray see 7.2 Agent crash investigation)



Under UNIX, UNIX, while being logged with the relevant Diagnostics Diagnostics Agent OS User, use use the following commands to operate the Diagnostics Agent.

Co m m an d starts ap SMDA [-v ] stops ap SMDA umber> [-v ]

Ac t i o n Start the Diagnostics agent. Stop the Diagnostics agent.

The sapstartsrv process of Diagnostics Agent is automatically started during the boot sequence of system if the sapinit program has been installed on the system. The Diagnostics Agents could later be also stopped and started remotely with the help of t he SAP MC or SAP MMC.

Information about former Diagnostics Diagnostics Agents 7.0x 7.0x 

On Windows platforms the SAP MMC allows to start or stop stop the Diagnostics Agent. The Windows service of of the Diagnostics Agent is confi gured to start automatically during Windows startup.

The green status in SAP MMC only indicates that the Diagnostics Agent process is running. It does however NOT mean that the Agent is operational / connected to the Solution Manager system. Only the Solution Manager Agent Administration UI allows to confirm that the Diagnostics Agent is successfully connected to the Solution Manager system and operational. (In the status is gray see 7.2 Agent crash investigation) 

On UNIX platform, the commands commands smdstart.sh, smdstart.sh, smdstop.sh smdstop.sh and smdadmin.sh smdadmin.sh allow to operate the Diagnostics Diagnostics Agent. Since 7.0 release of the Diagnostics Agent SAPInst, those commands are included in the PATH

©SAP AG 2013


14

variable of the OS agent user. In that case, the commands can be called from every folder (not only in the Agent script subfolder). Co m m an d smd start.sh smd sto p.sh smd admin.sh

Ac t i o n Starts the Diagnostics agent and the sapstartsrv process. Stops the Diagnostics agent and the sapstartsrv sapstartsrv process Launches jcmon tool to administrate t he Diagnostics agent instance

On Solaris platform , an error err or (/usr/sap/SMD/exe/smdstart.sh: usage: ulimit [ -HSacdfnstv ] [ limit ]) occurs when the smdstart.sh is executed. Check the solution in sap note1109366 note 1109366.. On HP-UNIX HP-UNIX platfo rm, rm, an error (!! ERRROR !! current user is not the owner of this instance ) occurs when the smdstart.sh is executed. Check the solution in sap note1109366 note 1109366..

Ex i t co cod e o f sm sm d st art .s h 0 2 5 4 3

©SAP AG 2013

Reas o n Success to launch the agent process (This is not the confirmation that the agent is connected to SMD). Input Parameters are invalid. Agent is already running. User limit are not set correctly. Current OS user is invalided to start the agent.


15

For UNIX system, the Diagnostics Agents do not start automatically when the system reboots because the Diagnostics Agent startup scripts are not included in the system startup scripts. The following command line can be used i n the system startup scripts:

su - c / usr / sap sap/ SMD/ J 98/ scr i pt / smdst ar t . sh > / dev/ nul l 2>&1

- s / bi n/ sh - smdadm

And in the shutdown sequence:

su - c / usr / sap/ sap/ SMD/ J 98/ scr i pt / smdst op. sh > / dev/ ev/ nul l 2>&1

- s / bi n/ sh - smdadm adm

The following procedure describes how to insert the agent in the boot sequence for a Linux SUSE 10 system:    

Create a file di agnost i cs_age cs_agent s in folder / et e t c/ c / i ni t . d Determine the init run level in file / et c / i ni t t ab Create a symbolic link in folder / et c/ c / i ni t . d/ r c . d/ to file di agnost i cs_age cs_agent s with name S80di agn agnost i cs_ agen agentt s Create a symbolic link in folder / et c / i ni t . d/ r c6 c 6. d to file di agnost i cs_ag cs_agent s with name S80di agn agnost i cs_ agen agentt s . (runlevel 6 is System reboot)

#! / bi n/ sh ### BEG BEGI N I NI T I NFO # Provi des: di agn agnost i cs_age cs_agent s # Requ equi r eded- St ar t : $net wor k # Requ equi r eded- St op: op: $net wor k # Def aul t - St ar t : 2 3 5 # Descr i pt i on: st ar t di agnost i cs ag agent s ### END END I NI T I NFO case case "$1" "$1" i n ' s t ar ar t ' ) su - c / usr / sap sap/ SMD/ J 98/ scr i pt / smdst ar t . sh > / dev/ nul l 2>&1 - s / bi n/ sh - smdadm ;; ' s t op op' ) su - c / usr / sap sap/ SMD/ J 98/ scr i pt / smdst op. sh > / dev/ nul l - s / bi n/ sh - smdadm ;; ' s t at at us us ' ) su - c / usr / sap/ sap/ SMD/ J 98/ scr i pt / smdadmi n. sh - s / bi n/ sh - smdadm adm ;; *) ech echo " Usage sage: $0 { st ar t | st op | st at us }" ;; esac exi t 0

When smdstart.sh and smdstop.sh commands are embedded in the system startup scripts, the ex it code of the smd scripts cannot be retrieved. Further more it is important to redirect the standard error output into the standard output with a syntax like 2>&1 to avoid that the smdstart.sh and smdstop.sh are blocking the parent script.

©SAP AG 2013


16

About sapstartsrv sapstartsrv process process SA P No t e 9276 92 7637 37 - Web Web servi ce authenti cation in sapstarts rv as o f Release 7.00 7.00 Note 823941 - SAP Start Service on Unix platforms

©SAP AG 2013

Des c r i p t i o n The Web service interface of sapstartsrv requires a user authentication from the client (MMC, SAP MC, sapcontrol and so on) for protected operations such as starting and stopping the SAP instance. After an SAP upgrade, the SAP Start Service (sapstartsrv) on UNIX platforms is not automatically started when the operating system is started.


5

17

Agent co connectivity 5.1 5. 1

Managi Ma naging ng sys tem

This section describes the different primary connection modes used by the Diagnostics Agents to connect to the Solution Manager System. Two modes are available: Connection through the Solution Manager SAPJ2EE Message server Direct connection with the Solution Manager SAPJ2EE P4 dispatcher.  



Connection through the Solution Manager SAPJ2EE Message server It is RECOMMENDED to connect the Diagnostics Agent to Solution Manager Java stacks using the J2EE Message Server. To establish this connection the hostname and port of t he Solution Manager J2EE Message J2EE Message server (Java SCS instance) have to be found out using for example the SCS instance profile.

The Agent connection mechanism can be described mainl y as follow: A. During the During the installation of a Diagnostic Agent the SAPJ2EE Message server connection mode has to be selected. B. This information is stored in the Agent Agent configuration (file runtime.properties) runtime.properties) and the credentials are are stored in the Agent secure store (file secstore.properties). C. When the Diagnostics Agent starts, the P4 connection framework will will retrieve from the SCS SCS Message Message Server (on HTT port ex: 81xx) all J2EE entry points that are available. D. Once the list of of J2EE entry points points is collected, one P4 entry entry point is chosen chosen from the list. E. The Diagnostics Agent will will then establish a P4 connection to the selected J2EE entry point with with the host name and the port corresponding to that P4 connection.

©SAP AG 2013


18

Troubleshooting Message Message server server connection Before starting the Agent installation or before executing the smdsetup script, you can double check the content of the J ava SCS Message Server with the following description: 1. Open the URL: http://msgserve http://msgserver:81 r:81xx/msgserver/ xx/msgserver/text/logon text/logon (xx (xx corresponding to instance number of the SCS instance) 2. Check if a P4 entry point is available available with the adequate hostname 3. On the managed system system host, open a shell as adm OS user (from Managed Managed system or Agent) Agent) 4. Execute the command “pi ng ”. Double check that the host is accessible 5. Execute also the command “t el net ”. If no error message occurs (blank screen) the connection is OK. Most common agent connection probl ems All connection problems are logged in a file under SMDAgent/log/SMDSystem.X.log. Check this log file to find the root exception of the issue: Once the error exception is found also locate the root cause of the exception by looking at the last “caused by” sentence.  No

Entry point found

com. sap sap. engi ne. ser ser vi ces. ces. j ndi . per si st ent . exce xcept i ons. ons. Nami ngExcep cept i on: Except Except i on : com Excep cept i on whi l e t r yi ng t o ge get I ni t i al Cont ext . [ Root excep cept i on i s com com. sap sap. engi ne. i nt er f ace aces. cross. Dest i nat i onExcep cept i on: on: t her e a arr e no no acce access ss poi poi nt s f or ser vi ce: P4 r egi egi st er ed on on the message essage ser ser ver ver ]

No P4 entry point has been reported by the Message Server. This is probably due to the fact that it is not the appropriate Message Server. It has to be the Solution Manager Java stack and not the ABAP stack Message Server. Also check the text display of the Message server content using the URL http://msgserver:81xx/msgserver/text/logon.. Verify if a P4 entry point is available (as shown below) http://msgserver:81xx/msgserver/text/logon 

Unknown host

sapms: / / yup yupxxxx xxxxx. x. domai n, com com: 8136] Checki ng ser ser ver ver avai vai l abi abi l i t y. . . Except Except i on: [ sapm Conne onnect ct i ng t o SMD ser ver ver sapm sapms: / / yupx yupxxx xxxx xx.. domai n, com: 8136 f ai l ed - er r or coun count er : 0 - com com. sap sap. engi ne. ser ser vi ces. ces. j ndi . per si st ent . exce xcept i ons. Nami ngExcep cept i on: Excep cept i on whi l e t r yi ng t o ge get I ni t i al Cont ext . [ Root excep cept i on i s com com. sap sap. engi ne. i nt er f ace aces. cross. Dest i nat i onExcep cept i on: on: can cannot est abl abl i sh con connect i on wi t h any of t he ava avai l abl abl e i nst ances ces #Nest ed except except i ons ons ar e: #j ava. net net . Unknow nknownHost Except Except i on: on: yupxx yupxxxxx: xxx: yupxxxxx yupxxxxx ]

It is a network resolution problem due to the fact that the selected P4 hostname retrieved from the SCS Message server list cannot be resolved from the Managed system host. Retrieve the list of the P4 entries in the Message server which l ooks like:

ver si on 1. 0 J 2EE9 2EE933 3389 8945 4500 00 J 2EE 2EE NCED6018 6018X XXXXA. dhcp. dhcp. ncel . sap. cor p J 2EES 2EES NCED6018 6018X XXXXA. dhcp. dhcp. ncel . sap. cor p P4 NCED6018XXXXA. dhc p. p. nc el el . s ap ap. c or or p P4S P4S NCED6018XXXXA. dhcp. cp. ncel . sap. sap. cor p P4HTTP NCED6018XXXXA. dhc p. p. nc el el . s ap ap. c or or p

5930 59300 0 5930 59301 1 59304 59306 59305

LB=1 LB=1 LB=1 LB= LB=1 LB=1

Execute the command “pi ng _5xx04>” in a Managed system shell. If the Solution Manager host cannot be reached, then change the host definition in your DNS or in the /etc/hosts. You should ideally fix you engine installation to report the right host in the Message Server.

©SAP AG 2013


Agent Ag ent Mainten Main ten anc e Mode (requ (r equir ir ed du ring ri ng LM-SERVICE depl oy ment) men t) As documented in various notes and guides, the Maintenance mode need to be enabled before deploying a new LM-SERVICE Release, Support Package or Patch, as described also in SAP note1483508 note 1483508.. If this operation is not performed the Diagnostics Agent runtime behavior will in most cases be affected. As a consequence the Diagnostics Agent logs might for example contain errors like: [SMDJNDILookupAgentSideImplementation] lookup /smdserverctx/ObjAutoRegistered/SMD_DataEnrichment_Registrator failed […] Object not found i n lookup of SMD_DataEnrichement_Registrator



Solution Manager SAPJ2EE P4 dispatcher (SAP does NOT NOT recom recom mend to use thi s mod e )

Solution Manager Manager in cluster environment If the Solution Manager system is installed as a cluster (having Dialog Instances), the direct P4 Agent connection should not be used as the Diagnostics Agent would else always establish a connection to the same Solution Manager Java instance corresponding to the selected P4 host/port. Also from a somehow High Availability point of view, you hav e to use a connection through t hrough Solution Manager J2EE Message server. In that situation, the connection of the Diagnostics Agent will be each time loadbalanced to the different available Java instances.

5.2 5. 2

Managed Ma naged s yst em

All connections from the Diagnostics Agent to the Managed systems are executed locally on the same host. It is the reason why it is really imperative to install the Diagnostics Agent on t he same Physical or Virtual host than t he SAP Managed system. The Agent connects to different SAP System layers as follow: Java stack on port P4 (5xx04)  Sapstartsrv http port of Managed System (5xx13)  J2EE Message server (39xx and 81xx)  Local RFC call to Managed ABAP stack  Webservice call call to SAP host agent on port 1128 for agent 7.1x 

©SAP AG 2013

19


20

Troubleshooting P4 connection probl ems with Manage Managed d s ystem 1.

Check on which which host the P4 socket is bound on the Managed Managed system a. Run the OS command “netstat –an” to find the entry line containing the P4 port of of the managed system: TCP TCP

Or t cp

2.

0. 0. 0. 0: 59304 59304

0

0. 0. 0. 0: 0

0

*. 50304

LI STENI STENI NG

*. *

LI STEN TEN

In this example we see that the P4 port is bound for all hosts where a p4 request can be emitted (* or 0.0.0.0), if the port is bound like “10.45.192.204:50304” only the p4 requests emitted from the host 10.45.192.204 will be accepted. It explains why the Diagnostic Agent must be installed on Logical hostname 10.45.192.204 with SAPInst. The situation can be fixed by changing the profile property proper ty SAPLOCALHOST (see 6.1.4 Verifying Logical hostname). hostname). b. After verifying the port in the OS configuration, check in the J2EE J2EE dispatcher configuration configuration that the property “bindHost” is set to 0.0.0.0. This should be visible with the J2EE configuration tool of the managed system->Instance_xxx->dispatcher->services->p4. If the v alue is not configured to 0.0.0.0 (which accepts all hosts), add the Agent Agent IP or install the Agent using on one of the list ed IP corresponding to logical hostnames running on t he SAME SAME underlying physical or virtual host . Check if the Agent Agent can access the J2EE Message Message Server (SCS) of the Managed system system with the host host and the http port stored in the Agent configuration (see the next section). For example :  



Open a shell with with the agent OS user (adm) on the Managed Managed System Execute the command “mspr ot - l - mshost - mss er v 39< 39 SCS> or 36 SCS> - d J 2EE – s ”, the values of host and port must be retrieved from the Agent configuration (see the next section) The command command must return the following content : SAP SAP Message essage Server Server Pr otocol Pr ogr ogr am, Ver Ver si on 1. 3 (bui l t : J an 24 2008 008 01: 01: 27: 27: 19) 19) I NFO CLUSTER CLUSTER-- I D HOST

GROUP- I D

STATE

J 2EE- SERVI SERVI CE- MASK

- - - - - - - - - - - - - - - - - - - - - - -- - - - - - - - - - - - - - - - - - - - - - - - - - - -- - - - - - - - - - - - - - - - - - - - - - - - - - - -- - - - - - - - - - - - - - -

©SAP AG 2013


3.

4.

©SAP AG 2013

21

- - - > LI ST

SER SERVI CE: 0000 00000 0 NAME: Conf onf i gur gur at i onM onManag anager er

- - - > LI ST

SERVI SERVI CE: 00001 00001 NAME: _CacheManager anager

- - - > LI ST

SER SERVI CE: 0000 00003 3 NAME: {servi ce_manag anager_de er_depl pl oy_di oy_di st r i butor}

- - - > LI ST

SER SERVI CE: 0000 00004 4 NAME: {ser {ser vi ce_manag anager_ er_ i nt ernal _connecti _connecti on} on}

- - - > LI ST

SERV SERVII CE: 0000 00005 5 NAME: shel l

- - - > LI ST

SER SERVI CE: 0000 00006 6 NAME: j mx_noti x_noti f i cati on

- - - > LI ST

SER SERVI CE: 0000 00007 7 NAME: p4

- - - > LI ST

SERV SERVII CE: 0000 00008 8 NAME: webservi ces

- - - > LI ST

SER SERVI CE: 0001 00010 0 NAME: i i op

- - - > LI ST

SER SERVI CE: 0001 00011 1 NAME: r 3st art up

- - - > LI ST

SER SERVI CE: 0001 00012 2 NAME: t el net net

- - - > LI ST

SER SERVI CE: 0001 00013 3 NAME: l og_con og_conff i gur gur ator

- - - > LI ST

SERV SERVII CE: 0001 00014 4 NAME: j mx

- - - > LI ST

SER SERVI CE: 0001 00016 6 NAME: htt p

- - - > LI ST

SER SERVI CE: 0002 00022 2 NAME: com. sap. sap. securi t y. cor e. ume. ser vi ce

- - - > LI ST

SER SERVI CE: 0002 00023 3 NAME: secur secur i t y

- - - > LI ST

SER SERVI CE: 0002 00024 4 NAME: t c~sec~cer t r evoc~ evoc~ser vi ce

- - - > LI ST

SER SERVI CE: 0002 00025 5 NAME: ser vl et_ j sp

- - - > LI ST

SERVI SERVI CE: 00026 00026 NAME: webdynpr ebdynpr o

LI ST 0004105250 0004105 250 ncev nc evhos hostt . mydom ydo mai n

0000041052 0000041 052 RUN RUNNI NG

XXXXXXXXO XXXXXXXXOXXOXXXXXO XXXXXOXXXXXXXXXXX XXXXXXXXXXXX X

LI ST 0004105200 0004105 200 ncev nc evhos hostt . mydom ydo mai n

0000041052 0000041 052 RUN RUNNI NG

XXXXXXO XXXXXXOXXXXXXXXXX XXXXXXXXXXO OOOOOOOOOOOOO

In this content, the P 4 service is listed as well as the J2EE nodes with hostnames (e.g. ncevhost.mydomain). Check if those hostnames can be resolved on your network by executing the command: ping ncevhost.mydomain Open the J2EE Visual Visual admin tool of of the Managed system and connect with with the J2EE user user used by the Diagnostics Agent (see in the next section the Agent configuration) to check if the user can be authenticated and if it has the Administrator role. Check also the Managed system system defaulttrace file to find some exception exception related to P4 connections.


22

Connection c onfigur ation for m ana anaged ged systems The following table lists all properties generated and updated by the Managed Setup Wizard or solman_setup with Solution Manager 7.0 EHP1 and higher. These properties allow the Diagnostics Agent to connect to the Managed system in order to complete the E2E E2E RCA operations. They are accessible using the Agent Administration UI / Application configuration. Those properties are stored by server name (scope) of the Diagnostics Agent. Connection Conne ction Type to Po r t Co n f i gu r at i on st o red i n ag el et Managed Ma naged sy stem P4 port port http port J2EE J2EE Message Message server Intern Port J2EE Message Message s erver Http port J2EE Message Message server h ost

Pr o p er t y

5xx04 5xx00 39xx

com.sap.smd.age com.sap.smd.agent.app nt.application lication.globa .global.configu l.configuration ration com.sap.smd.age com.sap.smd.agent.app nt.application lication.globa .global.configu l.configuration ration com.sap.smd.age com.sap.smd.agent.app nt.application lication.globa .global.configu l.configuration ration

[SID]/[INB]/sapj2e [SID]/[INB]/sapj2ee/P4/por e/P4/portt [SID]/[INB]/sapj2e [SID]/[INB]/sapj2ee/http/port e/http/port [SID]/sapj2ee [SID]/sapj2ee/msgserve /msgserver/port r/port

81xx

com.sap.smd.age com.sap.smd.agent.app nt.application lication.globa .global.configu l.configuration ration

[SID]/[INB]/sapj2e [SID]/[INB]/sapj2ee/http/port e/http/port

com.sap.smd.age com.sap.smd.agent.app nt.application lication.globa .global.configu l.configuration ration

[SID]/sapj2ee [SID]/sapj2ee/msgserve /msgserver/host r/host

For the credentials : Connection Conne ction Type to Managed Ma naged sy stem

Co n f i g u r at i on s t o r ed i n ag el et

Pr o p er t y u ser

Pro p er t y p as sw o r d

J2EE credentials

com.sap.smd.agent.application.global.configuration

[SID]/sapj2 [SID]/sapj2ee/c ee/com/u om/user ser

[SID]/sapj2 [SID]/sapj2ee/c ee/com/p om/pwd wd

Connection Type Connection to Ma Managed naged system

Con onfi figu gura ratio tion n stor store ed in age gele lett

Pro rope perty rty client

Pr o p er t y u ser

Pr o p er t y p assw o r d

ABAP AB AP c red enti als 7.0x ABAP AB AP c red enti als 7.1x

com.sap.smd.agent.application.global .configuration com.sap.smd.agent.application.global .configuration

[SID] [SID]/its/ /its/cl clien ientt

[SID] [SID]/ab /abap/c ap/com om/us /user er

[SID] [SID]/ab /abap/c ap/com om/pw /pwd d

[SID] [SID]/ab /abap/ ap/cl clien ientt

[SID] [SID]/ab /abap/c ap/com om/us /user er

[SID] [SID]/ab /abap/c ap/com om/pw /pwd d

It is recommended to change t hose values in the Managed system setup wizard or solman_setup with Solution Manager 7.0 EHP1 and higher instead of directly editing them in the Agent Admin tool. Those values could however be changed using the Agent Administration tool in case of troubleshooting/debugging.

5.3

Self-Management

During a Self-management setup of Solution Manager, some special configurations in the different setup steps are needed due to some restrictions about the local Diagnostics Agent connection sharing. Possible error in Manage Managed d sy stem Configuration 

End of stream stream is reached unexpectedly during input from Socket

Except Except i on : com. sap. sap. smdage agent . pl ugi ns. con connect ect or s. p4. excep xcept i ons. ons. P4A P4Aut hori zati onExcep xcept i on: on: Access ccess i s deni deni ed: ed: Pl ease ease make sur e t he and and par amet er s ar e cor r ect . ; nest nest ed exce xcept i on i s: com com. sap sap. engi ne. ser ser vi ces. ces. j ndi . per si st ent . exce xcept i ons. Nami ngExce xcept i on: Excep cept i on dur i ng get I ni t i al Cont ext ope oper at i on. No ser ser ver i s r unni ng. [ Root oot excep cept i on i s com. sap. sap. engi ne. servi ces. ces. secu securr i t y. exce excep pt i ons. BaseLo aseLog gi nExcep xcept i on: on: Excep cept i on i n cr eati ng new Remot eLog eLogi nCont ont ext ext i nst ance. ance. ] at com. sap. sap. smd. age agent. pl ugi n. conn connect or s. p4. P4C P4Conn onnect i onS onServi ce. get P4C P4Conn onnect i on( on( P4C P4Conne onnect ct i onS onServ i c e. e. j ava: 69) ... at j ava. va. l ang ang. Thr Thr ead. ad. r un( Thr ead. j ava: va: 838) com. sap sap. engi ne. ser ser vi ces. ces. j ndi . per si st ent . exce xcept i ons. ons. Nami ngExcep cept i on: Exce xcept i on Caused by: com dur i ng get I ni t i al Cont ext ope oper ati on. No ser ser ver ver i s r unni ng. Root exce excep pt i on i s

©SAP AG 2013


23

com. sap. sap. engi ne. servi ces. ces. secu securr i t y. exce excep pt i ons. BaseLo aseLog gi nExcep xcept i on: on: Excep cept i on i n cr eati ng new Remot eLog eLogi nCont ont ext ext i nst ance. ance. at com. sap. sap. engi ngi ne. ser vi ces. secur secur i t y. r emot e. l ogi ogi n. Remot eLogi Logi nCon nContt ext Ext . ( Remot eLog eLogii nCont ont e xt Ext. j ava: 34) ... at j ava. va. l ang ang. Thr Thr ead. ad. r un( Thr ead. j ava: va: 838) va. i o. EOFExc FExce ept i on: on: End of st r eam Caused by: j ava. eam i s r each ached unexp expect edl edl y du dur i ng i nput f r om addr =/ 56. zzz . xx. xx. xx, xx, por por t =50204 0204,, l ocal ocal por por t =223 22326] Socket [ add at com. sap. sap. engi ne. servi ces. ces. r mi _p4. _p4. Conn onnect i on. r un( Conn onnect i on. j ava: va: 423) 

It is a problem with a Java server server level. level. Two possibil ities to f ix the issu e: (See (See also also SA SAP P Note Note 1473530 1473530))

1. Apply the fix ( see SAP note ) on java server and use at least the LM-SERVICE 7.0 EHP1 SP7 P1 2. Use the workaround describes in SAP note (adapt the property bindHost). In a Self-M Self-Manage anagement ment scenario th ere exis exis ts anoth er solut ion Since Solution Manager 7.0 EHP1 SP21, the Diagnostics Agent configuration property – enabling the selfmanagement scenario – is added aut omatically by transaction solman_setup / Managed System Configuration to avoid such a problem. I t is named /selfmonitoring/enabled (value: true) and v isible in the Agent Administration under “com…global.configuration”.

5.4

SAP ro ut uter er

The Diagnostics Agent can be connected to the Managing system via a SAP Router. The 7.30 standalone installers and SWPM based installers support confi guring the Diagnostics Agent connection via a SAP Router. Alternatively the SAP route can be maintained aft er the Diagnostics Agent installation, using t he smdsetup script. (See chapter6.1.5 chapter6.1.5 Usage of smdsetup script). script). This feature is supported in the context of Solution Manager 7.10 and higher. Also, the automatic aut omatic activities “Introscope Host Adapter” and “Introscope Byte Code Adapter installation” in solman_setup transaction / scenario: Managed system Configuration will use this route provided at the Diagnostics Agent i nstallation/maintenance level. In the context of Solution Manager 7.0 EHP1 this type of connectivity is not supported. Therefore please contact the Diagnostics Agent support team to analyze the infrastructure and evaluate the setup possibilities.

©SAP AG 2013


6

24

Agent Maintenance 6.1

Configuration

This section describes all procedures to update Diagnostics Agent configurations that could not be changed/updated centrally using the Solution Manager system.

6.1.1 Cha Chang ng e Path Path o f th e JDK In the eventuality that the JDK installation path changes after the Diagnostics Agent has been installed, you might need to update this JDK path pat h information within the Agent profile file, as f ollow.

For 7.00 and 7.01 7.01 

  

Open the profile _ J BER>_ in folder / usr / sap sap/ / sys/ pr of i l e. Update the parameter j s t ar t up/ up/ vm/ home by putting the absolute path to the JDK home. Remove the file / usr / sap/ sap/ / J XX/ pr of i l e/ smd. pr ope oper t i es. vmpr op. Restart the Diagnostics agent.

For 7.1x , 7.2x and 7.3x 7.3x 

As the Diagnostics Agent Agent 7.1x and higher run using their own SAPJVM, it is not recommended to change the path of the JDK.

6.1.2 Change JVM Parameters If the JVM parameters need to be changed, you have to update the Diagnostics Agent instance properties file.

For 7.00 and 7.01 7.01 

  

Open the file smd. pr ope sap/ / J / er>/ prof i l e. oper t i es in folder / usr / sap/ Update the parameter “smdage parameters). dagent nt . j avaPar avaPar amet er s ” (see 4.3 JVM parameters). Remove the file / usr / sap/ sap/ / J XX/ pr of i l e/ smd. pr ope oper t i es. vmpr op. Restart the Diagnostics Agent.

For 7.1x, 7.2x and and 7.3x    

Open the file smdage agent . pr ope oper t i es in the folder

/ usr us r / s ap/ / SMDA/ SMDAgent Agent . Update the parameter “smdage parameters). dagent nt . j avaPar avaPar amet er s ” (see 4.3 JVM parameters). Remove the file / usr / sap/ sap/ / J XX/ pr of i l e/ smd. pr ope oper t i es. vmpr op. Restart the Diagnostics Agent.

6.1.3 Re Repl placing acing JDKs 1.4.2 wit h SAP JVM 4 Some Partner JDK 1.4.2 implementations are going out of maintenance. In this context SAP proposes to replace those JDKs with the SAP JVM 4. The manual procedure to be followed concerning t he Diagnostics Agents is described in the

©SAP AG 2013


25

“SAP JVM Switch Tool” guide available under https://service.sap.com/instguides under https://service.sap.com/instguides.. Especially, consider the chapter 5. “Performing “Performing the Switch Switch to SAP JVM 4 for the Diagno Diagnostics stics Agent”. Agent”.

6.1.4 Verifying Logical hostname Verify the logical hostname that was used during the Diagnostics Agent installation, in case a Diagnostics Agent is not reporting the expected logical hostname within the Agent Administration UI.

For 7.01, 7.1x, 7.1x, 7.2x and 7.3x 7.3x  

Open the profile _ J _ in the folder / usr / sap sap/ / sys/ pr of i l e. The parameter SAPLOCALHOST has to be present and contain a short hostname (not (not t he FQN) FQN)

6.1.5 Usage of smdsetup scr ipt The script ’smdsetup’ allows to update some configuration parameters for Diagnostics Agent. Some options of the script only exist since Solution Manager 7.0 EHP1. Note that this script is automatically patched at the time the Diagnostics Agent connects to the Solution Manager server. This feature has been implemented since Solution Manager 7.0 SP15. To determine the options supported by the smdsetup script, just execute the command “smdsetup.sh” or “smdsetup.bat”, without “smdsetup.bat”, without any parameters 

If the output proposes some action like “managingconf” “managingconf” (in case the smdsetup version is 7.01 or higher). Example of smdsetup 7.01 or higher output: >> Hel p I nf or mat i on ---------------------Scr i pt whi ch al al l ows t o up updat e som some di di agn agnost i cs age agent con conf i gur at i on. on. s mdset up L i s t o f ac t i ons : * smdset up secst or ej 2ee 2ee

user : " ue>" pwd: " ue>"

* smdset up secst or esl d

user : " " pwd: " ue>"

* smdset up sl dconf conf

host name: " " por por t : " " [ opt opt i onal onal user : " ue>" pwd: " ue>" ]

* smdset up manag anagii ngconf ngconf

host host nam name: " sapms: / / qn>" por por t : " " [ opt opt i ona onal user ser : "<. "<. . . >" pwd: "<. "<. . . >"

ser ver ver name: " <. . . >" ] * s mdset up managi anagi ngconf ngconf ser ver nam name: " ue>" ]

Script Location

©SAP AG 2013

host name: " qn>" por t : " " [ opt opt i onal onal user : " ue>" pwd: " ue>"


26

In 7.0 SR3 and 7.0x the smdsetup script in this version is located in:

/ usr / sap/ / J / scr sc r i pt In 7.1x and higher the smdsetup script is located under:

/ usr / sap/ / SMDA / sc r i pt Acti Ac ti on managingconf Sldconf secstorej2ee secstoresld changeservername

Descri Desc ri pt i on

Av ail abi abilili ty

Changes the Solution Manager System (also known as managing system) used by the Diagnostics Agent to connect to. Specify the host and port. Changes the SLD server where the Agent will perform its registration. Provide the SLD host and port. Changes only the J2EE user used to connect to the Solution Manager system. (restart after the changes) Changes only the J2EE user used to register the Agent in the configured SLD system. (restart after the changes) Change the server name of agent. (restart after the changes)

7.0 SP14 or higher

Full Implementation in 7.1, see chapter 5.4

7.01 or higher

addsaprouter saprouterpass removesaprouter supportlogs

Execute some Agent checks and generate a zip with all Agents logs (including a Agent check report)

7.0 SP17 patch 3 or higher

MANAGINGCONF SYNTAX MANAGINGCONF SYNTAX

To execute this command you need to stop the agent. The command updates the connection settings to the Solution Manager system: 



To connect the agent via J2EE Message server http http port: N.B.: - Do not confuse the message server HTTP port with the message server internal port which is like 36xx) - Do not forget to prefix the hostname with “sapms://” s mdset up managi ngconf host name: ” sapms://myhost yhost . domai n. cor p” por por t : ”8101 ”8101” ” (In rare circumstances) To connect connect the agent with a P4 direct connection: connection: s mdset up managi anagi ngconf ngconf host name: ” myhost ” por t : ” 5300 53004” 4”

Optionally, you can add the j2ee user information in the command line to update also the agent secure store: s mdset up managi ngconf host name: ” sapms://myhost yhost . dom domai n, cor p” por por t : ”8101” ”8101” user user : ”SMD_ ADMI N” pwd: ”i ni t 1234 1234” ”

SLDCONF SYNTAX SLDCONF SYNTAX

To execute this command you need to stop the agent. The command updates the connection settings to the SLD system of the agent registration: Register/connect the Agent to the SLD system (http port port of of SLD java stack). stack).  smdset up sl dconf dconf host host nam name: ”mySLDhost host . dom domai n. cor p” por por t : ”50000 ”50000” ” Optionally, you can add the SLD user information in the command li ne to update the agent secure store: smdset up sl dcon dconf host host nam name: ”mySLD ySLDh host . dom domai n. cor p” por t : ”5000 ”50000” 0” user user : ”s l ddsuser dsuser ” pwd: ”xxxxx” ”xxxxx”

SECSTOREJEE SYNTAX SECSTOREJEE SYNTAX

To execute this command you need to stop the agent. The command updates the credentials of the connection to the Solution Manager system:

©SAP AG 2013




27

Update the agent secstore file: smdset up secs t or ej 2ee 2ee user : ”SMD_ADM _ADMI N” pwd: ”i ni t 1234 1234” ”

SECSTORESLD SYNTAX SECSTORESLD SYNTAX

To execute this command you need to stop the agent. The command updates the credentials of the connection to the SLD system: 

Update the agent secstore file smdset up secst or esl d user: user: ”sl ddsuser suser ” pwd: ”xxxxx ”xxxxxx” x”

CHANGESERVERNAME SYNTAX CHANGESERVERNAME SYNTAX

To execute this command you need to stop the agent. The command updates the “servername” attribute of the agent: 

Update the agent configuration file s mdset up changeser ver name ser ver name: " myser ys er ver name"

SUPPORTLOGS SYNTAX SUPPORTLOGS SYNTAX

This command retrieves all agent log files and executes some basics tests to check the agent pre-requi sites as:        

jdk version check jvm parameters check hostname resolution check tcp port check p4 protocol check message server connectivity check … Execute the command smdset up s uppor ppor t l ogs ogs At the end of the execution of the command, a Zip file is generated in at the following location:

/ usr / s ap/ / ER>/ SMDAgent Agent / s uppor t Logs _ . z i p. The zip content is the following:      

Check results (AgentSupportToolReport.log in SMDAgent/log folder) Diagnostics Agent log files Diagnostics Agent profile SAPinst logs work directory content All properties files contains in SMD agent folder

©SAP AG 2013


28

6.1.6 Change Agent log level Main Diagnostics Agent log files are stored in folder: / usr / sap/ / / SMDAgent gent / l og and organized as follow: Fi l en am e SMDSystem.log

Def au l t s et t i n g s 10 f ililes of 1MBytes

Def au l t l ev el INFO

SMDAgentApplication.log

10 fi file les s of of 1Mb 1Mbyt yte es

WARN WA RNIN ING G

P4.log

10 f ililes of 1M 1MBytes

NONE

Des c r i p t i o n This file reports all inf or ormation about agent framework like connection to managing system, SLD report, agelets loading and patch sequence. This Th is fi file le re repo port rts s al all in info forrma mati tion on ab abou outt the the age agele lets ts used by RCA applications, Setup Wizards and Wily IS setup. This file reports all inf or ormation about ag agent connection layer with managing system.

In case the Diagnostic Agent is currently connected to the Solution Manager System, the procedure to change the log level of these log location is: 

Go in Agent Administration UI Select the tab “Agent Log viewer” Select the relevant Agent in the dropdown list  Configure the expected level.  With Solution Manager 7.1, click on the link “Log configuration” to access the Agent log levels configuration UI. In general, avoid changing the log levels of all Diagnostics Agents at the same time (using button “Save on All Agents”). 

In case the Diagnostics agent is not connected to the Solution Manager System, the procedure to change the log level is:     

At OS level login as adm OS user Stop the agent Go in folder / usr / sap/ sap/ / / SMDAgent ent / conf conf i gurat i on Open the file l og- con conf i gur ati on. xml Adapt the level (attribute effective-severity) in following line (level values: ALL,DEBUG,INFO,WARNING,ERROR,NONE): ALL,DEBUG,INFO,WARNING,ER ROR,NONE):

For Filename SMDSystem.log SMDAgentApplication.log P4.log



m">

Save the file and restart the agent.

©SAP AG 2013


6.2

29

Managing Managin g J2EE us user er

Information about the Solution Manager J2EE user needed by the Diagnostics Agent to connect to the Solution Manager System, is described: For Solution Manager 7.0 / 7.0 EHP1 in the “User Administration guide”, guide”, section 2.3.3 [SOLMAN.AGT.COM]: [SOLMAN.AGT.COM]:  Diagnostics agent Communication User. For Solution Manager 7.1 in the “Solution Manager Security Guide”.  Starting with Solution Manager 7. 1 SP5, the Diagnostics Agent can connect t o Solution manager server using an authentication via username / certificate, instead of using a basic authentication (user/password). Note that with the certificate authentication the user is not getting locked in case of multiple attempts (performed by several Diagnostics Agents). Pay Pay attention to the followin g situ ation For Diagnostics Agents that have not been connected so far to a Solution Manager 7.1 SP05 system or higher, remind that t he Agent will stop trying to connect/login on the Solution Manager system in case wrong user credentials have been provided (or the user is locked). In that context also be aware that in case the SMD_ADMIN user credentials are changed, “offline” Diagnostics Agent will not get notified of that change. At the time these Agents will go “online” again they will lock the SMD_ADMIN user, in case at least 3 or more Agents are in this situation. Since Solution Manager 7.1 SP05, consider using the “Non-authenticated” Agent list, as described by: http://wiki.sdn.sap.com/wiki/display/SMSETUP/Non-Authenticated+Diagnostics+Agent s Most common agent agent conn ection problems  Authentication failed all authentication problems are logged in file SMDSystem.log.

sap. engi ne. cor cor e. t hr ead ead. i mpl 3. Si ngl eTh eThr ead ead. r un Except Except i on: at com. sap. ( Si ngl eThr Thr ead. ad. j ava: ava: 170) Caused by: com. sap. sap. engi ne. ser ser vi ces. ces. secu secur i t y. exce excep pt i ons. BaseLo aseLog gi nExcep xcept i on: on: . Aut hent i cat cat i on d dii d n no ot succe succee ed at com com. sap sap. engi ne. ser ser vi ces. ces. secu securr i t y. l ogi n. Modul esProcessA sProcessAct ct i on. r un ( Modu odul esPr esPr ocessA ocessAct ct i on. j ava: va: 175) . . . 13 mor e

Before restarting the agent It is important to execute some checks about the J2EE user as follow: 1. Connect to the ABAP transaction SU01, to check if the J2EE user (SMD_ADMIN) is not locked and if the Role assigned is compliant to SAP recommendations done in the “User Administration guide”. 2. Logon to t he Agent administration tool with the J2EE user (SMD_ADMIN) to c heck that the logon module of the Java stack works correctly. 3. Call the smdsetup to update only the credentials to be sure the password is up-to-date. 4. Restart the Diagnostic agent. In dev_smdagent.log, if you find the exit exit code -11112 it is a authentication problem as described above [ Thr 880] Thr 880] Wed Feb 04 09: 09: 28: 28: 47 2009 2009 [ Thr Thr 880] 880] J Laun LaunchI St ar t Func: Func: end end of mai n met hod ( r c = 0) [ Thr 6000] J Lau LaunchI chI Exi t J ava: va: exi t hook ook i s cal cal l ed (r c = - 11112) [ Thr 6000] ************************************************************* * ** ERROR => The The J ava VM t er mi nat nat ed wi t h a non non-- zer o exi exi t code. code. ** * Pl ease see see SA SAP No Not e 94 943602 602 , sect i on ' J 2EE 2EE En Engi ne exi exi t codes' codes' *** f or add addi t i onal i nf or mat i on and and t r oub oubl e shoo shoott i ng. ********************************************************************** good bye bye ( exi t code = - 1111 11112) 2) [ Thr Thr 6000 6000]] J Launch LaunchC Cl osePr osePr ogr ogr am: good

©SAP AG 2013


7

30

Tracking Agent issues 7.1 7. 1

OSS OS S message pre-requi si sites tes

Before forwarding any OSS messages to the Dev support organization, it is important to check the following points (as far as concerning Diagnostics Agents):      

The Diagnostics agent has been installed with SAPinst SAPinst 7.01 EHP1, 7.11, 7.20, 7.3x or Solution Manager Manager 7.0 EHP1 SR1 or 7.1. The OS limit for Unix platform platform are compliant with with the agent OS user user (adm) (adm) JDK vendor and version is supported by the Diagnostics Agent JVM parameters of Diagnostics Agent are compliant to recommendations The Diagnostics Agent connection connection to the Solution Manager system i s of type J2EE Message Ser ver connection The Solution Manager Java dispatcher dispatcher is correctly sized to avoid Agent disconnections (see part 8.1 Solution Manager J2EE stack settings.)

In case the check points cannot solve the problem, problem, follow the SAP Note 1076573 1076573 to create the OSS message. Also Al so in case cas e th e LM-SERVICE vers ion io n i s at l east NW04S SPS17 Patch 3, att ach the support logs Zip file generated generated by t he smdsetup s crip t (see part 6.1.5 6.1.5 Usage Usage of smdsetup scri pt).

7.2 7. 2

Agent crash in vestigatio n

When the Diagnostics agent process shutdowns abnormally, the procedure to t o determine the cause is the foll owing:   

At OS level login as adm OS user Go in folder / usr / sap/ / J ER>/ wor k Open the log file dev_j con cont r ol (This is the log file of the jcontrol process which is controlli ng the Diagnostics Agent java process) to check if the jcontrol has started the jlaunch process as described below: [ Thr Thr 3288] J Cont ont r ol I CheckPr ckPr oce ocessLi st : pr ocess cess SMDAgent st ar t ed ( PI D: 3616)  If

the process jlaunch cannot be start, the error is probably located in SAP profile parameters

The following message is NOT a critical error: J St ar t upI upI CheckFr amewor kPackage: can' t f i nd f r amewor k package C: \ usr \ s ap\ SMD\ exe\ j vmx. j ar The Diagnostics Agent does not use specific SAP kernel functions and this message can be i gnored. 

The process jlaunch is started, open the file dev_ dev_ SMDAgent which corresponds to the jlaunch process output :

[ Thr 3860] J Launc hI hI Set St at e: c ha hange s t a t e f r om [ I ni t i al ( 0) ] t o [ I ni t i al ( 0 ) ] [ Thr 3860] J Launc hI hI Set St at e: c ha hange st st at e f r om [ I ni t i al ( 0) ] t o [ Wai t i ng f o r s t a r t ( 1) ] [ Thr 3860 3860]] Thu Dec 25 12: 12: 11: 11: 35 2008 2008 [ Thr 3860] J Lau LaunchI chI Set St at e: cha change st st at e f r om [ Wai t i ng f or st ar t ( 1) ] t o [St [St ar t i ng ( 2) ] [ Thr Thr 3860] J Lau LaunchI chI Set St at e: chan chang ge st st at e f r om [ St ar t i ng ( 2) ] t o [ Runni ng ( 3) ]

In this file, all information about agent java process is stored like JVM loading, process state, crash of VM or bad JVM parameters. It is important to have a look this fil e to find out Exit Codes or Err or messages.

JVM Loading problem

©SAP AG 2013

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide all JVM loading problems are logged in file dev_SMDAgent. Thr 108 10842381 238144 44]] ** * ERR ERROR => Dl Load LoadLi Li b: dl ope open() = l i bj vm. so: cann cannot open open shar shar ed Error : [ Thr obj ect f i l e: No such such f i l e or di r ectory - > DLEN LENOACCESS ( 0, Succe ccess) [ dl ux_mt . c 314] [ Thr Thr 1084 108423 2381 814 44] *** ERROR => Can' an' t l oad oad VM VM shar shar ed l i br ar y ( l i bj vm. so) ( r c=- 2) [ j hvmxx_mt . c 1576 1576]] [ Thr Thr 1084 108423 2381 8144 44]] ** * ERR ERROR => Canno annott l oad oad DLL f or J ava ava VM VM [ j l nchxxi chxxi _mt . 717] 17]

Here you can find potential JDK installation issues like :  Wrong JDK version for Agent c ompatible with SAP Kernel (32bits/64bits)  JDK is not compatible  Diagnostics Agent OS user have not rights to access to JDK Libraries The SAP Note 940893 - Analysis of errors within the startup of J2EE Engine 6.40 can help to troubleshoot JVM loading problems. 

If the JVM is loaded correctly, now check the file std_SMDAgent.out std_SMDAgent.out and jvm_SMDAgent.out to find some errors like OutOfMemory, StackOverflow, Nullpointer … which stop t he agent process.

©SAP AG 2013

31


7.3

32

Thread du dump mp

It is important to do t hread dumps on the Diagnostics Agent process to have an understand of the context, in case a deadlock or not responding Diagnostic Agent situation is faced.

Make a Thread Thread dump under Windows Windows The Diagnostics Agent is a SAP system and available in SAP MMC.

In SAP MMC, open the node “AS “ AS Java Process Table” of Diagnostics Agent identified by the SID and instance number. Only one process is displayed in the right part: It is the agent process. Select the process and right click on it to display the context m enu, press the item “Dump stack Trace”.

Make a Thread Thread dump under Unix Unix For 7.0x Logon in UNIX shell with the t he OS user of the Diagnostics Agent and execute the command “smdadmin.sh XX” (XX is the instance number of Diagnostics agent). For old agent installation, it is necessary to execute the command “. . / smdadm sap/ / J / scr i pt . admi n. sh” from the folder / usr / sap/ Once the command is executed, the JCMON menu is displaying. Select the entry “20: Local administration menu”. Then execute the action “10: Dump stacktrace” on process 0 (This is the process of the SMDAgent).

For 7.1x, 7.2x 7.2x and 7.3x Logon in UNIX shell with the t he OS user of the Diagnostics agent and go to ex e folder C:\usr\sap\DAA\SMDAXX\exe C:\usr\sap\DAA\SMDAXX\exe sapcontrol.exe -user adm adm -nr -function J2EEControlProcess SMDAgent DumpStackTrace example: /usr/sap/DAA/SMDA97.exe > ./sapcontrol -user smdadm abc123 -nr 98 -function J2EEControlProcess SMDAgent DumpStackTrace

Where is the thread dump stored? Once the thread dump is done, the output is written in file / usr / sap/ / J / wor k/ s t d_SMDAgent Agent . out with Sun and HP-UNIX JDK.

©SAP AG 2013


33

For IBM JDK, the thread dump output is written in j ava. cor e. . t xt in folder usr us r / sap/ / J / SMDAgent Agent . To find the absolute path of an IBM thread dump, open the file s t d_SMDAgent gent . out and find the line beginning with J “J ava Dump wr i t t en t o”

7.4 7. 4

Act iv ivate ate P4 log s

If a p4 connecti on problem (like timeout, connection reset) persists with the Solution Manager system or the Managed System after applying the different checks of this document, you should enable the P4 logs before sending the OSS message to the Development Support organization.

Connection problem to Managing System or Managed system The procedure to enable the P4 logs on the Diagnostics agent is the following:     

Stop the agent Clean the log folder under SMDAgent directory Set the log level (effective-severity) of SMDSystem location to ALL (see6.1.6 (see6.1.6 Change Agent log level) level) Set the log level (effective-severity) of P4 location to ALL (see6.1.6 (see6.1.6 Change Agent log level) level) Add the Java parameter -Ddebug=true in agent configuration (6.1.2 6.1.2 Change JVM Parameters) Parameters)

Once the p4 log is enabled on the Agent, go to the Solution Manager system, or the Managed system to execute the following instructions:      

Open the Visual Administrator of Java stack Select the server node Under SID node Select the service “Log Configurator” Go in tab Locations Select the location com.sap.engine.services.rmi_p4 Increase the log level to DEBUG and save the modification.

Once all P4 locations are configured, restart the agent to reproduce the problem and attach the agent logs (use the command ‘smdsetup supportlogs’) and the current defaultrace.log of the Managing or the Managed system to the OSS message. Perfo Perfo rmance Impact It is important to restore All log locations changed to their default level because the above described procedure has a high performance impact on the Diagnostics Agent and therefore indirectly on the Managed system or the Solution Manager system.

©SAP AG 2013


8

34

Scalability 8.1 8. 1

Solut io ion n Mana Manager ger J2EE stac k sett settin ings gs

SAP recommends adapting the J2EE configuration of Solution Manager Java stack especially in case you plan to connect over time more than 40 Diagnostics Agents to Solution Manager. The Java dispatcher has an important role in the Agent connection stability. Therefore, you need to adapt - using the Solution Manager system ConfigTool - the thread and connection settings, as well as the heap size configuration of the dispatcher node to node to avoid some timeout connection issues. Find below a summary of all parameters which have to be adapted. Also remind that a restart of the Solution Manager Java stack will be required. Di sp at c h er Par am et er Max Heap Size (in MB) (JVM Parameter -Xmx) ThreadManager->MaxThreadCount ThreadManager->InitialThreadCount ThreadManager->MinThreadCount ConnectionManipulator ->maxSoTimeOutConnections

Rec o m m en d ed Val u e Total number of Diagnostics Agents * 0.5 (Minimum 128MB) Total number of Diagnostics Agents * 2 + 50 Total number of Diagnostics Agents * 1.5 Total number of Diagnostics Agents Total number of Diagnostics Agents * 1.35

Ex am p l e f o r 600 A g en t s 300 MB 1250 900 600 810

Defaul Defaul t v alue of MaxThreadCount MaxThreadCount After an installation of Solution Manager, the default value of parameter MaxThreadCount in Java Dispatcher configuration is set to 70. These settings allow by default to co nnect a maximum of 40 Diagnostic Diagnostic s Agents .

©SAP AG 2013


9

35

Telnet SMD serv rve er commands

Some telnet commands are available in t he Solution Manager J2EE to check the installation and configuration of the SMD server. Connect with telnet program to SAP J2EE server of Solution Manager (default port 5xx18) and go on server node with command ‘jump ’. Add the group smdserver to enable the commands (use the command ‘add smdserver’). Once the group smdserver added, execute the command ’ man –g sm s mdser ver ’ to have access to all available “smd server” commands. SMD ser v er t el n et c o m m an d Smdstatus smdstatus –a list_agents fin d_agent or mm ON mm OFF kill_agent or start_agent or sto p_agent p_agent or host_mapping known_agents -c known_agents -n As Es

©SAP AG 2013

Des c r i p t i o n Display the version and running mode of SMD server. Check each agent connectivity to detect some connection problem with one or several agents. Display all Diagnostics Agents connected to SMD server with their status. Retrieve an Agent using a servername, fully qualified domain name or IP. Enable the maintenance mode of SMD server Disable the maintenance mode of SMD server Reboot the Diagnostics Agent process Start the Agent applications of selected Diagnostics agent Stop the Agent applications of selected Diagnostics agent Display the mapping between servername and network names for all agents. Displays the list of known Agents. (Agents that have been connected at least once to the Solution Manager system) Displays the list of known Agents by FQN. Provides ping statistics for all Diagnostics Agents. Provide method statics for the Solution Manager server.


36

10 Fu Full ll Qua Quali lifi fied ed Nam Name e reso resolu lutio tion n iss issue ue The list of SAP notes following can help to solve some network resolution probl ems in different part of SAP products.

SA P No t e Note 611361 - Hostnames of SAP servers

Resu m e The use of hostnames that are too long or contain illegal characters may result in undetermined errors

Note 962955 962955 - Use Use of o f virt v irt ual TCP/IP host name names s

You want to use one or more “logical” host names for SAP software within an SAP server landscape to hide the physical network identities for the pieces of software from each other. It is not clear which fully qualified host name (FQHN) the Internet Communication Manager (ICM) is using.

Note 773830 - FQHN determination in ICM Note 1282 128297 975 5 - Use of vi rtual TCP/I TCP/IP P host n ame ames s in Windows

You want to use one or more logical host names for SAP software within an SAP server landscape to hide the physical network identities for the pieces of software from each other.

Note 129 12999 997 7 - Hostname and IP address addre ss looku p

Information about IP address lookup.

Note 105 10521 2122 22 - Host names names in SLD

You wonder if a hostname that occurs in the System Landscape Directory (SLD) is correct. In the SLD, hostnames primarily occur in technical systems and destinations In computer systems registered by the ABAP data supplier in the SLD, attributes may be missing in certai n cases (for example, details of the operating system).

Note 97 9792 9277 77 - Incorrect registration registration of comupt er systems in the SLD SLD

©SAP AG 2013

Diagnostics Agent Troubleshooting

Recommend Documents