1
2
3
4
5
In transaction SMQR and SMQS you can see the resources available according to the settings of the rdisp/rfc* parameters. Call transaction SMQR and select ‚Goto„ >> ‚QRFC resources„. The result displays displays the number of work processes that can be used for processing the RFC request.
If DIA W Ps for tRFC/qRFC are constantly exhausted (DIA-WPs for tRFC/qRFC = 0), this indicates a resource problem. Either the RFC resources are not sufficient suff icient to accommodate the load or the qRFC processing is too slow. Note that the number of available resources in the system is a snapshot which relates to the load status of the system.
For tRFC and qRFC calls, the tRFC layer reacts r eacts by switching to synchronous RFCs RFCs instead of tRFCs or qRFCs. When the RFC is executed synchronously synchronously no further processes are needed for RFC processing. After finishing the processing for asynchronous tRFCs the program may again obtain free fr ee resources for further asynchronous tRFC calls.
To avoid overload situation the application can check the currently available resources using function module TH_ARFC_REQUESTS before calling RFCs.
6
The profile parameter rdisp/rfc_check rdisp/rfc_check can be used to strengthen the usage of quotas. Commonly, the problem is that only asynchronous asynchronous RFC calls will heed to the quotas being set. If there is a synchronous RFC call which is placed from within an asynchronous RFC, RFC, the quotas will not be adhered to by the synchronous RFC call.
By setting the parameter rdisp/rfc_check rdisp/rfc_check to 2, this will change. Any RFC cascade that starts with an asynchronous RFC call will be handled as if all of the RFCs in the cascade where asynchronous asynchronous RFC calls. You can even increase the value to 3 which would result in ALL RFCs being forced f orced to adhere to the quotas being set. However, this setting must be tested carefully, because it may result in resource r esource shortage by means of free dialog work processes.
RFC parameters may be changed dynamically (transaction RZ11 or via RFC server groups transaction RZ12) if resources are continuously exhausted. However, the changes are lost during restart.
Wrong configuration of CIF setting/parameters and lack of resources can slow down the CIF transfer/process or even worst, block the whole system.
7
The parameter 'rdisp/rfc_min_wait_dia_wp' 'rdisp/rfc_min_wait_dia_wp' indicates how many dialog work processes cannot be blocked using RFC. This prevents all dialog processes being occupied by parallel RFC requests. The default value is 1.
If 10 dialog work processes are configured on an instance (rdisp/wp_no (r disp/wp_no_dia _dia = 10) and the parameter rdisp/rfc_min_wait_dia_w rdisp/rfc_min_wait_dia_wp p is set to 2, maximal 8 dialog processes can be used for processing tRFC/qRFC call. In either case, 2 dialog processes are kept free for „real‟ dialog activities.
In the system, this can be verified as follows:
Determine the AS group which is assigned to the QIN scheduler (transaction SMQR)
Verify RFC parameter settings for this AS group (transaction RZ12 => choose corresponding AS group)
Determine the number of configured DIA work processes (Min. no of free W Ps) Attention: This number is taken from the active operation mode, not necessarily from the instance profile !
These numbers are visible in transaction SMQR => Goto => QRFC Resources.
8
To avoid that all available RFC resources are used by one user, the parameter rdisp/rfc_max_own_used_w rdisp/rfc_max_own_used_wp p can be set. When a user issues an RFC call it is checked how many processes the user has already occupied (RFCs or online dialog steps). The value is specified as a percentage of the configured dialog work processes. The default value is 75.
Example: There are 10 dialog work processes configured. If parameter rdisp/rfc_max_own_used_w rdisp/rfc_max_own_used_wp p is set to 50, maximal 5 dialog processes can be used by a certain RFC user / application at the same time. This is the minimum of the number of dialog work processes than can be used for tRFC/qRFC (10-2=8) and the share defined by rdisp/rfc_max_own_used_wp rdisp/rfc_max_own_used_wp (50 % of 10 = 5).
In the system, this can be verified as follows:
Determine the AS group which is assigned to the QIN scheduler (transaction SMQR)
Verify RFC parameter settings for this AS group (transaction RZ12 => choose corresponding AS group)
Determine the number of configured DIA work processes (Max. no of WPs used) This number is taken from the active operation mode, not necessarily from the instance profile ! Attention:
The available resources are visible in transaction SMQR => Goto => QRFC Resources. Note that the number of available resources in the system is a snapshot which relates to the load status of the system.
It is reasonable and recommended to restrict the resources for one RFC user / application because there may be other applications working with RFC calls and occupy dialog work processes, for example IDoc processing.
9
If the parameter rdisp/rfc_use_quotas is set to 1 the RFC resource parameters are used. You should NEVER change the default value. If the parameter is set to 0, then you can no longer work with the parallel RFC since no server can be determined for the next RFC.
The parameter rdisp/rfc_max_queue is percentage of the RFC entries that are allowed in the dispatcher queue until no further resources are given to RFC processing. However, However, the elements in the dispatcher queue are only increasing significantly if all work processes are used. Vice versa, as long as work processes are free, the dispatcher queue is (almost) empty. Therefore, as long as other RFC parameters are set, this parameter is not effectively ef fectively controlling controlling RFC load.
The parameter rdisp/rfc_max_login and rdisp/rfc_max_own_login are percentages of the logins of a single RFC user and the total of all RFC users compared to the maximum number of logins allowed. allowed. A dialog user usually stays logged on for a long time, usually all the time while working with the SAP System. Therefore, the number number of total connections allowed is usually much higher than the work processes configured. An RFC user however, however, usually logs off, of f, when the RFC is processed. The total connections of RFC users is close to the number of active work processes processing RFCs. Therefore, this parameter is not effectively controlling RFC load as long as other parameters are set.
For more information on these parameters see SAP note 74141.
10
11
12
The data constellation inside the CIF queue varies widely and depends highly on the objects types and on the business process triggering the transfer.
Number of queues = Number of entries:
1. possibility: Each object is sent in a separate queue and and one LUW uses exactly one queue. There are no (or only few) dependencies between between the queues so that the QIN scheduler can start the LUWs with a high parallel degree. There There is no (very low) risk that errors in queue processing block each other (no serialization). If the processing is too slow, a resource bottleneck can be assumed.
2. possibility: There is one LUW containing a huge number of objects that are using separate queue names each. In this case, the QIN scheduler cannot start the LUW in parallel (1 LUW => 1 work process). The processing of this large LUW may be not successful (timeout).
Number of queues << Number of entries:
The queues contain a high number of objects or one LUW owns objects in multiple queues. There are many dependencies between between the queues as the LUWs are containing the same objects. The queue monitor does not show that LUWs share queues.
The QIN scheduler determines which which LUW can be processed first to keep the right sequence. In case of highly dependent LUWs this step needs more time. The parallel degree of processing is limited even though enough resources are available.
The risk that errors in queue processing block each other is very high. In opposite to the situation above, additional resources resources have only limited effect on CIF processing speed .
13
To double check whether a huge LUW is processed ( and a serialization problem could be assumed ) , use SMQ2 2x bell and check the number of entries with same TID ( same LUW ).
A LUW is uniquely uniquely defined by the same TID. The LUW LUW may contain contain several objects objects that are transferred via different queue names. One LUW can only be processes in exactly one work process.
An error in in transferring or processing of the LUW LUW causes the whole queue to be stopped. Such a queue block not only affects the LUW containing the faulty queue entry, but also all LUWs containing subsequent queue entries. This is called serialization serialization effect.
Due to this the data transfer may be severely restricted and some data cannot be transferred at all. Consequently, there are inconsistencies between source and target system. For that reason, it is of utmost importance to rectify incorrect queue entries in time. Monitoring concept/handbook suitable suitable to the Best Practices is absolutely necessary and has to be established before go-live involving system administration AND business business department as well.
14
Status SYSFAIL: A serious error error occurred in the target system while while executing the LUW. For those queue entries, no automatic re-processing re -processing occurs through the QIN/QOUT scheduler. When you double-click on this status, the system displays an error text. SYSFAIL errors may have various reasons. They can be caused by missing or incomplete master data, liveCache errors (e.g. scheduling), termination of function modules / reports responsible for LUW processing. Additional information information about error reason can be found using using the following following transactions: Application
log log /SAPAPO/C3 (APO system) or CFG1 (R/3 system): Errors are recorded in the application log independent of the user settings (No logging, Normal, Detailed logging).
Short dump analysis ST22: In case of short dumps, no application log is recorded as this is done after LUW processing is finished.
System log SM21 and dev_* trace files f iles
15
16
The RETRY status occurs in general when another user or process already locks data (usually on location product level); theref ore the queue processing is stopped. The queue gets the status RETRY and the inbound scheduler triggers the scheduling of a background job named QRFC: with with a fixed, f ixed, hard coded period of 2 minutes. The maximum number of retries can be configured by the parameter “Attempts” / NRETRY during the registration of inbound queues in transaction SMQR.
Frequent queue entries in status retry should be avoided because they lead to enormous delay in inbound processing and to t o heavy system load (CPU utilization).
17
Locking problems can occur in both systems – systems – R/3 and APO. They T hey are usually recorded with error category 2 (Important) or error category 1 (Very important), visible visible as yellow or red lights.
Locking problems are mostly application problems and can have various reasons. Basis can provide some details, but application team needs to find reason.
Application logs logs can be viewed viewed with transaction transaction /SAPAPO/C3 (APO) or /CFG1 /CFG1 (R/3).
18
From the error message in the application log information can be obtained who (user) is locking what (object).
Example: The inspection lot is being modified at the same time via CIF (send order from APO to R/3) and an interactive user (dialog).
19
The background job only triggers a report that starts the CIF LUW again with the appropriate function module. Hence, the status Finished (GREEN) does not mean that the CIF LUW was successfully processed. As long as the processing fails the CIF LUW is visible with status RETRY in the queue q ueue monitor and a background job QRFC: with with status Released can be found in SM37.
If the maximum number of attempts (NRETRY, default 30) is hit the CIF LUW remains as is (status RETRY) without being touched anymore!
Failed processing issues a rollback of the entire CIF LUW. LUW . Consequently, Consequently, repeated processing leads to a higher work load. In particular, the CPU load increases significantly in case locking situations occur frequently.
20
Further bottlenecks are likely to occur because the number of background work processes in the system is limited and the background scheduler assigns them work with a period of 60 seconds (default value for f or rdisp/btc_time) though the runtime of the qRFC jobs is usually very short. Between 2 starts of the background scheduler a lot of new entries might be created which leads to the consequence that the jobs are delayed up to several hours.
Solution / work around: SAP Note 923228 provides a dynamic background scheduler which re-acts to the fact that a job has been completed and releases its background work process.
21
In some cases, due to complex logic of applications that create the t he queues, multiple calls in the sending system can be combined into one LUW independently of the queue names, which can generate interdependences between different queues. All the calls that belong to one LUW must be processed in the target system again as an unit.
Example: For a change of a purchase order and the corresponding inventory posting, two different queues are used, but these two changes are put in to one LUW and may only be processed together.
22
Since in queue CFPL000006003325 a SYSFAIL occurs and another LUW contains CFPL000006003325 CFPL000006003325 and CFSTKL10000S-FINISH CFSTKL10000S-FINISH--1 this LUW can„t be processed.
Queue CFSTKL10000S-FINISH-1 CFSTKL10000S-FINISH-1 is WAITING for queue CFPL000006003325 CFPL000006003325 and cannot be processed until the error (SYSFAIL) is resolved.
The display of a dependency chain (jumping from one W ait queue entry to the predecessor) is limited by the maximum number of internal modes allowed.
23
CFPL0000060003325 CFPL0000060003325 and CFSTKL1000OS-FINISH-1 are in the same LUW. LUW is distributed via 2 different queue names, means 2 different objects are processed.
The current situation is : the first entry in the queue failed to be processed, and blocks the second and third entry due to serialization demand. The third entry is in the same LUW with another entry in a different queue, and so blocks the other queue also.
24
25
26
27
Determine the RFC user which is defined for CIF connection (SM59) in the sending system. Check its activity using transactions SM66 / SM50 and find out whether there are any noticeable problems.
Work process in status running
Look in column Action/Reason column Action/Reason
Database actions in column Action/Reason for waiting
R/3 actions in column Action/Reason for waiting
No action in column Action/Reason for waiting
Work process in status stopped Look in column Reason column Reason
For both columns, detailed information can be obtained from the F1 help.
28
The workload monitor (transaction ST03n, expert mode) provides an overview about response time. To access the overall response time choose TOTAL and then specify a period to be analyzed. As CIF processing is recorded r ecorded as RFC step, this task type should be analyzed.
Within the time the response time and their shares can be evaluated.
Check for average response times during peaks (many dialog steps per hour)
Determine which which part is exceptional high (for example high DB DB Time)
The following profiles could be useful as well:
Transaction profile Server profile History of response times
29
The analysis of single statistic records is possible using transaction STAD. There are online parameters available influencing the statistic collection.
The following parameters might be useful to get an idea about the reason for bad performance. stat/dbprocrec :
If this parameter is set, the most expensive DB procedure calls and their execution time is recorded.
s tat/tabrec tabrec : stat/ If this parameter is set, table accesses and the time needed to access them is recorded.
The runtime parameter of the statistics collection can be changed online, see SAP note 579462 for details.
30
With the help of statistical records it is possible to find f irst indicators for the reason of performance problems. problems. SAP standard provides provides a couple of tools for different kind of analysis (ST05, SE30 etc.). In most cases, a detailed analysis by SAP experts is required. An OSS message should be opened (distribution list for performance problems SV-BO-SCM). For detailed bottleneck analysis SAP training tr aining BC315 Workload analysis is recommended. If liveCache performance is not sufficient, a detailed analysis using DB analyzer or analyzer or sapact may sapact may be reasonable to find the root cause. SAP t raining TEWA60 provides the necessary knowledge to deal with.
31
32
Performance/Applications Performance/Applications (direction Backend R/3 system => APO) shows data concerning the data volume and the performance on the timely basis specified in the user settings (per minute, hour, day or month).
The data is shown for the following documents: purchase documents (purchase orders and purchase requisitions), in-house production (planned orders and production orders), planned independent requirements, stocks, sales documents, inspection lots, reservation items, GI-posted document items, location products and locations (master data).
33
The data from CIF cockpit can be downloaded into MS excel file. Based on this format, the data can be prepared in various ways. Doing so, a good overview about the object types transferred per time frame (hours, day) is obtained. Peaks in CIF traffic are clearly visible.
34
The transaction ST13 provides an extended qRFC m onitoring included in tool CMO system monitoring. The transaction ST13 is contained in the component ST-A/PI - Application Servicetools.
Note that this is an expert tool which requires the preconditions mentioned above. It is preferable to be used for VTO tests and not recommended recomm ended for permanent usage.
35
The CMO system monitoring can be started upon request with a default runtime for the next 7 days. The system ID is filled by default with the current RFC destination. There are a couple of key figures measured and recorded (CPU utilization, m emory consumption, number of active work processed etc.).
The TRFCQIN snapshot counts and records the following figures:
Number of entries in status SYSFAIL / CPICERROR
Number of entries in status READY
Number of entries in status RUNNING
Beside this, the number of entries are counted per object type for the m ost common types: CFCO, CFEP, CFIP, CFLD, CFPO, CFTO, CFPLO, CFFCC, CFPPO, CFRSV, CFSHP, CFSLS, CFSTK.
The extended qRFC monitoring itself is activated by flag ‚with QRFC -counter„. This will force the QIN and QOUT scheduler respectively to count the processed LUW s (inbound and outbound) and the number of queue entries per LUW .
All key figures are recorded will be done with a period period of 60 seconds.
36
The figures recorded by CMO system monitoring can be displayed upon request on a daily basis. The flags for the corresponding c orresponding key figures have to be activated; otherwise they will not be displayed.
If several application servers (instances) are c onfigured, the figures can be displayed separatly by activating the flag ‚details of all application servers„.
If several systems involved in qRFC processing, these can be monitored m onitored and displayed displayed seperatly.
37
This example shows the result of a volume test for f or CIF outbound processing. The figures were downloaded to Excel.
In this slide you can see the real outbound scheduler throughput.
38
39
Traditional behavior: The algorithm the QIN scheduler is using to activate LUWs that do not have a predecessor does not allow a uniform usage of the maximum number of work processes that are available available for f or rfc. Once the quota is reached the QIN scheduler waits until at most 10 work processes are still active with rfc-processing before the next LUWs are activated. This leads to the following structure of active rfc work processes when the quota allowed 363 work processes at maximum. The time interval from one peak to the next is around 20 s. The threshold was defined with the absolute value 10
Improved behavior: behavior: The threshold value can be set using the static profile prof ile parameter rfc/inb_sched_resource_threshol rfc/inb_sched_resource_threshold. d. This parameter is available available as of certain kernel patch levels (see the Note 1115861).
SAP Note 1115861 provides a correction to get a more uniform population of the work processes after resource bottlenecks considering considering the threshold. That will enhance the speed of the QIN scheduler up to 50%.
40
If performance problems occur occur when a large number of CIF queue q ueue entries in the inbound queue is processed by the standard QIN scheduler, you can use the /SAPAPO/CIF_EMRG_QINSCHED /SAPAPO/CIF_EMRG_QINSCHED program to process these CIF queues.
Note that you should only use the program in emergencies ( e.g. post processing CIF entries after upgrade ). Do not use this program as standard to substitute the QIN scheduler.
Report /SAPAPO/CIF_EMRG_QINSCH /SAPAPO/CIF_EMRG_QINSCHED ED is in standard as of SCM 5.0. See SAP note 869399 for details.
For CIF outbound emergency scheduler see SAP Note 1055902. It is currently
available as of SCM 2007.
41
42
43
44
45
46
47
