CLOT KPI Analysis Guideline(RJIL)_v1.0

CLOT KPI Analysis Guideline (RJIL) March, 2015 Solution Lab.

Responsible for this document: 

배경 • • •



작성범위 • •



본 문서를 통해, RJIL CLOT 시험결과에 대한 원활한 KPI Pass/Fail 분석 업무 수행을 위해 초기 KPI Fail 사례 등을 바탕으로 작성 하였음 작성시점 기준 ActixOne을 활용한 분석이 용이하지 않아, 향후 AxtixOne을 활용한 분석 방법 기술 필요(VoLTE 관련 항목 등) 본 문서는 프로젝트 목표 달성을 위해 관련자 모두에게 공유 가능하며, 수정사항 필요 시 수정 및 추가 가능함

본 문서는 KPI 기준 및 ActixOne/XCAP/Anyplan-Optimizer활용 분석 방안을 제시 함 Reference CLOT 진행 시 발생 한 KPI Fail 사례를 통해 이슈를 정의 함

Change History: Changes

Author

Date

Version

Initial Version

Joohyun Kim/Jaewook Jung

29.03.2015

1.0

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

1

Table of contents  CLOT KPIs  How to Analyze KPI Items • • • • • • • • • • • •

RSRP SINR (Info) Overlapping Servers within 5dB PCI for all Sectors Observed (Info) Mac DL Throughput >= 2Mbps Mac UL Throughput >= 256kbps Average Mac DL Throughput Average Mac UL Throughput Call Drop TAU Success Rate (Info) Handover Success Rate Handover Interruption Time < 100ms

2

Table of contents  How to Analyze KPI Items • • • • • • • • • •

RACH Success Rate Attach Success Rate VoLTE Setup Success Rate Attach Setup Time < 1.5s IMS Registration Time < 1.5s (Report Only) VoLTE Setup Time < 1.5s VoLTE Jitter < 40ms VoLTE RTP Packet Loss Rate Average BLER DL (Info) Average BLER UL (Info)

 Appendix

3

CLOT KPIs No

KPIs

Call Model

Category

Target Value

NonExclusion

1

RSRP

LDL1, LUL

KPI

> 95 %

Don’t Consider

2

SINR >= -2 dB

LDL1, LUL

Info

-

Don’t Consider

3

Overlapping Servers within 5 dB <= 1

LDL1, LUL

KPI

> 60 %

Don’t Consider

4

Overlapping Servers within 5 dB <= 2

LDL1, LUL

KPI

> 90 %

Don’t Consider

5

PCI for all Sectors Observed

LDL1, LUL

Info

-

Don’t Consider

6

MAC Throughput DL >= 2 Mbps

LDL

KPI

> 95 %

Don’t Consider

7

MAC Throughput UL >= 256 kbps

LUL

KPI

> 95 %

Don’t Consider

8

Average MAC Throughput DL (kbps)

LDL

KPI

> 16,500

Don’t Consider

9

Average MAC Throughput UL (kbps)

LUL

KPI

> 2,500

Don’t Consider

10

Call Drop Rate

LDL1, LUL, VoL

KPI

<1%

Don’t Consider

11

TAU Success Rate

LDL1, LUL, VoL

Info

-

Don’t Consider

12

Handover Success Rate

LDL1, LUL, VoL

KPI

> 98 %

Don’t Consider

13

Handover Interruption Time < 100ms

LDL1, LUL, VoL

KPI

> 95 %

Don’t Consider

14

RACH Success Rate

VoS

KPI

> 99 %

Don’t Consider

15

Attach Success Rate

VoS

KPI

> 99 %

Don’t Consider

16

VoLTE Setup Success Rate

VoS

KPI

> 99 %

Don’t Consider

17

Attach Setup Time < 1.5 s

VoS

KPI

> 95%

Don’t Consider

18

IMS Registration Setup Time < 1.5 s

VoS

Info

> 95%

Don’t Consider

19

VoLTE Setup Time < 1.5 s

VoS

KPI

> 95%

Don’t Consider

20

VoLTE Jitter < 40 ms

VoL

KPI

> 95 %

Don’t Consider

21

VoLTE Latency < 100 ms

VoL

KPI

> 95 %

Don’t Consider

22

VoLTE RTP Packet Loss Rate

VoL

KPI

<= 1 %

Don’t Consider

23

Average BLER DL

LDL1, LUL

Info

-

Don’t Consider

24

Average BLER UL

LDL1, LUL

Info

-

Don’t Consider


Exclusion

4

How to Analyze KPI Items

5

1. RSRP 

Description •

Ratio of RSRP which meet ‘Coverage Objective RSRP Threshold’ within exclusion zone — ‘Coverage Objective RSRP Threshold’ value for each sector is referred from Cell Ref File ※ Sectors with the same clutter category could have different threshold values — BSP for the on-air sites would be only considered as exclusion zone

Clutter Category DU DU DU DU DU MU MU MU MU MU RU RU RU RU SU SU SU SU SU SU

Coverage Objective RSRP Threshold -106.8 -107.4 -108.2 -109.2 -110.5 -108.8 -107.2 -110.2 -110.3 -110.4 -113.6 -114.8 -116.1 -116.2 -111.8 -112.2 -113.2 -113.3 -113.4 -116.3

[Coverage Outage Threshold according to Clutter Category]

[BSP as Coverage Polygon]


6

1. RSRP 

Target Value •



Used Call Model •



> 95%

LDL1, LUL (Merge)

Formula •

(The number of passing RSRP samples / The total number of RSRP samples) within exclusion zone — Passing RSRP samples : the binned RSRP samples which meet ‘Coverage Objective RSRP Threshold’ within exclusion zone — The total number of RSRP samples : the all binned RSRP samples within exclusion zone


7

1. RSRP 

Analysis using ActixOne and AnyPlan-Optimizer •

Check KPI value in ActixOne Report

•

Find poor RSRP area using ActixOne and AnyPlan-Optimizer — Check RSRP plots with BSP in AnyPlan-Optimizer ※ Poor RSRP area outside BSP does not have to be considered in the analysis

•

Find the issue eNBs related with poor RSRP area — Check PCI plots in ActixOne (or AnyPlan-Optimizer)


8

1. RSRP 


Check the coverage of the issue eNBs — Check PCI plots in ActixOne(or AnyPlan-Optimizer) — Execute ‘Sector Swapped Site’ and ‘Tx Off Site’ in AnyPlan-Optimizer — Check ‘Sector Plot’ for each issue eNB in AnyPlan-Optimizer


9

1. RSRP 


Check the issue eNBs alarm status during drive test from LSM — Check the alarms such as RRH/Site/SNMP Down, ECP/UMP/GPS Function Fail , VSWR etc.

I-OR-SUNA-ENB-9005 Alarm History in eNB No.

Severity

Code

541

Critical

2007

556

Critical

2007

ProbableCause

Alarm time

Clear time

Clear type

Protocol Status Alarm ( Communication Fail : PING )

2015-02-03 12:37

2015-02-03 12:43

Auto

Protocol Status Alarm ( Communication Fail : PING )

2015-02-03 11:49

2015-02-03 11:49

Auto


10

1. RSRP 

Analysis using ActixOne and AnyPlan-Optimizer • • •

Execute ‘AOP’ in AnyPlan-Optimizer Fine-tune ‘AOP’ recommendation manually if necessary Check the prediction results — RSRP as well SINR and Serving Sector area should be also checked together

※ Refer to ‘AnyPlan-Optimizer User Guideline’ for the details ※ If RSRP KPI is not improved after finishing ‘AOP’/’Modify Sector Parameter’, it should be checked UE’s normality or the measurement environments such as obstacles Baseline

Prediction


11

1. RSRP 

Analysis Results •

Antenna Recommendation — Azimuth, M-Tilt and E-Tilt

•

The issue eNBs lists ※ All alarms not only for issue eNBs but also for all eNBs in the cluster should be checked and cleared before drive test


12

2. SINR (Info) 

Description •



Target Value •



-

Used Call Model •



Ratio of SINR that meets ‘SINR Outage Threshold’ value (-2dB) within exclusion zone

LDL1, LUL (Merge)

Formula •

(The number of passing SINR samples / The total number of SINR samples) within exclusion zone — Passing SINR sample : the binned SINR samples which meet ‘SINR Outage Threshold’ value (-2dB)’ within exclusion zone — The total number of SINR samples : the all binned SINR samples within exclusion zone


13

2. SINR (Info) 



•

Analyze poor SINR area : Analysis methodology would be the similar with RSRP analysis ※ Refer to ‘RSRP analysis’ and ‘AnyPlan-Optimizer User Guideline’ for the details — Find Poor SINR area — Find issue eNBs and check alarms — Execute ‘AOP’/’Modify Sector Parameters’ in AnyPlan-Optimizer etc. ex) removing dominant interference or enhancing the serving signal — Check the prediction results





•

The issue eNB lists ※ All alarms not only for issue eNBs but also for all eNBs in the cluster should be checked and cleared before drive test © SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

14

2. SINR (Info) 

Improve Point • Even through RSRP is good, SINR was very low due to missing neighbor ▶ Check Neighbor relation between two cells

SINR

RSRP

Despite of relatively high RSRP, SINR is very low due to missing neighbor


15

3. Overlapping Servers within 5 dB 

Description •



Target Value •



> {60% | 90%} for 1 server and 2 servers respectively

Used Call Model •



The average number of sectors having RSRP difference with serving best server within 5 dB, including best server signal within exclusion zone ※ The number of overlapping sectors is calculated first, then it is averaged by binning

LDL1, LUL (Merge)

Formula •

(The number of passing overlapping samples / The total number of bin samples) within exclusion zone — Passing overlapping samples : the binned samples which meet the average number of overlapping sectors within 5 dB <= {1 | 2} within exclusion zone — The total number of binned samples : the all binned samples within exclusion zone


16




•

Find overlapping area in ActixOne and AnyPlan-Optimizer — Check Serving and Neighbor RSRP in ActixOne (or AnyPlan-Optimizer) — Check ‘Pollution Plots’ in AnyPlan-Optimizer ※ If ‘AOP’ recommendation was already derived, check ‘Pollution Plots’ in Prediction layer


17



Analyze overlapping area : Analysis methodology would be the similar with RSRP analysis ※ Refer to ‘RSRP analysis’ and ‘AnyPlan-Optimizer User Guideline’ for the details — Find Overlapping area — Find issue eNBs and check alarms — Execute ‘AOP’/’Modify Sector Parameters’ in AnyPlan-Optimizer etc. ex) removing dominant interference or enhancing the serving signal — Check the prediction results





•

The issue eNB lists ※ All alarms not only for issue eNBs but also for all eNBs in the cluster should be checked and cleared before drive test


18


Analysis using XCAP • •

Analyze overlapping area from XCAP Find issue overlapping area — By XCAP Analyze, try to find the overlapping area

5dB<=2

5dB<=1


19

4. PCI for all Sectors Observed (Info) 

Description •



Target Value •



-

Used Call Model •



The number of observed PCIs during drive test compared with the number of planned PCIs for the particular cluster

LDL1, LUL (Merge)

Formula •

The number of observed PCI during drive test / The number of planned PCIs for the particular cluster


20

4. PCI for all Sectors Observed (Info) 



•

Check measured PCIs in ActixOne (or AnyPlan-Optimizer) — Check PCI Plots in ActixOne (or AnyPlan-Optimzier) — Check ‘Sector Plot’ and ‘Display Type’ function in AnyPlan-Optimizer

•

Check the un-measured eNBs alarm status during drive test from LSM — Check the alarms such as RRH/Site/SNMP Down, ECP/UMP/GPS Function Fail , VSWR etc.



Analysis Results • •

The unmeasured on-air eNB lists The issue eNB lists ※ All alarms not only for issue eNBs but also for all eNBs in the cluster should be checked and cleared before drive test


21

5. DL MAC Throughput >= 2Mbps 

Description •



Target Value •



> 95%

Used Call Model •



Ratio of summed MAC DL Throughput from device1 and device2 which meets the criteria (2Mbps) within exclusion zone

LDL1, LDL2 (Separate)

Formula •

(The number of passing DL Tput samples (LDL1) + The number of passing DL Tput samples (LDL2)) / (The total number of DL Tput samples (LDL1) + The total number of DL Tput samples (LDL2))

within exclusion zone — Passing DL Tput samples : the binned MAC DL Throughput samples which meet 1Mbps within exclusion zone — The total number of DL Tput samples : the all binned MAC DL Throughput samples within exclusion zone


22


Analysis using ActixOne, AnyPlan-Optimizer and XCAP •


•

Find poor DL Throughput area using ActixOne and AnyPlan-Optimizer — MAC Layer Throughput Download could be checked in ActixOne report ※ Only ‘PayloadRate’ can be displayed in ActixOne — Check DL Throughput plots with BSP in AnyPlan-Optimizer ※ Poor DL Throughput area outside BSP does not have to be considered in the analysis


23



Find the issue eNBs related with poor DL Throughput area — Check PCI plots in ActixOne (or AnyPlan-Optimizer)

•

Check the issue eNBs alarm status during drive test from LSM — Check the alarms such as RRH/Site/SNMP Down, ECP/UMP/GPS Function Fail , VSWR etc. ※ Refer to ‘RSRP analysis’ for the details

•

Modify Antenna Parameters in AnyPlan-Optimzier (Improve DL Edge SINR) — Execute ‘AOP’ in AnyPlan-Optimzier — Execute ‘Modify Sector Parameters’ in AnyPlan-Optimzier if necessary ※ Refer to ‘RSRP analysis’ and ‘AnyPlan-Optimizer User Guideline’ for the details

I-OR-SUNA-ENB-9005 Alarm History in eNB No. Severity Code 541

Critical

556

Critical

ProbableCause

Alarm time Clear time

Protocol Status Alarm 2015-02-03 2015-02-03 2007 ( Communication Fail : 12:37 12:43 PING ) Protocol Status Alarm 2015-02-03 2015-02-03 2007 ( Communication Fail : 08:53 11:49 PING )

Clear typ e Auto

Auto


24

5. DL MAC Throughput >= 2Mbps •

Check other issues in XCAP — Check missing Neighbors (No HO after multiple MR transmission) — Check Backhaul issues ‐ Server Drop (Error message from FTP server) ‐ Socket Close ‐ Initial attach or IdletoActive fails due to no receiving ‘dlInformationTransfer’ message — Check FTP Server issues





•

The issue eNB lists ※ All alarms not only for issue eNBs but also for all eNBs in the cluster should be checked and cleared before drive test

•

Missing NBR list etc.


25


Fail Cases •

Poor RF (SINR) — DL MAC Throughput was low due to the low SINR

▶ RF Optimization DL MAC Throughput fail


26


Fail Cases •

Missing Neighbors — DL MAC Throughput was low due to no HO procedure ‐ No HO procedure after multiple MR transmission due to missing Neighbors ‐ RACH procedure by ‘UL Data arrival’ after multiple MR transmission due to missing Neighbors

▶ Check Neighbor list in LSM (check missing neighbor)


27


Fail Cases •

Interference issue — DL MAC Throughput was low due to UL interference ‐ eNB didn’t receive ACK/NACK for FTP DL data on PUSCH due to high IoT in this eNB

▶ Check Interference

LDL PUSCH Tx Power LDL Power Headroom

LDL BLER

LDL Low DL Throughput © SAMSUNG Electronics Co., Ltd. Confidential and Proprietary © SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

28


Fail Cases •

Late Service Request Triggering — UE triggered service request too late after RRC Connection Release, so there happened server drop

▶ Check Backhaul and S1 interface

Server Drop Release DL Low Throughput


29

6. UL MAC Throughput >= 256kbps 

Description •



Target Value •



> 95%

Used Call Model •



Ratio of MAC UL Throughput which meets the criteria (256kbps) within exclusion zone

LUL

Formula •

(The number of passing UL Tput samples / The total number of UL Tput samples) within exclusion zone — Passing UL Tput samples : the binned MAC UL Throughput samples which meet 256kbps within exclusion zone — The total number of UL Tput samples : the all binned MAC UL Throughput samples within exclusion zone


30




•

Analyze poor UL Throughput area : Analysis methodology would be the similar with DL Edge Throughput analysis — — — —

Find poor UL Throughput area Find issue eNBs and check alarms Execute ‘AOP’/’Modify Sector Parameters’ in AnyPlan-Optimizer etc. Check other issues such as missing neighbor, FTP Server issues in XCAP

BSP


31


Fail Cases •

Poor RF (RSRP) — UL MAC Throughput was low due to the low RSRP

▶ RF Optimization UL MAC Throughput fail


32


Fail Cases •

Missing Neighbors — UL MAC Throughput was low due to no HO procedure ‐ After multiple MR transmission, UE didn’t start HO procedure due to missing Neighbors ‐ After multiple MR transmission, UE started RACH procedure by ‘UL Data arrival’



33


Fail Cases •

Backhaul issue — UL MAC Throughput was low due to several backhaul related issues ‐ FTP Server Drop ‐ Socket Close ‐ Small Buffer Size of UE due to small TCP window size of eNB ‐ Late service request after RRC Connection Release ‐ No receiving ‘Security protected NAS message’ message from MME after RRC connection setup complete in normal RF environment

▶ Check Backhaul and S1 interface


34


Fail Cases •

Inherence issue — UL MAC Throughput was low due to interference(Tx Power : Max, Power Headroom : low, RF environment : Not bad)

▶ Check interference

UL Throughput


35

7. Average DL MAC Throughput 

Description •



Target Value •



> 16,500kbps

Used Call Model •



Summation of the averaged MAL DL Throughput from device 1 and device 2 within exclusion zone

LDL1, LDL2 (Separate)

Formula •

(Average MAC DL Throughput for LDL1 + Average MAC DL Throughput for LDL2) within exclusion zone — Average MAC DL Throughput : (summation of the binned MAC DL Throughput samples / the total number of MAC DL Throughput samples) within exclusion zone


36

7. Average DL MAC Throughput 



•

Analyze poor DL Throughput area : Analysis methodology would be the similar with DL Edge Throughput analysis — — — —



37

8. Average UL MAC Throughput 

Description •



Target Value •



> 2,500kbps

Used Call Model •



The averaged MAL UL Throughput within exclusion zone

LUL

Formula •

Average MAC UL Throughput for LUL within exclusion zone — Average MAC UL Throughput : (summation of the binned MAC UL Throughput samples) / (the total number of MAC UL Throughput samples) within exclusion zone


38

8. Average UL MAC Throughput 



•

Analyze poor UL Throughput area : Analysis methodology would be the similar with DL Edge Throughput analysis — — — —


BSP


39

9. Call Drop Rate 

Description •



Target Value •



<1%

Used Call Model •



Ratio of Dropped VoLTE Calls and Abnormal Release of Data Call

LDL1, LUL and VoL (Separate)

Formula •

(The number of VoLTE Call Drops + The number of Abnormal Release for LDL1, LUL and VoL) / (Summation of the number of Calls Attempts for LDL1, LUL and VoL) — VoLTE Call Drop : Call is terminated by receiving BYE message from network side — Abnormal Release : UE is released without ‘rrcConnectionRelease’ message (RRC Connection Request without RRC Connection Release) — The number of Call Attempts for LDL1 : Roundup(Sum(MAC DL Bytes(LDL1))/52,428,800Bytes) — The number of Call Attempts for LUL : Roundup(Sum(MAC UL Bytes(LUL))/10,485,760Bytes) — The number of Call Attempts for VoL : Roundup(Sum(VoLTE Call Duration(VoL))/90sec)


40


Analysis using ActixOne and XCAP •


•

Check call drop events in ActixOne — ActixOne does not provide the items which can check VoLTE Call Drop and RRC Drop yet


41



Check Call drop events in XCAP — Check the event times for all call drops

•

Check the detailed messages in XCAP for the fail events


42


Fail Cases •

Poor RF Environments — UE didn’t received RTP packets due to poor RF quality

▶ Check RF environment (RSRP/SINR)


43


Fail Cases •

Service Reject — UE didn’t received RTP packets due to service reject from core network

▶ Check Core network (including IMS Server)

Service Reject


44


Fail Cases •

Poor RF Environments — UE was released from eNB without ‘rrcConnectionRelease’ message due to Out-of-Sync.



45


Fail Cases •

Abnormal release by S1 Interface issue — UE was released from eNB without ‘rrcConnectionRelease’ message due to S1 interface

▶ Check Backhaul and S1 Interface

No ‘rrcConnectionRelease’ message


46

10. TAU Success Rate (Info) 

Description •

Ratio of Tracking Area Update (TAU) Success — TAU procedure : from Tracking Area Update Request to Tracking Area Update Success in NAS L3 message



Target Value •



Used Call Model •



-


Formula •

The number of TAU Success / The number of TAU Attempts — TAU Success : Receiving ‘Tracking area update accept’ (DL EPS MM) — TAU Attempt : Sending ‘Tracking area update request’ (UL EPS MM)


47


Message from XCAP •

From Tracking area update request (UL EPS MM) to Tracking area update success (DL EPS MM)

Attempts

Complete


48




•

Check TAU fail events in ActixOne — ActixOne does not provide the items which can check VoLTE Call Drop and RRC Drop yet

•

Check TAU fail events in XCAP — Check the specific event time for TAU fails


49





50


Fail Cases •

No TAU Update Message — After sending the TAU Message, UE didn’t receive the TAU accept

▶ Check Core network (including MME)

TAU Update Fail

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary © SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

51

11. Handover Success Rate 

Description •

Ratio of Handover Success for the intra-frequency LTE Handover Procedure — Handover procedure : from RRC Connection Reconfiguration (contains HO command) to LTE MAC RACH Attempt Success for RRC Connection Reconfiguration Complete



Target Value •



Used Call Model •



>98%


Formula •

The number of Handover Success / The number of Handover Attempts — Handover Success : LTE MAC RACH Attempt Success for ‘rrcConnectionReconfigurationComplete’ — Handover Attempts : Sending ‘rrcConnectionReconfiguration’ (contains HO command)


52



From ‘rrcConnectionReconfiguration’ message (with HO Command) to LTE MAC RACH attempt Success

Attempts

Complete


53




•

Check ‘LTE Handover Fail’ events in ActixOne


54



Check Handover fail events in Macro and XCAP — Check the specific event time for HO fail

•



55


Fail Cases •

Failure at MSG2 — UE didn’t receive RAR(MSG2) from eNB due to poor RF environments



56


Fail Cases •

RLF during HO — UE didn’t succeed Handover due to Radio link failure during HO procedure

▶ Check RF Environments (RSRP/SINR)


57


Fail Cases •

PCI Duplication — UE didn’t receive RAR(MSG2) due to referring the wrong cell

▶ Check PCI Collision and Confusion in NBR list from LSM

RACH info

SIB2


58


Fail Cases •

No Measurement Report — Measurement parameters are not configured in eNB

▶ Check eNB Parameter or configuration

Measurement parameters are not configured


59

12. Handover Interruption Time < 100ms 

Description •

Ratio of Handover Success which meets the criteria (100ms) among the intra-frequency LTE Handover Success Procedure — Handover procedure : from RRC Connection Reconfiguration (contains HO command) to LTE MAC RACH Attempt Success for RRC Connection Reconfiguration Complete



Target Value •



Used Call Model •



>95%


Formula •

The number of Handover Success within 100ms / The number of Handover Success — Handover Success within 100ms : LTE MAC RACH Attempt Success for ‘rrcConnectionReconfigurationComplete’ within 100ms — Handover Success : LTE MAC RACH Attempt Success for ‘rrcConnectionReconfigurationComplete’


60



From ‘rrcConnectionReconfiguration’ message (with HO Command) to LTE MAC RACH attempt Success

Attempts

Complete


61




•

Check ‘LTE Slow Handover’ events in ActixOne ※ Handover Interruption Time fail could be different with ‘LTE Slow Handover’ events in ActixOne


62



Check Handover Interruption time fail events using Macro — Check the specific event time for HO interruption time fail

•



63


Fail Cases •

Multiple MSG2 re-transmission — UE re-transmitted MSG2 due to poor RF environments



64

13. RACH Success Rate 

Description •

Ratio of Contention-based RACH Success — RACH Procedure (Contention based): from Random Access Request message(MSG1) to Contention Resolution message(MSG4)



Target Value •



Used Call Model •



>99%

VoS

Formula •

The number of RACH Success / The number of RACH Attempts(CT Based) — RACH Success : LTE MAC RACH Attempt Success (Success of Message 4) — RACH Attempts(CT Based) : ‐ LTE MAC RACH trigger followed by ‘rrcConnectionRequest’ ‐ LTE MAC RACH trigger followed by ‘rrcConnectionReestablishmentRequest’


65



Initial RRC Connection Setup — From LTE MAC RACH trigger by Connection request to LTE MAC RACH attempt Success Attempts

Complete

•

RRC Connection Re-establishment — From LTE MAC RACH trigger by Radio Link failure to LTE MAC RACH attempt Success Attempts

Complete


66




•

Check Contention-based RACH fail events using Macro and XCAP — Check the event times for RACH fails ※ RACH triggered by RLF should be also counted

•

Check the detailed messages in XCAP for the fail event


67


Fail Cases •

Failure at MSG2 — UE didn’t receive Random Access Response (RAR) from eNB due to poor RF environments — RACH will be aborted when T300 timer is expired



68


Fail Cases •

Failure at MSG4 due to CT Timer expired — UE didn’t receive MSG4 from eNB within the pre-defined CT timer(56sf)



69

14. Attach Success Rate 

Description •

Ratio of LTE Attach Success — Attach Procedure : from RRC Connection Setup Complete with Attach Request to IMS Registration



Target Value •



Used Call Model •



>99%

VoS

Formula •

The number of Attach Success / The number of Attach Attempts — Attach Success : Receiving SIP 200 OK for IMS Registration after ‘rrcConnectionSetupComplete’ with Attach Request — Attach Attempts : Sending ‘rrcConnectionSetupComplete’ with Attach Request


70


Attach Procedure


71



From ‘rrcConnectionSetupComplete’ message (with Attach Request) to SIP 200 OK message for IMS REGISTER

Attempts

Complete


72




•

Check Attach fail events in ActixOne — ActixOne does not provide the items which can check Attach fails yet


73



Check Attach fail events using Macro — Check the event times for attach setup fails

— Check the event times for IMS registration fails

• •

Check the detailed messages in XCAP for the fail events Define the issues whether the issue is comes from core network(IMS) or not — Refer IMS issue code in Appendix © SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

74


Fail Cases •

Initial Attach Fail by S1 interface issue — UE didn’t receive “dlInformationTransfer" message after RRC Connection Setup Complete

▶ Check the stability of S1 interface

No message from MME after rrcConnectionSetupComplete


75


Fail Cases •

No Receiving TCP ACK for Sync — UE didn’t receive TCP Acknowledgement (SYNC+ACK) for IMS Register after Default Bearer Setup

▶ Check Configuration of N7K router ▶ Check the stability of Backhaul and S1 interface

No receiving ACK message for TCP Sync.


76


Fail Cases •

403 Forbidden — UE didn’t receive SIP 200 OK message for IMS registration due to IMS server error

※ Another core side issue codes could be analyzed with the similar way ▶ Check Core network (including IMS server)

No 200 OK for Registration


77


Fail Cases •

Initial Attach Fail — After RRC Connection Setup Complete, UE was released due to ‘PDN CONNECTIVITY REJECT’ message from MME ※ Reject Cause : #30 (Request rejected by Serving GW or PDN GW)

▶ Check Core network (including IMS server)

Attach Reject : PDN Connectivity Reject


78

15. VoLTE Setup Success Rate 

Description •

Ratio of VoLTE Call Setup Success — VoLTE Setup Procedure : from INVITE to 200 OK for INVITE



Target Value •



Used Call Model •



> 99%

VoS

Formula •

The number of VoLTE Setup Successes / The number of VoLTE Setup Attempts — VoLTE Setup Success : Receiving SIP 200 OK message for INVITE (After a dedicate bearer with QCI 1 is activated) — VoLTE Setup Attempt : Sending SIP INVIVTE message (with originating SDP)


79


VoLTE Setup Procedure


80



From SIP INVITE message to SIP 200 OK message for INVITE

Attempts

P-RACK

Complete 200 OK for INVITE


81




•

Check VoLTE Setup fail events in ActixOne ※ ActixOne does not implement analyzing function for VoLTE Call in detail Network Optimization

Call Event Explorer


82



Check VoLTE Setup fail events using Macro — Check the event time for VoLTE setup fails ‐ Long Setup time more than 10 seconds ‐ No Setup time ※ The number of fails found from Macro file could be different with ActixOne (Refer to Appendix)

• •

Check the detailed messages in XCAP for the fail events Define the issues whether the issue is comes from IMS or not — Refer IMS issue code in Appendix © SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

83


Analysis using ActixOne and XCAP • • •

Check the fail events found in ActixOne in XCAP also Check the detailed messages in XCAP for the fail events Define the issues whether the issue is comes from core network or not — Refer IMS issue code in Appendix


84


Fail Cases •

487 Request Terminated — UE didn’t receive SIP 200 OK message for invite due to IMS server error

※ Another core side issue code error could be analyzed with the similar way ▶ Check Core network (including IMS Server)

No 200 OK for INVITE


85


Fail Cases •

500 Server Internal Error — UE didn’t receive SIP 183 Session Progress message from IMS due to server error


No 200 OK for INVITE


86


Fail Cases •

No 200 OK for PRACK — UE didn’t receive SIP 200 OK message for P-RACK when RF environment was good ※ SIP 200 Status Code : The 200 status code is the ACK message which represents the request was received and processed successfully


No 200 OK for PRACK


87


Fail Cases •

No 100 Trying — UE didn’t receive 100 Trying message after invite even through RF Environment is good



88


Fail Cases •

No 100 Trying — UE didn’t receive 100 Trying message which comes from IMS Server


No 100 Trying between INVITE and PRACK


89

16. Attach Setup Time < 1.5s 

Description •

Ratio of Attach Setup Success which meets the criteria (1.5s) among the Attach Setup Success Procedure — Attach Setup Procedure : from RRC Connection Setup Complete with Attach Request to Attach Complete



Target Value •



Used Call Model •



>95%

VoS

Formula •

The number of Attach Setup Success within 1.5s / The number of Attach Setup Success — Attach Setup Success within 1.5s : Sending ‘Attach complete’ message (UL EPS MM) after ‘rrcConnectionSetupComplete’ with Attach Request within 1.5s — Attach Setup Success : Sending ‘Attach complete’ message (UL EPS MM) after ‘rrcConnectionSetupComplete’ with Attach Request


90


Attach Setup Procedure


91



From ‘rrcConnectionSetupComplete’ (with Attach Request) to Attach Complete (UL EPS MM) Attempts

Complete


92




•

Check Attach setup time fail events in ActixOne — ActixOne does not provide the items which can check Attach setup time fail event yet


93



Check Attach setup time fail events using Macro — Check the event time for attach setup time fails : Filter out the cases having more than 1.5s

• •

Check the detailed messages in XCAP for the fail events Define the issues whether the issue is comes from core network or not — Refer IMS issue code in Appendix


94


Fail Cases •

Security protected NAS message Delay — UE received Security protected NAS message from MME too late

▶ Check the stability of S1 Interface 16:59:59.819

16:00:02.088


95

17. IMS Registration Time < 1.5s (Report Only) 

Description •

Ratio of IMS Registration Success within the criteria (1.5s) among IMS Registration Success Procedure — IMS Registration Procedure : from PDN Connectivity Request to 200 OK for IMS Registration



Target Value •



Used Call Model •



>95% (Report Only)

VoS

Formula •

The number of IMS Registration within 1.5s / The number of IMS Registration Success — Attach Setup Success within 1.5s : Receiving SIP 200 OK for IMS Registration after ‘PDN Connectivity Request’ (UL EPS SM) within 1.5s — IMS Registration Success : Receiving SIP 200 OK for IMS Registration after ‘PDN Connectivity Request’ (UL EPS SM)


96


IMS Registration Procedure


97



From ‘PDN connectivity request’ (UL EPS SM) to SIP 200 OK for IMS registration

Attempts

Complete


98




•

Check IMS registration setup time fail events in ActixOne — ActixOne does not provide the items which can check IMS registration setup time fails yet


99


Fail Cases •

504 Server Time-out — UE received 200 OK for IMS registration too late due to server error

※ Another IMS error code could be analyzed with the similar way ▶ Check Core network (including IMS Server)


100

18. VoLTE Setup Time < 1.5s 

Description •

Ratio of VoLTE Setup Success which meets the criteria (1.5s) among the VoLTE Setup Success Procedure — VoLTE Setup Procedure : from INVITE to 200 OK for IVITE



Target Value •



Used Call Model •



>95%

VoS

Formula •

The number of VoLTE Setup Success within 1.5s / The number of VoLTE Setup Success — VoLTE Setup Success within 1.5s : Receiving SIP 200 OK for INVITE after SIP INVITE within 1.5s — VoLTE Setup Success : Receiving SIP 200 OK for INVITE after SIP INVITE


101


VoLTE Setup Procedure


102



From SIP INVITE message to SIP 200 OK message for INVITE

Attempts

P-RACK

Complete

200 OK for INVITE


103




•

Check VoLTE setup time fail events in ActixOne — ActixOne does not provide the items which can check VoLTE setup time fails yet


104



Check VoLTE setup time fail events using Macro — Check the event time for VoLTE setup time fails : Filter out the cases having more than 1.5s

• •

Check the detailed messages in XCAP for the fail events Define the issues whether the issue is comes from core network or not — Refer IMS issue code in Appendix


105


Fail Cases •

QCI1 Activation Delay by IMS — QCI1 Bearer was activated too late

▶ Check Core network (including IMS Server) Time: 16:55.25.687

Time: 16:55.31.093


106


Fail Cases •

183 Session Progress Delay by IMS — UE received 183 Session Progress message with delay more than 20000ms

▶ Check Core network (including IMS Server) Time: 17:25.13.799

Time: 17:25.18.623


107

19. VoLTE Jitter < 40ms 

Description •

Ratio of VoLTE Jitters which meets the criteria (40ms) — VoLTE Jitters : Statistical variation of the RTP data packet inter-arrival time (RFC 3550)



Target Value •



Used Call Model •



>95%

VoL

Formula •

The number of VoLTE Jitters within 40ms / The number of VoLTE Jitters — Lost VoLTE RTP Packets will be calculated based on the received RTP Packet Sequence Number


108




•

Check VoLTE Jitter fail events in ActixOne — ActixOne does not provide the items which can check VoLTE Jitter yet


109


Analysis using XCAP • •

Check Jitter value in XCAP Check the detailed messages in XCAP for the fail events


110

20. VoLTE RTP Packet Loss Rate 

Description •

Ratio of Lost VoLTE RTP Packet — Causes for packet loss : network congestion, late arrivals etc.



Target Value •



Used Call Model •



<1%

VoL

Formula •

The number of Lost VoLTE RTP Packets / The number of Transmitted VoLTE RTP Packets — Lost VoLTE RTP Packets will be calculated based on the received RTP Packet Sequence Number


111




•

Check RTP Packet Loss in ActixOne


112



Check RTP Packet Loss events using XCAP — Check the event time for RTP Packet Loss ‐ Packet count/Packet Loss Count


113


Fail Cases •

Poor RF Environments — RTP Packets were lost due to poor RF environments(RSRP/SINR)

▶ RF Optimization


114

Packet Loss Point


114


Fail Case •

Handover Fail — RTP Packets were lost due to HO Fail


H/O Fail

Packet Loss


115

21. Average BLER DL (Info) 

Description •



Target Value •



-

Used Call Model •



Average NACK rate in DL for the initial transmission within exclusion zone

LDL1, LUL (Merge)

Formula •

Average of DL BLER for the binned samples within exclusion zone


116

21. Average BLER DL (Info) 



•

Check BLER DL in XCAP — Check Graph or Table> Qualcomm LTE/LteAdv> PDSCH BLER Info> Total Info> Total BLER


117

22. Average BLER UL (Info) 

Description •



Target Value •



-

Used Call Model •



Average NACK rate in UL for the initial transmission within exclusion zone

LDL1, LUL (Merge)

Formula •

Average of UL BLER for the binned samples within exclusion zone


118




•

Check BLER DL in XCAP — Check Graph or Table> Qualcomm LTE/LteAdv> PDSCH BLER Info> Total Info> Total BLER


119


Fail Cases •

High UL BLER due to Interference — High UL BLER was detected withtin specific area due to interference ※ UL would be affected more severely than DL due to high sensitivity and antenna gain of eNB

▶ Check IoT level in LSM ▶ Detect Interference source using spectrum analyzer at the field ※ Interference source should be removed before CLOT

High UL BLER area


120

Appendix    

CLOT Report Download from ActixOne CLOT Test Scenarios Issue Code – IMS & EMM Automation Process

CLOT Report Download from ActixOne 

Drive log cycle and Cluster name are required to download CLOT report



Log On to ActixOne(http://49.40.2.69/ActixOne/) with provided login credentials •

ID/PW : samsung1/samsung1


122


Follow the below steps sequentially

1. Click on Network optimization module

2. Click on Task Management tab

3. Click on filter tab

4. Select & click create filter


123



5. Put the filter Name you want

6. Keep one row for filter , select other row and delete with clicking “delete row” b utton


124



7. From “Field “ column sel ect “description” and from “operator” column select ‘ contains’ and put Cluster n ame of required report whi ch you want to download in “Value” column and the click on “Save “


125



8. Right click on Task management window

9. click on Reports


126



10. Double click on “R JIL CLOT Acceptance R eports “ template


127



11. Fill the Cluster Name , Task ID detail and click on “Run”

Task ID

Cluster Name


128



12. Save the CLOT report


129

CLOT Test Scenario 

Drive Test • •

It will be used 5 handsets to perform five tests using Accuver DM tool installed in mobile handset Each handset will perform one of the test given below 1. 2. 3. 4. 5.

•

FTP DL 1 FTP DL 2 FTP UL VoLTE Long VoLTE Short

All test scenarios are already predefined in Accuver DM tool — Drive test engineer will select different test scenarios per handset


130

Issue Codes 

IMS Issue Code for VoLTE

NO

Codes

Description

1

403

Forbidden

2

487

Request Terminated

3

500

Internal Server Error

4

501

Not Implemented

5

502

Bad Gateway

6

503

Service Unavailable

7

504

Service Time-out

8

505

SIP Version Not Supported

9

513

Message Too Large


131

Issue Codes 

PDN Connectivity Reject •

If the PDN CONNECTIVITY REQUEST cannot be accepted by the network, then MME shall send a PDN CONNECTIVITY REJECT message to the UE. — This message shall contain an ESM cause value which indicates the reason for rejecting the UE requested PDN connectivity. — The ESM cause IE typically indicates one of the following ESM cause values; ‐ Refer to section 6.5.1.4 in the 3GPP TS 24.301 for detailed information #

EMM Cause

#

EMM Cause

#8

operator determined barring

#35:

PTI already in use

#26

insufficient resources

#38

network failure

#27

missing or unknown APN

#50

PDN type IPv4 only allowed

#28

unknown PDN type

#51

PDN type IPv6 only allowed

#29

user authentication failed

#53:

ESM information not received

#30

request rejected by Serving GW or PDN GW

#54

PDN connection does not exist

#31

request rejected, unspecified

#55

multiple PDN connections for a given APN not allowed

#32

service option not supported

#95

111: protocol errors

#33

requested service option not subscribed

#112

APN restriction value incompatible with active EPS bearer context.

#34

service option temporarily out of order


132

Automation Process 

Automation Process Improvement •

XCAL-Mobile → FMS → ActixOne ※ XCAL-Mobile : UE Diagnostic Monitor App. in S4mini, FMS : Fleet Management System, ActixOne : Automatic field test log analysis and report generation tool

XCALMobile Content



Collect Log file

File Type



DRX

Content

•

Management Call Scenario, IMEI & Log file

File Type

•

Convert DRX → DRM

Storage

•

600GB → Upgrade 6TB (10DEC ~ 15DEC2014)

Processing

•

Processing 1 = max 2,000 Devices

Save file

•

The type of Log file is DRX saved 3 month (User option)

FMS

DRX

DRM

(Storage 1 / Processing 1)

A

(Storage 1 / Processing 28)

B DRX

Backup Server

Point

ActixOne

Content

•

Save Log files

File Type

•

DRX

Storage

•

10TB

Save file

•

The type of Log file is DRX saved (Not delete)

Content

•

Generation SCFT & CLOT report Provide Plot & Map

File Type

•

DRM

Storage

•

7TB → upgraded 60TB (15JAN2015)

Processing sever

• •

8 → 28 upgraded 1 = 16 files processing

Improvement Plan

•

Target

Owner

A

 To add file check and feedback process in FMS when engineer upload log files.

End of Feb.

Accuver

B-1

 To enhance link bandwidth between FMS and ActixOne to raise file transfer speed up.

ASAP

RJIL

B-2

SAMSUNG Electronics Co., Ltd. Confidential and Proprietary TBD  Need to add status managing ©system between FMS and ActixOne.

Accuver, RJIL

133

Copyright and Confidentiality Copyright © 2014 © SAMSUNG Electronics Co., Ltd. SAMSUNG Electronics reserves the right to make changes to the specifications of the products detailed in this document at any time without notice and obligation to notify any person of such changes. Information in this document is proprietary to SAMSUNG Electronics Co., Ltd. No information contained here may be copied, translated, transcribed or duplicated by any form without the prior written consent of SAMSUNG Electronics.

CLOT KPI Analysis Guideline(RJIL)_v1.0

Recommend Documents