Transition from KPI to KQI -- Huawei Servi Service ce Quality Quality Improveme Improvement nt Solution Solution
HUAWEI TECHNOLOGIES CO., LTD.
Content
What is Service Quality Improvement
Case Study
Technology Trend
Cafe
Working
Office
Watching TV
Subway
User
Device Video telephony
Email
Learning
Home
Driving
Multimedia Apps
Video/Voice Messaging
Social Networking
SMS
Mobile TV
PTT Location Based Services
IMS
User culture drives Technology Trend HUAWEI TECHNOLOGIES CO., LTD.
Huawei Confidential
IP Transformation
Page 3
End-user Experience
CamGSM’s Challenges in 3G Era
To understand end user Demand
Behavior
-Understand end user interest & service distributions
Maintaining network
Performance during rapid growth Performance
-Maintain VIP urban 3G network During 300 NodeB launch
Total Value of Ownership
C omplexity
V isibility
Increasing network C omplexity after new services introduction
Services
on offered services
-Maintain 2G&3G networks simultaneously
-KPI ≠ User Experience (QoE), visibility to manage user experience, E2E, All-IP etc
Defend and grow top line Services
-Support new service to launch & grow HUAWEI TECHNOLOGIES CO., LTD.
Huawei Confidential
V isibility of user experience
Page 4
What is Service Quality Improvement about?
Service Quality Improvement
Is about End-To-End
Visualization of real end-user experience
Improve application service quality
Root cause analysis of service quality issues
Network KPI RRC Setup Success Rate Call Drop Rate
User Experience Call Setup Success rate
Slow to web
No service Poor voice quality
…
…
≠ Customer Hotline
HUAWEI TECHNOLOGIES CO., LTD.
Huawei Confidential
Page 5
From KPI to Service Quality Improvement As is
To be
Good KPI, but Poor User Experience
Build SQI System, Improve End-User Experience
CTO’s Experience: • Full signal bar, but difficult to call out • Poor voice quality, One-way audio • Connecting to internet, but can’t browse internet
√
√
Cell KPI Value RAN Access Success >99% Rate Call Drop Rate <0.3% Cell Average KPI is Good Sample DT KPI Value Coverage >99% Voice Quality (MOS) 3.6 Sample DT KPI is Good
X
Level 1 QoE: Quality of Experience, ranking from 1-5;
Level 2 KQI: Key Quality Index, measure end user experiences;
Level 3 Measurement KQI: Measurable, Manageable, Assessable, Improvable KQIs;
QoE WEB page access satisfaction
KQI
Measurement KQI
WEB page access success ratio
E2E access success ratio
WEB page access delay
E2E page access delay
Good KPI ≠ Good Service Quality
HUAWEI TECHNOLOGIES CO., LTD.
DT access success rate
DT page Access delay
Huawei Confidential
Page 6
From Network layer to User layer As is
To be
VIP Care ---Quick Response to User Complaint
Indicator Care ---Monitoring Network KPI Call Setup Success Ratio-NCR[%] 99.20
99.25
99.30
99.30
Week 40
Week 41
Week 42
KPI Normal Week 39
Call Set-up Success Rate
10:00 am
10:30 am
11:00 am
11:10 am
At same time: Call information Cause types:
…..
Call record
Access/release
Access failure
CS service
Handover failure Call drop
10:00 am
18:00 pm
Several Key users’ worse experience will not indicate from KPI, then complaint exists… HUAWEI TECHNOLOGIES CO., LTD.
Subscriber
Normal
PS service Signaling
Web refresh delay Other exceptions
Huawei Confidential
Page 7
Handover
Huawei SQI Capability - Tools & Platform Accessibility Service Quality KQI System
E2E access completion rate
Retainability
E2E access duration
Integrity -One-way Audio -Echo -Time delay
Speech Quality
Web offline ratio …
User perception of the entire network
-Chopping -Noise -Crosstalk
VIP&VAP user perception
Monitoring (tool)
Assessment & Delimiting & Identification (Service)
Tool ＋Service M2000 (traffic/log)
PRS
U2520
Nastar
MR
Nastar
Resource-based refined tracking
End-to-End Network
E2E Active/passive joint testing
Assessment tool set
User-based refined tracking
IP BTS Abis
HUAWEI TECHNOLOGIES CO., LTD.
BSC
A
UMG
MSC
Huawei Confidential
MSC
UMG
A
BSC
Abis
Page 8
BTS
Equipment capability
Huawei SQI Capability – User Experience Research Team Special Experts
Biography
Dr. Dong
The User Experience Principle at Huawei. Ph.D from Purdue University in the USA. More than 15 years of research and Design experience in top companies, such as IBM and eBay. He is now in charge of the user research and consumer programs in the Department of User Centered Design.
Dr. Wan
Ph.D. degree in psychology from the Zhejiang University, Zhejiang, China. More than 6-year research experiences on human perception and mental processing. Expert of QoE modeling- establishing quantitative relation between KQIs and QoE.
Dr. Cui
Ph.D in Cognitive Psychology from the Zhejiang University, China. Senior user researcher at UCD center, Huawei R&D. Expert of QoE research User Experience Design and Human-Computer Interaction.
Tina
Zhang Tingting received her master of Arts (Sociology) in Nanjing University, accumulated wealth of experience in user research. She organized and participated in lots of large sample social research and focused on quantitative analysis.
From Technologies to Research in End User Experience & Human Perception HUAWEI TECHNOLOGIES CO., LTD.
Huawei Confidential
Page 9
Q1: How to Establish SQI System? Example Project Kick-off
Tool & Service Setup
KQI Data Collection
KQI Monitoring & QoE Survey
KQI Analysis & Optimization
QoE – KQI Modeling
Project End
Step 5： KQI Monitoring
Step5： QoE Survey
1.Precision KQIs for use of each specific service 2.Time Span: 2~4 weeks, 7*24 hours 3.Sample Size About 20,000 4.Sample Structure User groups of different user behavior, and from different regions.
1.Method Phone survey 2.Questionnaire Satisfaction of the quality of Service, including QoE and sub-QoE. 3.Performer Call center (Trained by QoE Experts) 4.Sample Size: 900~1,000 users from those monitored . o en a s : g annoy e en users, ow o con ro
5
5
Objective
Subjective
Data from KQIs monitoring
KQI monitored
QoE score from phone survey
QoE surveyed
6
Mapping KQI to QoE Step6： QoE-KQI Modeling Method
HUAWEI TECHNOLOGIES CO., LTD.
Regression analysis+ QoE = f(KQI1, KQI2, KQI3 ……)
Huawei Confidential
Page 10
Q2: How to Measure QoE? Mapping Category
KQI name
KQI value
QoE Weight score
E2E access completion rate (Passive Test) Accessibility Front page access delay
Example QOE score
QoE grade
User perception
≥ 4.5
Excellent
Very satisfied
4~4.5
Very good
Satisfied
3.5~4
Good
Neutral
3~3.5
Fair
Very unsatisfied
<3
Poor
Extremely unsatisfied
QoE Retainability
Web browsing data transfer cut-off ratio
4 * 40% + 3 3* *10% + 4*10% + 3*25% + 4*15%
3.65 (Good) Web page refresh success ratio Integrity
• Initialized with default weight value & benchmark for KQIs , yet need to get agreement with CamGSM first. Web page refresh delay
HUAWEI TECHNOLOGIES CO., LTD.
Huawei Confidential
Page 11
Content
What is Service Quality Assurance
Case Study
Data Service Quality Improvement – China Mobile Customer Needs Measurable – Establish WAP/MMS/FTP/Stream data service measurement system, which is tailored for China Mobile Hangzhou; Visible – Quality monitoring of WAP/MMS/Stream data services; Manageable – Establish methodology to identify key factors and improve KQI; Improvement 3G Quality – Improve 3G network quality. (E.g. WEB front page access delay/Unstable MMS downloading speed/Frequent 3G complaints etc)
Hangzhou, China Mobile
Solutions Establishment - China Mobile Hangzhou SQI system through partnership
Analysis & Optimization – - Data Service Quality Analysis - User complaints Analysis - 2G/3G Hotspot Analysis - User Behavior Analysis - P2P Service Analysis & Restriction - PS Core Network Parameter Audit & Resource Evaluation
HUAWEI TECHNOLOGIES CO., LTD.
Delivery Period: Aug.2010 – Dec.2010 Organization: An End-to-End project team including: Wireless / PS / SQI PDT team / Nastar Tool Team Tools: - Nastar - Wireless tool set - Probe /CDS - U2520
Huawei Confidential
Page 13
Data Service Quality Improvement – China Mobile •
TOP region’s WEB Delay comparison Page Refresh Delay – Before & After (s)
Front Page Access Delay – Before & After (s)
Refresh delay
Connection access delay
BF AF
BF
•
BF
AF
BF
AF
BF
AF
BF
AF
BF
AF
BF
AF
AF
TOP region’s WEB access failure comparison TOP Cell Access Failures – Before & After
Access failure rate
BF
AF
BF
HUAWEI TECHNOLOGIES CO., LTD.
AF
BF
AF
BF
Values: - Established customized SQI solution for Hangzhou, China Mobile. - New O&M methodology for long-term benefits; - WEB Data Service Quality was remarkably Improved; - Joint Benefits Analysis is undertaken.
AF
Huawei Confidential
Page 14
Thank you www.huawei.com
