KPI Definitions in TEMS Products
NT13-25359, NT13-25359, ver 2.0, 9/25/2013
© Ascom. All rights reserved. TEMS is a trademark of Ascom. All other trademarks are the property of their respective holders. NT13-25359, ver 2.0, 9/25/2013
Contents 1
Introduction
1
2
General Aspects of KPI Data Collection and Computation
1
3
2.1
KPI Reporting...................................................................... .............................................. 1
2.2
ETSI Compliance .............................................................................................. ................ 1
2.3
Complete vs. Timed Measurements ................................................................................. 1
2.4
IP Capture Options ....................................... ............................................................... ..... 2
KPIs by Service 3.1
3.2
3.3
KPIs for FTP Download ...................................................... .............................................. 2 3.1.1
Timed Measurements .................................................................................................... 3
3.1.2
SFTP (Secure Shell FTP) .............................................................................................. 3
KPIs for FTP Upload ........................................................... .............................................. 3 3.2.1
Timed Measurements .................................................................................................... 3
3.2.2
SFTP (Secure Shell FTP) .............................................................................................. 4
KPIs for HTTP Get/Download ........................................................ ................................... 4 3.3.1
3.4
4
Timed Measurements .................................................................................................... 5
KPIs for Ping ............................................................ ......................................................... 5 3.5.1
3.6
Timed Measurements .................................................................................................... 4
KPIs for HTTP Post/Upload ........................................................... ................................... 4 3.4.1
3.5
2
Timed Measurements .................................................................................................... 5
KPIs for CS Voice ............................................................... .............................................. 5
Notes on Other Services: Email, Iperf, Streaming, CS Fallback
6
4.1
Email ............................................................. ............................................................... ..... 6
4.2
Iperf................................................................ .............................................................. ..... 7
4.3
4.4
NT13-25359, ver 2.0, 9/25/2013
4.2.1
TCP ............................................................................................................................... 7
4.2.2
UDP .............................................................................................................................. 7
Streaming over HTTP ......................................................... .............................................. 8 4.3.1
Streaming Player Download .......................................................................................... 8
4.3.2
Start of Streaming Video Replay...................... ..................... ...................... ................... 9
4.3.3
Streaming Video Session .............................................................................................. 9
CS Fallback.................................................................................... ................................. 13
1
Introduction
This document describes the implementation of ETSI circuit-switched and packetswitched service KPIs, as well as some closely related non-ETSI KPIs, in TEMS products. The document is valid for the following TEMS product versions:
TEMS Automatic 10.1 and later
TEMS Investigation 16.0 and later
TEMS Symphony 7.4 and later
2
General Aspects of KPI Data Collection and Computation
2.1
KPI Reporting
For PS data services, current TEMS products report KPI events, each of which contains either
an instance of the KPI itself (such as a session duration or throughput figure), or an instance of an occurrence whose probability is measured by the KPI (such as a session failure).
These in turn underlie the aggregation of KPI statistics in TEMS Discovery or some other post-processing tool. The “probability” KPIs obviously need to be based on a fair number of sessions for a good estimate of the probability in question to be obtained. The data-collecting TEMS products do not in themselves output values of these latter KPIs; rather, they need to be calculated during post-processing. TEMS Symphony users should note that earlier versions of TEMS Symphony (version 6.x and older) did not report data in this fashion, but only trigger points on which to base the KPI computation: for example, session start and end times. Regarding CS voice, see section 3.6.
2.2
ETSI Compliance
In TEMS products, timestamps for KPI computation are captured as close to t he source as possible: that is, from the packet capture driver within the operating system kernel. Calculations adhere to the specification ETSI TS 102 250-2 V2.2.1 (2011-04) unless otherwise stated. This specification is hereafter referred to as “ETSI 102 250-2”.
2.3
Complete vs. Timed Measurements
The standardized KPIs have been defined with complete transactions in mind. Transactions that are aborted after a fixed duration specified by the TEMS product user (timed measurements) have not been considered. In case of timed measurements, a best-effort solution is provided. Where triggers do not fully match ETSI 102 250-2, they have been aligned as closely as possible to their ETSI counterparts in a manner NT13-25359, ver 2.0, 9/25/2013
1
compliant with ETSI TS 102 678 V1.2.1, except that the following item stated in section 4.1 of that specification has not been implemented: The connection should be checked to be still alive at the end of the transfer period Δt d. The reception of any data packet sent by the server on the data connection after the end of the transfer period is a valid indicator that the data connection is still alive. An appropriate timeout of e.g. 3 to 5 multiples of typical RTT in the measured network might be used while waiting for the desired packet.
“Session Time” KPIs, of course, make sense only for sessions that have been allowed to run to completion. For details, see the various subsections of chapter 3.
2.4
IP Capture Options
The default IP data capture that is done by TEMS products during PS data service testing is designed to obtain sufficient input for KPIs. No special activity needs to be used in Service Control scripts for this purpose. It is however also possible to conduct a more thoroughgoing IP sniffing by means of the “Start IP Sniffing” script activity. This activity has a “Filter” parameter whos e possible settings are given here for convenience. Filter: Type of filtering to apply to IP packets. o
o
o
3
Optimized Performance: Capture of IP packets is reduced to the minimum needed to compute KPIs. (Packets are filtered with respect to IP address, protocol [TCP/UDP], and port.) Not supported for on-device measurement. Headers Only: The first 54 bytes of every IP packet are captured. None: No filtering of IP packets. Note: This means that all packets belonging to any service used by the PC will be captured; packets are filtered with respect to IP address only. Be aware that full packet capturing can cause crashes on high speed networks where sustained high throughput is achieved.
KPIs by Service
This chapter deals with TEMS product KPIs for each service.
3.1
KPIs for FTP Download ETSI-to-TEMS Mapping Table ETSI KPI “FTP {Download|Upload}”
Sect.
TEMS KPI Event
Service Non-Accessibility
6.1.1
FTP Download Service Not Accessible
Setup Time
6.1.2
FTP Download Setup Time
IP-Service Access Failure Ratio
6.1.3
FTP Download IP Service Access Failure
IP-Service Setup Time
6.1.4
FTP Download IP Service Setup Time
Session Time
6.1.6
FTP Download Data Transfer Time
NT13-25359, ver 2.0, 9/25/2013
2
ETSI-to-TEMS Mapping Table ETSI KPI “FTP {Download|Upload}”
Sect.
TEMS KPI Event
Mean Data Rate
6.1.7
FTP Download Mean Data Rate
Data Transfer Cut-off Ratio
6.1.8
FTP Download Data Transfer Cutoff
Note: “Method A” and “Method B” values are differentiated wherever applicable. See ETSI 102 250-2 section 4.2.
3.1.1
Timed Measurements
For timed measurements, the end trigger is defined as “Last incoming packet containing content”. This trigger is valid for both completed and aborted transfers. Since this definition conforms to the ETSI standard, timed FTP downloads will have ETSI compliant KPIs.
3.1.2
SFTP (Secure Shell FTP)
If SFTP is used for the file transfer rather than FTP proper, no KPIs are obtained. ETSI 102 250-2 does not define any KPIs for SFTP, and some metrics would not be possible to obtain owing to the SSH encryption.
3.2
KPIs for FTP Upload ETSI-to-TEMS Mapping Table ETSI KPI “FTP {Download|Upload}”
Sect.
TEMS KPI Event
Service Non-Accessibility
6.1.1
FTP Upload Service Not Accessible
Setup Time
6.1.2
FTP Upload Setup Time
IP-Service Access Failure Ratio
6.1.3
FTP Upload IP Service Access Failure
IP-Service Setup Time
6.1.4
FTP Upload IP Service Setup Time
Session Time
6.1.6
FTP Upload Data Transfer Time
Mean Data Rate
6.1.7
FTP Upload Mean Data Rate
Data Transfer Cut-off Ratio
6.1.8
FTP Upload Data Transfer Cutoff
Note: “Method A” and “Method B” values are differentiated wherever applicable.
3.2.1
Timed Measurements
For timed FTP uploads, the end trigger is defined as “Reception of ACK for last packet containing content”. This trigger is not ETSI compliant: ETSI 102 250-2 (section 6.1.6.3) defines the trigger as “Stop: Reception of the [FIN, ACK] for the last data packet containing content”. That is, the “FIN” indicator has been removed from the
NT13-25359, ver 2.0, 9/25/2013
3
condition. This is because the FIN indicator is sent during connection teardown, a procedure which does not necessarily take place during a timed measurement.
3.2.2
SFTP (Secure Shell FTP)
If SFTP is used, no KPIs are obtained; compare section 3.1.2.
3.3
KPIs for HTTP Get/Download
This section deals primarily with HTTP downloads conducted with the TEMS browser. The same KPI values apply also for Internet Explorer browser usage. ETSI-to-TEMS Mapping Table ETSI KPI “HTTP”
Sect.
TEMS KPI Event
Service Non-Accessibility
6.8.1
HTTP Service Not Accessible
Setup Time
6.8.2
HTTP Setup Time
IP-Service Access Failure Ratio
6.8.3
HTTP IP Service Access Failure
IP-Service Setup Time
6.8.4
HTTP IP Service Setup Time
Session Time
6.8.6
HTTP Data Transfer Time
Mean Data Rate
6.8.7
HTTP Mean Data Rate
Data Transfer Cut-off Ratio
6.8.8
HTTP Data Transfer Cutoff
Note: “Method A” and “Method B” values are differentiated wherever applicable.
3.3.1
Timed Measurements
There is no difference between timed and complete measurements; the same triggers can be used in both scenarios.
3.4
KPIs for HTTP Post/Upload
ETSI does not define KPIs for HTTP Post/Upload. The TEMS product KPIs have been designed to closely resemble the ETSI KPIs for other services, such as HTTP Get and FTP (compare sections 3.1 –3.3). ETSI-to-TEMS Mapping Table ETSI KPI “HTTP” (Get)
Sect.
TEMS KPI Event
Service Non-Accessibility
6.8.1
HTTP Service Not Accessible
Setup Time
6.8.2
HTTP Setup Time
IP-Service Access Failure Ratio
6.8.3
HTTP Post IP Service Access Failure
IP-Service Setup Time
6.8.4
HTTP Post IP Service Setup Time
Session Time
6.8.6
HTTP Post Data Transfer Time
Mean Data Rate
6.8.7
HTTP Post Mean Data Rate
NT13-25359, ver 2.0, 9/25/2013
4
ETSI-to-TEMS Mapping Table ETSI KPI “HTTP” (Get)
Sect.
Data Transfer Cut-off Ratio
6.8.8
TEMS KPI Event HTTP Post Data Transfer Cutoff
Trigger points for the HTTP Post session are as follows:
Start: First packet containing content.
End: Last incoming ACK packet for content.
No “Method A” vs. “Method B” distinction exists for this service.
3.4.1
Timed Measurements
Since ETSI does not define any KPIs for HTTP Post, the session end trigger was defined in such a way as to work for both timed and complete measurements.
3.5
KPIs for Ping ETSI-to-TEMS Mapping Table ETSI KPI “Ping”
Sect.
Round trip time
TEMS KPI Event
6.3.1
Ping Roundtrip Time
Trigger points for Ping round trip:
Start: ICMP Echo Request sent.
End: ICMP Echo Reply received by the sender.
The round-trip time is that calculated by the Windows API function IcmpSendEcho2Ex. This is the lowest-latency way available in Windows to send Ping requests and capture replies. Reference: msdn.microsoft.com/en-us/library/windows/desktop/aa366050(v=vs.85).aspx The difference between the round-trip time measured by Windows and that derived from the network trace is in the range ±0.5 ms. Since the round-trip time is reported in milliseconds in the KPI, the difference after rounding is not noticeable. TEMS products also define a timeout error message “Ping Timeout” signifying a failed Ping request.
3.5.1
Timed Measurements
Not applicable.
3.6
KPIs for CS Voice ETSI-to-TEMS Mapping Table ETSI KPI: “Telephony”
Service Non-Accessibility
NT13-25359, ver 2.0, 9/25/2013
Sect. 6.6.1
TEMS Data (see also below) Blocked Call event
5
ETSI-to-TEMS Mapping Table ETSI KPI: “Telephony”
Sect.
TEMS Data (see also below)
Setup Time
6.6.2
Call Setup event with call setup time
Speech Quality on Sample Basis
6.6.4
PESQ/POLQA information elements
Cut-off Call Ratio
6.6.5
Dropped Call event
For CS voice, no special KPI events are generated. However, call setup time is carried as extra information by the Call Setup event, and speech quality scores are reported in information elements belonging to the “Media Quality” category. Supported speech quality measures are PESQ, POLQA NB and POLQA SWB. TEMS products also have an additional, non-ETSI speech quality measure called Speech Quality Index (SQI). CS voice KPIs are computed for mobile-originated (MO) as well as mobile-terminated (MT) calls.
4
Notes on Other Services: Email, Iperf, Streaming, CS Fallback
For the following services, TEMS products do not have any KPIs defined. However, they do exhibit certain information elements and/or events with similar content. Average throughputs and all percentages apply to the period following the latest network connect (triggering of Network Connect event) and are reset at network disconnect (when the Network Disconnect activity has completed).
4.1
Email IE Name
Range/Unit
Description
Email Receive Average Throughput
0 ... 350000 kbit/s
Average throughput for receiving of email.
Email Receive Transfer Time
0 ... 172800 s (= 48 h)
Elapsed time for current email receive session. (Not an average.)
Email Send Average Throughput
0 ... 350000 kbit/s
Average throughput for sending of email.
Email Send Transfer Time
0 ... 172800 s
Elapsed time for current email send session. (Not an average.)
NT13-25359, ver 2.0, 9/25/2013
6
4.2
Iperf
Iperf testing is conducted with the Network Bandwidth activity in Service Control scripts. The testing can be done over either TCP or UDP.
4.2.1
TCP IE Name
Range/Unit
Description
TCP Download Average Throughput (kbit/s)
0 ... 350000 kbit/s
Average throughput for TCP download.
TCP Download Transfer Time
0 ... 172800 s
Elapsed time for current TCP download session. (Not an average.)
TCP Packet Loss
0 ... 100 %
Retransmissions (in percent) on the downlink over the TCP protocol during the last second. (Not an average.) Note: For this percentage to be correct, IP sniffing must not be set to “Optimized Performance”. See section 2.4.
TCP Upload Average Throughput (kbit/s)
0 ... 350000 kbit/s
Average throughput for TCP upload.
TCP Upload Transfer Time
0 ... 172800 s
Elapsed time for current TCP upload session. (Not an average.)
4.2.2
UDP IE Name
Range/Unit
Description
UDP Download Average Throughput (kbit/s)
0 ... 350000 kbit/s
Average throughput for UDP download.
UDP Download Jitter
Text
UDP jitter: the mean deviation (in ms) of the difference in packet spacing at the receiver compared to the sender, for a pair of packets.
UDP Download Packet Loss
0 ... 100 %
NT13-25359, ver 2.0, 9/25/2013
Retransmissions (in percent) on the downlink over the UDP protocol. This measurement is obtained once every second. (Not an average.)
7
IE Name
Range/Unit
Description
UDP Download Transfer Time
0 ... 172800 s
Elapsed time for current UDP download session. (Not an average.)
UDP Upload Average Throughput (kbit/s)
0 ... 350000 kbit/s
Average throughput for UDP upload.
UDP Upload Transfer Time
0 ... 172800 s
Elapsed time for current UDP upload session. (Not an average.)
4.3
Streaming over HTTP
4.3.1
Streaming Player Download
These information elements relate to the task of downloading the streaming player. Please note that not all of the “Streaming Player” elements are obtained if the streaming server is an HTTPS server, since the relevant packets are then encrypted. IE Name
Range/Unit
Description
Streaming Player Service IP Access Failure Ratio
0 ... 100 %
Percentage of attempts to download the streaming player that failed before the first packet of the download was received (IP access failure).
Streaming Player Service IP Access Time
0 ... 172800 s
Streaming Player Download Data Transfer Failure Ratio
0 ... 100 %
Streaming Player Download Data Transfer Time
0 ... 172800 s
Streaming Player Session Failure Ratio
0 ... 100 %
Percentage of attempts to download the streaming player that failed at some stage (IP access or data transfer).
Streaming Player Session Time
0 ... 172800 s
Total session time for streaming player download (from user request to receipt of last packet).
NT13-25359, ver 2.0, 9/25/2013
Time taken for IP access when downloading the streaming player (from user request to receipt of first packet). Percentage of attempts to download the streaming player that failed at the data transfer stage (after receipt of first packet and before receipt of last packet). Time taken to download the streaming player (from receipt of first packet to receipt of last packet).
8
4.3.2
Start of Streaming Video Replay IE Name
Range/Unit
Description
Streaming Service Access Time
0 ... 172800 s
Streaming Service Non Accessibility
0 ... 100 %
Percentage of attempts to request streaming that failed (never receive response from streaming host).
Streaming Reproduction Start Delay
0 ... 100 %
The duration of the delay after streaming request was responded and before the video replay was started.
Streaming Reproduction Start Failure
Text
The duration of a service access from requesting the stream at the portal until the reception of the first stream data packet.
Indicate that there is error occurs after streaming request was responded and before the video replay was started.
Streaming Video Play Start Failure Ratio
0 ... 100 %
Percentage of attempts to start streaming video replay that failed (playing never started).
Streaming Video Play Start Time
0 ... 172800 s
Time from user video replay request until playing started.
Streaming Reproduction Cut off Ratio
0 ... 100 %
4.3.3
Percentage of failures throughout the video playing session.
Streaming Video Session
Overall IE Name
Range/Unit
Streaming Video Session Failure Ratio
0 ... 100 %
Streaming Video Session Time
0 ... 172800 s
Description Percentage of streaming video replay session that failed at some point before the last video packet was received. Total time of streaming video replay session from user request to receipt of last video packet.
VQmon Metrics: Video and Audio Perceptual Quality IE Name
NT13-25359, ver 2.0, 9/25/2013
Range/Unit
Description
9
IE Name
Range/Unit
Description
Streaming Absolute MOS-V
0 ... 5 MOS
Average absolute video MOS for the stream.
Streaming MOS-A
0 ... 5 MOS
Average audio MOS for the stream.
Streaming MOS-AV
0 ... 5 MOS
Average audio –video MOS for the stream.
Streaming Relative MOS-V
0 ... 5 MOS
Average relative video MOS for the stream. The difference between relative and absolute video MOS is that the relative metric does not consider frame resolution, thus constituting a score which is relative to the ideal for the current video format.
These average scores are also reported in a “ Streaming Quality MOS” event at the end of a streaming session. Bandwidth IE Name
Range/Unit
Description
Streaming Average Audio Received Bandwidth
0 ... 350000 kbit/s
Average audio bandwidth excluding transport packet overhead, FEC, and retransmissions.
Streaming Average Video Received Bandwidth
0 ... 350000 kbit/s
Average bandwidth for video transport packets, excluding IP overhead, FEC, and retransmissions.
Packet Loss, Packet Errors IE Name
Range/Unit
Description
Streaming Average Audio Effective Packet Loss Rate
0 ... 100 %
Percentage of audio packets lost or discarded (minus those corrected).
Streaming Average Video Effective Packet Loss Rate
0 ... 100 %
Percentage of video packets lost or discarded (minus those corrected).
NT13-25359, ver 2.0, 9/25/2013
10
IE Name
Range/Unit
Description
Streaming Audio Packet Corrected Rate
0 ... 100 %
Percentage of audio packets received with errors, but corrected using error correction algorithms.
Streaming Audio Packet Discarded Rate
0 ... 100 %
Percentage of audio packets discarded by the receiving jitter buffer.
Streaming Audio Packet Out Of Sequence Rate
0 ... 100 %
Percentage of audio packets identified as out-of-sequence, possibly due to high jitter levels or the use of load-sharing devices.
Streaming Video Packet Corrected Rate
0 ... 100 %
Percentage of video packets received with errors, but corrected using error correction algorithms.
Streaming Video Packet Discarded Rate
0 ... 100 %
Percentage of video packets discarded by the receiving jitter buffer.
Streaming Video Packet Out Of Sequence Rate
0 ... 100 %
Percentage of video packets identified as out-of-sequence, possibly due to high jitter levels or the use of load-sharing devices.
Buffering and Rebuffering IE Name
Range/Unit
Streaming Video Interruption Count
Text
Description Number of times the video stream replay was interrupted for rebuffering.
Streaming Video Interruption Duration
0 ... 172800 s
Total duration of video stream replay interruptions for reasons of rebuffering.
Streaming Playout Buffer Delta Rate
0 ... 350000 kbit/s
Equal to Streaming Playout Buffer Empty Rate minus Streaming Playout Buffer Fill Rate.
Streaming Playout Buffer Empty Proportion
0 ... 100 %
Streaming Playout Buffer Empty Rate
0 ... 350000 kbit/s
Rate at which data is emptied from the streaming player’s playout buffer.
Streaming Playout Buffer Fill Rate
0 ... 350000 kbit/s
Rate at which the streaming player’s playout buffer is filled with new data.
NT13-25359, ver 2.0, 9/25/2013
Percentage of the total session time during which the playout buffer was empty.
11
IE Name
Range/Unit
Streaming Playout Buffer Rebuffering Proportion
0 ... 100 %
NT13-25359, ver 2.0, 9/25/2013
Description Percentage of the total session time that was spent rebuffering.
12
4.4
CS Fallback
The following events report on various aspects on CS fallback procedures: Event Name CSFB Blocked Call
Description A CS fallback call was blocked. This can happen in several ways:
The Extended Service Request timed out without any response from network, or the network responded with Service Reject. In this case the CS fallback procedure never reaches the RAT change stage. RAT change to UTRAN/GERAN failed, as indicated by the event EUTRAN RRC Connection Release Redirected Failure. After successful RAT change to UTRAN/GERAN, the CS call setup failed (CS Blocked Call event generated).
Extra information:
CSFB Call Setup
Call direction (MO/MT)
Block type
A CS fallback call was set up. Triggered by the Layer 3 message Alerting. Extra information:
Call direction (MO/MT)
Target technology (WCDMA/GSM)
EUTRAN Reselection Time After CSFB Call
Call setup time (measured from CSFB Call Attempt, i.e. the first call attempt) User setup time (measured from CSFB Call Initiation, thus more accurately reflecting the userperceived setup time. Obtained only for MO calls in scripts. If no CSFB Call Initiation event was generated, the user setup time cannot be computed.)
This event carries performance information on the switch back to LTE after hangup of a CS fallback call. Extra information:
NT13-25359, ver 2.0, 9/25/2013
Idle to LTE time: Time in seconds from entering idle mode (in UMTS) to reception of System Information Block on LTE. SIB 19 to LTE time: Time in seconds from reception of System Information Block Type 19 (UMTS) to reception of System Information Block on LTE.
13
Event Name PS Data Interruption Time Due To CSFB
Description This event reports the IP interruption time during RAT change to UTRAN/GERAN due to initiation of a CS fallback call. Extra information: Interruption time in ms. Measured from last received IP packet in EUTRAN to first received IP packet in UTRAN/GERAN.
NT13-25359, ver 2.0, 9/25/2013
14
