Use the percentiles to detect and make sense of problematic quality of experience metrics

Knowledge
1

Knowing that your median video streaming user is watching a 3mbps rendition and has a video startup time of 2.23s is great a great indication that the playback experience is smooth for most of your users.

However, this doesn’t give us any indication of what the experience looks like on both sides of the average : for instance if 30% of your users have an extremely good (low) startup time and are able to watch an extremely high rendition in average, while a smaller portion of your users suffer from an unacceptable quality of experience, the average might still be satisfying, but it hides the disparities and there is potentially still a lot of users who might not experience an adequate quality of experience.

Also, when getting a user report about a high rebuffering time or high startup time, using the percentiles can help realize whether their experience is more likely an isolated problem, or if many other users have the same experience.

A solution is to look at percentiles: The 90th percentile STARTUP_TIME, for instance, is a value that separates the 90% of the users with the lowest startup time and the 10% with the highest one.

Example :

queryBuilder
  .percentile("VIDEO_STARTUPTIME",'90')
  .licenseKey('YOUR_ANALYTICS_KEY')
  .between(fromDate, toDate)
  .filter('STARTUPTIME', 'GT', 0)
  .groupBy("OPERATINGSYSTEM")
  .query();

Result :

Operatingsystem Video startuptime (ms)
Android 16213.811620427201
iPadOS 7131.540056818183
iOS 4312.540567818183
Tizen 3602.76545678
macOS 2601.76545678

In this scenario, on Android, the 10% of users with the highest startup time are experiencing 16s startup time, which is an suspicious value. that is worth investigating further (for instance, what is special about the Android Player configuration, stream, DRM etc, that would cause the startup time to be so much higher than on other mobile platforms).

A follow up query can for instance look at which player, player version, CDN, countries to orient the investigations.

queryBuilder
  .count("IMPRESSION_ID")
  .licenseKey('YOUR_ANALYTICS_KEY_HERE')
  .between(fromDate, toDate)
  .filter('STARTUPTIME', 'GT', 16213) // only look at values over the 90th percentile
  .filter("OPERATINGSYSTEM", "EQ", "Android")
  .groupBy("CDN_PROVIDER")
  .groupBy("PLAYER_VERSION")
  .groupBy("COUNTRY")
  .query();

Example

CDN provider Player version Country Impression id / Play id
AKAMAI bitmovin-3.50.0 FR 108
AKAMAI bitmovin-2.67.0 US 16549
AKAMAI bitmovin-8.140.0 BR 203

This very simplistic result, can lead us to think that the older version of the Bitmovin player might be the problem, or a geographical factor. To realise the real proportion, one would need to rerun the above request after removing the STARTUPTIME filter, to compare the results with the total amount of users and get a percentage