Configure and use Charles proxy as powerful debugging tool for playback issues

INTRODUCTION

Charles Proxy is a great and very powerful tool that mainly allows to:

  • Capture, filter and inspect HTTP(s) traffic → Very useful when you want to inspect what happens behind the scenes at network/http level in a given website, mobile app, API call, etc…
  • Modify HTTP(s) traffic → Very useful to force specific scenarios that are required in order to trigger an issue, and also to test how certain modifications affect occurrence of an issue during the troubleshooting/debugging phase. Some of these modifications (request/response headers, bodies, etc…) can also be accomplished via Bitmovin’s Player Network API, but Charles offers some advantages over it, such as no need for coding and deeper options than the Network API.

When opening a Support ticket to Bitmovin team for an issue you or your viewers are experiencing, our Support Engineers might ask you for a Charles capture if they don’t manage to reproduce the same issue on their end.

SETUP

1 - Download it from here and install it on your laptop

2 - Open it and navigate to Help → SSL Proxying. Then click on any of the following, depending on your needs:

  • Install Charles Root Certificate → For usage on your laptop
  • Install Charles Root Certificate in iOS Simulators → For usage with XCode simulators
  • Install Charles Root Certificate in Java VM’s → For usage within any Java environment
  • Install Charles Root Certificate on a Mobile Device or Remote Browser → For usage with real mobile devices (instructions will show up on screen upon clicking)

3 - Go to Proxy → Proxy Settings → Proxies → HTTP Proxy, and fill port 8888 with support for HTTP/2 and transparent proxying enabled

4 - Go to Proxy → SSL Proxying Settings → SSL Proxying. There, enable SSL Proxying and add the following location so it will apply to all domains using the default HTTPS port(443) → *:443

5 - Restart your laptop for changes to apply. Upon re-opening Charles, click on the jar symbol on the top-right menu of your laptop and ensure “Recording” and “macOS Proxy” are enabled. If your setup is correct, you’ll start seeing any HTTP(s) traffic happening on your laptop there, as shown in below screenshot:

Note: When loading any HTTPS traffic on your client (ie: browser) for 1st time with Charles enabled, you’ll need to manually trust the certificate: SSL certificate warnings appear in my browser or other client • Charles Web Debugging Proxy

6 - In order to capture traffic from iOS/Android simulators and/or real mobile devices, you’ll need to first know the local IP where Charles is running on your laptop, which can be obtained by going to Help → Local IP Address. Then, with your mobile plugged via USB into your laptop, go to the Wifi settings on the mobile device → Proxy → switch from None to Manual → select port 8888 and fill the IP with the Local one from your laptop obtained before. Charles on laptop will show a pop-up asking you whether to allow incoming traffic from an external device. Allow it, and you should see any HTTP(s) traffic from the mobile logged on your Charles desktop. More info here: Using Charles from an iPhone • Charles Web Debugging Proxy

(7 - If you’re running your HTTP(s) traffic on localhost, bear this in mind: Localhost traffic doesn’t appear in Charles • Charles Web Debugging Proxy )

HOW TO READ .CHLS LOGS

A typical Charles log session (.chls file), for example capturing a streaming session, will look like this:

providing lots of relevant and useful information:

  • On the left side of the screen we’ll see all the HTTP endpoints/servers being called, such as: website’s host, player .js and .css scripts, analytics, licensing, stream manifests/chunklists/segments, etc…
  • On the right side of the screen there are two main tabs to look at:
  • Overview : For a given endpoint selected on the left, provides complete information about the request, such as full url being called, http status returned, TLS protocol/version being used, client/remote IP addresses, latencies between request/response, etc…

  • Content : For a given endpoint selected on the left, provides information about headers&body sent on the request and headers&body received on the response

TOOLS AND USAGE

Charles has, literally, an almost infinite list of use cases where it becomes a great and useful tool. Here are the most relevant for Bitmovers and anybody involved in video streaming:

CLIENT SIDE LOGS : For any issue involving HTTP(s) traffic (ie: playback of http(s) streams, REST API’s, etc…) that somebody can reproduce you can’t, a .chls session captured when reproducing the issue provides great visibility of what’s happening on the affected side, which can easily pinpoint where is the issue coming from

THROTTLING : This tool allows simulation of specific network conditions, such as low bandwidth, high round-trip-time, network un-reliability, network un-stability, which becomes very handy when we want to test ABR (adaptive bitrate switching) behaviour, measure latencies under poorer network conditions, or simulate a given environment (ie: 3G vs broadband). It can be configured and enabled/disabled under Proxy → Throttle Settings. Bandwidth Throttle / Bandwidth Simulator • Charles Web Debugging Proxy

  • BREAKPOINTS : This tool allows interception of requests/responses in order examine and edit them and decide whether to allow them or to block them. A practical use could be to force timeouts on certain requests by leaving the request pending without approval/denial until it times out, or to execute HTTP requests sequentially in a controlled way where we can edit them on the fly. It can be configured and enabled/disabled under Proxy → Breakpoint Settings. Breakpoints Tool • Charles Web Debugging Proxy

NO CACHING : Prevents local caching of any resource. Ideally this should be enabled for all your tests in order to ensure you always get fresh requests/responses and nothing pre-cached locally. It can be configured and enabled/disabled under Tools → No Caching → Enable No Caching. No Caching Tool • Charles Web Debugging Proxy

BLOCK COOKIES : Blocks usage of cookies, very useful for testing purposes when cookies are involved, or for simulating your browser incognito mode (along with no-caching). It can be configured and enabled/disabled under Tools → Block Cookies → Enable Block Cookies. Block Cookies Tool • Charles Web Debugging Proxy

REWRITE : One of the most powerful tools Charles has to offer. It literally allows you to rewrite any HTTP(s) request/response as per your needs, which becomes extremely valuable for simulating multiple scenarios during troubleshooting. It can be configured and enabled/disabled under Tools → Rewrite → Enable Rewrite. Rewrite Tool • Charles Web Debugging Proxy . A few examples would be:

  • forcing a certain HTTP code on a certain resource (ie: force a 403 forbidden on a given .ts segment)
  • adding/removing/modifying certain headers (ie: simulate different userAgent, add DRM headers, add CORS, etc…)
  • rewriting playlists/segment paths (ie: when requesting path1/segment_1.ts, grab path2/segment_2.ts instead)
  • adding/removing/modifying URL query params (ie: append ?token=mytoken to the URL)
  • modify bodies (ie: add #EXT-X-ENDLIST to an HLS Live playlist to make it VOD, modify timestamps or content in WebVTT subtitles body, etc…)
  • and so many more….

REPEAT : Useful for repeating a given request without the need of refreshing browser or creating/running any script for that purpose. There’s an advanced option too for defining number of iterations and concurrency, in a similar fashion to load testing. It’s located under Tools → Repeat. Repeat Tool • Charles Web Debugging Proxy

BLOCK-LIST/ALLOW-LIST : It blocks/allows certain domains, giving you two options for the blockage: closing the connection or enforcing a 403-forbidden. It can be configured and enabled/disabled under Tools → Block List (or Allow List) → Enable Block/Allow List. Block List Tool • Charles Web Debugging Proxy

DNS SPOOFING : It manipulates DNS records in order to map a certain domain to a different IP (the same effect as modifying your Mac Hosts file). This becomes very useful for testing CDN staging configurations before pushing to production.

  • LOCAL/REMOTE MAPPING : It maps resources either locally or remotely. For example, if you want to load a player instance hosted online but you’d like to use locally hosted .CSS files. It can be configured and enabled/disabled under Tools → Map Remote/Local → Enable Map Remote/Local. Map Remote Tool • Charles Web Debugging Proxy

Hope you’ll find this useful! And please feel free to contribute asking questions and sharing with others how you use it for your specific use cases :slight_smile:

1 Like

Another powerful use of Charles Proxy is to save a stream locally, which can be useful for debugging purposes when streams expire, or to capture a part of a live stream. How to proceed :


1- Turn Charles proxy on and make sure it is recording requests


2- Make sure caching is disabled, either via the browser network tab option, or via the No Caching Charles tool explained above


3- Load the stream in the player (eg. https://cdn.bitmovin.com/content/assets/art-of-motion-dash-hls-progressive/mpds/f08e80da-bf1d-4e3d-8899-f0f6155f6efa.mpd)


4- Make sure the player is downloading everything you would like to capture. For instance play all the renditions you would like to capture in order for them to be recorded and use all the subtitle tracks, thumbnails etc. that you need.


5-Back to Charles, once everything is recorded, make sure to select all the relevant files that belong to the stream (manifest, segments, VTT etc…). Filtering only the files that are served from a certain hostname (eg cdn.bitmovin.com) might be useful


6- Right-click, select Save All… and choose your destination


7- The whole arborescence should be recreated locally, which can be checked via ls -lR terminal command


8- If you don’t record all the renditions, make sure to remove them from the Manifest, not to encounter 403s/404s. We could for example save only 4 segments of the audio and HD renditions, so adjusting the mediaPresentationDuration accordingly and removing the other renditions makes the stream fully useable for the player


9- Now you should be ready to create a simple web page, host it at the root of the directory you just saved, add the manifest relative URL as source and play

    var source = {
        dash: 'mpds/f08e80da-bf1d-4e3d-8899-f0f6155f6efa.mpd'
    };