Automating WebPageTest via the WebPageTest API

webpagetest robots

WebPageTest is incredible. It allows us to visit a web page, enter a few values and then produce performance results from any destination around the world. Best of all, you can do this in many different possible browser configurations; even on many different real devices.

If you’re doing this a lot, then using that simple web form can become the bottleneck to rapidly iterating on your web performance improvements.

In this article I’ll show you how to easily execute your web performance tests in a simple, repeatable, automated way using the WebPageTest API.

WebPageTest API

1) Executing a test using runtest.php

WebPageTest has its own API which allows you to submit tests via GET or POST to runtest.php; you can specify more options than are available via the usual WebPageTest web interface, giving your greater control over your performance testing.

The options are seriously extensive; from the url and the number of test runs, through to bandwidth throttling and choosing which metric to use for calculating the median run.

To get the full list of available options, check out the documentation

For example:

Test www.microsoft.com (url) 10 times (runs), first view only (fvonly), capture a chrome devtools timeline (timeline), and redirect to the results page:

http://www.webpagetest.org/runtest.php
?url=www.microsoft.com
&runs=10
&fvonly=1
&timeline=1

Since this is via an API then perhaps you don’t want to be redirected to the results page, and instead you’d like to get some test metadata that you can use to check back on the test.

To do this, pass a format (f) parameter set to either xml or json, e.g.:

http://www.webpagetest.org/runtest.php
?url=www.microsoft.com
&runs=10
&fvonly=1
&timeline=1
&f=json

Important note: neither of these example URLs will actually work, since you need an API key in order to use the public instance of WebPageTest; you can get one for free from http://www.webpagetest.org/getkey.php, but it’s limited to 200 page loads each day.

Just append your API key to the URLs above, e.g.:

http://www.webpagetest.org/runtest.php
?url=www.microsoft.com
&runs=10
&fvonly=1
&timeline=1
&f=json
&k=<YOUR API KEY>

This will now execute the tests on the public WebPageTest instance.

If you’ve got a private WebPageTest instance then you don’t have the usage limit, and will just need to change the URL to match your private instance’s URL. It’s under your control as to whether you need to specify an API key too.

Don’t have a private WebPageTest instance? Let me help you out with that!

Using f=xml will give a response like this:

<response>
    <statusCode>200</statusCode>
    <statusText>Ok</statusText>
    <data>
        <testId>190616_YG_95</testId>
        <ownerKey>afde544cb0dd68a22d103b4059659690d9dd22f8</ownerKey>
        <xmlUrl>http://www.webpagetest.org/xmlResult/190616_YG_95/</xmlUrl>
        <userUrl>http://www.webpagetest.org/result/190616_YG_95/</userUrl>
        <summaryCSV>http://www.webpagetest.org/result/190616_YG_95/page_data.csv</summaryCSV>
        <detailCSV>http://www.webpagetest.org/result/190616_YG_95/requests.csv</detailCSV>
        <jsonUrl>http://www.webpagetest.org/jsonResult.php?test=190616_YG_95</jsonUrl>
    </data>
</response>

f=json will look like more this:

{
    "statusCode": 200,
    "statusText": "Ok",
    "data": {
        "testId": "190616_F1_92",
        "ownerKey": "2d588dff122bda2a04845d9b6d5c5695a7872d1c",
        "jsonUrl": "http://www.webpagetest.org/jsonResult.php?test=190616_F1_92",
        "xmlUrl": "http://www.webpagetest.org/xmlResult/190616_F1_92/",
        "userUrl": "http://www.webpagetest.org/result/190616_F1_92/",
        "summaryCSV": "http://www.webpagetest.org/result/190616_F1_92/page_data.csv",
        "detailCSV": "http://www.webpagetest.org/result/190616_F1_92/requests.csv"
    }
}

A good start; now what? What do those URLs do and what do the properties mean?

Well, I’m glad you asked – now we can use the testStatus.php endpoint to check how the test is coming along.

2) Getting the status of a test with testStatus.php

The testStatus.php endpoint will let us know if the test has finished yet:

http://www.webpagetest.org/testStatus.php
?f=<format>
&test=<test id>

# XML
http://www.webpagetest.org/testStatus.php
?f=xml
&test=190616_YG_95

# JSON
http://www.webpagetest.org/testStatus.php
?f=json
&test=190616_YG_95

The various responses are shown below:

Test Not Started

<response>
  <statusCode>101</statusCode>
  <statusText>Waiting behind 27 other tests...</statusText>
  <data>
    <statusText>Waiting behind 27 other tests...</statusText>
    <id>190616_WH_46</id>
    <testId>190616_WH_46</testId>
    <location>Dulles</location>
  </data>
</response>
{
    "statusCode": 101,
    "statusText": "Waiting behind 27 other tests...",
    "data": {
        "statusCode": 101,
        "statusText": "Waiting behind 27 other tests...",
        "id": "190616_YG_95",
        "testInfo": {
            "url": "http://www.microsoft.com",
            "runs": 9,
            "fvonly": 1,
            "web10": 0,
            "ignoreSSL": 0,
            "priority": 5,
            "location": "Dulles",
            "browser": "Chrome",
            "connectivity": "Cable",
            "bwIn": 5000,
            "bwOut": 1000,
            "latency": 28,
            "plr": "0",
            "tcpdump": 0,
            "timeline": 1,
            "trace": 0,
            "bodies": 0,
            "netlog": 0,
            "standards": 0,
            "noscript": 0,
            "pngss": 0,
            "iq": 0,
            "keepua": 0,
            "mobile": 0,
            "scripted": 0
        },
        "testId": "190616_YG_95",
        "runs": 9,
        "fvonly": 1,
        "remote": false,
        "testsExpected": 9,
        "location": "Dulles",
        "behindCount": 27
    }
}

Test Started – No results yet

<response>
  <statusCode>100</statusCode>
  <statusText>Test Started 1 second ago</statusText>
  <data>
    <statusText>Test Started 1 second ago</statusText>
    <id>190616_YG_95</id>
    <testId>190616_YG_95</testId>
    <location>Dulles</location>
    <startTime>06/16/19 20:42:42</startTime>
  </data>
</response>
{
    "statusCode": 100,
    "statusText": "Test Started 10 seconds ago",
    "data": {
        "statusCode": 100,
        "statusText": "Test Started 10 seconds ago",
        "id": "190616_YG_95",
        "testInfo": {
            "url": "http://www.microsoft.com",
            "runs": 9,
            "fvonly": 1,
            "web10": 0,
            "ignoreSSL": 0,
            "priority": 0,
            "location": "Dulles",
            "browser": "Chrome",
            "connectivity": "Cable",
            "bwIn": 5000,
            "bwOut": 1000,
            "latency": 28,
            "plr": "0",
            "tcpdump": 0,
            "timeline": 1,
            "trace": 0,
            "bodies": 0,
            "netlog": 0,
            "standards": 0,
            "noscript": 0,
            "pngss": 0,
            "iq": 0,
            "keepua": 0,
            "mobile": 0,
            "scripted": 0
        },
        "testId": "190616_YG_95",
        "runs": 9,
        "fvonly": 1,
        "remote": false,
        "testsExpected": 9,
        "location": "Dulles",
        "startTime": "06/16/19 20:35:01",
        "elapsed": 10,
        "fvRunsCompleted": 0,
        "rvRunsCompleted": 0,
        "testsCompleted": 0
    }
}

Test Started – Partially Complete

<response>
  <statusCode>100</statusCode>
  <statusText>Completed 7 of 9 tests</statusText>
  <data>
    <statusText>Completed 7 of 9 tests</statusText>
    <id>190616_YG_95</id>
    <testId>190616_YG_95</testId>
    <location>Dulles</location>
    <startTime>06/16/19 20:35:01</startTime>
  </data>
</response>
{
    "statusCode": 100,
    "statusText": "Completed 7 of 9 tests",
    "data": {
        "statusCode": 100,
        "statusText": "Completed 7 of 9 tests",
        "id": "190616_YG_95",
        "testInfo": {
            "url": "http://www.microsoft.com",
            "runs": 9,
            "fvonly": 1,
            "web10": 0,
            "ignoreSSL": 0,
            "priority": 5,
            "location": "Dulles",
            "browser": "Chrome",
            "connectivity": "Cable",
            "bwIn": 5000,
            "bwOut": 1000,
            "latency": 28,
            "plr": "0",
            "tcpdump": 0,
            "timeline": 1,
            "trace": 0,
            "bodies": 0,
            "netlog": 0,
            "standards": 0,
            "noscript": 0,
            "pngss": 0,
            "iq": 0,
            "keepua": 0,
            "mobile": 0,
            "scripted": 0
        },
        "testId": "190616_YG_95",
        "runs": 9,
        "fvonly": 1,
        "remote": false,
        "testsExpected": 9,
        "location": "Dulles",
        "startTime": "06/16/19 20:42:42",
        "elapsed": 27,
        "fvRunsCompleted": 0,
        "rvRunsCompleted": 0,
        "testsCompleted": 7
    }
}

Test Finished

<response>
  <statusCode>200</statusCode>
  <statusText>Test Complete</statusText>
  <data>
    <statusText>Test Complete</statusText>
    <id>190616_YG_95</id>
    <testId>190616_YG_95</testId>
    <location>Dulles</location>
    <startTime>06/16/19 20:29:04</startTime>
    <completeTime>06/16/19 20:29:21</completeTime>
  </data>
</response>
{
    "statusCode": 200,
    "statusText": "Test Complete",
    "data": {
        "statusCode": 200,
        "statusText": "Test Complete",
        "id": "190616_YG_95",
        "testInfo": {
            "url": "http://www.microsoft.com",
            "runs": 9,
            "fvonly": 1,
            "web10": 0,
            "ignoreSSL": 0,
            "priority": 5,
            "location": "Dulles",
            "browser": "Chrome",
            "connectivity": "Cable",
            "bwIn": 5000,
            "bwOut": 1000,
            "latency": 28,
            "plr": "0",
            "tcpdump": 0,
            "timeline": 1,
            "trace": 0,
            "bodies": 0,
            "netlog": 0,
            "standards": 0,
            "noscript": 0,
            "pngss": 0,
            "iq": 0,
            "keepua": 0,
            "mobile": 0,
            "scripted": 0
        },
        "testId": "190616_YG_95",
        "runs": 9,
        "fvonly": 1,
        "remote": false,
        "testsExpected": 9,
        "location": "Dulles",
        "startTime": "06/16/19 20:29:04",
        "elapsed": 17,
        "completeTime": "06/16/19 20:29:21",
        "testsCompleted": 9,
        "fvRunsCompleted": 0,
        "rvRunsCompleted": 0
    }
}

Alright, so we can check up on the test to see if it’s done yet. Isn’t it interesting that you get more data back in the json response than the xml one?

But what about those URLs in the original request to runtest.php? What do they do?

3) Get Test Results using jsonResult / xmlResult

Look back at the response from runtest.php near the start and you’ll notice a few URLs listed:

{
    "statusCode": 200,
    "statusText": "Ok",
    "data": {
        "testId": "190616_F1_92",
        "ownerKey": "2d588dff122bda2a04845d9b6d5c5695a7872d1c",
        "jsonUrl": "http://www.webpagetest.org/jsonResult.php?test=190616_F1_92",
        "xmlUrl": "http://www.webpagetest.org/xmlResult/190616_F1_92/",
        "userUrl": "http://www.webpagetest.org/result/190616_F1_92/",
        "summaryCSV": "http://www.webpagetest.org/result/190616_F1_92/page_data.csv",
        "detailCSV": "http://www.webpagetest.org/result/190616_F1_92/requests.csv"
    }
}

The userUrl will just take you to the web page that refreshes to show the status of the test – the GUI.

The CSV URLs will give you the test data in CSV format; summary (page_data.csv) gives the overall per run split by first and repeat views, whereas detail (requests.csv) contains the huge amount of information available for every single request made within every test run.

Hitting the jsonUrl or xmlUrl will give you similar results to testStatus.php:

{
  "data": {
    "statusCode": 100,
    "statusText": "Test Started 4 seconds ago",
    "id": "190616_F1_92",
    "testInfo": {
      "url": "http://www.microsoft.com",
      "runs": 9,
      "fvonly": 0,
      "web10": 0,
      "ignoreSSL": 0,
      "priority": 5,
      "location": "Dulles",
      "browser": "Chrome",
      "connectivity": "Cable",
      "bwIn": 5000,
      "bwOut": 1000,
      "latency": 28,
      "plr": "0",
      "tcpdump": 0,
      "timeline": 1,
      "trace": 0,
      "bodies": 0,
      "netlog": 0,
      "standards": 0,
      "noscript": 0,
      "pngss": 0,
      "iq": 0,
      "keepua": 0,
      "mobile": 0,
      "scripted": 0
    },
    "testId": "190616_F1_92",
    "runs": 9,
    "fvonly": 0,
    "remote": false,
    "testsExpected": 9,
    "location": "Dulles",
    "startTime": "06/16/19 20:50:20",
    "elapsed": 4,
    "fvRunsCompleted": 0,
    "rvRunsCompleted": 0,
    "testsCompleted": 0
  },
  "statusCode": 100,
  "statusText": "Test Started 4 seconds ago"
}

The big – and I mean big – difference between testStatus.php and jsonResult.php (or xmlResult) is what happens when the test is finished:

{
  "data": {
    "id": "190616_Y7_c8",
    "url": "http://www.microsoft.com",
    "summary": "https://www.webpagetest.org/results.php?test=190616_Y7_c8",
    "testUrl": "http://www.microsoft.com",
    "location": "Dulles:Chrome",
    "from": "Dulles, VA - <b>Chrome</b> - <b>Cable</b>",
    "connectivity": "Cable",
    "bwDown": 5000,
    "bwUp": 1000,
    "latency": 28,
    "plr": "0",
    "mobile": 0,
    "completed": 1560718259,
    "tester": "VM02-07-172.16.20.224",
    "testRuns": 9,
    "fvonly": false,
    "successfulFVRuns": 9,
    "successfulRVRuns": 9,
    "average": {…},
    "standardDeviation": {…},
    "median": {…},
    "runs": {…}
  },
  "statusCode": 200,
  "statusText": "Test Complete",
  "webPagetestVersion2": "19.04"
}

That data section is huge; every detail about every request made within every test run is tucked within the average, standardDeviation, median, and runs sections. This is the payload that contains every single thing about your test, including masses of information that you don’t see in the usual web interface.

4) Pingback

So far we’ve been able to run a test, check on the status of that test, and then retrieve the full results for the test. Each of these has been a separate API call, which isn’t great for automation; we’d have to build in polling for test results for every test we queue up. Not at all impossible, but it doesn’t feel like the best implementation.

The absolute killer feature in WebPageTest for performance test automation is the ability to specify a pingback URL when calling runtest.php, e.g.:

http://www.webpagetest.org/runtest.php
?pingback=<your own endpoint>
&url=www.microsoft.com
&runs=10
&fvonly=1
&timeline=1
&f=json
&k=<YOUR API KEY>

With this in place you will queue up the test as before, but this time it will call your pingback URL when the test has completed and the results are available. You can now fetch and process the results in a totally separate process!

PingBack Demo

I’m going to use the incredible ngrok to demonstrate what happens with the pingback. ngrok is an incredibly powerful tool that provides a secure tunnel to localhost. This means you can expose ports on your dev PC on various web ports temporarily, and have a packet inspector in a web interface. If you don’t already have it, definitely check it out; I find it invaluable for learning what comes back from a web hook..

Download ngrok and fire it up using a command like ngrok http 3000 (expose a local process running on port 3000 over HTTP – we don’t have a process running, but just want to capture the request that gets made by the webpagetest-api process) to get something like this:

ngrok by @inconshreveable                           (Ctrl+C to quit)

Session Status      online
Session Expires     7 hours, 59 minutes
Version             2.3.30
Region              United States (us)
Web Interface       http://127.0.0.1:4040
Forwarding          http://18e2a650.ngrok.io -> http://localhost:3000
Forwarding          https://18e2a650.ngrok.io -> http://localhost:3000

That means we now have a temporary end point on http(s)://18e2a650.ngrok.io – let’s plug this into a runTest.php request:

http://www.webpagetest.org/runtest.php
?pingback=http://18e2a650.ngrok.io
&url=www.microsoft.com
&k=<YOUR API KEY>

Run that and after a few minutes and you’ll see the following appear in your ngrok window:

HTTP Requests
-------------

GET /                          502 Bad Gateway

This looks bad, but it’s just because we don’t actually have a process running on port 3000, hence a "Bad Gateway". Let’s head over to the ngrok web interface on http://127.0.0.1:4040 to get a bit more information about that request:

ngrok web interface with wpt pingback result

We can see that a test ID is passed back as a querystring param "id"!

We can now use this to programmatically retrieve the details about the test in another process using the WebPageTest API‘s jsonResult.php or xmlResult endpoints, passing in the value of id as the testId parameter.

Summary

A review of what we can now do:

  • We can queue up a website performance test, made up of a combination of detailed configuration options, using a particular WebPageTest API url.
  • We can check how that test is coming along via the API.
  • We can get the resulting full details of that test via the API.
  • We can send the test ID to an endpoint of our choice once it’s finished via the API.
  • We can use that test ID to retrieve results in a separate process via the API.

Trouleshooting at scale

If, like me, you end up queueing up a few thousand tests at once, be aware of the following:

The AWS AMI that the WebPageTest server runs as PHP on can and will run out of resources, especially if the runtest requests all come in at once. A method that works for me is to randomly jitter the runTest requests within a window of ten seconds or so, which gives the server breathing room.

I hope that was useful; let me know how you get on and shout if you have any questions.

Leave a Reply

Your email address will not be published. Required fields are marked *