Update: November 2021
This article is out of date; there are no longer WebPageTest Server AMIs, so you need to install WPT on a base OS. The updated article is here WebPageTest Private Instance: 2021 Edition

If you have any interest in website performance optimisation, then you have undoubtedly heard of WebPageTest. Being able to test your websites from all over the world, on every major browser, on different operating systems, and even on physical mobile devices, is the greatest ever addition to a web performance engineer’s toolbox.

The sheer scale of WebPageTest, with test agents literally global (even in China!), of course means that queues for the popular locations can get quite long – not great when you’re in the middle of a performance debug session and need answers FAST!

Also since these test agents query your website from the public internet they won’t be able to hit internal systems – for example pre-production or QA, or even just a corporate intranet that isn’t accessible outside of a certain network.

In this article I’ll show you how to set up your very own private instance of WebPageTest in Amazon AWS with autoscaling test agents to keep costs down

Once you have this you can optionally extend the setup, for example:

• Always-on test agent(s) – get your test result faster
• Your own test agent within the China firewall – get realistic test results from the perspective of a user behind the Great Firewall
• Automation and scripting – schedule regular tests
• Reporting and visualisation – graph your tests to find trends

We’ll get on to those topics later, but first up let’s focus on building a solid WebPageTest foundation!

Getting familiar with AWS

You don’t need to be an AWS guru to get this all running. AWS can be confusing, however this setup only requires a few clicks throughout the AWS console. Make sure you have your own AWS account ready to go.

There are 2 or 3 main areas we will need to use in AWS for setting up our private WebPageTest instance:

1. A user with permissions to create new EC2 instances which the WebPageTest server uses to spin up test agents, and optionally S3 permissions to archive the tests.
2. A place to archive test results, freeing up space on the WebPageTest server itself (optional, but highly recommended – and cheap).
3. A VM instance on which to host your WebPageTest server which orchestrates the whole process.

Let’s start off with –

1. Create a WebPageTest User

IAM is the AWS Identity and Access Management (aka IAM – "I am"…) area; we need to create a new programmatic user for the WebPageTest server to run under.

Log in to your AWS console and head over to IAM by finding it in the Services menu:

From there, click Add user, enter a name, and select Programmatic Access:

Now we need to Set permissions. Select Attach Existing Policies and search for and select "AmazonEC2FullAccess" (so it can start test agents)…

… and search for and select "AmazonS3FullAccess" (to archive tests into S3).

The tests can easily fill up even a large EC2 volume quickly, so archiving to S3 is strongly recommended – S3 is super cheap, EBS volumes are not.

Archiving tests to S3 won’t change how WebPageTest looks and feels; it will fetch the zipped test from S3 on demand incredibly quickly.

Copy the AWS Access Key ID and Secret Access Key for this new user somewhere as we’ll need them in a moment.

2. Configure Test Storage in S3

EBS volumes (the hard drive on a EC2 instance) will fill up quickly and, although expanding them is possible, it isn’t easy and it can’t be reversed. S3 is the super cheap, almost limitless, alternative.

Head over to the S3 area of your AWS console – again, you can find this under the Services menu:

Give your bucket a unique name – your bucket name cannot already exist in the same region, even in another user’s AWS account – and set the region where you’ll be creating your WebPageTest server:

With that done, now you’re ready to create the server!

3. Setting up the WebPageTest server on EC2

Now let’s create the actual WebPageTest server itself; this is quite a long set of steps so get that coffee ready!

Head over to the EC2 area within the AWS console and select Launch Instance:

Search for "webpagetest" and select Community AMIs:

Select the top webpagetest-server result.

Choose instance size: t2.micro is ok to start with as you can always scale up if necessary, and t2.micro is currently free:

This is an important step: instead of launching the server and logging in to edit settings, we can actually define the settings in "user data" which is passed in to the instance at start-up.

If you prefer, you can connect to your WebPageTest instance after launching and configure these settings directly in the file /var /www /webpagetest /www /settings/ settings.ini; you can copy the sample and edit it.

Tap on Configure instance details, then Advanced, and paste in user data similar to this, filling in the blanks:

# User to start/stop agents and save tests to s3
ec2_key=<the IAM user access key ID>
ec2_secret=<the IAM user secret access key>

# Set the API key to use when programmatically enqueuing tests
# Can't think of one? check out http://new-guid.com/
api_key=<choose a super secret api key>

# show user timing marks in waterfalls - very handy
waterfall_show_user_timing=1

# better images for screenshots - you're using S3, right?
# so you have the storage!
iq=75
pngss=1

# archiving to s3
archive_s3_server=s3.amazonaws.com
archive_s3_key=<the IAM user key>
archive_s3_secret=<the IAM user secret>
archive_s3_bucket=<the WPT archive bucket>

# number of days to keep tests locally before archiving
archive_days=1

# run archive script hourly automatically as agents poll for work
cron_archive=1

The full list of options that can be set are over on the github repo for WebPageTest, as the settings.ini.sample file; as mentioned earlier, you could skip the user data step and set these options in a ini file to be created in /var /www /webpagetest /www /settings /settings.ini

Now we need to make sure we can access the server over port 80; for this you need to select Configure security group, then add rule and choose HTTP:

Alright! Ready to rock! Let’s hit Review and Launch and after you’ve selected or created a keypair (used to log in to the instance), and after it’s finished initialising we’ll be given a URL, where it says Public DNS:

Head over to that URL and you should see the familiar WebPageTest homepage:

Try it out – pop in a URL, select a location, and submit the test. If you then check the EC2 area of your AWS account, after a moment you’ll notice a new instance starting up called "WebPagetest Agent":

A new test agent can take a few minutes to actually connect and start testing – sometimes up to 10 minutes – but once connected it’ll automatically pick up the enqueued test and run it:

That was easy, right? The server will update itself from the WebPageTest github repo regularly, as will the test agents. Your tests will be automatically archived to, and retrieved from, S3.

Your private WebPageTest foundation has been laid! We will build on this over a few more articles.

BONUS) Scaling Up and Staying There

By default, only one test agent per 100 tests will be created per region. One of the reasons I wanted a private WebPageTest instance was my impatience at queueing up to find out how slow my sites are. If you can throw money at the problem, then change this in your user data:

# This will create one new test agent for every 5 tests queued
# *up to the location max*:
EC2.ScaleFactor=5

# The default max per location is 1, so you need to override it per
# location to enable scale out:
EC2.us-east-1.max=10
EC2.us-west-1.max=20
EC2.eu-west-1.max=15
...
# etc

If there are no more tests queued which that agent can pick up for an hour (or however long you want; an hour is the default) then it will be terminated. This means that you’ll have to wait for the agent to be recreated next time, but it also means you’re not paying for an EC2 instance that you’re not using; these agents should be c4.large, so they’re not free tier eligible.

To avoid this you can alter the user data (you’ll need to stop the WebPageTest server instance before you can edit it) to add in a line for each location you want to keep an agent always available, e.g.:

 EC2.us-east-1.min=1
 EC2.us-west-1.min=1
 EC2.eu-west-1.min=1

It will now scale down to a minimum of one test agent for each of the specified regions.

Debugging

Wondering why your test isn’t starting? Getting impatient? Check /install to make sure you have green everywhere, except at the bottom where no test agents will exist (since they’re spun up on demand):

You can also check /getTesters.php?f=html to see what is connected:

The test agents are only created on demand, so let’s ensure there’s a test actually registered at /getLocations.php?f=html:

You can also log in and check what’s happening on the server:

ssh -i "<your keypair>.pem" ubuntu@<the url of your instance>

e.g.

ssh -i "webpagetest.pem" [email protected]

Now let’s check nginx for requests to getwork.php – this is the URL that test agents poll to pick up the next test in their queue:

tail -f /var/log/nginx/access.log | grep getwork

e.g.

ubuntu@blah:~$ tail -f /var/log/nginx/access.log | grep getwork
127.0.0.1 - - [25/Apr/2019:20:42:01 +0000] "GET /work/getwork.php HTTP/1.1" 200 5 "-" "Wget/1.15 (linux-gnu)"
127.0.0.1 - - [25/Apr/2019:20:43:01 +0000] "GET /work/getwork.php HTTP/1.1" 200 5 "-" "Wget/1.15 (linux-gnu)"
127.0.0.1 - - [25/Apr/2019:20:44:01 +0000] "GET /work/getwork.php HTTP/1.1" 200 5 "-" "Wget/1.15 (linux-gnu)"
127.0.0.1 - - [25/Apr/2019:20:45:02 +0000] "GET /work/getwork.php HTTP/1.1" 200 5 "-" "Wget/1.15 (linux-gnu)"

hmm… not a lot happening there. That’s the server pinging itself for some reason; you can tell this since the requests are from "127.0.0.1". Usually after a few more minutes you’ll see:

172.31.20.24 - - [25/Apr/2019:20:45:37 +0000] "GET /work/getwork.php?f=json&shards=1&reboot=1&location=eu-west-1&pc=EC2AMAZ-CO5OM1I&key=4d446cb0d60d76dced79ffa39cf3c1e953db594b&ec2=i-0323a543b58e9ba00&ec2zone=eu-west-1a&version=190221.200223&screenwidth=1920&screenheight=1200&freedisk=7.464&upminutes=9 HTTP/1.1" 200 282 "-" "python-requests/2.21.0"
127.0.0.1 - - [25/Apr/2019:20:46:01 +0000] "GET /work/getwork.php HTTP/1.1" 200 5 "-" "Wget/1.15 (linux-gnu)"
172.31.20.24 - - [25/Apr/2019:20:46:39 +0000] "GET /work/getwork.php?f=json&shards=1&reboot=1&location=eu-west-1_IE11&pc=EC2AMAZ-CO5OM1I&key=4d446cb0d60d76dced79ffa39cf3c1e953db594b&ec2=i-0323a543b58e9ba00&ec2zone=eu-west-1a&version=190221.200223&screenwidth=1920&screenheight=1200&freedisk=7.404&upminutes=10 HTTP/1.1" 200 31 "-" "python-requests/2.21.0"
172.31.20.24 - - [25/Apr/2019:20:46:39 +0000] "GET /work/getwork.php?f=json&shards=1&reboot=1&location=eu-west-1&pc=EC2AMAZ-CO5OM1I&key=4d446cb0d60d76dced79ffa39cf3c1e953db594b&ec2=i-0323a543b58e9ba00&ec2zone=eu-west-1a&version=190221.200223&screenwidth=1920&screenheight=1200&freedisk=7.404&upminutes=10 HTTP/1.1" 200 284 "-" "python-requests/2.21.0"

Notice the timestamps show there was nothing going on for over 3 minutes after I had started to get impatient; AWS EC2 test agents can take a while to wake up and connect, so bear this in mind.

Interesting point: notice the querystring parameters in the request; the test agent informs the server a lot about itself, even including available disc space.

Things to check:

• User Data – did you set the correct IAM details for EC2? If not, then the WebPageTest server will not be able to spin up the agents
• IAM – did you give EC2 permissions to the IAM user you created? If not, same issue as above.
• WPT logs – you can check the logs for issues in the WebPageTest logs, which can be found in /var/www/webpagetest/www/log/ (not /logs/, as this is where the submitted tests are logged, not errors).
• nginx logs – you can check if the agents are able to connect to your the WebPageTest server at all.

Summary

Hopefully you followed along and successfully set up your own private WebPageTest instance, and can now queue up all the tests you like!

If you have issues, then hit me up on twitter or head over to the WebPageTest forums – seriously, given how many times the same question must be asked in there, the gang are exceptionally patient and helpful.

Good luck!