Automating WebPageTest via the WebPageTest API

webpagetest robots

WebPageTest is incredible. It allows us to visit a web page, enter a few values and then produce performance results from any destination around the world. Best of all, you can do this in many different possible browser configurations; even on many different real devices.

If you’re doing this a lot, then using that simple web form can become the bottleneck to rapidly iterating on your web performance improvements.

In this article I’ll show you how to easily execute your web performance tests in a simple, repeatable, automated way using the WebPageTest API.

Continue reading

Smart TV 101 : Part #4 – Creating packages without the SDK

I’m committing to doing 12 months of “101″s; posts and projects themed at beginning something new (or reasonably new) to me. January was all about node development awesomeness. February is all about Smart TV apps.

To IDE or not to IDE

I’ve mentioned how I’m not the greatest fan of Eclipse, so working on a development method that doesn’t rely on it intrigued me.

Given that all the Smart TV apps consist of are pretty standard web pages, then surely it’s possible to do this without the integrated IDE and webserver?

Starting at the end and working backwards:

Web server

The SDK bundles Apache for serving the apps. I don’t really have any problem with Apache; it’s currently the most commonly used web server  on the interwebs, free, and stable. I just don’t see why I’d need it to serve up an XML page and some zip files!

Looking into the contents of the widgetlist.xml from previous posts we can see that it’s just listing the contents of a Widget subdirectory. That should be easy enough to manage ourselves. I’ve decided to dive back into nodejs for a lightweight alternative.

The code I’ve used is the same as that from most of January. The one that I’ve changed is requestHandlers.js to serve the listing xml and the zip files:

requestHandlers.js[js]var fs = require(‘fs’);

// build the full xml file
function widgetlist(response, notused, request) {
console.log("Request handler ‘widgetlist’ was called");
var packageDir = "packages";

BuildPackageXml(__dirname, packageDir, request, function(packageXml){
var content = ‘<?xml version="1.0" encoding="UTF-8" standalone="no"?>\r\n<rsp stat=\’ok\’>\r\n<list>\r\n’ + packageXml + ‘\r\n</list>\r\n</rsp>’;
var headers = {
"Content-Type": "application/xml",
"Content-Length": content.length
};
response.writeHead(200, headers);
response.end(content);
});
}

// build the xml for each package, getting the stats for each zip
function BuildPackageXml(directory, packageDir, request, callback){
var filesData =”;
var host = request.headers.host;

fs.readdir(‘packages’, function(err, files){
files.forEach(function(file){
console.log(‘found: ‘+ file);
var stats = fs.statSync(directory + ‘\\’ + packageDir + ‘\\’ + file)
filesData += ‘<widget id="’ + file + ‘">\r\n’ +
‘<title>’ + file + ‘</title>\r\n’ +
‘<compression size="’ + stats.size + ‘" type="zip" />\r\n’ +
‘<description>’ + file + ‘</description>\r\n’ +
‘<download>http://’ + host + ‘/Widget/’ + file + ‘</download>\r\n’+
‘</widget>’;
});
callback(filesData);
});
}

// serve the zip file
function widget(response, path) {
console.log("Request handler ‘widget’ was called for " + path);

var packageDir = "packages";
var packagepath = __dirname + ‘\\’ + packageDir + ‘\\’ + path.split(‘/’)[2];
var widget = fs.readFile(packagepath, ‘binary’, function(err, data){

var headers = {
"Content-Type": "application/zip",
"Content-Length": data.length // to avoid the "chunked data" response
};

response.writeHead(200, headers);
response.end(data,’binary’);
});
}

exports.widgetlist = widgetlist;
exports.widget = widget;[/js]

This will give us the same XML and also serve the Zip files; they don’t even need to be in the Widgets subdirectory since we’ve implemented basic routing here.

widgetlist_xml

 

Problems encountered

  • my general inability to comprehend node’s inherently async implementation caused me much confusion throughout this development
  • xml generation node modules over-complicates what is a very basic file; hence why I went for inline
  • getting the content-length header is important if you want to avoid the "content chunked" response in your http request for the zip file; the smart tv isn’t so smart in this scenario.

 

Generating the packages

Moving backwards another stage we get to the generation of the zip files themselves. This should be easy, but again I over-complicated things by trying to implement js-zip using node-zip to recursively traverse the directory containing my work files. Async recursive archive creation was a bad idea for a Sunday evening so I should have instead opted for firing a command line call to the OS’s built-in archive-er.

Luckily my code had at least abstracted this functionality out so I could easily replace it with another implementation. The code in my git repo uses this implementation, which appears to create the archive, but that file is apparently corrupt/invalid; patches/forks/pull requests welcome!

pack.js
[js]var fs = require(‘fs’);

// main function – loop through the root package dir and create one archive per sub directory
// (assumption is that each sub dir contains one entire project)
function createPackages(rootDirectory)
{
fs.readdir(rootDirectory, function(err, files)
{
files.forEach(function(item){
if (item.indexOf(‘.’) != 0)
{
var file = rootDirectory + ‘\\’ + item;
fs.stat(file, function(err,stats){
if (stats.isDirectory()){
console.log(‘** PACKAGE **\n’ + item);
createPackage(item, file, rootDirectory);
}
});
}
});
});
}

// create each zipped archive
function createPackage(packageName, path, rootPath)
{
console.log(‘* PACKING ‘ + packageName);
var zip = new require(‘node-zip’)();

var archive = zipMe(path, zip);
console.log(‘** ARCHIVING’)
var content = archive.generate({base64:false,compression:’DEFLATE’});

fs.writeFileSync(rootPath + ‘\\’ + packageName + ‘.zip’, content);
console.log(‘saved as ‘ + rootPath + ‘\\’ + packageName + ‘.zip’);
}

// recursive function to either add a file to the current archive or recurse into the sub directory
function zipMe(currentDirectory, zip)
{
console.log(‘looking at: ‘ + currentDirectory);
var dir = zip.folder(currentDirectory);

var files = fs.readdirSync(currentDirectory)

files.forEach(function(item){
if (item.indexOf(‘.’) != 0)
{
var file = currentDirectory + ‘\\’ + item;
var stats = fs.statSync(file);
if (stats.isDirectory())
{
console.log(‘directory; recursing..’)
return zipMe(file, dir);
}
else
{
console.log(‘file; adding..’)
dir.file(file, fs.readFileSync(file,’utf8′));
}
}
});

return dir
}

exports.createPackages = createPackages;[/js]

 

Using a different IDE

This is slightly more difficult; the SDK creates a bunch of files automatically (.widgetinfo, .metadata, that sort of thing). This does add an extra manual step, but isn’t impossible.

One thing I couldn’t actually get around is the debugging and testing locally; the commands being passed to the emulator aren’t easy to manipulate. When you choose to run the emulator from within Eclipse the only parameter passed is something which tells it you’re running it from Eclipse; nothing handy like a path or filename, dammit!

 

Summary

I realise I went off on a tangent in this post and I’ll explain more in the next one. However, we’re now at a point where we can save our project files *somewhere* (locally, on the LAN, on the interwebs – so long as you have the IP on their location) and spin up a nodejs script to serve them upon request to our TV.

The code from this post is over on github here https://github.com/rposbo/basic-smart-tv-app-server

 

Next up

Conclusion of February – why I had that huge gap in the middle, and why it went off on a massive tangent!

Smart TV 101 : Part #3 – Deploying to TV

I’m committing to doing 12 months of “101”s; posts and projects themed at beginning something new (or reasonably new) to me. January was all about node development awesomeness. February is all about Smart TV apps.

Deploying to a TV

Now that we’ve got a basic Smart TV app this post will investigate how to get that app on to the TV itself.

Packaging using the IDE

During the initial installation of the IDE you will have been asked to installed Apache; this is what it’s all been leading up to! You actually just need a web server on your home network somewhere; doesn’t have to be apache, doesn’t have to be on your developer pc.

Prerequisites

Make sure you’ve configured your Server settings within the IDE preferences:
samsung-packaging-server-prefs
samsung-packaging-server-prefs-root

The packaging process will drop a zip file into a Widget/ subdirectory of this directory.

Initiating package creation

As for actually creating the package, if you’re using the Eclipse IDE then you’re spoiled for choice: highlight the project in your project explorer and then either

  1. Click the Samsung App Packaging button
    samsung-app-packaging-1
  2. Click the Samsung Smart TV SDK menu, then click App Packaging
    samsung-smart-sdk-packaging-menu
  3. Right click the project in project explorer, Samsung Smart TV SDK, Packaging
    samsung-sdk-packaging-context-menu

Whichever you do you’ll end up with the same results:
samsung-packaging-dialog
samsung-packaging-confirmation

Results

Assuming you’ve set up the server settings in your preferences then you’ll end up with:

  1. a zip file placed within the SDK installation’s Package/ directory
    samsung-package-sdk-dir
  2. the same zip file placed in a Widget/ subdirectory on your configured server
    samsung-package-widget-dir
  3. a new (or updated) widgetlist.xml file in the root of your configured server’s directory

    samsung-widgetlist-xml

Make sure that you can browse to this file and that Apache is running by opening a browser and putting in http://<your development pc’s IP>/widgetlist.xml

Anatomy of a package and a widgetlist

So what is a package made of? Looking at the image above for the the zip file that’s created you’ll see that it looks almost identical to the contents of your application within the workspace:
samsung-app-workspace

So essentially the packaging step is zipping up your project directory, putting it into a specified web server subdirectory, and updating an XML file. Obviously, you shouldn’t need an IDE or SDK to do this sort of thing and I’ll be getting on to this development & deployment process without using Eclipse or installing Apache in a later post.

Deploying!!

Now that we have a package it’s time to load it on to your Smart TV. For this post I’ll be talking about deploying from the development pc via your home network, and in a later post will be talking about loading in packages externally.

TV setup

Make sure your TV is connected to your network and that your development pc’s Windows Firewall is off (or at least configured to allow local network traffic).

  • Turn on the TV
  • Go to your app hub/Smart Hub
  • Press the Login button
  • Create an account using the username “develop” and set a password

developer account

After you’ve successfully created the develop user you need to

  • Open the Settings menu
  • Open the new Development sub menu
  • Choose Setting server IP and enter the IP of your development PC
  • Choose User application synchronisation to check the apps that are listed in widgetlist.xml and install (or update) them all

download dev app

You should now find your application on the App Hub screen with a little red “user” banner over it; select it to run it, just like any other app.

asos-app-running-on-smart-tv-1