Category Archives: Asides

Creating a 4G router using a Raspberry Pi and a mobile phone

A few days ago these workmen were using cutting machinery dangerously close to my broadband cables:

Shortly after this picture was taken – bang! No internet! They cut the cables while doing their work!

Two adults working from home on back to back video calls, one high-schooler also on video classes, and one primary-schooler with streaming classes – all suddenly disconnected from the world!

That afternoon we huddled around the kitchen table, mobile phones on with hotspots enabled to get through the rest of the day – but this wouldn’t work for regular use.

The broadband company said it wouldn’t be fixed for weeks due to how badly everything was damaged; the pavement would have to come up! I had to think of a pragmatic solution.

In this article I’ll go through the steps I took to completely swap my home broadband for a Raspberry Pi and a spare mobile phone – and show the results!

Continue reading →

Involved in a startup? Read this!

Having been the VP of Engineering at a startup, I understand a lot of the challenges. The technical ones relating to the solution you think you need to build, more technical ones relating to the solutions the investors want you to build, the development process to best fit a rapidly changing product, team, requirements, and priorities, as well as managing the team through uncertain terrain.

They’re the fun ones. The easy ones! Especially given how talented my dev team was.

The founder had the difficult challenges; define a product that could be a success, iterate that idea based on extensive user testing, and most importantly, ensure there was funding.

Luckily, our founder was as talented at soliciting funds as we were at building epic tech!

If you are involved in a startup, perhaps Just Eat’s Accelerator programme can help with both types of challenge!

Continue reading →

Receiving Images to Your Skype Botframework Bot (v2!)

If you’re getting a “403” HTTP error when attempting to receive an image sent to your Skype bot, and the previous use of message.ServiceUrl to create a ConnectorClient didn’t work, try this more verbose version which explicitly sets the authorization header:

byte[] data;

if (image.ContentUrl != null)
{
    using (var connectorClient 
        = new ConnectorClient(new Uri(message.ServiceUrl)))
    {
        var token = 
            await (connectorClient.Credentials as MicrosoftAppCredentials)
                .GetTokenAsync();

        var uri = new Uri(image.ContentUrl);

        using (var httpClient = new HttpClient())
        {
            if (uri.Host.EndsWith("skype.com") 
                && uri.Scheme == Uri.UriSchemeHttps)
            {
                httpClient
                    .DefaultRequestHeaders
                    .Authorization = 
                        new AuthenticationHeaderValue("Bearer", token);

                httpClient
                    .DefaultRequestHeaders
                    .Accept
                    .Add(new MediaTypeWithQualityHeaderValue("application/octet-stream"));
            }

            // Get the image in a byte[] variable
            data = await httpClient.GetByteArrayAsync(uri);
        }
    }
}

Rate-Limiting, Powershell, Pester, and the ZenDesk API

Have you ever had to utterly hammer an API via a little Powershell script, only to find you’re getting rate limited, and lose all that previously downloaded data before you could persist it somewhere?

I have. I’ve recently put together a script to query a customer support ticketing system’s API, get a list of all tickets within a given time period, then query again to get to get the full content for each ticket.

All of this data is being added to a custom Powershell object (as opposed to churning out to a file as I go) since I convert it all to Json right at the end.

I’d rather not get half way through a few thousand calls and then have the process fail due to being throttled, so I’ve chosen to create a little function that will check for – in my case – an HTTP 429 response (“Too Many Requests”), get the value of the “Retry-After” header, then wait that many seconds before trying again.

This particular implementation is all quite specific to the ZenDesk API, but could easily be adapted to other APIs which rate limit/throttle and return the appropriate headers.

Continue reading →

Achievement Unlocked: OSCON 2015

I’ve recently returned from a fantastic week in Amsterdam for the O’Reilly OSCON conference

OSCON in Amsterdam celebrates, defines, and demonstrates the best that open source has to offer. From small businesses to the enterprise, open source is the first choice for engineers around the world.

Gruesome twosome

I thoroughly enjoyed presenting my Automating Web Performance workshop with the ever-epic Dean Hume at the Amsterdam RAI.

Amsterdam from our OSCON workshop room

Having a dedicated conference venu instead of just using a hotel was a master stroke; the venue was amazing, the wifi always worked, the food was plentiful and regular, our workshop room had a glorious view, and the general colour scheme changed from OSCON red to Velocity Conf turquoise as the conferences switched over mid week.

Continue reading →

Is that image a photo?

A very quick hack to check if an image is a photo in C#, using image properties, is to see if there’s a “camera manufacturer” property set:

[csharp]
foreach (var img in Directory.GetFiles(imagedump))
{
using (var fs = new FileStream(img, FileMode.Open))
using (var i = Image.FromStream(fs))
{
foreach (var p in
from p in i.PropertyItems
where p.Id == 271 // manufacturer property
select p)
{
var encoding = new ASCIIEncoding();
var manufacturer = encoding.GetString(p.Value);
Console.WriteLine("Camera make:{0}", manufacturer);
}
}
}
[/csharp]

Learned from a little msdn article; the value 271 is the integer version of the hex “0x010F” in the table of common IDs.

Learning By Doing: Java, Maven, Seyren, Hipchat, Github

If you use a graphing system, waiting all day long to see something spike can be painful. And boring.

Spike

Ouch! Seeing those spikes can be annoying, but missing them can be even more annoying. Who can be bothered to stare at graphs all day and night?

Seyren

That’s why recently I had the opportunity to try out Seyren. I’ve been meaning to try it out for a while; it’s a Java-based solution to monitor a graphite endpoint and react to various configured thresholds by alerting external services.

Seyren

Creating a check

These external services are extensible and currently there is support for systems such as plain old email, Flowdock, Hubot, Irc, PagerDuty, Slack, and – the subject of my interest – Hipchat.

Unfortunately, Seyren only supports Hipchat API v1 (which is deprecated) and as such I couldn't use it. Also it’s written in Java and I’ve never written anything in Java. However, I did do a degree in C++ and that's pretty close, right?…
Right?..

This is the short story of how I contributed to a Java-based open source project, adding support for Hipchat V2 and generally saving the day! (possible exaggerations may exist in this post.)

First up, how I managed to set up a minimal Java development environment on Windows.

Java

Installing Java on Windows

You have two main options for getting the Java Development Kit running on your system:

One:

  1. Head over to the JDK download page on oracle.com
  2. Download and install the Java Development Kit for your OS

Two:

  1. If you haven't already got the amazing Windows package manager chocolately, go and get it!
  2. choco install jdk8

For either one, you may still need to add the Java root dir to your PATH (not the bin subdir) as JAVA_HOME, like this:
JAVA_HOME environment variable

Maven

What is it?

Maven is a build automation tool, used a lot in Java projects. The build configuration is in an xml file – usually named "pom.xml" for Project Object Model – which defines the dependencies and project configuration.

Maven has the concept of plugins, goals, and phases to extend and link stages into build lifecycles. The Build lifecycle is a list of named phases that can be used to give order to goal execution.

One of Maven's standard lifecycles is the default lifecycle, which includes the following phases:

  1. validate
  2. generate-sources
  3. process-sources
  4. generate-resources
  5. process-resources
  6. compile
  7. process-test-sources
  8. process-test-resources
  9. test-compile
  10. test
  11. package
  12. install
  13. deploy

So running mvn test will result in the execution of phases 1 through 10; mvn install will execute phases 1 though 12. You get the idea.

Installing Maven on Windows

Again, a couple of options;

One:

  1. Head over to https://maven.apache.org/
  2. Download the zip
  3. Place the contents in a seriously high level directory such as C:\mvn (Maven doesn't like spaces in pathnames)
  4. Append the bin subdir to your PATH

Two:

  1. choco install maven

Heh.

Either route needs you to open a fresh command line in order to get the updated PATH values maven configures.

Right, now on to the main event!

Seyren

What is it?

Seyren "is an alerting dashboard for Graphite. It supports the following notification channels: Email, Flowdock, HipChat, HTTP, Hubot, IRCcat, PagerDuty, Pushover, SLF4J, Slack, SNMP, Twilio"

You configure it to point at a graphite instance, tell it what metrics to monitor, what thresholds to watch out for, how to notify you of these events, and it will ping graphite every few seconds; should any of those thresholds be met, it will notify you.

Simple as that. Its power is in that simplicity.

Getting the code

Head over to the github repo at https://github.com/scobal/seyren and clone the repo locally.

If you just want to run it, then you can just download the latest release as a Java jar file.

Running Seyren locally

Seyren has a dependency on mongodb where it saves the checks (points at which a configured threshold has changed state)

So, let's set that up.

  • choco install mongodb

Easy. If everything has worked so far, you can open a terminal in the repo directory and run the following maven command to check it builds and the tests pass:

  • mvn clean verify

If all went well, you will need to set up an environment variable or two, such as your graphite instance's url and in my case my hipchat API key. Again, these are just environment variables, like JAVA_HOME.

Once that's done you can run Seyren temporarily within maven for a quick play.

Happy with it? You can use maven to create a jar file using

  • mvn package

    or if you want to skip running the tests again

  • mvn package -DskipTests

That will generate you a bunch of jar files in various target subdirectories; the one we're interested in is in seyren-web – the others are dependencies for it.

You can now start this puppy up within a tomcat instance using (substitute the name of your "war-exec.jar" file in here):

  • java -jar /seyren/seyren-web/target/seyren-web-1.4.0-SNAPSHOT-war-exec.jar

Checkpoint 1

You should now have Java, Maven, MongoDB, and Seyren running happily. Now here's how I managed to implement Hipchat v2 support and get the PR accepted!

Java IDEs

Seriously? Eclipse? I've looked at it before, and couldn't stand it. I even downloaded and installed it, but gave up. Since I'm not building a Java project from scratch, all I needed was a half decent text editor.

As such, I fired up my current favourite editor – SublimeText. I like the simplicity of this editor, and you can easily get it yourself with choco install sublimetext2, naturally.

Having a pretty good understanding of the Hipchat v2 API, I was able to guess and googlebing the necessary Java syntax for updating the existing HipchatNotificationService, as can be seen in this pull request: https://github.com/scobal/seyren/pull/294/files

Being able to easily switch back to command line and run mvn clean verify to get the build feedback and the test results was pretty painless. I got it running by pointing it at the company Hipchat account and checked everything worked as expected, then proceeded to submit a pull request, expecting to receive all manner of awards for my contribution and epic skillz.

Contributing

messy comits

Unfortunately I made a few messy commits in my initial pull request and someone had their own PR merged in the meantime (i.e., whilst I spent several days trying to work out how to generate a "jar" file..), but it didn't do anything of value and was even incorrect. Instead of doing the right thing and merging with it, fixing it, reverting their stuff, and submitting a full PR, I was lazy; I submitted a messy list of commits with a message along the lines of "I know it doesn't merge, but that's your fault not mine".

I was tired, I apologise.

I received a request to rebase and squash the commits, then resubmit the PR. I've never done that; I can
git clone
like a pro, but after something like
git mergetool --tool=kdiff3
(my current fave), I'm out of my depth. I had to learn quickly!

In short; rebase appears to rewind your commits, take another branch, and replay the commits one by one over that branch. If any of the commits result in a merge conflict you can fix it before deciding to git rebase --continue. Pretty nice.

Squashing commits is pretty obvious; takes a set of individual commits and merges them into one commit. There are several ways of doing this that I've read about, but the one I find easiest is to "soft reset" your local branch using
git reset HEAD~3 --soft

In this example I will effectively uncommit the latest 3 commits, but the key is the --soft option – your changes are not undone (as with --hard), merely uncommited.

You can now commit all of the changes from those 3 commits into one, and give it a nice, clear, commit message.

That's what I did, and that's how my first Java OSS contribution was accepted!

References

Aside

A Question: What’s Your Third Place?

Cheers!

I’ve been thinking recently about the concept of the Third Place, which is usually defined by:

  • The “first place” is the home and those that one lives with;
  • The “second place” is the workplace where people may actually spend most of their time;
  • “Third places”, then, are “anchors” of community life and facilitate and foster broader, more creative interaction.

With hallmarks such as those defined by Ray Oldenburg in his book The Great Good Place which explores such hangouts at the heart of the community:

  • Free or inexpensive
  • Food and drink, while not essential, are important
  • Highly accessible: proximate for many (walking distance)
  • Involve regulars – those who habitually congregate there
  • Welcoming and comfortable
  • Both new friends and old should be found there.

[wiki]

Apparently, even Starbucks was shaped by someone wanting to create a third place as a coffee shop in the style of the traditional Italian coffeehouse.

How I Define It

For me, a third place is somewhere physical or virtual (I’ll get on to this again later) that’s not work and not home, that allows me to mentally relax and defocus from the concerns of everyday life usually associated with those places.

It must be somewhere that affords me the luxury of uninterrupted thought to consider whatever worry, concern, or even slight niggle that has been bugging me about places one and two. By gaining that mental distance I’m able to more easily find solutions or just generally be at peace.

Many years ago my “third places” would have been friend’s houses (massive all-nighter gaming sessions!) or pubs, since that’s all I really did outside of home, work/study, commute.

Physical

Coffee shop

Then I grew older and it became almost entirely pubs and bars.

Thankfully, London being a major city has a significant number of social places, and a few of these are reasons for me loving this city as much as I do; if you’ve never spent a lazy afternoon wandering around the Tate Modern, Royal Festival Hall, or just exploring Southbank in general, or the British Museum, British Library, The Barbican, then I thoroughly recommend you do. (*)

That’s not even scratching the surface of the available resources for public Zen; the museum district around South Kensington offers the V&A, Science Museum, and the Natural History Museum – all amazing, all fascinating, all full of people just milling around (all with free wifi!)

The benefit of these sorts of places for my psychological well-being is immense.

However, since becoming a parent my third places have significantly changed; I’ve no money for pubs and bars and very little time for visiting London’s great spaces so for a few years that Third Place became my cycle commute; a chance to spend an hour not thinking about usual day to day concerns, instead focussing solely on staying alive in central London’s rush hour traffic!

I’d often find that a good, fast – safe – cycle ride would result in my brain having deciphered that coding, architecture, or management problem I’d just mentally left behind in the office, possibly more efficiently than finding a space to walk around in and force myself to think it through.

Virtual

I have previously spent much time working from coffee shops or those great places mentioned above, moving from one to another as I need to; I no longer have that luxury.

Having moved both house and job recently, I am no longer able to fit cycling into my daily routine and I don’t spend enough time on one single form of public transport to achieve that same state of Zen – constantly changing from bus to tube to tube to bus interrupts any such defocusing.

Several years ago a few articles claiming the internet as a third place were published; a couple that stood out for me were Scott Hanselman’s The Developer Theory of the Third Place (2007) and The internet as the 3rd place from Advomatic.

The time people invest in Facebook, Twitter, Tumblr, Pinterest, and any other forums of social media allow them to treat these as their social interaction outside of the work and the home (presumably whilst still physically inside of work or home) and for some people this brings similar benefits as a physical location and have almost all of the same hallmarks;

  • Free or inexpensive
  • Highly accessible: proximate for many (walking distance)
  • Involve regulars – those who habitually congregate there
  • Welcoming and comfortable
  • Both new friends and old should be found there.

The same is true of most well known techie online hangouts; StackOverflow or Reddit as great examples of this.

So What’s My Point?

I’m writing this as I’ve realised why I’m gradually more uneasy these days; I think I’ve lost my third place. No bars, pubs, gaming sessions, time to just wander around, time to cycle.

The virtual places are no longer that engaging; Facebook is just something to look at whilst patting my kids to sleep at night. No time to just hangout on SO, Reddit, FB, other.

I think for a while I immersed myself in learning new things or working all hours on one or many projects, perhaps just blogging; this was probably more of a distraction than escapism and only succeeded in exhausting me such that I haven’t had any energy for a few months!

As such, I’m slightly at a loss, dear reader. What’s your third place? What could mine be?..

Disclaimer

(*) Other cities are available, and are equally amazing.

Thoughts on learning to become a web developer

I was recently asked by a friend for advice on a career change; he wants to get into web development and wasn’t quite sure the best place to start.

Requirements

  • Where do I start??
  • How do I decide between being a front end or back end developer?
  • Attend a course with an instructor or self-learning online?

My thoughts on learning to become a developer

Instructor-led courses or workshops are great for intense, short, periods of studying, where you can ask questions to get explanations for things you may not have understood. I sometimes teach via frameworktraining.co.uk and love how I can focus on whatever areas the attendees want me to; I’ve previously taught an Advanced MVC5 course, but it morphed into an Advanced MVC5/6 course with a focus on security.

I would suggest starting with Pluralsight – you get a free trial of something like 10 days or 200 minutes’ worth, then if you stick with it you pay either $30 or $50 a month for unlimited access to the excellent training videos (and coursework if you pay the $50); I use this all the time to seriously cram before I train a course – usually watching it at double speed!

Pluralsight is extremely well respected as a learning resource for all levels of ability. There are fantastic beginner courses – and even courses for teaching kids to program! I’d suggest checking out the Learning to Program Part 1 and 2 by Scott Allen – his stuff is usually great.

Personally I prefer to use pluralsight to an in-person course, but that’s just how I best learn. You may be different and want to get some in-person help as well.

Check out egghead.io for angularjs (popular javascript framework) videos; again, some free, some paid.

As for Back End vs Front End; just learn the basic concepts, then work out what you prefer.

So. How does that sound to you? Any suggestions?