Create your first botframework bot in Azure!

At //Build 2016 Microsoft unleashed something known as the Bot Framework; a set of three products to help you build conversational bots and connect them with services, like Slack, Skype, and even Email and SMS.

The Bot Framework has a number of components including the Developer Portal, Bot Builder, and the Bot Directory:

  • Developer Portal is where you wire your bot up to existing services like Skype and Slack
  • Bot Directory is where you publish your bot to so that others can search for and connect it to their Skype and Slack
  • Bot Builder is what you use to implement a more complex and intelligent conversational flow, optionally leveraging the awesome power of some of the Microsoft Research’s Cognitive Services for language interpretation, image recognition, and much more.

In this article I’m going to take you through the process of creating, deploying, and configuring your own Skype bot.

Continue reading

Learning By Doing: Java, Maven, Seyren, Hipchat, Github

If you use a graphing system, waiting all day long to see something spike can be painful. And boring.

Spike

Ouch! Seeing those spikes can be annoying, but missing them can be even more annoying. Who can be bothered to stare at graphs all day and night?

Seyren

That’s why recently I had the opportunity to try out Seyren. I’ve been meaning to try it out for a while; it’s a Java-based solution to monitor a graphite endpoint and react to various configured thresholds by alerting external services.

Seyren

Creating a check

These external services are extensible and currently there is support for systems such as plain old email, Flowdock, Hubot, Irc, PagerDuty, Slack, and – the subject of my interest – Hipchat.

Unfortunately, Seyren only supports Hipchat API v1 (which is deprecated) and as such I couldn't use it. Also it’s written in Java and I’ve never written anything in Java. However, I did do a degree in C++ and that's pretty close, right?…
Right?..

This is the short story of how I contributed to a Java-based open source project, adding support for Hipchat V2 and generally saving the day! (possible exaggerations may exist in this post.)

First up, how I managed to set up a minimal Java development environment on Windows.

Java

Installing Java on Windows

You have two main options for getting the Java Development Kit running on your system:

One:

  1. Head over to the JDK download page on oracle.com
  2. Download and install the Java Development Kit for your OS

Two:

  1. If you haven't already got the amazing Windows package manager chocolately, go and get it!
  2. choco install jdk8

For either one, you may still need to add the Java root dir to your PATH (not the bin subdir) as JAVA_HOME, like this:
JAVA_HOME environment variable

Maven

What is it?

Maven is a build automation tool, used a lot in Java projects. The build configuration is in an xml file – usually named "pom.xml" for Project Object Model – which defines the dependencies and project configuration.

Maven has the concept of plugins, goals, and phases to extend and link stages into build lifecycles. The Build lifecycle is a list of named phases that can be used to give order to goal execution.

One of Maven's standard lifecycles is the default lifecycle, which includes the following phases:

  1. validate
  2. generate-sources
  3. process-sources
  4. generate-resources
  5. process-resources
  6. compile
  7. process-test-sources
  8. process-test-resources
  9. test-compile
  10. test
  11. package
  12. install
  13. deploy

So running mvn test will result in the execution of phases 1 through 10; mvn install will execute phases 1 though 12. You get the idea.

Installing Maven on Windows

Again, a couple of options;

One:

  1. Head over to https://maven.apache.org/
  2. Download the zip
  3. Place the contents in a seriously high level directory such as C:\mvn (Maven doesn't like spaces in pathnames)
  4. Append the bin subdir to your PATH

Two:

  1. choco install maven

Heh.

Either route needs you to open a fresh command line in order to get the updated PATH values maven configures.

Right, now on to the main event!

Seyren

What is it?

Seyren "is an alerting dashboard for Graphite. It supports the following notification channels: Email, Flowdock, HipChat, HTTP, Hubot, IRCcat, PagerDuty, Pushover, SLF4J, Slack, SNMP, Twilio"

You configure it to point at a graphite instance, tell it what metrics to monitor, what thresholds to watch out for, how to notify you of these events, and it will ping graphite every few seconds; should any of those thresholds be met, it will notify you.

Simple as that. Its power is in that simplicity.

Getting the code

Head over to the github repo at https://github.com/scobal/seyren and clone the repo locally.

If you just want to run it, then you can just download the latest release as a Java jar file.

Running Seyren locally

Seyren has a dependency on mongodb where it saves the checks (points at which a configured threshold has changed state)

So, let's set that up.

  • choco install mongodb

Easy. If everything has worked so far, you can open a terminal in the repo directory and run the following maven command to check it builds and the tests pass:

  • mvn clean verify

If all went well, you will need to set up an environment variable or two, such as your graphite instance's url and in my case my hipchat API key. Again, these are just environment variables, like JAVA_HOME.

Once that's done you can run Seyren temporarily within maven for a quick play.

Happy with it? You can use maven to create a jar file using

  • mvn package

    or if you want to skip running the tests again

  • mvn package -DskipTests

That will generate you a bunch of jar files in various target subdirectories; the one we're interested in is in seyren-web – the others are dependencies for it.

You can now start this puppy up within a tomcat instance using (substitute the name of your "war-exec.jar" file in here):

  • java -jar /seyren/seyren-web/target/seyren-web-1.4.0-SNAPSHOT-war-exec.jar

Checkpoint 1

You should now have Java, Maven, MongoDB, and Seyren running happily. Now here's how I managed to implement Hipchat v2 support and get the PR accepted!

Java IDEs

Seriously? Eclipse? I've looked at it before, and couldn't stand it. I even downloaded and installed it, but gave up. Since I'm not building a Java project from scratch, all I needed was a half decent text editor.

As such, I fired up my current favourite editor – SublimeText. I like the simplicity of this editor, and you can easily get it yourself with choco install sublimetext2, naturally.

Having a pretty good understanding of the Hipchat v2 API, I was able to guess and googlebing the necessary Java syntax for updating the existing HipchatNotificationService, as can be seen in this pull request: https://github.com/scobal/seyren/pull/294/files

Being able to easily switch back to command line and run mvn clean verify to get the build feedback and the test results was pretty painless. I got it running by pointing it at the company Hipchat account and checked everything worked as expected, then proceeded to submit a pull request, expecting to receive all manner of awards for my contribution and epic skillz.

Contributing

messy comits

Unfortunately I made a few messy commits in my initial pull request and someone had their own PR merged in the meantime (i.e., whilst I spent several days trying to work out how to generate a "jar" file..), but it didn't do anything of value and was even incorrect. Instead of doing the right thing and merging with it, fixing it, reverting their stuff, and submitting a full PR, I was lazy; I submitted a messy list of commits with a message along the lines of "I know it doesn't merge, but that's your fault not mine".

I was tired, I apologise.

I received a request to rebase and squash the commits, then resubmit the PR. I've never done that; I can
git clone
like a pro, but after something like
git mergetool --tool=kdiff3
(my current fave), I'm out of my depth. I had to learn quickly!

In short; rebase appears to rewind your commits, take another branch, and replay the commits one by one over that branch. If any of the commits result in a merge conflict you can fix it before deciding to git rebase --continue. Pretty nice.

Squashing commits is pretty obvious; takes a set of individual commits and merges them into one commit. There are several ways of doing this that I've read about, but the one I find easiest is to "soft reset" your local branch using
git reset HEAD~3 --soft

In this example I will effectively uncommit the latest 3 commits, but the key is the --soft option – your changes are not undone (as with --hard), merely uncommited.

You can now commit all of the changes from those 3 commits into one, and give it a nice, clear, commit message.

That's what I did, and that's how my first Java OSS contribution was accepted!

References

Node.js 101: Wrap up

Year of 101s, Part 1 – Node January

Summary – What was it all about?

I set out to spend January learning some node development fundementals.

Part #1 – Intro

I started with a basic intro to using node – a Hello World – which covered what node.js is, how to create the most basic of all programs, and mentioned some of the development environments.

Part #2 – Serving web content

Second was creating a very simple node web server, which covered using nodemon to develop your node app, the concept of exports, basic request routing, and serving various content types.

Part #3 – A basic API

Next was a simple API implementation, where I proxy calls to the Asos API, return a remapped subset of the data returned, reworked the routing creating basic search functionality and a detail page, and touched on being able to pass in command line arguements.

Part #4 – Basic deployment and hosting with Appharbor, Azure, and Heroku

Possibly the most interesting and fun post for me to work on involved deploying the node code on to three cloud hosting solutions where I discovered the oddities each provider has, various solutions to the problems this raises, as well as some debugging cleverness (nice work, Heroku!). The simplicity of a git-remote-push-deploy process is incredible, and really makes quick application development and hosting even more enjoyable!

Part #5 – Packages

Another interesting one was getting to play with node packages, the node package manager (npm), the express web framework, jade templating engine, and stylus css pre-processor, and deploying node apps with packages to cloud hosting.

Part #6 – Web-based development

The final part covered the fantastic Cloud9IDE, including a (very) basic intro to github, and how Cloud9 can still be used in developing and deploying directly to Azure, Appharbor, or Heroku.

What did I get out of it?

I really got into githubbing and OSSing, and really had to try hard to not over stretch myself as I had starting forking repos to try and make a few tweaks to things whilst working on the node month.

It has been extremely inspiring and has opened up so many other random tangents for me to explore in other projects at some other time. Very motivating stuff.

I’ve now got a month of half decent blog posts – I had only intended to do a total of 4 posts but including this one I’ve done 7, since I kept adding more information as it turned up and needed to split a few posts into two.

Also I’ve learned a bit about blogging; trying to do posts well in advance allowed me to build up the details once I’d discovered more whilst working on subsequent posts. For example, how Appharbor and Azure initially track master – but can be configured to track different branches. Also, debugging with Heroku only came up whilst working with packages in Heroku.

Link list

Node tutorials and references

Setting up a node development environment on Windows
Node Beginner – a great article, and I’ve also bought the associated eBooks.
nodejs.org – the official node site, the only place to go for reference

Understanding Javascript better

Execution in The Kingdom of Nouns
Object Orientation and Inheritance in Javascript

Appharbor

Appharbor and git

Heroku

Heroku toolbelt download and reference
node on Heroku

Azure

Checkout what Azure can do!

February – coming up, Samsung Smart TV App Development!

Yeah, seriously. How random is that?.. 🙂

Node.js 101: Part #6 – Web-Based Development

Web-Based Development

Following on from my recent post about doing something this year, I’m committing to doing 12 months of “101”s; posts and projects themed at begining something new (or reasonably new) to me

January is all about node, and I started with a basic intro, then cracked open a basic web server with content-type manipulation and basic routing, created a basic API, before getting stuck into some great deployment and hosting solutions and then an intro to using node packages including cloud deployment

In my previous posts I’ve been developing code locally, committing to a local git repo and pushing to a remote git repo. This is fine for the particular situation, but what about when I’m not at my own pc and feel the need to make some changes? Maybe I’m at my dad’s place using his netbook with no dev tools installed?

Cloud9IDE

Cloud9 is an incredible web-based development environment that is so feature-rich you’d usually expect to fork out wads of cash for the opportunity to use it: LIVE interactive collaborative development in the same shared IDE (see multiple people editing a file at once), code completion, syntax highlighting, an integrated console for those useful commands like ssh, git, npm.

It’s frikkin open source too, so you could install it on your own servers and have your own private IDE for your own code, based in a web browser. How amazing is that?

It’s built on Node.js in the back-end and javascript and HTML5 at the front. I’ve been playing around on there for the past year, and it’s been improving all the time – it’s just the best thing around. Go and start using it now. There are still some bugs, but if you find something you can always try to fix it and send a pull request!

c9-demo-1

So. That’s great for my web-based development, so how about if I need to collaborate on this project with people who I’m not sharing my C9 environment with?

GitHub

If you’re not already using github but are already using git (what the hell are you playing at?!), go and sign up for this exceptionally “powerful collaboration, review, and code management for open source and private development projects.”

You configure github as your git remote, push your code to it, and other users can pull, fork, edit, and send pull requests, so that you’re still effectively in charge of your own code repository whilst others can contribute to it or co-develop with you.

github-demo-1

Great. So how do I deploy my code if I’m using this sort of remote, web-based development environment?

Azure/AppHarbor/Heroku

Deploying to an existing Azure/Appharbor/Azure site from Cloud9IDE is the same as from your local dev pc; set up a remote and push to it! C9 has a built in terminal should the bare command line at the bottom of the screen not do it for you.

As for creating a new hosting environment, C9 also includes the ability to create them from within itself for both Azure and Heroku! I’ve actually never managed to get this working, but am quite happy to create the empty project on Heroku/Azure/Appharbor and use git from within C9 to deploy.

c9-azure-setup-1

Coming up

Next post will be the last for this first month of my Year of 101s: January Wrap-Up – Node.js 101; a summary of what I’ve learned in January whilst working with Node, as well as a roundup of the useful links I’ve used to get all of the information.

What’s in February’s 101?.. wait and see..!

Node.js 101: Part #5 – Packages

Following on from my recent post about doing something this year, I’m committing to doing 12 months of “101”s; posts and projects themed at begining something new (or reasonably new) to me

January is all about node, and I started with a basic intro, then cracked open a basic web server with content-type manipulation and basic routing, created a basic API, before getting stuck into some great deployment and hosting solutions

Node Packages

Up until now I’ve been working with node using the basic code I’ve written myself. What about if you want to create an application that utilises websockets? Or how about a Sinatra-inspired web framework to shortcut the routing and request handling I’ve been writing? Maybe you want to have a really easy to build website without having to write HTML with a nice look without writing any CSS? Like coffeescript? mocha? You gaddit.

Thanks to the node package manager you can easily import pre-built packages into your project to do alllll of these things and loads more. This command line tool (which used to be separate but is now a part of the node install itself) can install the packages in a ruby gem-esque/.Net nuget fashion, pulling down all the dependencies automatically.

Example usage:

npm install express -g

The packages (compiled C++ binaries, just like node itself) are pulled either into your working directory (local node_modules folder) or as a global package (with the “-g” parameter). You then reference the packages in your code using “requires”.

Or you can install everything your project needs at once by creating a package.json e.g.:

{
  "name": "basic-node-package",
  "version": "0.0.1",
  "dependencies": {
    "express": "*",
    "jade": "*",
    "stylus": "*",
    "nib": "*"
  }
}

And then call

npm install

A great intro to using these four packages can be found on the clock website

I’ve decided to write a wrapper for my basic node API using express, jade, stylus, and nib. All I’m doing is call the api and displaying the results on a basic page. The HTML is being written in jade and the css in stylus & nib. Routing is being handled by express.

app.js

var express = require('express')
  , stylus = require('stylus')
  , nib = require('nib')
  , proxy = require('./proxy')

var app = express()
function compile(str, path) {
  return stylus(str)
    .set('filename', path)
    .use(nib())
}
app.set('views', __dirname + '/views')
app.set('view engine', 'jade')
app.use(express.logger('dev'))
app.use(stylus.middleware(
  { src: __dirname + '/public'
  , compile: compile
  }
))
app.use(express.static(__dirname + '/public'))

var host = 'rposbo-basic-node-api.azurewebsites.net';

app.get('/products/:search/:key', function (req,response) {
console.log("Request handler 'products' was called");

  var requestPath = '/products/' + req.params.search + '?key=' + req.params.key;
    
  proxy.getRemoteData(host, requestPath, function(json){
    var data = JSON.parse(json);
    
    response.render('products',
        {
            title: 'Products for' + data.category,
            products: data.products,
            key: req.params.key
        }
    );
  })
});

app.get('/product/:id/:key', function (req,response) {
console.log("Request handler 'product' was called");

  var requestPath = '/product/' + req.params.id + '?key=' + req.params.key;
    
  proxy.getRemoteData(host, requestPath, function(json){
    var data = JSON.parse(json);
    
    response.render('product',
        {
            title: data.title,
            product: data
        }
    );
  })
});

app.get('/', function (req,response) {
    console.log("Request handler 'index' was called");
    response.end("Go");
});

app.listen(process.env.PORT);

So that file sets up the express, jade, and stylus references and wires up the routes for /products/ and /product/ which then make a call using my old proxy.js to the API; I can probably do all of this with a basic inline http get, but I’m just reusing it for the time being.

Notice how the route “/products/:search/:key” which would actually be something like “/products/jeans/myAp1k3Y” is referenced using req.params.search and req.params.key.

Then all I’m doing is making the API call, parsing the returned JSON and passing that parsed object to the view.

The views are written in jade and have a main shared one:
layout.jade

!!!5
html
  head
    title #{title}
    link(rel='stylesheet', href='/stylesheets/style.css')
  body
    header
      h1 basic-node-packages
    .container
      .main-content
        block content
      .sidebar
        block sidebar
    footer
      p Running on node with Express, Jade and Stylus

Then the route-specific ones:

products.jade:

extend layout
block content
  p
    each product in products
        li
            a(href='/product/' + product.id + '/' + key)
                img(src=product.image)
                p
                    =product.title

and

product.jade:

extend layout
block content
  p
    img(src=product.image)
    li= product.title
    li= product.price

The stylesheet is written in stylus & nib:

style.styl

/*
 * Import nib
 */
@import 'nib'

/*
 * Grab a custom font from Google
 */
@import url('http://fonts.googleapis.com/css?family=Quicksand')

/*
 * Nib provides a CSS reset
 */
global-reset()

/*
 * Store the main color and
 * background color as variables
 */
main-color = #fa5b4d
background-color = #faf9f0

body
  font-family 'Georgia'
  background-color background-color
  color #444

header
  font-family 'Quicksand'
  padding 50px 10px
  color #fff
  font-size 25px
  text-align center

  /*
   * Note the use of the `main-color`
   * variable and the `darken` function
   */
  background-color main-color
  border-bottom 1px solid darken(main-color, 30%)
  text-shadow 0px -1px 0px darken(main-color, 30%)

.container
  margin 50px auto
  overflow hidden

.main-content
  float left

  p
    margin-bottom 20px

  li
    width:290
    float:left

p
  line-height 1.8

footer
  margin 50px auto
  border-top 1px dotted #ccc
  padding-top 5px
  font-size 13px

And this is compiled into browser-agnostic css upon compilation of the app.

The other files used:

proxy.js:

var http = require('http');

function getRemoteData(host, requestPath, callback){
  
    var options = {
        host: host,
        port: 80,
        path: requestPath
    };
 
    var buffer = '';
    var request = http.get(options, function(result){
        result.setEncoding('utf8');

        result.on('data', function(chunk){
            buffer += chunk;
        });

        result.on('end', function(){
            callback(buffer);
        });
    });

    request.on('error', function(e){console.log('error from proxy call: ' + e.message)});
    request.end();
};
exports.getRemoteData = getRemoteData;

package.json

{
  "name": "basic-node-package",
  "version": "0.0.1",
  "dependencies": {
    "express": "*",
    "jade": "*",
    "stylus": "*",
    "nib": "*"
  }
}

web.config

<configuration>
    <system.web>
        <compilation batch="false" />
    </system.web>
    <system.webServer>
        <handlers>
            <add name="iisnode" path="app.js" verb="*" modules="iisnode" />
        </handlers>
        <iisnode loggingEnabled="false" />

        <rewrite>
            <rules>
                <rule name="myapp">
                    <match url="/*" />
                    <action type="Rewrite" url="app.js" />
                </rule>
            </rules>
        </rewrite>
    </system.webServer>
</configuration>

All of these files are, as usual, on Github

Deployment with Packages

Something worth bearing in mind is that deploying something which includes packages and the result of packages (e.g. minified js or css from styl) requires all of these artifacts to be added into your git repo before deployment to certain hosts such as Appharbor and Azure; Heroku will actually run an npm install as part of the deployment step, I believe, and also compile the .styl into .css, unlike Azure/Appharbor.

The files above give a very basic web interface to the /products/ and /product/ routes:
asos-jade-products-1

asos-jade-product-1

Coming up

Web-based node development and deployment!

Node.js 101 : Part #4 – Basic Deployment and Hosting with Azure, Heroku, and AppHarbor

Following on from my recent post about doing something this year, I’m committing to doing 12 months of “101”s; posts and projects themed at beginning something new (or reasonably new) to me.

January is all about node, and I started with a basic intro, then cracked open a basic web server with content-type manipulation and basic routing, and the last one was a basic API implementation

Appharbor, Azure, and Heroku

Being a bit of a cocky git I said on twitter at the weekend:

It’s not quite that easy, but it’s actually not far off!

Deployment & Hosting Options

These are not the only options, but just three that I’m aware of and have previously had a play with. A prerequisite for each of these – for the purposes of this post – is using git for version control since AppHarbor, Azure, and Heroku support git hooks and remotes; this means essentially you can submit your changes directly to your host, which will automatically deploy them (if pre-checks pass).

I’ll be using the set of files from my previous API post for this one, except I need to change the facility to pass in command line args for the api key to instead take it from a querystring parameter.

The initial files are the same as the last post and can be grabbed from github

Those changes are:

app.js (removed lines about getting value from command line):

var server = require("./server"),
    router = require("./router"),
    requestHandlers = require("./requestHandlers");

// only handling GETs at the moment
var handle = {}
handle["favicon.ico"] = requestHandlers.favicon;
handle["product"] = requestHandlers.product;
handle["products"]  = requestHandlers.products;

var port = process.env.PORT || 3000;
server.start(router.route, handle, port);

server.js (added in querystring param usage):

var http = require("http"),
    url = require("url");

function start(route, handle, port) {
  function onRequest(request, response) {
    var pathname = url.parse(request.url).pathname;
    var apiKey = url.parse(request.url, true).query.key;
    route(handle, pathname, response, apiKey);
  }

http.createServer(onRequest).listen(port);
  console.log("Server has started listening on port " + port);
}

exports.start = start;

The “.query” returns a querystring object, which means I can get the parameter “key” by using “.key” instead of something like [“key”].

Ideal scenario

In the perfect world all I’d need to do is something like:

git add .
git commit -m "initial node stuff"
git push {azure/appharbor/heroku/whatever} master
.....
done
.....
new site deployed to blahblah.websitey.net
.....
have a lovely day

and I could pop off for a cup of earl grey.

In order to get to that point there were a few steps I needed to take for each of the three hosts.

Appharbor

appharbor-home-1

Getting started

First things first; go and sign up for a free account with AppHarbor.

Then set up a new application in order to be given your git remote endpoint to push to.

I’ve previously had a play with Appharbor, but this is the first time I’m using it for more than just a freebie host.

Configuring

It’s not quite as simple as I would have liked; there are a couple of things that you need to bear in mind. Although Appharbor supports node deployments they are primarily a .Net hosting service and use Windows hosting environments (even though they’re on EC2 as opposed to Azure). Running node within iis means that you need to supply a web.config file and give it some IIS-specific info.

The config file I had to use is:

<configuration>
    <system.web>
        <compilation batch="false" />
    </system.web>
    <system.webServer>
        <handlers>
            <add name="iisnode" path="app.js" verb="*" modules="iisnode" />
        </handlers>
        <iisnode loggingEnabled="false" />

    <rewrite>
        <rules>
            <rule name="myapp">
                <match url="/*" />
                <action type="Rewrite" url="app.js" />
            </rule>
        </rules>
    </rewrite>
  </system.webServer>
</configuration>

Most of that should be pretty straightforward (redirect all calls to app.js), but notice the lines about compilation and logging; the permissions under which the appharbor deployment process runs for node projects doesn’t have access to the filesystem so can’t create anything in a “temp” dir (precompilation) nor write any log files upon errors. As such, you need to disable these.

You could also enable file system access and disable precompilation within your application’s settings – as far as I can tell, it does the same thing.

appharbor-settings-1

Deploying

Commit that web.config to your repo, add a remote for appharbor, then push to it – any branch other than master, default, or trunk needs a manual deploy instead of it happening automatically, but you can specify the branch name to track within your appharbor application settings; I put in the branch name “appharbor” that I’ve been developing against and it automatically deploys when I push that branch or master, but not any others.

You’ll see your dashboard updates and deploys (automatic deployment if it’s a tracked branch):

appharbor-deploy-dashboard-1

And then you can browse to your app:

appharbor-deploy-result-1

Azure

azure-home-1

Getting started

Again, first step is to go and sign up for Azure – you can get a free trial, and if you only want to host up to 10 small websites then it’s completely free.

You’ll need to set up a new Azure website in order to be given your git remote endpoint to push to.

Configuring

This is pretty similar to the AppHarbor process in that Azure Websites sit on Windows and IIS, so you need to define a web.config to set up IIS for node. The same web.config works as for AppHarbor.

Deploying

Although you can push to Appharbor from any branch and it will only deploy automatically from the specific tracked branch, you can’t choose to manually deploy from within azure, so you either need to use

git push azure {branch}:master

(assuming your remote is called “azure”) or you can define your tracked branch in the configuration section:

azure-settings-1

Following a successful push your dashboard updates and deploys:

azure-deploy-dashboard-1

And then your app is browsable:

azure-deploy-result-1

Heroku

heroku-home-1

Getting started

Sign up for a free account.

Configuring

Heroku isn’t Windows based as it’s aimed at hosting Ruby, Node.js, Clojure, Java, Python, and Scala. What this means for our node deployment is that we don’t need a web.config to get the application running on Heroku. It’s still running on Amazon’s EC2 as far as I can tell though.

However, we do need to jump through several other strange hoops:

Procfile

The procfile is a list of the “process types in an application. Each process type is a declaration of a command that is executed when a process of that process type is executed.” These can be arbitrarily named except for the “web” one which handles HTTP traffic.

For node, this Procfile needs to be:

Procfile:

web: node app.js

Should I want to pass in command line arguments, as in the previous version of my basic node API code, I could do it in this file i.e.

web: node app.js mYAp1K3Y

Deploying

Heroku Toolbelt

There’s a command line tool which you need to install in order to use Heroku, called the Toolbelt; this is the Heroku client which allows you to do a lot of powerful things from the command line including scaling up and down, and start and stopping your application.

Instead of adding heroku as a git remote yourself you need to open a command line in your project’s directory and run

heroku login

and then

heroku create

Your application space will now have been created within Heroku automatically (no need to log in and create one first) as well as your git remote; this will have the default name of “heroku”

Deploying code is still the same as before

git push heroku master

In Heroku you do need to commit to master to have your code built and deployed, and I couldn’t find anywhere to specify a different tracking branch.

Before that we need to create the last required file:
package.json:

{
  "name": "rposbo-basic-node-hosting-options",
  "author": "Robin Osborne",
  "description": "the node.js files used in my blog post about a basic node api being hosted in various places (github, azure, heroku)",
  "version": "0.0.1",
  "engines": {
    "node": "0.8.x",
    "npm": "1.1.x"
  }
}

This file is used by npm (node package manager) to install the module dependencies for your application; e.g. express, jade, stylus. Even though our basic API project has no specifc dependencies, the file is still required by Heroku in order to define the version of node and npm to use (otherwise your application simply isn’t recognised as a node.js app).

Something to consider is that Heroku doesn’t necessarily have the same version of node installed as you might; I defined 0.8.16 and received an error upon deployment which listed the available versions (the highest at time of writing is 0.8.14). I decided to define my required version as “0.8.x” (any version that is major 0 minor 8).

However, if you define a version of node in the 0.8.x series you must also define the version of npm. A known issue, apparently. Not only that, it needs to be specifically “1.1.x”.

Add these settings into the “engines” section of the package.json file, git add, git commit, and git push to see your dashboard updated:

heroku-deploy-dashboard-1

And then your app – with a quite random URL! – is available:

heroku-deploy-result-1

If you have problems pushing due to your existing public keys not existing within heroku, run the following to import them

heroku keys:add

You can also scale up and down your number of instances using the Heroku client:

heroku ps:scale web=1

Debugging

The Heroku Toolbelt is a really useful client to have; you can check your logs with

heroku logs

and you can even leave a trace session open using

heroku logs --tail

, which is amazing for debugging problems.

The error codes you encounter are all listed on the heroku site as is all of the information on using the Heroku Toolbelt logging facility.

A quick one: if you see the error “H14”, then although your deployment may have worked it hasn’t automatically kicked off a web role – you can see this where it says “dyno=” instead of “dyno=web.1”; you just need to run the following command to start one up:

heroku ps:scale web=1

Also – make sure you’ve created a Procfile (with capitalised “P”) and that it contains

web: node app.js

Summary

Ok, so we can now easily deploy and host our API. The files that I’ve been working with throughout this post are on github; everything has been merged into master (both heroku files and web.config) so it can be deployed to any of these hosts.

There are also separate branches for Azure/Appharbor and Heroku should you want to check the different files in isolation.

Next Up

Node packages!

Your New Year Resolution 2013 – JOIN ME!

I’m bored. Let’s do something. You and me. Let’s make something fun and educational. Something you’re proud of and enjoyed being a part of.

What on Earth am I talking about?!

For example:

Code52

Downmarker

I really liked the Code52 idea, but that succumbed to everyone involved being human and not able to actually commit to a new coding project every single week for a whole year. Very very ambitious.

What amazing things came out of code52?

  • some really wonderful pieces of open source software (Markpad, one of the first, got me into Markdown which resulted in my sublimemarkpress sublimetext plugin)
  • explained concepts such as OSS and git (or github) in a very easy to understand and welcoming way; if this didn’t get you to fork a repo and learn git, nothing will
  • got people enthusiastic about collaborating on interesting and small projects; big projects are hard to commit time to and hard to join later on

I applaud their efforts and the fact that they couldn’t commit to literally 52 projects in one year in no way detracts from their achievements. Bravo, code52, bravo.

7DFPS

What about the 7 Day FPS challenge? Create a first person shooter in 7 days!

7DFPS - Sub Rosa

Sub Rosa is a fantastic (if extremely blocky) multiplayer game, with a pretty complex city to wander around and an actual 3 team “plot” (get item – from team 1 – into briefcase – held by team 2. Team 3 must stop them. Or something)

The Plan

Why don’t we start something like that? Something a bit educational and for future reference; the sort of thing you can point your new junior developer to when they have a few spare minutes to help build their skillz up, but nothing too taxing for the contributors

Potential projects

Year of GoF

How about a project for example/reference implementations of the Gang of Four Design Pattern in various languages; one a month for all of 2013 (and almost all of 2014, if you like).

For example, January could be Strategy Month, February is Observer Month. Contributers submit example implementations in a language of their choice, and the we try to document the best in each language. The examples should all be hosted on github/codeplex

Kinect 2013

Got a Kinect for Windows? What stuff can we make with it, one a month?

RaspberryPi’13

RaspberryPi’s – what are they good for? Let’s find out. 12 projects over 2013.

Anyone fancy it? Any suggestions?

TeamCity + Git + NuGet + AppCmd= automated versioned deployments V1

Attempting to implement a Continuous Deployment workflow whilst still having fun can be tricky. Even more so if you want to use reasonably new tech, and even more if you want to use free tech!

If you’re not planning on using one of the cloud CI solutions then you’re probably (hopefully) looking at something like TeamCity, Jenkins, or CruiseControl.Net. I went with TeamCity after having played with CruiseControl.Net and not liked it too much and having never heard of Jenkins until a few weeks ago.. ahem..

So; my intended ideal workflow would be along the lines of:

  • change some code
  • commit to local git
  • push to remote repo
  • TeamCity picks it up, builds, runs tests, etc (combining and minifing static files – but that’s for another blog post)
  • creates a nuget package
  • deploys to a private nuget repo
  • subscribed endpoint servers pick up/are informed of the updated package
  • endpoint servers install the updated package

Here’s where I’ve got with that so far:

 

1) On the development machine

a. Within your VisualStudio project ensure the bin directory is included

I need the compiled dlls to be included in my nuget package, so I’m doing this since I’m using the csproj file as the package definition file for nuget instead of a nuspec file.

Depending on your VCS IDE integration, this might have nasty implications which are currently out of scope of my proof of concept (e.g. including the bin dir in the IDE causes the DLLs to be checked into your VCS – ouch!). I’m sure there are better ways of doing this, I just don’t know them yet. If you do, please leave a comment!

In case you don’t already know how to include stuff into your Visual Studio project (hey, I’m not one to judge! There’s plenty of basic stuff I still don’t know about my IDE!):

i. The bin dir not included:

vs2010_bin_excluded

ii. Click the “don’t lie to me“ button*:

vs2010_dltm

iii. Select “include in project”:

vs2010_bin_include_dialog

iv. The bin dir is now included:

vs2010_bin_included

 

b. Configure source control

Set up your code directory to use something that TeamCity can hook into (I’ve used SVN and Git successfully so far)

That’s about it.

 

2) On the build server

a. Install TeamCity

i. Try to follow the instructions on the TeamCity wiki

b. Install the NuGet package manager TeamCity add in

i. Head to the JetBrains TeamCity public repo

ii. Log in as Guest

iii. Select the zip artifact from either the latest tag (or if you’re feeling cheeky the latest build):

teamcity_nuget_plugin_1

iv. Save it into your own TeamCity’s plugins folder

v. Restart your TeamCity instance

(The next couple of steps are taken from Hadi Hariri’s blog post over at JetBrains which I followed to get this working for me)

vi. Click on Administration | Server Configuration. If the plug-in installed correctly, you should now have a new Tab called NuGet

adminpanelnuget

vii. Click on the “Install additional versions of the NuGet.exe Command Line”. TeamCity will read from the feed and display available versions to you in the dialog box. Select the version you want and click Install

nugetversion

 

c. Configure TeamCity

i. Set it up to monitor the correct branch

ii. Create a nuget package as a build step, setting the output directory to a location that can be accessed from your web server; I went for a local folder that had been configured for sharing:

teamcity_nuget_1

In addition to this, there is a great blog post about setting your nuget package as a downloadable artifact from your build, which I’m currently adapting this solution to use; I’m getting stuck with the Publish step though, since I only want to publish to a private feed. Hmm. Next blog post, perhaps.

 

3) On the web server (or “management” server)

a. Install NuGet for the command line

i. Head over to http://nuget.codeplex.com/releases

ii. Select the command line download:

nuget_cmdline_dl

iii. Save it somewhere on the server and add it to your %PATH% if you like

 

b. Configure installation of the nuget package

i. Get the package onto the server it needs to be installed on

Using Nuget, install the package from the TeamCity package output directory:

nuget install "<MyNuGetProject>" –Source <path to private nuget repo>

e.g.

nuget install "RposboWeb" –Source \\build\_packages\

This will generate a new folder for the updated package in the current directory. What’s awesome about this is it means you’ve got a history of your updates, so breaking changes notwithstanding you could rollback/update just by pointing IIS at a different folder.

Which brings me on to…

ii. Update IIS to reference the newly created folder

Using appcmd, change the folder your website references to the “content” folder within the installed nuget package:

appcmd.exe set vdir "<MyWebRoot>/" /physicalpath:"<location of installed package>\content"

e.g.

appcmd.exe set vdir "RposboWeb/" /physicalpath:"D:\Sites\RposboWeb 1.12\content"

So, the obvious tricky bit here is getting the name of the package that’s just been installed in order to update IIS. At first I thought I could create a powershell build step in TeamCity which takes the version as a parameter and creates an update batch file using something like the below:

param([string]$version = "version")
$stream = [System.IO.StreamWriter] "c:\_NuGet\install_latest.bat"
$stream.WriteLine("nuget install \"<MyNugetProject>\" –Source <path to private nuget repo>")
$stream.WriteLine("\"%systemroot%\system32\inetsrv\appcmd.exe\" set vdir \"<MyWebRoot>/\" /physicalpath:\"<root location of installed package>" + $version + "\content\"")
$stream.Close()

However, my powershell knowledge is miniscule so this automated installation file generation isn’t working yet…

I’ll continue working on both this installation file generation version and also a powershell version that uses IIS7 administration provider instead of appcmd.
 

In conclusion

  • change some code – done (1)
  • commit to local git – done (1)
  • push to remote repo – done (1)
  • TeamCity picks it up, builds, runs tests, etc – done (2)
  • creates a nuget package – done (2)
  • deploys to a private nuget repo – done (2)
  • subscribed endpoint servers pick up/are informed of the updated package – done/in progress (3)
  • endpoint servers install the updated package – done/in progress (3)

 
Any help on expanding my knowledge gaps is appreciated – please leave a comment or tweet me! I’ll add another post in this series as I progress.
 

References

NuGet private feeds
http://haacked.com/archive/2010/10/21/hosting-your-own-local-and-remote-nupack-feeds.aspx
http://blog.davidebbo.com/2011/04/easy-way-to-publish-nuget-packages-with.html
 
TeamCity nuget support
http://blogs.jetbrains.com/dotnet/2011/08/native-nuget-support-in-teamcity/
 
Nuget command line reference
http://docs.nuget.org/ | http://docs.nuget.org/docs/reference/command-line-reference
 
IIS7 & Powershell
http://blogs.iis.net/thomad/archive/2008/04/14/iis-7-0-powershell-provider-tech-preview-1.aspx
http://learn.iis.net/page.aspx/447/managing-iis-with-the-iis-70-powershell-snap-in/
http://technet.microsoft.com/en-us/library/ee909471(WS.10).aspx
 

* to steal a nice quote from Scott Hanselman

AppHarbor, Heroku, Git, and the Sweet, Sweet CI Process

The background: I thought that my Mobile TFL Bus Countdown site might be suddenly very popular for a very short time (for about a weekend perhaps) and didn’t want to pay for the potential sudden jolt in hosting costs from my own servers. As such, I developed it locally using git as VCS, pushed it to my newly acquired Appharbor account, and just saw it suddenly available to browse at rposbo.apphb.com

The pitch: For your own small website/app you probably edit it locally on your PC, maybe you even have source control like a good dev, you’ll compile the code and then you’ll copy it to your hosting provider, probably using FTP/ via a web interface/ SCP/ SSH.

Then at work you’re probably shouting about how awesome CI builds are and how to introduce continuous deployment as part of a branching and build strategy.

You might even use Azure or EC2 at work, maybe for your own little home projects too. Maybe you’ve learned a bit of git but your office uses TFS (ugh) or SVN (meh).

So why not do this for your own stuff? For free? In the cloud?

Imagine the ideal workflow: make some code changes –> commit them to (D)VCS –> push them to a (remote) repo –> the push kicks off a build the committed project (git hook) –> run any associated tests, then if they pass –> deploy the app to the cloud.

That’s exactly what Appharbor and Heroku do! Let’s start with the pretty one:

Heroku

Heroku says it’s a “cloud application platform” for running scalable Ruby, Node.js, Clojure, and Java sites/apps. To create and deploy a new site is, apparently, as easy as:

$ heroku create
Created sushi.herokuapp.com | [email protected]:sushi.git

$ git push heroku master
-----&gt; Heroku receiving push
-----&gt; Rails app detected
-----&gt; Compiled slug size is 8.0MB
-----&gt; Launching... done, v1
http://sushi.herokuapp.com deployed to Heroku

So here the flow is: write some code –> commit to git –> push to Heroku –> code is built –> code is deployed. Done.

heroku homepage

The Heroku website is fantastically full of all the information you’d want to get started, and their pictorial representation of how their solution works and the various levels of databases you can buy are geek-awesome:

heroku databases

“This app needs a BAKU DATABASE!! GRRAARRRR!!” Go and have a look and bask in the beautiful piccies and animations. No wonder this is (apparently) the place to go to write and deploy cloud hosted Facebook apps.

Thanks to Heroku I’m finally beaing pushed to learn Ruby, but haven’t managed anything quite yet, hence no demo of the Heroku flow – wait a few more posts and I’ll have something Ruby-fied and certainly some Node.js as I’ve been meaning to get into that for a while, possibly even Clojure (sounds fun) and Java (old school!).

Next up is one for the .net crowd:

appharbor

Appharbor sells itself as “Azure done right” which confused me. The website itself is verrrry low on information so I just assumed it would deploy my app to Azure. Turns out I was wrong:

appharbor chat on twitter

Despite my being pedantic over their homepage tagline I took the dive and just signed up. Only once you’ve done this do you get to see the money shot – the intro video; a new MVC app in Visual Studio to EC2 cloud via git + appharbor in a matter of minutes:

Now I have my account and I have a great intro vid I just hop into my code directory;

git init
git add .
git commit –m "init"
git remote add appharbor <git repo url appharbor gave me>
git push appharbor master

And that’s it. Committed code is checked out on their servers, built, any associated tests are executed, if everything passes then it gets deployed – and you can see all this from your Appharbor account:

appharbor deployment

(mine didn’t actually have anything to build, as it was a single html page and that really basic asmx web proxy I wrote).

In conclusion; you now have absolutely no excuse to not write and deploy whatever applications you feel like writing. There is no hosting to worry about, no build server – it just works. Use Appharbor for your .Net and use Heroku as an excuse to look at their pretty pictures and learn something that’s not .Net.

I know I will.

Comments appreciated.