botframework_receive_attachment

Receiving files sent to your botframework chatbot

We’ve already looked at how a botframework bot receives messages, and even how to save those messages.

In this article I’ll show you how to handle files that are sent to your botframework chatbot.

When a user interacts with your bot, unless they’re responding to a prompt, they will cause the Message controller’s Post method to fire with an activity.

This will send a message through to your underlying IDialog or LuisDialog implementation.

MessageReceived

The method that receives the message will have the signature (though the parameter names and method name could be different):

public async Task MessageReceived(
                    IDialogContext context, 
                    IAwaitable<IMessageActivity> argument)

We can get the actual message by awaiting the IAwaitable<IMessageActivity>:

var message = await argument;

Now let’s check if the message has any attachments:

if (message.Attachments != null &&
     message.Attachments.Any())

Assuming the message has attachments we can now get the file contents; due to limitations on the total message size, the preferred way to send a file attachment is by using the ContentUrl property:

var attachmentUrl = 
   message.Attachments[0].ContentUrl;

If the file is being passed directly then you can get the data directly from the Content property, which is of type object:

var attachmentData = 
   message.Attachments[0].Content;

Depending on the type of the attachment, this object could be pretty much anything..

The attachment type is in the the ContentType property, which should be a valid mime type, e.g. “image/jpeg” or “audio/mp3”:

var attachmentType = 
   message.Attachments[0].ContentType;

Getting the file!

Most clients will save the file in their own remote hosting service (possibly backed by Amazon S3 or Azure blob storage) and send that hosting service’s URL as the ContentUrl for the attachment.

var attachmentUrl = 
   message.Attachments[0].ContentUrl;

var httpClient = new HttpClient();

var attachmentData = 
    await httpClient.GetByteArrayAsync(attachmentUrl);

Skype (and other client) Attachment Trick

If the client that’s talking to your bot has uploaded the image into a location that requires some form of authentication (such as Skype’s CDN does), then you’ll find a direct HTTP request to the attachment’s URL results in an “access forbidden” error.

You need to set the HTTP request headers to include your bot’s current credentials by creating a ConnectorClient from the message’s ServiceUrl. That ConnectorClient will have a HttpClient property which will be authenticated to access the attachments referenced from the message:

var attachmentUrl = 
   message.Attachments[0].ContentUrl;

var connector = 
    new ConnectorClient(new Uri(message.ServiceUrl));

var attachmentData = 
    connector.HttpClient.GetByteArrayAsync(attachmentUrl).Result;`

Hope you find that useful!

sqlmessagerepo

Example for Saving Botframework Messages to Sql Server

In my previous article I showed how to override various methods within botframework such that you could run some logic to save your bot’s dialogues.

In this article I’ll give an example implementation of the IMessageRepository Interface.

IMessageRepository and IMessageActivity

In the last article I introduced an IMessageRepository concept in order to show how it would be possible to hook into botframework’s various MessageReceived methods, and create our own version of PostAsync to save outgoing messages also.

Saving both incoming and outgoing messages passed an IMessageActivity object (“context”) to a MessageRepositorys AddMessageAsync method.

Given a connection string, this implementation could persist the useful parts of a message:

public class SqlMessageRepository : IMessageRepository
{
    private readonly string _connectionString;

    public SqlMessageRepository(string connectionString)
    {
        _connectionString = connectionString;
    }

    public async Task AddMessageAsync(IMessageActivity message)
    {
        using (var conn = new SqlConnection(_connectionString))
        using (var cmd = new SqlCommand())
        {
            cmd.Connection = conn;
            cmd.CommandType = CommandType.StoredProcedure;
            cmd.CommandText = "CreateMessageEntry";
            cmd.Parameters.AddWithValue("conversationId", message.Conversation.Id);
            cmd.Parameters.AddWithValue("messageId", message.Id);
            cmd.Parameters.AddWithValue("timestamp", message.Timestamp);
            cmd.Parameters.AddWithValue("content", message.Text);
            cmd.Parameters.AddWithValue("sender", message.From.Name ?? message.From.Id);
            cmd.Parameters.AddWithValue("recipient", message.Recipient.Name ?? message.Recipient.Id);
            cmd.Parameters.AddWithValue("activityType", message.GetActivityType());
            cmd.Parameters.AddWithValue("channel", message.ChannelId);

            await cmd.ExecuteNonQueryAsync();
        }
    }
}

An alternative approach using entity framework can be found over on Michael Washington’s blog.

Hopefully you found that useful!

transcribing

Transcribing messages in BotFramework

Now that you’ve deployed your well structured, exception handling, language understanding chatbot, how do you know just what people are saying to it?

Sure, you could copy and paste some logging code all over the place, but there must be a cleaner way.

In this article I’ll show you a few simple tricks to be able to save each message going to and from your botframework chatbot.

IDialog

Let’s start off by saving messages going to any dialog that implements IDialog<T>.

In order to implement IDialog<T> you only need to implement the StartAsync method; however, this isn’t much use on its own, so let’s get the dialog into a conversation loop by adding in a MessageReceivedAsync method and calling that from StartAsync and from itself:

public class TranscribingDialog : IDialog<object>
{
    public async Task StartAsync(IDialogContext context)
    {
        context.Wait(MessageReceivedAsync);
    }

    public async Task MessageReceivedAsync(IDialogContext context, IAwaitable<IMessageActivity> argument)
    {
        var message = await argument;
        context.Wait(MessageReceivedAsync);
    }
}

Right now this doesn’t do much, so let’s add in a repository for saving those messages using the ServiceResolver, as introduced in the previous article:

public class TranscribingDialog : IDialog<object>
{
    private static IMessageRepository MessageRepository
        = ServiceResolver.Get<IMessageRepository>();

    public async Task StartAsync(IDialogContext context)
    {
        context.Wait(MessageReceivedAsync);
    }

    public async Task MessageReceivedAsync(IDialogContext context, IAwaitable<IMessageActivity> argument)
    {
        var message = await argument;
        // save it!
        await MessageRepository.AddMessageAsync(message);
        context.Wait(MessageReceivedAsync);
    }
}

Let’s assume for now that the MessageRepository implementation actually persists the message somewhere; you can put it in table storage, dynamo db, mongo db, mysql; even SQL server – entirely up to you and your own bot situation. I’ll fill in this blank in another article.

That’s the easiest part done; you can do the same for any of your dialogs that implement IDialog.

Now on to the other dialog type, LuisDialog:

LuisDialog

Let’s take a look at the dialog we created in a previous article:

[Serializable]
[LuisModel("modelID", "subscriptionID")]
public class WeatherLuisDialog : LuisDialog<object>
{
    [LuisIntent("")]
    public async Task DidntUnderstand(IDialogContext context, LuisResult result)
    {
        await context.PostAsync("Could you try that again?");
        context.Wait(MessageReceived);
    }

    [LuisIntent("local.weather")]
    public async Task StartWeatherDialog(IDialogContext context, LuisResult result)
    {
        await context.PostAsync("Most likely!");
        context.Wait(MessageReceived);
    }

}

LuisDialog has a MessageReceived method which handles the parsing of your user’s input. When you use a LuisDialog your class will tend to call the underlying LuisDialog’s MessageReceived method.

See the StartWeatherDialog method above as an example of this; the WeatherLuisDialog itself has no MessageReceived method, so it’s using the MessageReceived method on the LuisDialog instead.

I’m going to create a new base class that overrides LuisDialog’s MessageReceived:

[Serializable]
public class TranscribingLuisDialog<T> : LuisDialog<T>
{
    private static IMessageRepository MessageRepository => ServiceResolver.Get<IMessageRepository>();

    protected override async Task MessageReceived(IDialogContext context, IAwaitable<IMessageActivity> item)
    {
        var message = await item;
        // save it!
        await MessageRepository.AddMessageAsync(message);
        await base.MessageReceived(context, item);
    }
}

This will get the contents of the user input, persist it, then call the base class’s (LuisDialog’s) MessageReceived method.

To have our own dialogs use this new class is simply a case of changing the class definition to use TranscribingLuisDialog instead of LuisDialog:

public class WeatherLuisDialog : TranscribingLuisDialog<object>
{
    ...
}

That’s all! Now all dialogs that need to use LUIS will have their incoming messages saved. This is good progress! We’ve now got a solution for both IDialog and LuisDialog classes.

The next section is a big one – it’s going to take a while, so take a deep breath; here we go:

Prompt

When you send the user a prompt their response doesn’t fire the same MessageReceived flow as before; instead they’re kept in a mini flow where their input must be a valid response to the prompt. Capturing this input is a little trickier, but not impossible.

If we look at the Prompt class itself, we can see the closure that handles the response is in an abstract class – yay! Unfortunately, all of the classes that extend it are sealed – boo! (All in that same file)

This limits our options; since I can’t override the implementations, I will create duplicate classes that implement our own logic – and also extend the abstract class – to do the message saving.

There are a couple of approaches to this, and I’m going to go with basically overriding Prompt with duplicate code but add in one extra line to save the message:

[Serializable]
public abstract class TranscribingPrompt<T, TOptions> : Prompt<T, TOptions>, IDialog<T>
{

    private static IMessageRepository MessageRepository => ServiceResolver.Get<IMessageRepository>();

    protected TranscribingPrompt(PromptOptions<TOptions> promptOptions) : base(promptOptions) { }

    async Task IDialog<T>.StartAsync(IDialogContext context)
    {
        await context.PostAsync(this.MakePrompt(context, promptOptions.Prompt, promptOptions.Options, promptOptions.Descriptions));
        context.Wait(MessageReceived);
    }

    private async Task MessageReceived(IDialogContext context, IAwaitable<IMessageActivity> message)
    {
        var content = await message;
        // save it!
        await MessageRepository.AddMessageAsync(content);

        T result;
        if (TryParse(content, out result))
        {
            context.Done(result);
        }
        else
        {
            --promptOptions.Attempts;
            if (promptOptions.Attempts > 0)
            {
                await context.PostAsync(this.MakePrompt(context, promptOptions.Retry ?? promptOptions.DefaultRetry, promptOptions.Options));
                context.Wait(MessageReceived);
            }
            else
            {
                //too many attempts, throw.
                await context.PostAsync(this.MakePrompt(context, promptOptions.TooManyAttempts));
                throw new TooManyAttemptsException(promptOptions.TooManyAttempts);
            }
        }
    }
}

Since this implements IDialog I need to implement StartAsync. Since it extends Prompt I can use Prompt’s existing virtual MakePrompt method to actually generate the prompt message.

The class above is a direct copy of Prompt from that huge PromptDialog.cs file.

I’ve used the same approach to get a MessageRepository (i.e., the ServiceResolver) and I’m calling AddMessageAsync inside of the MessageReceived method so that the response to the prompt is captured.

To use this I need to replace usages of Prompt with TranscribingPrompt; this is the time-consuming bit – since the usages are all in the sealed classes we need to get copy-pasting…

TranscribingPrompt Example: Int64

I’m not going to paste the entire file that needs duplicating in here, since it’s HUGE, but here’s an example of what needs doing for each prompt type – the example is for PromptInt64:

// BEFORE
[Serializable]
public sealed class PromptInt64 : Prompt<Int64, Int64>
{
    public PromptInt64(string prompt, string retry, int attempts)
        : base(new PromptOptions<long>(prompt, retry, attempts: attempts))
    {
    }

    protected override bool TryParse(IMessageActivity message, out Int64 result)
    {
        return Int64.TryParse(message.Text, out result);
    }
}

The only real difference between the various types of Prompt is their override of Prompt’s abstract method TryParse.

We can create our new TranscribingPromptInt64 by merely extending the TranscribingPrompt abstract class in a new file which duplicates PromptInt64:

// AFTER
[Serializable]
public class TranscribingPromptInt64 : TranscribingPrompt<Int64, Int64>
{
    public TranscribingPromptInt64(string prompt, string retry, int attempts)
        : base(new PromptOptions<long>(prompt, retry, attempts: attempts))
    {
    }

    protected override bool TryParse(IMessageActivity message, out Int64 result)
    {
        return Int64.TryParse(message.Text, out result);
    }
}

Notice the class is extending TranscribingPrompt instead of Prompt

Repeat this process for – wait for it:

  • PromptString
  • PromptConfirm
  • PromptDouble
  • PromptChoice
  • PromptAttachment

Boom. Lots of copy-pasting.

Since Resources are internal in resx files, you’ll find a few errors in your PromptConfirm after you copy and paste.

Just create your own Resources using use some static properties on a new Resources class:

public class Resources
{
    public static string TooManyAttempts = "too many attempts";
    public static string PromptRetry = "I didn't understand. Say something in reply.";
    public static string[] MatchYes = { "yes", "y", "sure", "ok", "yep", "1" };
    public static string[] MatchNo = { "no", "n", "nope", "2" };
}

You’ll also need to remove the uses of SplitList from Options and Patterns in your new version of PromptConfirm.

Prompt dialog

You’ll also need your own PromptDialog class, so that you can change the usual static methods that look like this:

public static void Number(IDialogContext context, ResumeAfter<long> resume, string prompt, string retry = null, int attempts = 3)
{
    var child = new PromptInt64(prompt, retry, attempts);
    context.Call<long>(child, resume);
}

To methods that look like this:

public static void Number(IDialogContext context, ResumeAfter<long> resume, string prompt, string retry = null, int attempts = 3)
{
    // Call our transcribing version instead
    var child = new TranscribingPromptInt64(prompt, retry, attempts);
    context.Call<long>(child, resume);
}

Notice I’ve changed PromptInt64 to TranscribingPromptInt64

As such, I’ll create a new TranscribingPromptDialog and create my own versions of the following methods, changing the type of the underlying Prompt they use:

  • Text
  • Confirm
  • Number (see above for one of the implementations)
  • Choice
  • Attachment

Using the TranscribingPrompt

Now instead of calling the static method Prompt.Number I can call TranscribingPrompt.Number, e.g.:

PromptDialog.Number(context, AfterConfirming, "think of a number");

becomes

TranscribingPromptDialog.Number(context, AfterConfirming, "think of a number");

Saving messages sent by the Bot

We’ve now captured the messages coming in to our bot, but what about the bot’s replies? If we include these then it’s easy enough to export an entire conversation.

To send a message to a user we use the context.PostAsync method.

Again, a couple of approaches for altering this, but I’m just going to create a couple of extension methods on IBotToUser:

public static class BotToUserEx
{
    private static IMessageRepository MessageRepository => ServiceResolver.Get<IMessageRepository>();

    public static async Task PostToUserAndTranscribeAsync(this IBotToUser botToUser, string text)
    {
        var message = botToUser.MakeMessage();
        message.Text = text;

        await MessageRepository.SaveMessage(message);
        await botToUser.PostAsync(message);
    }


    public static async Task PostToUserAndTranscribeAsync(this IBotToUser botToUser, IMessageActivity message)
    {
        await MessageRepository.SaveMessage(message);
        await botToUser.PostAsync(message);
    }
}

To use these, simply replace your calls to context.PostAsync with context.PostToUserAndTranscribeAsync.

Summary

You should now have a solution for saving user messages coming in to your IDialog dialogs, your LuisDialogs, and your Prompts, and you should also be able to save the messages sent by the bot.

Hopefully you’ve found this botframework message transcribing walkthrough useful and interesting.

I’d love to know how you’re solving this problem yourself, so please let me know.

ldnbotframework_2_intro

LDNBotFramework Meetup #2 Retrospective: From Zero to Echo via BotFramework

I had an awesome time presenting my session at the second LDNBotframework meetup!

This time I ordered the correct amount of pizza, and let people loose on it at the right time; so no fridge full of pizza for breakfast the next day!

From Zero to Echo via BotFramework

LDNBotFramework #2 - From Zero to Echo via BotFramework

Tight deadline

Having scheduled this session weeks ago, unfortunately my preferred presenters were not able to make it; as such I had just over a week to decide what to do about the already scheduled meetup: should I cancel it and lose momentum, or should I do a session myself?

After a couple of days of uncertainty I decided to take the plunge and commit to doing the evening on my own, leaving me with less than a week to come up with some content!

I’ve spoken at plenty of conferences and meetups in the past, but I’ve always had months of notice to prepare and practice.

Echo… Echo.. Echo..

I decided to investigate my newly acquired Amazon Echo Dot’s capabilities, but didn’t fancy using AWS and lambdas, since they don’t make for a fun presentation.

As such, I committed to developing an Alexa Skill without writing any code, and then hook that skill into an existing botframework chatbot. No slides, no presentation, just working through the process, live.

What could possibly go wrong??

Photos and Video

Given that it was me doing the talking, no one else was taking pictures, tweeting, nor checking the video feed was actually working and recording.. which meant that there is no record of the event ever having taking place. Ugh.

I’m gutted about that, since the session went well otherwise. I’m going to write it all up as a separate article instead of go into too much detail here.

If you run a meetup yourself, I’ll happily go through the session again – so long as you can record it for me!

Summary

Live-developing an Alexa skill, complete with intentional “failures”, and using my laptop to proxy calls to an existing botframework chatbot was an interesting concept and it seems that the attendees enjoyed it as well.

Given that this meetup only had one session instead of three, it was much more relaxed and had plenty of time to chat and network whilst enjoying the pizza and beer provided by the generous hosts, JustEat.

I’m looking forwards to the next meetup; don’t forget to join the group so you know when it will be!

Botframework + Alexa = a fun meetup session!

serializable_error

BotFramework: Avoiding have to make everything [Serializable]

In the last article I touched on how IoC is used within botframework using Autofac.

If you’re comfortable with IoC, you probably started to enhance your dialogs using constructor injection, so that you can put your conversation logic outside of your entry point and just start coding without needing an implementation to exist yet, e.g.,

[Serializable]
public class ResolvingDialog : IDialog<object>
{
    private readonly IMessageRepository _messageRepository;

    public ResolvingDialog(IMessageRepository messageRepository)
    {
        _messageRepository = messageRepository;
    }

    public async Task StartAsync(IDialogContext context)
    {
        await context.PostAsync($"Hi!");
        context.Wait(MessageReceivedAsync);
    }

    public async Task MessageReceivedAsync(IDialogContext context, IAwaitable<IMessageActivity> argument)
    {
        var message = await argument;
        await _messageRepository.SaveMessage(message);

        await context.PostAsync($"working now");
        context.Wait(MessageReceivedAsync);
    }
}

Then you can just wire up the interface resolution by defining the default implementation for the interface in your IoC configuration e.g.

var builder = new ContainerBuilder();
builder.RegisterType<MessageRepository>().As<IMessageRepository>();

...

var container = builder.Build();

Easy, right? Exactly how you’d do it for a Controller in a WebAPI or MVC project.

However..

serializable all the things please!

Ah. This appears to be due to the dialog and all their dependencies needing to be serialized such that the conversation can avoid needing “sticky sessions” to keep the context between messages.

ye olde service locator pattern

One way to avoid this that I’ve been using is something like a Service Locator/Factory pattern.

Yes, people don’t like the Service Locator pattern, so if you’re one of them please do suggest alternative solutions! I’d love to hear them.

Instead of using constructor injection, we can define a static property on the dialog which is resolved via a factory class to retrieve the concrete implementation via our IoC configuration.

Lots of long words there, so let’s look at some code instead!

The dialog now has a property for each interface we need an implementation for, resolved using another class:

[Serializable]
public class ResolvingDialog : IDialog<object>
{
    private static readonly IMessageRepository _messageRepository
       = ServiceResolver.Get<IMessageRepository>();

    public async Task StartAsync(IDialogContext context)
    {
        await context.PostAsync($"Hi!");
        context.Wait(MessageReceivedAsync);
    }

    public async Task MessageReceivedAsync(IDialogContext context, IAwaitable<IMessageActivity> argument)
    {
        var message = await argument;
        await _messageRepository.SaveMessage(message);

        await context.PostAsync($"working now");
        context.Wait(MessageReceivedAsync);
    }
}

That ServiceResolver class merely hooks into AutoFac and returns a concrete implementation:

public static class ServiceResolver
{
    public static IContainer Container;

    public static T Get<T>()
    {
        using (var scope = Container.BeginLifetimeScope())
            return scope.Resolve<T>();
    }
}

Now we just set the resolver’s static Container in a bootstrapping class (or just in the Global.asax.cs):

var builder = new ContainerBuilder();
builder.RegisterType<MessageRepository>().As<IMessageRepository>();
...
var container = builder.Build();

ServiceResolver.Container = container;

Now running that again gives us:

not serializable? not a problem

static?! Service Locator?!

Yeah, the service resolution class is static. The properties in the dialog are static. I’m using a Service Locator pattern.

This may not seem clean, but it means these dependencies do not need to be [Serializable]. Having a Container property on ServiceResolver means we can pass in a container full of mocks/ stubs/ fakes/ substitutes to facilitate testing whilst using this static class.

Summary

We’ve now got a way of keeping the dialog classes reasonably empty, merely handling initial routing like a Controller would.

This should hopefully allow for a nicer structure in your bot code. I’d be interested in hearing how you’ve solved this problem in your own projects, since – let’s face it – I’m using a Service Locator and static all over the place..

botframework_error_custom_message

BotFramework Exception Wrapping and Custom Error Message

During your time building and debugging your botframework chatbot, you’ve probably already come across the annoyance of seeing a huge error message with a ridiculous stack trace that doesn’t seem to relate to your error, or even just a plain old HTTP 500 and big red exclamation mark..

emulator red exclamation mark

Perhaps you’ve deployed to a production environment and are stuck with the stock error message:

Sorry, my bot code is having an issue

In this article I’ll show you how to 1) display useful exception information, and 2) override the default error message sent to the user

In order to capture the short, top level, exception only, we need to wrap the root Dialog with an exception handling one; let’s start with that.

Exception Handler Dialog

By default, when you start a conversation your MessagesController will look a bit like this:

public async Task<HttpResponseMessage> Post([FromBody]Activity activity)
{
    if (activity.Type == ActivityTypes.Message)
    {
        await Conversation.SendAsync(activity, () => new MyRootDialog());
    }
    else
    {
        HandleSystemMessage(activity);
    }
    var response = Request.CreateResponse(HttpStatusCode.OK);
    return response;
}

When an exception occurs within your root dialog you’ll either see nothing (i.e. an HTTP 500 error) or a huge stack trace. To try to handle both of these scenarios, we need to wrap the root dialog with an exception catching one.

This dialog wrapping code isn’t mine – it came from a GitHub issue, but I use it a lot so am sharing the love:

This dialog will enable us to wrap our root dialog’s StartAsync and ResumeAsync methods in try catch blocks:

[Serializable]
public class ExceptionHandlerDialog<T> : IDialog<object>
{
    private readonly IDialog<T> _dialog;
    private readonly bool _displayException;
    private readonly int _stackTraceLength;

    public ExceptionHandlerDialog(IDialog<T> dialog, bool displayException, int stackTraceLength = 500)
    {
        _dialog = dialog;
        _displayException = displayException;
        _stackTraceLength = stackTraceLength;
    }

    public async Task StartAsync(IDialogContext context)
    {
        try
        {
            context.Call(_dialog, ResumeAsync);
        }
        catch (Exception e)
        {
            if (_displayException)
                await DisplayException(context, e).ConfigureAwait(false);
        }
    }

    private async Task ResumeAsync(IDialogContext context, IAwaitable<T> result)
    {
        try
        {
            context.Done(await result);
        }
        catch (Exception e)
        {
            if (_displayException)
                await DisplayException(context, e).ConfigureAwait(false);
        }
    }

    private async Task DisplayException(IDialogContext context, Exception e)
    {

        var stackTrace = e.StackTrace;
        if (stackTrace.Length > _stackTraceLength)
            stackTrace = stackTrace.Substring(0, _stackTraceLength) + "…";
        stackTrace = stackTrace.Replace(Environment.NewLine, "  \n");

        var message = e.Message.Replace(Environment.NewLine, "  \n");

        var exceptionStr = $"**{message}**  \n\n{stackTrace}";

        await context.PostAsync(exceptionStr).ConfigureAwait(false);
    }
}   

Now we use this dialog in our MessagesController instead of our previous “root” dialog. Change the conversation initiation from (assuming your main dialog is called MyRootDialog and implements IDialog<object>):

await Conversation.SendAsync(activity, () => new MyRootDialog());

to

await Conversation.SendAsync(activity, () =>
   new ExceptionHandlerDialog<object>(
      new MyRootDialog(),
      displayException: true));

This will actually capture the top level exception, and optionally return it as a message; just configure displayException to be true when debugging and false when deployed.

You can also add in logging in the catch section of the ExceptionHandlerDialog.

Now your real exception should be both exposed and summarised.

shortened botframework error message

Sorry, my bot code is having an issue

Next let’s override the default error message. This requires a little detective work, digging into the code in the GitHub repo. Let’s get our Sherlock hats on!

The exception message is sent by the PostUnhandledExceptionToUserTask class, which has this method:

async Task IPostToBot.PostAsync<T>(T item, CancellationToken token)
{
    try
    {
        await this.inner.PostAsync<T>(item, token);
    }
    catch (Exception error)
    {
        try
        {
            if (Debugger.IsAttached)
            {
                await this.botToUser.PostAsync($"Exception: {error}");
            }
            else
            {
                await this.botToUser.PostAsync(this.resources.GetString("UnhandledExceptionToUser"));
            }
        }
        catch (Exception inner)
        {
            this.trace.WriteLine(inner);
        }

        throw;
    }
}

This class can be found in the DialogTask file in the BotBuilder github repository

Notice the reference to this. resources. GetString("UnhandledExceptionToUser"), which gets the message from the resource file for the current context’s language; for English, this is set to:

<data name="UnhandledExceptionToUser" xml:space="preserve">
   <value>Sorry, my bot code is having an issue.</value>
</data>

As can be found in the Resources.resx file

The PostUnhandledExceptionToUserTask class is wired up using Autofac for IoC (Inversion of Control, out of scope for this article, but essentially allows resolution of an interface to an implementation via configuration in the “hosting” code) – notice the last few lines of the following code from the DialogModule.cs Autofac module:

builder
.Register(c =>
{
    var cc = c.Resolve<IComponentContext>();

    Func<IPostToBot> makeInner = () =>
    {
        var task = cc.Resolve<DialogTask>();
        IDialogStack stack = task;
        IPostToBot post = task;
        post = new ReactiveDialogTask(post, stack, cc.Resolve<IStore<IFiberLoop<DialogTask>>>(), cc.Resolve<Func<IDialog<object>>>());
        post = new ExceptionTranslationDialogTask(post);
        post = new LocalizedDialogTask(post);
        post = new ScoringDialogTask<double>(post, stack, cc.Resolve<TraitsScorable<IActivity, double>>());
        return post;
    };

    IPostToBot outer = new PersistentDialogTask(makeInner, cc.Resolve<IBotData>());
    outer = new SerializingDialogTask(outer, cc.Resolve<IAddress>(), c.Resolve<IScope<IAddress>>());

    // -- we want to override the next line:
    outer = new PostUnhandledExceptionToUserTask(outer, cc.Resolve<IBotToUser>(), cc.Resolve<ResourceManager>(), cc.Resolve<TraceListener>());

    outer = new LogPostToBot(outer, cc.Resolve<IActivityLogger>());
    return outer;
})
.As<IPostToBot>()
.InstancePerLifetimeScope();

We need to override this and have our own class resolved for the IPostToBot interface instead. However, we still need all of the other “wrapper” dialog tasks, so it’s actually easier to copy and paste this into our own AutoFac module, just changing that one line.

Default Exception Message Override Module

Let’s create a DefaultExceptionMessageOverrideModule AutoFac module that looks like this:

public class DefaultExceptionMessageOverrideModule : Module
{
    protected override void Load(ContainerBuilder builder)
    {
        builder
            .Register(c =>
            {
                var cc = c.Resolve<IComponentContext>();

                Func<IPostToBot> makeInner = () =>
                {
                    var task = cc.Resolve<DialogTask>();
                    IDialogStack stack = task;
                    IPostToBot post = task;
                    post = new ReactiveDialogTask(post, stack, cc.Resolve<IStore<IFiberLoop<DialogTask>>>(),
                        cc.Resolve<Func<IDialog<object>>>());
                    post = new ExceptionTranslationDialogTask(post);
                    post = new LocalizedDialogTask(post);
                    post = new ScoringDialogTask<double>(post, stack,
                        cc.Resolve<TraitsScorable<IActivity, double>>());
                    return post;
                };

                IPostToBot outer = new PersistentDialogTask(makeInner, cc.Resolve<IBotData>());
                outer = new SerializingDialogTask(outer, cc.Resolve<IAddress>(), c.Resolve<IScope<IAddress>>());

                // --- our new class here
                outer = new PostUnhandledExceptionToUserOverrideTask(outer, cc.Resolve<IBotToUser>(),

                    cc.Resolve<ResourceManager>(), cc.Resolve<TraceListener>());
                outer = new LogPostToBot(outer, cc.Resolve<IActivityLogger>());
                return outer;
            })
            .As<IPostToBot>()
            .InstancePerLifetimeScope();
    }
}

Notice the line which previously registered PostUnhandledExceptionToUserTask now uses our own PostUnhandledExceptionToUserOverrideTask.

Post Unhandled Exception To User Override Task

That new file looks like this – should be very familiar!

public class PostUnhandledExceptionToUserOverrideTask : IPostToBot
{
    private readonly ResourceManager resources;
    private readonly IPostToBot inner;
    private readonly IBotToUser botToUser;
    private readonly TraceListener trace;

    public PostUnhandledExceptionToUserOverrideTask(IPostToBot inner, IBotToUser botToUser, ResourceManager resources, TraceListener trace)
    {
        SetField.NotNull(out this.inner, nameof(inner), inner);
        SetField.NotNull(out this.botToUser, nameof(botToUser), botToUser);
        SetField.NotNull(out this.resources, nameof(resources), resources);
        SetField.NotNull(out this.trace, nameof(trace), trace);
    }

    async Task IPostToBot.PostAsync<T>(T item, CancellationToken token)
    {
        try
        {
            await inner.PostAsync(item, token);
        }
        catch (Exception)
        {
            try
            {
                await botToUser.PostAsync("OH NOES! Put your own message here, or logic to decide which message to show.", cancellationToken: token);
            }
            catch (Exception inner)
            {
                trace.WriteLine(inner);
            }

            throw;
        }
    }
}

Wiring it all up

Now that we’ve got a new implementation for IPostToBot it has to be wired up in order to override the default one from DialogModule.cs. To do this, put the following code wherever you already do your IoC, or in the Global.asax.cs if you’re not doing it anywhere else yet.

var builder = new ContainerBuilder();
builder.RegisterModule(new DefaultExceptionMessageOverrideModule());
builder.Update(Conversation.Container);

botframework custom exception message

Summary

With these changes in place you can finally avoid scrolling through many screens of stack trace that doesn’t give you any useful information, and customise the friendly error message sent to your user.

ldnbotframework_1_team

Inaugural LDNBotFramework Meetup Retrospective

We recently wrapped up the first #LDNBotFramework meetup! There were a lot of lessons learned for me, as a first time meetup organiser, and overall I think it was a success.

The venue was great; the big video wall in JustEat’s Fleet Place House office combined with a mic and speaker system that “just works”, a stocked beer fridge and far too much pizza, all made for a perfect tech meetup setup.

LDNBotFramework Team #1!

Erdeniz Hassan, Simon Michael, Robin Osborne, and David Low

Thanks to @beanbaglabs for this group pic!

Sessions

We were very lucky to have representation from Microsoft to kick things off, then some great insights from SkyScanner’s case study, finishing off with a lightning talk on user expectations from JustEat.

Microsoft Bot Framework and Cognitive Services

Simon Michael (@simon_mich) – Senior Tech Evangelist at Microsoft UK – had the obligatory BotFramework 101 (it was the first meetup after all), where we learned more about “Conversation as a Platform” including Microsoft Bot Framework and adding intelligence to our bots/apps using cognitive services.

LDNBotFramework 1 - Simon Michael: 10 ZetaBytes!

LDNBotFramework 1 -

LDNBotFramework 1 -

LDNBotFramework 1 -

Conversational Interface Case Study

David Low (@daviddlow) – SkyScanner – guided us through a case study of their growing use of conversational interfaces, and how they used BotFramework with some nodejs magic sprinkles to create a whole new way of booking your next holiday.

LDNBotFramework 1 - David Low

LDNBotFramework 1 - David Low

LDNBotFramework 1 - David Low

LDNBotFramework 1 - David Low

LDNBotFramework 1 - David Low

LDNBotFramework 1 - David Low

User Expectations of Bots

Erdeniz Hassan (@justeat_tech) – Senior UX Researcher at JustEat – wondered if chat bots will soon reign supreme! He shared some of the findings from discussions with JustEat customers when developing the Facebook chat bot and the Help & Support chat bot, and shed some light on the challenges faced and the lessons learned.

LDNBotFramework 1 - Erdeniz Hassan

LDNBotFramework 1 - Erdeniz Hassan

Video

The evening was recorded, and although you can see our beautiful faces in the photos, the video is audio and the presentations only.

Summary

From my perspective, a fantastic first session with a great group of attendees creating a thought provoking atmosphere with their questions and general chat.

I’m very much looking forwards to the next one!

Which reminds me: please sign up and rsvp!

Ordered far too much pizza..

Next time, we’ll walk the streets of London and donate it to the needy..

LDNBotFramework 1 - Leftovers..

Whoops..

luisdialog_luisresult_1

Implementing LUIS Routing within BotFramework

In the previous LUIS article, I introduced how to set up and train (and publish) a LUIS language interpreting web service, getting an “intent” and extracting “entities” from a given “utterance”

In this article I’ll use LUIS to enhance your botframework chatbot

If you haven’t done so already, create your bot using botframework, and set up a LUIS application.

Now that we’ve laid the foundations, let’s build a house. A ..um. chatbot house.. yeah.

LUIS entry point

Botframework has a special type of Dialog especially for interacting with LUIS; the imaginatively titled LuisDialog!

Create a new class that extends LuisDialog<object> and add the LuisModel attribute, adding in your Model Id and Subscription Key from LUIS; an easy way to get these values is to hit “PUBLISH” and “Publish web service” for your LUIS app:

LUIS service URL

Then check the URL and copy the SubscriptionKey and id (for your ModelId) values.

[LuisModel("<LuisModelId>","<LuisSubscriptionKey>")]
public class RootLuisDialog : LuisDialog<object>
{
}

(You can get these values elsewhere within the LUIS pages, but this was the easiest method for me)

Now edit your MessagesController such that the conversation is now started using the static Conversation.SendAsync method, passing in the incoming message activity as the first param and your new dialog as the result of a function for the 2nd param:

await Conversation.SendAsync(activity, () => new RootLuisDialog());

Your MessagesController should now look a bit like this:


public async Task<HttpResponseMessage> Post([FromBody]Activity activity)
{
    if (activity.Type == ActivityTypes.Message)
    {
        // return our reply to the user
        await Conversation.SendAsync(activity, () => new RootLuisDialog());
    }
    else
    {
        HandleSystemMessage(activity);
    }
    var response = Request.CreateResponse(HttpStatusCode.OK);
    return response;
}

Right now there won’t be any actual functionality since our new dialog class is empty. Let’s rectify that!

As well as having an attribute to map our LUIS app to a Dialog in our codebase, there is another attribute to map a LUIS intent to a method; the attribute just needs the intent name from our LUIS app:

[LuisIntent("local.weather")]
public async Task StartWeatherDialog(IDialogContext context, LuisResult result)
{
}

The method signature needs to take an IDialogContext and a LuisResult and return a Task.

Assuming you’ve TRAINed and PUBLISHed your LUIS app, and you’ve got the Model, Subscription, and Intent attribute values entered correctly, then we can set a breakpoint in that empty method and start debugging.

Open up the bot emulator and enter a phrase that you know LUIS will map to your intent, e.g. “will it rain in London tomorrow?”

As per usual the message hits theMessagesController, but this time we start a Conversation with the RootLuisDialog.

Since you’ve got your LuisModel attribute set up this message will now be posted to your LUIS app’s endpoint.

The resulting json is used to route the response to a method; if the response from LUIS contains the intent local.weather and if I have a method with a matching LuisIntent attribute, then the json response will be mapped to a LuisResult and passed in as a parameter.

Let’s dig into the LuisResult a bit – I’ll ask “is it going to rain in bristol this weekend?”:

LuisResult properties

That’s our query – nice. There are our entities! How handy.

LuisResult entities

What else is in there?

LuisResult intents

Looks like we have the three intents as defined within the LUIS app:

LuisResult intent local.news

local.news gets the lowest score..

LuisResult intent None

Then None

LuisResult local.weather

And local.weather gets the highest score – as it should do. Which is why the method that gets called is the one with a `LuisIntent` attribute configured with the value `local.weather`

No match?

You should always be sure to add in a method that will be called if LUIS fails to match the input to any intent. Use the attribute with an empty string if you haven’t set up a “catch all” in your LUIS app, and add on you “catch all” intent as well if you have one, e.g.

[LuisIntent("")]
[LuisIntent("None")]
public async Task DidntUnderstand(IDialogContext context, LuisResult result){
}

This will ensure you’re always able to respond, perhaps with a “Sorry, I didn’t understand that – can you try saying it in a different way?”

Summary

Hopefully this article has shown you how to take a LUIS app and wire it up to your botframework code, extracting the entities from the utterance and using these to create a dynamic response.

Let me know how you get on!

highres_454824208

The Chatbot Revolution and The London BotFramework Meetup Group

If you’re one of the few people who have managed to avoid the onslaught of Chat Bot related articles over the past year, then let me start by way of an introduction; a chatbot is, in it’s most basic form, a computer program that can mimic basic human conversations.

This isn’t particularly new or exciting; this sort of chat bot has been around since the 70s. What is new and exciting is the recent development in systems and frameworks which make creating your own chat bot easy enough that you can focus on the quality of the interaction with the end user instead of wallowing in the technical considerations.

There is a website with a form to fill in that will give you a chat bot at the end of it, all the way through to an enterprise company’s framework for building your bespoke conversational interface from scratch.

The term “Conversational Interface” has become significantly more popular recently as well; all those Web Designers, UI Developers, UX Developers who are taking this opportunity to reinvent classic, almost stale, concepts from e-commerce into the limited flow of an instant message are paving the way for the next iteration of how we interact with a company or brand.

We’re already more than happy to throw a tweet at a company who may have wronged us, instead of calling their helpline or emailing their customer support; and we expect, and usually receive, a personalised and rapid response. Our expectations are high, and the next generation of chatbots and conversational interfaces can help to deliver on this.

Whilst there are a brave few who have ventured forth with a chat bot – Uber, Dominos, SkyScanner, Shopify (although they’re quick to tell you it’s not a chatbot, but a conversational interface to a product catalogue, and a fine one it is at that) – the rest of us have a downpour of resources, information, documentation, APIs, EULAs, T&Cs, and no idea where to start.

This is one of the reasons I decided to start the London (LDN) BotFramework Meetup group. BotFramework is Microsoft’s emerging platform for developers to create their bespoke chatbots, and it’s this technology – along with a general conversation on chatbot futurism and various UX/R&D considerations – that will be focused on at the meetups.

We already have a good relationship with Microsoft’s BotFramework product team, and even have a Senior Technical Evangelist from Microsoft UK as the opening speaker for the inaugural session on Wednesday 26th October.

I’ve reached out to companies who have blazed that conversational interface trail and have a few already lined up to take us through in-depth case studies of both their tech and design considerations, challenges, and results.

It is my hope that the meetup group will help to mould the burgeoning future of the chatbot, via the conversations during and after the meetups, into something both tangible and wonderful.

Anyone interested in meeting other bot enthusiasts, developers, designers, or conversational interface nerds, can join the group over at the LDNBotFramework meetup.com page.

Submissions for talks are more than welcome via the LDNBotFramework meetup.com page, @LDNBotFramework on twitter, or by contacting me (Robin Osborne, the group organiser) directly: @rposbo

highres_454824208

luis_retrained_6

LUIS Natural Language Service for BotFramework

Creating a hosted bot using Microsoft’s botframework couldn’t be easier; hopefully you’ve had a chance to create one already, and if not there’s a great introduction to creating your first bot right here.

In the previous article we saw how to create a QnA (aka FAQ/Knowledge Base) service using a little known QnA Maker service of the botframework.

In this post we’ll start to create a more intelligent bot; one which can appear to understand the intent of the incoming message and extract specific key variables from it.

Understanding the intent of a piece of text is a really tricky problem to solve; totally out of scope for this article, and for most bot projects! Luckily, the botframework has a friend called LUIS – the Language Understanding Intelligence Service.

LUIS exists as a standalone product, and can be called with a query (text input) to give a result (matching “intent” with a “confidence” score, as well as any extracted “entities” – we’ll get on to these later).

The botframework can support LUIS such that the intent response from LUIS can map directly to a class with a matching attribute value, such as

[LuisModel("<LuisAppId>","<LuisModelId>")]

It can even map down to a specific method within that class using another attribute:

[LuisIntent("<IntentName>"]

Given this magic, it’s no longer obvious what the entry points are for your application which can make developing and debugging the logic within your bot application really tricky..

We’ll get on to the C# and botframework magic later, but first let’s look at getting started with LUIS and how to use it.

Setting up LUIS

Head over to Luis.ai and log in with your Microsoft account. After a brief intro wizard thingy, you’ll see a screen like this

luis inital page

Here is where we can define our apps. Create a new app by clicking either “New App” -> “New Application” or “Build a new language understanding application” – both open the same overlay:

luis create app popup

  • Give it a name and choose from IoT, Bot, Mobile Application, or Other (honestly have no idea if this info changes anything; AFAIK it’s just data gathering); I just choose “Bot”.
  • Choose your “Application domains” – again, this doesn’t alter your experience, so just go with whatever you prefer; I chose “Entertainment”.
  • “Application description”? “rposbo’s demo luis bot thingy” is all I put 🙂
  • “Culture” might actually be used, but I’ll get on to that in another article.

Now that you’ve got an empty app, you need to set up some intents. You should see a screen that looks a bit like this:

created app

Click the plus sign next to INTENT to create a new intent; in terms of the botframework we can just think of an intent as a category which we’re going to teach LUIS to map sentences to, and extract meaningful terms from.

create intent

Give it a name – for example “local.news” – and enter an example “utterance”:

“Utterances” are the short sentences which are used to map the user’s input to an intent. You can define many different ways of saying (uttering) something which you intend to mean the same thing.

For my “local.news” example, I’m going to use the utterance “what’s happening in London today?”

Now I’ve got an intent name – “local. news” – and an example utterance – “what’s happening in London today?” – I can hit “Save” to create this intent.

I’m going to create one more intent – “local.weather” – and seed it with the utterance “what’s the weather like in London today?”

(Notice these utterances have some variables in them – “London” and “today” – which we’ll come back to later on)

The LUIS web service

We’re nearly ready to give this very basic logic an endpoint; to do this we have to TRAIN and PUBLISH the LUIS app. Every time you want to update your LUIS app – if you’ve added more utterances or intents for example – you always need to TRAIN and PUBLISH again.

Firstly, hit TRAIN in the bottom left and wait for the spinny loader thing to finish; the more utterances you have, the longer this process takes.

train fail

Uh oh. You do need to define a minimum of two utterances per intent. Add a couple more and hit TRAIN again, then once that’s done hit PUBLISH to get this overlay:

app publish pop-up

Now hit “publish web service” to create or update your LUIS app’s web service.

app published

Testing LUIS

Now that we’ve got a web service, let’s try it out! In that same overlay you should see a URL for testing and a query input box. If I type in “will it be sunny in Ipswich tomorrow?” and hit TEST, then I should see a new tab open with a json response that looks like this:

json response from LUIS endpoint

Notice the confidence scores are really low – they’re on a scale of 0 to 1. Basically, LUIS was not able to match that input to either intent with any degree of confidence, so the “None” intent got the highest score (0.28 – just above “local.weather” with 0.26, which is promising).

We can easily fix this by submitting the exact phrase into the LUIS app homepage and selecting our prefered intent from the dropdown:

retrain

After doing so that same sentence will now match with a confidence of “1” – i.e., super premo number one confidence!

retrained

Remember what I said earlier? Before any training changes (i.e., new or recategorised utterances) can take effect, you need to TRAIN and PUBLISH, otherwise the query result won’t change. So hit TRAIN then PUBLISH and “Update Published Application”. Let’s refresh that web service call again:

retrained republished

(Not quite “1”, but it rounds up!)

Checkpoint

We’ve now got a LUIS app with two intents, each with two mapped utterances, and a web service endpoint to send text to and see if LUIS can map that text to an intent.

Training LUIS

Hopping back to the LUIS app’s home screen, you’re presented with a text box where you can enter more utterances to see how well LUIS maps them to intents. You can then override or set the correct intent, hitting “Submit” each time, to help teach LUIS.

Once you’ve done this for a few more utterances, hit TRAIN and PUBLISH to update the web service endpoint.

This is obviously a pretty manual process. If you already have a good set of input data to hand (for example, extracted from chat logs elsewhere) then this method of training your LUIS app is very inefficient.

Luckily there’s the option to upload a text file of utterances directly! Hurrah! This must be in the form of a text file containing one utterance per line, each utterance being unconnected to any intent.

On your LUIS application list page (not the LUIS app homepage itself), hit “import utterances” for the app you want to update (in case you have more than one)

upload text file of utterances

You’ll then be presented with this:

import 'unlabled' utterances popup

Once the file has been uploaded, LUIS will attempt to map each one to the existing intents.

utterances imported

It takes a bit of clicking around to find these uploaded utterances; you need to head to the SUGGEST tab, where it will gradually show you the uploaded entries along with their mapped utterances and their confidence score.

It’s here that you can recategorise or confirm the terms one by one; each time you submit one the model is retraining and the subsequent utterances have higher confidence scores.

Remember to TRAIN and PUBLISH when you’re done, then try querying the endpoint again with some alternative input; if the intent match is wrong or has a really low score, you can enter the same input on the LUIS app’s home screen – NEW UTTERANCES – to retrain to the correct intent.

Entities

I mentioned variables in the input earlier, so let’s have a look at mapping and extracting those next.

Head back to your LUIS app’s homepage and hit the plus sign next to ENTITIES. That will open the entity overlay where you can create a new entity.

add new entity popup

Give your entity a name and hit “Ok”; since my utterances are like “what’s the weather like in London today?” I’m going to create a “where” and a “when” entity.

LUIS already has many built-in entity types which cover location and dates; I’m just creating these for demonstration purposes

Now we can go back to our utterances and improve them by highlighting the entities; go to “Review Labels” and tap “Show all labelled utterances” to start enhancing them.

Choose an utterance and click to highlight the entity, then select the entity type from the pop-up and hit “submit” to confirm

select entity in utterance

Do this a few times and hit TRAIN and PUBLISH when you’re done.

Now try searching for a new utterance, one that you haven’t explicitly defined the entities for; you should see LUIS has identified the entities for you! If not, highlight and set them yourself.

Submit this new utterance to help improve your LUIS model (…and TRAIN and PUBLISH again..)

Summary

You should now be able to query LUIS via the service url and receive a json response with the most relevant intent and also with the entities identified:

json response with entities

You could now call this endpoint directly – using a webclient for example – and parse the response to determine the code to execute and the variables (entities) that were passed in.

You can also use the built-in support for LUIS in the Botframework.

We’re going to look at both of these options in the next article!