Using IBM's Watson Conversation API with Cisco Spark in Node.js

IBM's Watson has a number of APIs but the one I am dealing with in this post is the Conversation API which has some interesting features. Even though IBM are clearly still adding features to this fairly new API it is pretty easy to work with once you understand how context can be used. I am not going to go over the basics of the API which if you have never worked with Watson before there are a million You Tube videos on the basic's. Seems even a 13 year old can put a video together on how to use Watson. I did watch this video BTW and its a handy starting place for the basics but one thing that is missing from all the videos is how to use the context data to get information back and forth from your bot to Watson. The video's mention it and the Watson documentation talks about it but it is a little confusing to start with.

What is Context?

Context is a pretty commonly used phrase in natural language processing(NLP) systems, it allows you to pass information back and forth between your bot and the NLP processor, in our case Watson. It does this using a JSON format which can include basically any attribute you want to put into it. Watson does have its own tracking with the conversation_id but tracking of the user and the conversation the user is currently in is up to you. I like to think of context as a cookie that gets passed back and forth. If you are working with a chat app in the browser the browser can take care of the cookie for you. But in our case with Cisco Spark you have no such luck. You as the developer have to track context which is included with every response from Watson. Watson will update the context with information from your users and information Watson uses to track the conversation. Without the mechanism to track the conversation context Watson assumes every transaction is a new conversation and the conversation_id is reset to a new value.

Building a Cisco Spark Bot with Watson

I recently built a Spark bot that started out using an array to track context but I quickly moved to MongoDB database (I wanted some persistence on restarts of node) . Seeing as Watson's context is already formatted in JSON it is easy to slip this straight into Mongo. Below is an example of some of the attributes I add to the context parameter before I send my message to Watson. Email, roomId, user and orderComplete (it is a pizza bot) are all created and added before the first message is sent to Watson. The combination of email and roomId from a Spark point of view makes every conversation unique as well as giving me some handy attributes. I, as a person can have multiple conversations with Watson in either a one-on-one room or multiple participant room so having a way to track that is important.

{ _id: '',

  email: '@cisco.com',

  roomId: '

  user: 'Christopher Norman',

  orderComplete: 'true',

  conversation_id: 'af4487c4-5543-4bf8-ab9c-f4611b3498bd',

  system: 

   { dialog_stack: [ 'node_9_1475525825835' ],

     dialog_turn_counter: 9,

     dialog_request_counter: 9 },

  pizzaType: 'vegetarian',

  hackerNumber: 5 }

PizzaType and hackerNumber are appended during the conversation by Watson, which I will cover in just a bit. Obviously there are some other attributes Watson will add like what node it is up to in your conversation with node_9 as an example. This is its own tracking system and as long as you relay this back to Watson during your conversation it will keep your place in the dialog exchange. The exception to the this is a flat dialog structure where every node in the Watson dialog is a conversation entry point. In this case you don't care where the conversation is because the dialog has no depth. A FAQ bot where every question is the start and end and there is no follow up question as an example. I was going to say data collection as well but you can still make context updates even in this more simple dialog structure.

Below is a sample of what I used to build the context request using the Node.js Watson module. You may notice this is different than the example from the module. That is because the module example doesn't show you how to pass context through your conversation, this does. Watson's Node.js module is pretty useless without a way to collect and pass context. I worked this out by looking at this code, go look its worth the time. Stefania is an IBM dev evangelist, she has written some great examples over on Github that helped me a great deal. Just remember you have to save the returned context and retrieve what you have saved to send back a reply when your user responds to Watson. Watsons NLP engine will append entity information to context as you dictate but your still need to save it.

If you are interested in what I did to write my context to a MongoDB see my gist below. I am not going to cover it blow by blow but some might find it handy so I have included it for completeness. There is nothing to complicated in there as I replace the previous context Watson sends me rather than update the one in the database. There are quite few fields that require updating in every transaction so replacing context was just simpler than updating fields in the stored JSON record. If you prefer another database go for it this is just a quick solution I put it together in a couple of days. I didn't have weeks to explore the perfect database or why one option may be better than another or even put much thought into the data structure. I am just throwing JSON data chunks into the MongoDB and retrieving them later. I only mention this after reading this article while doing some research. It is not so much the article, which is interesting (although not sure I agree with), but some of the comments. The blogger should have mentioned to check your ego at the door before writing your comment. Is it just me or are people on the Internet snarky? lol.


So now we have covered how to build the context but what else can we do with it? Obviously I am going to use the classic pizza bot example, doesn't everyone. As you build out your dialog you can use Watson to pick out and update information in the context as well as create some more complex logic for dialog decision making. In the first dialog box even though you can not quite see it, I am checking the content of an attribute in my context using $orderComplete.contains('new'). This allows me to see if this person has ordered yet. If they had that same attribute "orderComplete" would have read true and I could have put them in a different thread.

In the first dialog box we ask the user what Pizza type they want. In the second box we are accepting their response, relaying it back and prompting for more info. But first we must record their response in our context. In our case it is the pizza selection using NLP we identify with entities(sorry I never covered this earlier. Now is the time to go read about intents and entities on IBM's Watson documentation site if you don't know WTF I am talking about).

Quick tip. If you need access to info in your context at any stage to respond to your user, using $ will get it for you. Below I added $user to grab the user name attribute from my context to add a personal touch to the interaction but this can be anything stored in the context.

Watson has a very interesting NLP engine that is actually pretty easy to work with once you get past the initial knowledge gap. Not to say there are not a load of other possible alternatives such as api.ai etc. or a node module that does NLP this is just one engine. If you have tried others feel free to comment about it. Always interested to check out new stuff especially if you have done NLP using a node module.

Thanks for reading.

VoIPNorm