Bot Project

The Idea

A few years ago, I was working on trying to make life faster and easier for our service desk team. Since we communicated almost exclusively by chat, when Microsoft released the preview of the Bot Framework, it seemed like a really interesting option for enabling people who already had dozens of websites to use at any time.

Customer Development

I interviewed many of the senior technicians to get a better understanding of what they frequently had to do that required limited interaction. I also interviewed senior management and noticed that they frequently needed single facts throughout the day (e.g. how many users are online? what department is Sally in?) Things like looking up data or single line commands seemed like ideal things to wrap in a chat bot and to simplify.

Executive Buy-in

I put together a comprehensive, high level pitch presentation and got approval for a proof of concept. The easy integration with our enterprise Skype for Business platform and the fact that we already had a number of data sources mapped out and ready to consume were convincing for a limited trial of the technology.

Building it Out

Using a combination of the LUIS language comprehension rest api and high level logic code in C#, I was able to rapidly design a working bot. In addition to simple statements (how many people are logged in?), I was able to model a rudimentary entity identification model so we could ask for a type of information and provide a named entity. This seems simple but allowed for the same bot to answer 'how many people are logged into datacenter X?' and 'what virtual machine is Sally Jones logged into?'

Executives with access made frequent use of it and initially service desk were heavy users.

The Catch

Before my MBET started, I had the unfortunate (but seemingly common) belief that the value of a new product is equivalent to the awesomeness of the technology. While it was novel and awesome, users quickly reverted to what they had done before they had access to the bot. Within a few months, usage had dropped to one or two users per day and I couldn't justify investing further efforts in improving it. While users said they wanted a bot to help them out, it didn't get traction.

Reflections

While it may have been faster to use than websites, post-launch interviews illustrated three main problems:

1. Frustration with Phrasing: We have support users in 4 countries on opposite sides of the world, each with a different local dialect and slang. While I anticipated ongoing training for improving the model, having multiple effective language sets actually degraded accuracy as we continued to train.
2. Un-obvious Features: While a website is likely slower and involves a lot of clicks, it's visual nature makes it plain what can be accomplished and how. Chat-bots (at least based in Skype) lack any kind of menu and users must either know what they want to do specifically or access help dialogs. While I assumed users would quickly internalize their favorite bot skills, the combination of a huge number of features and varying workloads (a user might only need a feature once a week depending on what tickets they get), users didn't internalize many skills and the depth of skills remained hidden.
3. Painkiller vs. Vitamin Problems: While I learned this only long after the fact in the MBET program, there is a massive difference in problems. These users already had effective ways to accomplish their tasks, even if they were hard to learn and a bit slower to use. The bot offered a good speed improvement but was only a vitamin solution for the teams. Users had little incentive to push through adoption pains given the limited rewards.

Since it was a proof of concept project, I have think it was a success: the lessons learned were invaluable in both giving experience with validating demand (really making sure there is a problem that needs solving and not just an incremental improvement) and specific experience with bots and LUIS models.