Jeb Chatbot Conclusion (and Future Goals)

The spring semester is quickly drawing to a close, and I was not able to get a chatbot working with Jeb. However, though my end goal was not met, I don’t feel as if I’ve wasted my time trying. Shortly after starting to work on this project, I found an API that would let me interact with a few different chatbots using Java. I added this API as a dependency to Jeb.

At this point, I was only able to communicate with the chatbots by typing to them, but my speech synthesizer was able to successfully speak their responses back to me. The first bump in the road occurred when I tried to hook the chatbots up to the speech recognizer (though I expected this). As detailed in some of my previous blog posts, in the grammar file for the version of Sphinx4 that I’m currently using, you have to explicitly define what words or phrases Sphinx4 should listen for. Since a conversation with a chatbot is fairly dynamic, I had to find an alternative to this.

I reread some of the Sphinx4 documentation and saw that they had a new version out which supported something new: dictation. As opposed to explicitly listening for certain commands, with dictation, Sphinx4 would be able to listen for anything. This sounded like a perfect solution. I got the new version of Sphinx4 working with my project, but the accuracy was quite low. It made my program unusable. After reading some more of the documentation, I found out that I could do something called acoustic model adaptation to improve Sphinx4′s accuracy. After doing this adaptation, I did see a noticeable increase in the accuracy of the speech recognizer; however, it was still so low that it made my program unusable. I’m not sure why my accuracy was so bad. Sphinx4 is supposed to be the best free speech recognizer for Java, and many people online speak highly of it. It’s a definite possibility that I set something up incorrectly or did the acoustic model adaptation wrong, but it’s hard to be sure.

By this point, time was running out. It was clear that I wasn’t getting anywhere with Sphinx4, so I decided to look for other options. I didn’t find much. The few resources that I did find either didn’t work at all or weren’t free.

So where does that leave me now? Well, I think I’ve decided to look outside of Java. If I can find a better speech recognizer in another language, I think it would be beneficial to either attempt to hook that up to Jeb, or even to port Jeb to that language. By itself, Jeb’s code is extremely simple; the 3rd party API configuration is the hard part. My college career may be ending in five days, but I think my work with Jeb will continue for years to come. I think the biggest thing that I’ve learned in my work with Jeb is that, at least for free/open source options, speech recognition has a long way to come. There are some great commercial solutions out there, but, for now at least, those are kept behind locked doors. However, I don’t think it’ll be too long before the open source software catches up. I look forward to that day.

Posted in Uncategorized | Leave a comment

Jeb Chatbot Update #4

When I posted last, I decided that I was going to look for a new speech recognizer in Java. Sphinx4 worked well for single-word/single-phrase commands, but when I wanted to integrate chatbot functionality, it just wasn’t cutting it.

After scouring the Internet, however, there really isn’t much out there. Google has some really nice speech APIs (check out, but the code isn’t public. There’s a project where some people set up an interface in Java to send data from your microphone to Google’s servers, mimicking the request/response process from the URL above. The problem lies in the fact that, if I went that route, I’d have to send voice data over the Internet to Google’s servers for my voice recognition to work. I’m not sure that I want to take on that kind of overhead. There are functions in Jeb that depend on Internet connectivity to work, but I don’t want the entire speech recognition engine to depend on it.

Another project I found is called EvalDictator. It’s built on top of Sphinx4 and is preconfigured for dictation, which is the type of speech recognition that I’m looking for. Unfortunately, I couldn’t even get it to load, and it looks as if the project has been abandoned.

So what are my options going forward? Well, my chatbot idea just plain isn’t going to work with my current Sphinx4 setup. If I had more time, I’d see if any other programming languages had good speech recognizers, and port Jeb. Unfortunately, I’ve only got about a week left, and I’m not sure that that’s enough time to find a speech recognizer in a different language, learn enough of the language to write the core functionality of Jeb, and test and refine it. As a long term plan, however, I like the idea.

Posted in Uncategorized | Leave a comment

Jeb Chatbot Update #3

When I posted last, I had discovered a “general” acoustic + language model for Sphinx4 to use. This means that I don’t have to pre-define every word/phrase I’m expecting Sphinx4 to hear. However, the accuracy was really bad – it was unusable. After reading some tutorials, I discovered that I could do acoustic model adaptation to improve my accuracy. I followed this guide to do so.

After doing the acoustic model adaptation, I saw a noticeable improvement in the accuracy. However, it is still so low that it is unusable. At this point, I think I’m going to look for a new speech recognizer to use. I like Sphinx4, but I don’t think I have the time to learn all of the intricacies of it in order to get general word recognition to a stage where it is as accurate as it was when it was recognizing pre-defined words from the grammar file. I might not find anything that works better than Sphinx4, but I think that for now, it’s my best bet.

Posted in Uncategorized | Leave a comment

Jeb Chatbot Update #2

Just a short update this time. The people who develop Sphinx4 seem to have released a new version with an improved API. However, for some reason, they stopped building Sphinx4 with Maven. Instead, they’re using Ant now. After figuring out how to install a project built with Ant to my local Maven repository, I was finally able to use the new Sphinx4 code.

As mentioned in my previous post, this new method should allow my program to recognize anything that is said, not just single-word commands. However, after actually trying it out, I discovered that the accuracy is atrocious. After reading some more of the updated Sphinx4 tutorial, it seems I may be able to boost the accuracy quite a bit by adapting the acoustic model that Sphinx4 is using right now. This isn’t the same as training to a specific person’s voice; this will adapt Sphinx4 to my sound setup, the microphone I’m using, general background noise, etc. Acoustic model adaptation isn’t supposed to take very long, so here’s hoping to a success in my next blog post!

Posted in Uncategorized | Leave a comment

Jeb Chatbot Update #1

I’ve made some great progress on the chatbot functionality this week, which could also have some profoundly positive effects on the program as a whole.

First things first, though: the chatbot. I found a nice API called Chatter Bot that provides a Java interface for users to interact with a few different chatbots. Although the Chatter Bot Java project wasn’t built with Maven, Maven comes bundled with a nice plugin to install non-Maven projects to your local Maven repository. I made use of this plugin and included Chatter Bot as a dependency for Jeb.

The setup process, however, is the easy part. The hard part is going to be getting the speech recognizer to play nicely with Chatter Bot. If you’ve read my previous blog posts about Jeb, then you’ll know that the speech recognizer (Sphinx4) works using a grammar file and a dictionary file. The dictionary file contains a list of every possible word the speech recognizer can hear, while the grammar file defines what the speech recognizer is actively listening for. Problems arise when you consider that, with a chatbot, the conversation is very dynamic. When in chatbot mode, I can’t have Jeb listen for a predefined list of words because, well, what fun would that be?

I came across a forum thread on the Sphinx4 website in which the topic creator was in a situation similar to mine. Another user replied, telling him that the functionality he needed already existed in Sphinx4, and pointed him to a tutorial. I haven’t looked at the tutorial yet, but if it works like the person who replied to the thread said it did, this could solve one of the main problems with Jeb right now. Currently, if Jeb is running and you say the words “Hi Bob, how are you today?”, Sphinx4 will take that and match it to something in the grammar (whether that be “jeb”, “weather”, “mail”, etc.). However, I may be able to use some sort of “general” grammar file to get Sphinx4 to recognize any phrase for what it actually is. For example, if I say “Hi Bob, how are you today?”, Sphinx4 will actually recognize “Hi Bob, how are you today?”

If this works, it would solve the two biggest problems with Jeb at the moment:
1. Jeb would be able to be run in the background. It wouldn’t identify every phrase that it hears with a command, so it would be able to sit idle.
2. You would be able to speak to Jeb in a very natural manner, rather than giving it single word commands.

This is exciting. I should have my next blog post detailing how everything goes up within a week.

Posted in Uncategorized | Leave a comment

Jeb the Chatbot

For my final project in my artificial intelligence class, I will be attempting to add chatbot functionality to Jeb. For those who aren’t familiar with chatbots, it’s a program that you can type messages back and forth to as if it were a human being. With Jeb, this will be a spoken conversation.

There’s one potential problem that I foresee: the grammar file for the speech recognizer. With Sphinx4, you have to explicitly define each word/phrase/sentence that Sphinx4 should listen for. Since conversations with chatbots are very dynamic, the grammar file may prove to be a problem.

Posted in Uncategorized | Leave a comment

Jeb – Let There Be Light!

I’m going to preface this post with a YouTube video:

In the past month, I’ve achieved my goal of being able to turn on a light bulb using Jeb, an Arduino, and a PowerSwitch Tail II.

Here’s how I got it to work:

Arduino Uno
PowerSwitch Tail II
A lamp I had sitting around

The official Arduino IDE

Before Jeb, I had used an Arduino before for a small project, so I was a bit familiar with the Arduino IDE. I didn’t want to have to use a second piece of software with Jeb though. I wanted something that would allow me to control my Arduino in Java. After minutes of grueling Google searching, I stumbled upon JArduino.

JArduino was exactly what I was looking for. The setup was fairly easy. I had to download the official Arduino IDE in order to upload JArduino’s custom firmware to my Arduino board, but that was all I needed to do to get my Arduino working in Java. The JArduino Java library was built using Maven, so including it in my project was as simple as downloading it, installing it to my local repository, and adding a new dependency element to my pom.xml file.

Once I had included the JArduino library in my project, the implementation was simple as well. I created a class called Arduino, which extends the JArduino library’s conveniently named JArduino class. After overriding a basic setup() method, my code to turn the PowerSwitch (and thus the lamp) on and off looked like this:
protected void loop() {
digitalWrite(DigitalPin.PIN_13, on ? DigitalState.HIGH : DigitalState.LOW);

In the above code, “on” is a boolean variable dictating whether the light should be switched on or switched off. Pin 13 is the pin that connects the Arduino to the PowerSwitch.

This is how all the components connect to each other:

                                           [AC power outlet]
[Computer]----[Arduino]====[PowerSwitch Tail II]----[Lamp]

The computer is connected to the Arduino with a USB cable. The Arduino is connected to the PowerSwitch through two small wires (I’m not sure what they’re called, but they look like this). The PowerSwitch is connected to both the AC power outlet and lamp with a normal AC power plug.

Overall, once I had the hardware in-hand, it was a pretty straightforward process to get the Arduino working. JArduino has a lot of cool functionality built-in, including Bluetooth and Ethernet, so I’m already brewing up more Arduino + Jeb ideas.

Posted in Uncategorized | Leave a comment

Jeb – Moving Forward

Back in November when I wrote my last blog post, I didn’t expect it’d be almost two months before I actively worked on Jeb again. First came finals, and then the holidays (with starting a new job happening in between). Thought I haven’t written much code for Jeb lately, a lot has happened with the project.

First off, I’ve been accepted to present at the 2014 National Conference of Undergraduate Research in Lexington, Kentucky April 3-5. This will be an awesome opportunity to show Jeb off, and I’m pretty excited to go.

My undergraduate research fellows grant from Fall 2013 is extended through Spring 2014, so I get to keep working on Jeb on a regular basis at least until May. On top of that, I’ve got a couple of class projects that I think I’ll be able to use Jeb for. For my senior project in one of my networking classes, I’m going to add a networking component to Jeb (more on that below). I’m also taking an artificial intelligence class. I’m not very far into the class yet, but I’d like to take some of the knowledge that I gain from it and apply it to my dear digital butler.

So, moving forward, here are my goals from now until late April/early May:

In the next month or so, I’d like to program Jeb to control an AC power switch – to be able to turn a light bulb on and off. This was something I had originally planned for last semester, but I ran out of time. I have all of the physical equipment that I need, I just need to figure out how to use it.

From then until the beginning of April, the majority of the my time will probably be focused on improving the speech recognition and speech synthesizer. In one of my previous posts, I mentioned some of the issues I was experiencing with the speech recognition engine. I’m going to see if doing some manual training will help me be able to run Jeb in the background like I’m envisioning. The Sphinx4 documentation isn’t terribly helpful on this front, which is why I envision this portion taking up a majority of my time. As for the speech synthesizer, I’d just like to try to get it to sound less robotic. It doesn’t sound terrible as it is, but if I can tweak some settings to get it to sound a bit more natural, I’ll be a happy camper.

After that’s finished, I’d like to take a second look at reflection. I mentioned previously that I had read that it wasn’t the best way to handle dynamic method invocation, but I went with it anyways due to time constraints. Since then, I’ve had two different people show me better alternatives. I’d like to evaluate these alternatives and see if they’re a good fit for Jeb.

The above are my goals for the part of my research funded by URF for the semester. I get about 5 hours a week to work on it through URF, and I definitely think that these goals will be achievable. Since I have the backbone of the project done, adding a new feature is relatively easy. If I come up with a cool new function for Jeb to perform, there’s nothing stopping me from switching gears and adding it.

I did mention, however, that I’ve also got some opportunities outside of URF to work on Jeb – a networking class, and an artificial intelligence class. For the networking class, there are three things I’m going to attempt to accomplish. First, I’d like to be able to operate Jeb over a LAN. Computer A will run Jeb, and computer B will have a microphone attached to it and will be running a program to feed the data from the microphone to the computer running Jeb. Jeb will then send a voice response over the network which will play on computer B. The second thing I’d like to accomplish builds off of the previous – I’d like to do the same, but with multiple computers (clients) sending microphone data to a single computer running Jeb. Jeb would then process it and send a response to the appropriate computer. In the same fashion, I’d also like to try to get this to work over the Internet, so it can be operated from anywhere as long as you have a microphone and access to the server that it’s running on.

Hoping for an exciting semester!

Posted in Uncategorized | Leave a comment

Jeb Update #4

Just a short update this time. In the past week, I’ve looked into and implemented an API to check Gmail. The API is called Gmail4j. It was very easy to set up (it was built with Maven) and configure. Currently, when you have Jeb check your email, the program reads to you all of your unread messages. This may be something I’d like to change – maybe make it more interactive? Instead of “Here are your unread messages: First message, from…” etc., I could make it “Your first unread message is from [person], would you like me to read it?” It wouldn’t require much work and might make the user experience a bit better.

Other than the Gmail implementation, there’s isn’t much else to report on. My next move is to interface Jeb with a light bulb of some sort. This is going to require an Arduino and a PowerSwitch. I received the Arduino today; I’m just waiting on the PowerSwitch so I can move forward.

Posted in Uncategorized | Leave a comment

Jeb Update #3

Since my last post, I’ve accomplished quite a bit. After making the decision to prioritize functionality over speech recognition accuracy, I started to look at what I wanted to actually implement. In April, I’m hoping to present Jeb at the National Conference on Undergraduate Research in Lexington, Kentucky. I have to submit an abstract to be considered for the conference, and in my abstract I state that I will have these two things implemented at the bare minimum: the ability to check the weather and the ability to turn a light bulb on/off.

I chose those two specific tasks for a few reasons. First, they’re fairly simple and can demonstrate Jeb’s intended purpose easily. I’d like Jeb to run my house eventually, but that’s not easily demonstrated considering:

A. I don’t currently posses a house.
B. Even if I did, I couldn’t bring it to Kentucky.

Also, I want to demonstrate that Jeb will be able to interact with both physical and digital objects – hence the light bulb (physical) and the weather checking (digital). I decided to start by adding weather-checking functionality first; turning on a light bulb will involve additional hardware that I don’t have (yet).

When I started to code the weather-checking, I ran into a bit of a design/technical roadblock. My program currently works like this:

1. The speech recognizer listens for input and passes the result to a class to execute the command
2. The command is executed and the result is passed to the speech synthesizer
3. The speech synthesizer provides audial feedback on the result of the command

The problem occurred in point #2 above. I wasn’t sure how to translate the String object I was passing to the command executer to a method name. For example, if I say the word “Jeb”, I want the command executer to run the method called “jeb”. But the only way I knew how to do that was:

if (command.equals("jeb")){
// call the jeb() method
} else if (command.equals("somethingelse")){
// call the somethingelse() method

As you can see, for every command I add (and I envision adding quite a few), I’d have to go in and add another “else if” block. Which would get really annoying. Fast. Basically, what I was looking for was dynamic method invocation. “Does such a thing exist?” I asked myself. A quick Google search showed me that such a thing does, indeed, exist. It’s called reflection. However, the nice people on Stack Overflow (in the above link) mentioned that this was not good object-oriented coding practice and that one should avoid it. After some discussion with my project mentor, Dr. Sherri Harms, we decided that reflection may be my best course of action.

After implementing reflection, getting Jeb to check the weather was rather easy. I used; I’m able to use Java to open up a URL connection, passing in the name of the city and state to check the weather for, and I get back weather data in JSON format. As of right now, I’m only pulling temperature information from the JSON object, but it would be a simple task in the future to pull high and low temperatures, conditions (rain, snow, etc.) and weather for future dates.

In the coming weeks, there are two main tasks I’d like to work on: turning on a light bulb and checking email. I’ll have to research and purchase supplies for the light bulb part, but I’ve already found an API to check Gmail. All in all, Jeb is coming along nicely and I’m excited to see how far I can take it before April.

Posted in Uncategorized | Leave a comment