Archive for August, 2010

Scripting JSoar

August 30th, 2010 No comments

In simpler times (say 2001) Soar was just Tcl. That is to say, Soar was a module, dynamically loaded into a Tcl interpreter at run-time. When loaded, Soar added a bunch of useful commands to the interpreter. Like run, matches, preferences, and probably most importantly, the sp command. When you “sourced” a Soar file, the Tcl interpreter just executed commands, loading rules, setting watch levels, etc.

The main drawback to this whole situation was that Tcl didn’t always lend itself to friendly embedding in other programs. It had funny rules about threads and, if Tk was involved, demanded to have its message queue pumped. And, of course, very few people get to know Tcl enough to like it :)

On the other hand, you could create macros for repetitive Soar structures, define new RHS functions, manipulate I/O. In short, you had the power of a full programming language mixed in with your Soar code.

With Soar 8.6 Soar’s tight integration with Tcl was broken, replaced by SML and a stricter command interpreter. It still looked like Tcl commands, but there were no Tcl control structures, variables, etc. Way it goes. When I initially started work on JSoar, I needed to quickly bootstrap a command interpreter so I could load existing code into the kernel. I turned to Tcl, in the form of Jacl, a Java implementation of Tcl. It saved a lot of time and, since Soar’s syntax was still basically Tcl, no one would really notice.

Of course, as I mentioned, no one wants Tcl, so over the last couple weeks, I’ve added a new scripting layer to JSoar. This time, I’m taking advantage of the Java Scripting API, JSR-223. This allows any scripting language with a JSR-223 implementation to be pretty seamlessly accessed from Java (and vice versa). With this new capability, it’s now possible to automate Soar agents, implement simple agent environments, and extend SoarUnit testing to include I/O, all from within a Soar source file. All with a variety of languages including Ruby, JavaScript, Python, Clojure, Groovy, etc.

A scripting engine (a language implementation) is invoked with the script command:

script javascript {
   soar.onInput(function(e) {
      soar.wmes.add("my-input", "hello");
   soar.onOutputCommand("say", function(e) {
      soar.print("The agent says: " + e.greeting);

This little bit of code sets up an input phase callback and creates a WME on the agen’ts input-link. It also handles an output command called “say”. The equivalent Java code would be … more ceremonious. Not to mention setting up a new project, compiling, etc, etc is a major hassle.

As an example, I’ve implemented a simple waterjugs environment in JavaScript, Ruby, and Python. Here are some things you can do:

  • Generate input
  • Handle output commands
  • Auto-convert hashes (JavaScript objects, Python dicts, or Ruby hashes) to input structures
  • Install new RHS functions
  • Add new commands
  • and on and on

Also, with maybe a little more work, I might have a pretty good story for dealing with I/O in SoarUnit tests. Stay tuned.

More detailed info on JSoar scripting support can be found on the JSoar wiki.

Introducing SoarUnit

August 1st, 2010 No comments

The history of testing in Soar is short and not very happy. This is especially true for automated testing, where the tools have generally been ad hoc, proprietary, and hard to use. I am personally responsible for some of these tools. Despite all this, I’m foolishly taking another foray into the land of Soar unit testing. This new effort is called SoarUnit and is part of the JSoar suite. However, since the JSoar community is even tinier than the overall Soar community, I’ve made sure that SoarUnit is compatible with both Soar implementations.

(For more complete docs and examples, see the SoarUnit wiki page, and the Bebot source code. There’s a snapshot (20100801) of JSoar with the latest SoarUnit on the JSoar downloads page.)

SoarUnit UI

The one advantage that this effort may have over previous Soar testing attempts is that I developed it at the same time that I was actively working on Soar code. When I first started working on Soar99 and Bebot, my testing strategy was very ad hoc. I had a big Soar file that looks something like this:

   source "test1.soar"
   excise --all

   source "test2.soar"
   excise --all

   ... and so on ...

To run my tests, I’d just source the file and eyeball the trace for errors, which, if you’re familiar with Soar, you’ll know are usually pretty easy to spot. Of course, there were some drawbacks to this approach:

  • Manually checking for success/failure is painful. The “green bar” is so much nicer.
  • To add a single new test, I had to add another three lines to¬† my “master test” file.
  • There was no way to put more than one test in a file, necessitating either a bunch of duplicate code, or some confusing trickery with the source command.
  • When a test failed, I had to manually source it to debug.

Basically, all the same problems you get when manually testing code in any other language.

So, with 30 or so ad hoc tests already built, I started working on SoarUnit. Here’s the basic structure of a testcase file:

setup {
   ... Code run before every test. Here you can source code under test,
       setup test data, propose initialization opreators, etc ...

test "name of test" {
   ... Code for an individual test. Here you put any code
       you need for the test. When the test's success condition's
       have been detected, a rule should call the (pass) RHS function.
       Similarly, there's a (fail) RHS function for detecting failures ...

... More tests ...

Nice and simple.

I think the existing tests were key to how well development has gone. Every previous attempt I’ve made for testing Soar code has been based on imagining how someone might use such a framework, without any real-world requirements to build from. Although I’ve only been working on it a little more than a week, SoarUnit already has:

  • Test case discovery by file name pattern
  • Multiple tests per file
  • Setup blocks to handle code shared by multiple tests
  • A graphical interface similar to Eclipse’s JUnit view (see below)
  • Single-click debugging from the user interface
  • Support for both JSoar and CSoar 9.3.0 (CSoar requires that a SOAR_HOME environment variable be set so it can find native libraries and stuff)
  • Basic code coverage reporting

Best of all, I’ve used it a lot on Bebot and Soar99 and it’s honestly really nice. I did a major refactoring, basically renaming all of the major data structures and operators in the library, and it was a breeze. I guess that’s the point of having tests…

What’s Next?

I’m going to keep working on SoarUnit to support my own Soar development. There are still some obvious holes and open questions.

One area that’s always plagued talk of testing Soar code is I/O. How do you test an agent independent of the environment it will run in. I’m currently of the mind that, like unit testing in every other language, this is where mocking comes in. With a few rules, an agent can easily simulate static structures on the input-link to driver tests. If things get too complicated, one option is simple integration with soar2soar, where for each test, a helper agent would be created to simulate the environment. There are other options as well (plugins, external processes, etc), but none of them maintains the simplicity I want for SoarUnit. For every configuration parameter, you lose a user and with Soar there aren’t that many to start with.

The other open question is whether SoarUnit is effective for testing idiomatic Soar code. Soar has very little encapsulation or modularity which can make it difficult to isolate code for testing. The problems I’m solving with Bebot are very procedural, so they’re easy to test, but I’m not sure that’s true for most Soar code. I’d like to work through creating tests for some of the Soar tutorial problems and see how it goes.

Categories: soar Tags: , , ,