One of the issues with existing benchmarks such as waltz and manners is validation. The manners benchmark only runs in CLIPS with the depth conflict resolution strategy and the waltz benchmark executes different numbers of rules depending upon the conflict resolution strategy chosen.
Clearly this is an issue. I'm a proponent of having lots of benchmarks, rather than one or two, but in order to have lots of benchmarks, they need to be dead simple to translate and verify. In the case of the existing benchmarks, they're not.
What this means is that you have to design the benchmarks with this is mind. I thought it would be an interesting exercise to demonstrate how this can be done. So I wrote a CLIPS program to solve a variation of the river crossing problem and once I had it working, set the conflict resolution strategy to random to see if the same number of rules were executed with each run. There weren’t.
It took several iterations before I had a version that produced the same number of rule executions regardless of the order in which rules of the same priority/salience were placed on the agenda. The primary mechanism used to get an exact number of rules executed was to assign weights to each of the possible moves that could be made so that the search was always made in the same order. In total, there were 19 rules and 5 salience values for the different groups of rules. If I’d used modules (which would have made translation to other languages more difficult), there wouldn’t have been any need for salience values at all.
Like the manners and waltz programs, the river program runs considerably faster in version 6.3 of CLIPS than in version 6.24. On a 2.66 GHz Intel Core 2 Duo it completes in 0.7 seconds in the newer version as opposed to 29 seconds in the older version.
The river program is available here.
You mention that a benchmark should be dead simple to translate and verify. When I look at your river riddle, the description says: "The dog must be with either the mother or the sons", yet in the solution I see the line "father and dog move to shore 2". This means that the dog is on the raft without the presence of the mother or one of the sons: a clear violation of the rule. Am I missing something?
ReplyDeleteThe constraints for the cat and dog should begin with "while onshore." They can each move across by raft with either the mother or father, but once on the other side, the appropriate family members must be there or they will run away.
ReplyDelete