Only with precise specifications is it possible to teach the fundamental engineering practice of implementation to specification. We rely on simple implementation exercises to teach and evaluate the students' ability to implement to specification. To facilitate grading and to emphasize the role of automated testing, we use PGMGEN to automatically grade the execution behavior of the implementations.
In some cases we provide a specification and the students must develop the implementation, but often we give the students a specification and an implementation, and ask them to modify the implementation to use a different algorithm. For these exercises, 50% of the grade is based on the source code (style, simplicity, and efficiency) and the other 50% on the execution behavior. To determine the grade for the execution behavior, we use a PGMGEN test suite developed by the instructor. These test suites contain hundreds---sometimes thousands---of test cases and are not available to the students until after they have handed in their implementations. We evaluate the student solutions at zero-tolerance: the implementation must pass all test cases to get any credit for execution behavior. The zero-tolerance approach, combined with a large number of test cases that are executed automatically, reinforces implementation to specification: students realize that even ``minor'' errors in the implementation will incur substantial penalties.
Figure 2 shows the students' scores for an implementation to specification assignment. The students were given a specification and an array-based implementation of symtbl. They were asked to change the implementation to use a hash table. Over half of the students passed the assignment, indicating that their implementation passed all the test cases. Many of the students who did not pass all the test cases also lost marks on the source code.
Figure 2: Scores for the implementation to specification exercise
In the instructor's solution, only 15 lines were changed from or added to the original implementation. Many of the students perceived this exercise as hard, mainly because they found it hard to exploit the existing implementation. Many students developed their implementation from scratch, completely ignoring the array-based implementation.
Following on from implementation to specification exercises, we have the students writing their own test suites using PGMGEN. For a typical exercise, we give the students a module specification and implementation and ask them to write a test plan and test suite. We grade the execution behavior of the students' test suites by running them, fully automatically, on known faulty implementations. We have found that students find it easy to write test cases using PGMGEN, but that they do have problems with systematically selecting and documenting their test cases.