Mechanix: a sketch-based tutoring and grading system for free-body diagrams

Paper
Valentine, Stephanie, et al. "Mechanix: a sketch-based tutoring and grading system for free-body diagrams." AI Magazine 34.1 (2012): 55.
Publication Link: http://www.aaai.org/ojs/index.php/aimagazine/article/view/2437

Summary

This paper describes the implementation and deployment test results for a domain specific, sketch-based tutoring system, Mechanix. The system, unlike other systems, offers machine intelligence as well as free hand sketch input as solutions to instructor provided truss problems, which are a major feature in some large introductory engineering courses where student enrollment is high, and individual feedback is important but limited.

Discussion

Pros

The paper is very well organized and provides a very thorough motivation for the work and a fairly detailed literature review. The authors conducted some experiments testing the overall function of the system to get a sense of impact on student performance, and enrollment. They also provide performance metrics on recognition accuracy of the system.

Cons

I feel that, since the major advantage of said system is in grading and providing feedback, the authors should have spent a little more time in the paper discussing the system performance on these specific functions vs. the overall system performance. What are the problem areas / challenges in recognition that they encountered during deployment? Providing some idea on how the feedback mechanism was used, and metrics collected to evaluate its performance would be useful to read and as fodder for future work in system improvement.