The end of the 20th Century presented the United States Army a new technological and geopolitical environment. The Army has recognized this change and is adapting to operate in this new environment. The army has developed a number of new automated tools to assist leaders to command and control their organizations. One tool currently being developed, tested, and integrated into the Army is the Combined Arms Planning and Execution Monitoring System or CAPES. CAPES is designed to assist decision makers develop, coordinate, and produce operational plans. As this and other automation systems are developed, they must be evaluated. The Army uses evaluation to determine the merits of a system. The evaluation also reveals deficiencies a system must correct to become more useful. This monograph focuses on the evaluation process. Investigating the process identified a number of problems an evaluation might encounter during the assessment. These problems identified in the research for this paper are now documented. Assessors can now use this list to design aspects of future evaluations. Initially the research for this monograph attempted to evaluate the usefulness of CAPES. To determine whether CAPES was useful a test was organized to compare two planning efforts. Two military staffs planned operations based on a similar problem. The experiment defined one planning staff, using traditional planning tools, as the bases of comparison. The second group would have the benefit of CAPES. The early collection and analysis of observations from the two planning groups indicated the design of the experiment was inadequate to show either CAPES 19 benefits or faults. Initial scrutiny revealed significant problems with the experiment 's design. These problems prevented any comparison of the two planning efforts. Investigation of the difficulties discovered problems in the scenario selected, organization of the planning staff, and the exercise architecture. Analysis of the problems indicated greater attention must be placed on selecting scenarios that replicate command and control conditions intended for CAPES. The staff used in the experiment not only requires adequate training with the planning tools available, they also need sufficient practice working together as a staff. Command and control systems must interconnect and communicate across all Battlefield Automation Systems (BAS) and staffs must use all of the subsystems simultaneously. The problems identified during this study constitute a set of preventable deficiencies where system designers and evaluators can avoid when conducting software assessments. Incorporating the overlooked design parameters with those of the planned test will lead to a better assessment of automation planning tools during future research. Creating planning staffs from students enrolled in military schools initially appeared attractive; however, the students neither have the required training on new information systems nor the time to develop the organizational relationships required in an established functioning staff.

ADA429324 Approved for public release; Distribution is unlimited. The opinions and conclusions expressed herein are those of the student-authors and do not necessarily represent the views of the U.S. Army Command and General Staff College or any other governmental agency. (References to these studies should include the foregoing statement.)