Evaluative Efficacy
By Anne Diffendal
Regent for Examination Development and
Chair, Examination Development Committee, 2000-2004
Any single ACA exam is the result of a complex series of events that began nearly twenty years ago. Some of the following details are less immediately relevant to the question of efficacy than others, but it is important to have context for the process of exam development.
It is also important to remember that the archival certification examination is a certification exam. It is not a graduate school entrance exam, nor a comprehensive for a master’s or doctor’s degree, nor the final exam for a course (though it contains questions similar in form to those on other kinds of tests). Because most archivists have little or no experience with testing for certification or licensure, they may question the ACA exam from lack of understanding.
The Academy undertakes to present a good certification exam (in which “evaluative efficacy” is one among many elements) by:
- following an accepted process developed from practical experience and legal challenge;
- applying the best information available from the educational testing and measurement profession; and
- involving a large number of “job experts” (archivists) in a variety of ways at all stages of the process.
The Process
The accepted process of developing and maintaining an exam of this type is the result of 50-60 years of experience involving thousands of organizations that certify or license practitioners. In establishing the certification program, the Society of American Archivists hired one of the most respected national test development firms to guide archivists in the process. We have faithfully followed the recommended process and procedures.
The Practice
We have used the best information available from the discipline of psychometrics in developing and maintaining the item bank and exam. One issue, considered at the outset, was what kinds of questions to use on the exam. The choices were among what are known as “limited response” questions, which include multiple choice, matching, and true-false questions; and “constructed response” questions, which include essay, short-answer, and oral questions. Nearly all certification/licensure exams include multiple-choice questions; and it is very common that they contain only this type of question.
Multiple-choice questions are widely used for a number of reasons. They can be constructed to test many different levels of knowledge, including higher cognitive skills and problem solving. They offer an effective way achieve one of the goals of a certification exam, that is, to test a broad range of knowledge and skills. There is by far the most research on this type of question, making conclusions about the validity and the reliability of any given test quite firm, as well as permitting development of guidelines for writing good questions and avoiding poor ones to aid in question-writing. And finally, they are efficient and economical to score.
After each exam is given, statistical tests are applied to the test as a whole and to each question. Results are transmitted to the Examination Development Committee, as are any questions or comments by test takers that are reported by proctors, Academy officers and board members, etc. Exams are revised and questions are revised or eliminated based upon the statistical tests and reported questions and comments.
The ACA exam has continually tested well according to standard statistical measures. It generally contains only one or two individual questions that have tested badly; an exceptionally low number for a test of one hundred questions, according to my understanding.
The Archivists
At every step of the process, archival practitioners have made the decisions, prepared the role delineation statement, developed the questions, and constructed the exam. Under the direction of expert facilitators, panels of archivists wrote the role delineation statement; other panels reviewed it; it was “validated” (a test developers’ term) through a mailing to a random sample of SAA members. The same process using a different set of archivists was followed to assign weight to the various domains on the test. Panels of archivists have subsequently revised the preservation and management domains. Groups of archivists were convened to write questions to comprise the original item bank.
Thereafter, a standing committee of Certified Archivists has written new questions. This same committee has revised existing ones based on test results, changes in best practice, or new professional literature. Each question in the item bank is linked to a domain and to a task statement and a knowledge statement within that domain. The questions for each test are chosen to reflect the entire range of the role delineation statement. Examination Development Committee chairs have sought to ensure that the committee represents a wide variety of employing institutions, subject expertise, length of experience in the profession, and geography.
In the early-nineties, a panel of experienced archivists, including some skeptics, took the exam and participated in a panel discussion at an SAA Annual Meeting. Once, the entire ACA Board took the current exam. A few Certified Archivists from the initial class took the exam rather than submit a petition for certification maintenance. These folks were pleased, if not actually pleasantly surprised, with the test.
The process of exam development includes continuing evaluation and revision by archival practitioners for the purpose of producing an effective test.