Harold, Tnx for the insight. Jim Weaver, K8JE Director, Great Lakes Division ARRL; http://www.arrl.org/ Chairman, ARRL ad hoc Legislative Action Committee 5065 Bethany Rd., Mason, OH 45040 Tel.: 513-459-0142; E-mail: k8je@arrl.org ARRL: The reason Amateur Radio Is! MEMBERS: The reason ARRL Is! -----Original Message----- From: Kramer, Harold, WJ1B [mailto:wj1b@arrl.org] Sent: Wednesday, December 20, 2006 3:34 PM To: arrl-odv Cc: Green, Perry WY1O; Spencer,Mark WA8SME Subject: [arrl-odv:14985] Thoughts on Exams Hi, During the past few days, there have been a few emails on the ODV reflector about Amateur Radio examinations. As educator, of sorts, I thought that I would send along some of my comments about our exams. Warning: this is one of my pet peeves. To my thinking, amateur radio license exams have never been "subject knowledge" exams. They would be more properly classified as "barrier to entry" exams. You simply cannot test for much subject knowledge on a 35 question multiple choice exam, particularly when the questions and answers are available in advance. A barrier to entry exam often just tests the memory, desire and tenacity of the applicant. A competent instructor can overcome this and impart some real amateur radio knowledge, but it takes knowledge, effort and skill on the part of the instructor and a desire to learn beyond just passing the exam on the part of the student. As pointed out in previous discussions, the real learning takes place after the license is obtained. I agree with Joel and others that the subject matter in the question pool is excessively wide. I am currently teaching a General Class course here in HQ and the breadth of knowledge to pass the exam is extremely broad and not very deep. It would take me at least two semesters to cover that much material in the Engineering and Technology Program at the University of Hartford where I teach. Amateur Radio exam questions themselves are poorly written and constructed. There are double negatives, too many similar answers, unclear terminology, etc. To my experience, these exams are not easy for a newcomer. Unfortunately due to the nature of the QPC, we only have limited input into the exam content and the exam construction process. I believe that we have made progress with the latest question pools because of the extraordinary in-house efforts that were made by Mary Hobart and Perry Green. Much of their work was incorporated into the Technician Question Pool. Similarly, the new General Class pool has also been improved. Because of the nature of the QPC, changing the subject matter of the exams is a complex endeavor. However, one thing that we can improve is the exam itself. We could use the established processes that validate multiple choice exams. Unfortunately, Amateur Radio exams are not validated using this disciplined approach. I asked my friend, Dr. Dick Hezel, of Hezel Consulting, one of the top educational consultants in the country, <http://hezel.com/> http://hezel.com/ about this process. I offered to the QPC that the ARRL would pay for an analysis of the new General Class Question Pool before it was released. Unfortunately, the QPC refused our offer to improve these exams. In case you are curious, here's Dick Hezel's response about the type of exam validation that would be required. Dick had a Novice license when he was younger so he knows something about amateur radio: Developing a good test requires several levels of validity testing: * Face validity requires that people familiar with the subject make sure it asks the questions that it purports to ask and that there are no technical errors. * We would review the test for internal validity. We would want to make sure the questions all have only one possible answer and that all other choices are mutually exclusive. . * External validity has to do with whether the test overall does what it is supposed to do. Does it really test knowledge in the area and is that knowledge predictive of radio operator behavior? * Another issue is internal reliability. Does each question contribute to the overall test and to what extent? An item analysis of many tests taken will provide this. The one thing that all amateurs have in common is that we have passed a licensing exam. The Amateur Radio Service could create better constructed and more educationally valid examinations, but our hands are tied because of QPC politics, their lack of knowledge about educational testing and their unwillingness to have a third-party "outsider" educationally validate the exams. Sorry to get on the soapbox but I care a lot about this particular subject! 73 and Happy Holidays to everyone, Harold PS: If you have not received the new ARRL Instructor's Manual for Technician Class License Courses, it is on its way. Harold Kramer, WJ1B Chief Operating Officer ARRL - The National Association for Amateur Radio 225 Main Street Newington, CT 06111 860 594 0220 www.arrl.org/ hkramer@arrl.org