View All News

Exam Development Process and Response to State of Georgia Comments and Inquiry

May 3, 2019 - 03:59:51 PM

Exam Development Process and Response to State of Georgia Comments and Inquiry

ABC’s 2017 Standardized Exams were developed through a rigorous psychometric process that began with job analyses conducted in 2014-2015. Through this process, more than 7,000 industry stakeholders provided input on the significance of approximately 730 job tasks. This data was analyzed to prepare new exam content outlines (Need-to-Know Criteria) reflecting the most widely performed and significant job tasks. Compared to prior editions of the Need-to-Know Criteria, these new outlines include:

  • More clearly stated and streamlined job task statements;
  • A fixed and clearly communicated number of calculation items per exam form;
  • New information detailing the number of recall, application, and analysis level items included in each content area; and
  • New information indicating the basic, intermediate, or advanced level for each statement of supporting knowledge.

More than 1,000 new items (“test questions”) were written and closely scrutinized against the new Need-to-Know Criteria and best item writing practices to avoid common item flaws or bias over the course of 2016. Much of this new content was integrated into the 2017 Standardized Exams which, after extensive review and refinement, were made ready for implementation beginning May 1, 2017.

Each phase of exam development was carried out by diverse committees of highly-qualified subject matter expert volunteers from across the United States and Canada under the guidance of ABC’s psychometric partner, PSI Services LLC.

Standardized Exams

The development of a truly standardized exam that can be utilized by ABC members throughout North America and around the world was the primary objective of the 2017 standardized exams. ABC standardized exams are utilized by more than 60 North American authorities and regulations on single subjects sometimes differ between jurisdictions and/or levels of governance – federal limits may be less stringent than state/provincial limits, which may be less stringent than local or intra-utility limits. Given these issues, our experts’ determination was that regulations-based content was inappropriate for a standardized exam administered across so many jurisdictions.

To be clear, ABC strongly believes regulations are important for operators to know and follow; however, those regulations are subject to regular/arbitrary change whereas the fundamental physical, chemical, biological, and mechanical concepts that underlie operations are not. Testing an operator’s ability to recall specifics from regulations is an exercise in memory, whereas testing an operator’s understanding of concepts is an exercise in competency and effectiveness to operate their facilities/system within any constraints including those set by regulations. By eliminating diverging local, state/provincial, and federal regulations items and focusing on core concepts, the exams can be used by a wider audience and provide additional evaluation of an operator’s knowledge of fundamentals. This will promote and improve reciprocity between all certification entities in addition to giving a more accurate measurement of a candidate’s competency.

The certification process, which generally relies on a combination of education, experience, and examination, offers a number of opportunities for certifying authorities to emphasize the importance of locally-applicable regulations. Mandatory training prior to examination, acknowledgement of responsibilities at the time of application, and tailored continuing education requirements for recertification are all tools commonly implemented for this purpose.

An Exam for All Jurisdictions

In order to provide a standardized resource to all ABC members; address broad concerns on diverging local, state/provincial, and federal regulations; and best invest volunteer time and Association funds; the 2017 Standardized Exams were developed to serve both United States and Canadian certification programs. To accommodate this shift, weights and measures are provided in both US Customary and metric units.

Pre-test Items

The 2017 standardized exams include 10 extra items that have not been used on previous versions of the exam. These are known as “pre-test” items and allow ABC to gather valuable data about the new items before they are included as scored items on future exams. Pre-test items are unidentified and scattered throughout the exam to ensure candidates answer them with the same care in which they address scored items. The pre-test items are not included in the candidate’s final score.

All of these comments and issues are related to water and wastewater operator exams unless otherwise noted.

  1. Some people did not like the wholesale change out of the questions on these exams, but there is general understanding that there was too much knowledge of actual questions and answers on the exams because of a failure of the Certification Board or ABC to keep them refreshed on a more regular basis.
  2. Some esoterica on methods and procedures is important, but there may be a few too many questions about very outdated methods that are not utilized by the common operator.
    1. Every topic on the exam was essentially mentioned as important and being done on a regular basis by at least 50% of the operators that responded to the job task analysis.
  3. Agree with the addition of more equipment troubleshooting questions in the new format, but people were not initially prepared for this type of question.
    1. Troubleshooting is a key skill needed by competent operators. These types of items have always been included on the exam, but there are more technologies and treatment types to address now, potentially leading to more troubleshooting items than in the past.
  4. Class 3 exams included a good bit of advanced laboratory knowledge without any requirements for more intensive training in this knowledge area. This is a big flaw in the format of class 3 operator exams compared to the old format where this was only expected on class 2 and higher exams.
    1. ABC released the revised need to know several months ahead of exam implementation, and Georgia posted it to its web site as soon as it was available and sent it ot the major training organizations in the state.
  5. Microbiological methods seem to have greater prominence on new class 3 wastewater exam, but level of knowledge down to genus and species of organisms is a little too complex for entry level operator.
    1. We will take this under advisement and pass this comment to the appropriate examination scheme committee.
  6. Math is important for operators, but the complexity of problems presented on exams may be too high level compared the corresponding certification level.
    1. Based on feedback from the job task analysis, the math items are appropriate for the exam levels.
  7. Too many engineering/design questions in math versus more important process control calculations.
    1. We will take this under advisement and pass this comment to the appropriate examination scheme committee.
  8. There may be too many two normal equation (ratio) math questions on lower level exams.
    1. I am not sure how many of these are on the exam, but I don’t think it is more than two.
  9. Too many complex multilevel math problems on lower certification level tests where calculations are needed before choosing appropriate formulas.
    1. We tried to insure the appropriate formulas were on the formula/conversion sheet. There are instances where more than one formula from the sheet are needed.
  10. Is our certification board reviewing exam comments and taking appropriate actions such as limiting tricky questions that may be slightly flawed and have a high percentage of incorrect responses?
    1. Exam comments are reviewed by the ABC Scheme Committee for each exam. The exams are prepared by ABC, and the Board has no direct control over the items chosen. We have tried to make sure Georgia operators are involved in the various exam Scheme committees.
  11. There are no well questions on the class 3 water operator exam even though there are many class 3 level systems in Georgia that run well water exclusively.
    1. Well water items are on the GA Class IV exam, but GA EPD makes the decision as to what level operator a system must have.
  12. Too many anaerobic digestion questions on some wastewater operator exams.
    1. There are no anaerobic digester question on the Class II exams. They are now only on the upper level exams.
  13. Too many questions on class 3 wastewater exam that are about tertiary treatment techniques such as physical chemical treatment methods. Some of these questions are okay because a minority of plants in Georgia have these systems required to meet lower NPDES permit limits.
    1. Nationally, treatment requirements are increasing and including more tertiary treatment, which was reflected in the job task analysis results. Georgia is moving in that direction as evidenced by the new NPDES permits being issued, requiring utilities to upgrade treatment. In this instance, the exams made the change before the treatment plants.
  14. Why are there little to no land application system questions on Georgia wastewater operator exams when there are so many land application systems in Georgia?
    1. Land application system items are addressed on the Georgia Class IV exam, but EPD may be requiring these systems to obtain Class III licenses due to other technologies employed.
  15. Might be a little too much for a wastewater class 3 operator to understand all of the concepts of biological nutrient removal.
  16. Is Georgia certification board helping with selection of questions that is more relevant to the types of treatment [plants and unit processes that are more common in Georgia. We realize these certifications have a reciprocity aspect to them to they need to be nationally relevant, but is there a proper balance on this issue.
    1. There are Georgia operators on some of the ABC examination scheme committees who help select items for the exams. The Board has chosen to use the exams as developed by ABC because it does not have the resources to develop the exams from scratch and follow all of the psychometric principles required to develop psychometrically sound, legally defensible exams. Board members are allowed to review the exams; they are not allowed to change any of the questions.

Please note that ABC will be releasing new exam forms during the summer of 2019. These will be refreshed exams with new content that has been beta tested over the past two years.

View All News