Questions About CBT Linger
SED Stands By Pass/Fail Policy

By Simon Eskow

New York—The State Education Department is standing by its policy of reporting pass/fail results to candidates taking the computer-based Uniform CPA Examination.

New York is the only jurisdiction in the U.S. that does not report numeric scores to candidates, a position the New York State Society of CPAs has opposed.

The NYSSCPA has asked the state to switch to numeric reporting to help candidates develop a strategy for studying if they must retake one or more of the four parts of the Computer-Based Test (CBT). But in an April 18 letter to David Evangelista, Accountancy Board Executive Secretary Daniel Dustin wrote that changing to numeric reporting would require a lengthy process, and could be redundant in light of new diagnostics being developed by the CBT’s architects, the American Institute of CPAs.

“It is my understanding that the AICPA’s Board of Examiners is in the process of discussing enhancements to the diagnostics reports generated for (CBT),” Dustin wrote.

“It would appear more appropriate for (the Society and the SED) to express their concerns over the deficiencies in the current diagnostic reporting model to the AICPA and its Board of Examiners to ensure that the best, most comprehensive diagnostic reporting model is developed and provided to (candidates) as early as possible,” Dustin later stated in his letter.

While it remains steadfast on that one issue, the SED has seen eye-to-eye with NYSSCPA members on other problems that have emerged with the CBT. These have included difficulties students encountered at some testing facilities, problems with the registration process, the timeliness of issuing test results, and other issues.

“The state’s been trying to work on those other issues,” said Evangelista, a member of the Society’s Political Action Committee who has worked with Dustin on CBT. “They’re going to hold a firm line on this grade issue.”

Evangelista met with Dustin in April to discuss some of the Society’s reservations about the CBT, which were first raised in a letter submitted to Dustin in January regarding the pass/fail policy and other problems.

Evangelista said that it seemed as if the state actually had to go to some length to prevent candidates from receiving numeric grades. The CBT’s grader, NASBA, gives a numeric score to all CBT examinations, and automatically rescores grades of 70 to 74, to ensure that a student is given the benefit of the doubt in attaining a passing grade of 75. Grades below 70 are not regraded, leading the SED to believe that numeric grades at lower scores may be statistically flawed, and thus not give the student an accurate picture of how they did on a particular section.

Evangelista said he didn’t agree with this point of view.

“Nobody’s looking at that grade as the only factor,” Evangelista said. “They’re looking at that for a general idea of what to study and how to plan for the future. So, maybe I got a 50 on one part of the test and a 65 on another part. Knowing that, maybe I can spend more time on the part where I scored 50.”

The state accidentally distributed grades to some students in the first testing window this year. Evangelista said staff members at his firm who had taken the exam were happy to have received a grade. Normally, under current circumstances, the only way students can see how they have performed on the CBT is through a diagnostic report released with the test results, which Evangelista says is not a good indication of anything.

“Right now, that’s not working,” he said. “The only information they give now is some percentages in a bar chart that mean nothing.”

An improved diagnostic report is expected sometime in 2006.

Timeliness of reporting test results has also been an issue. Currently, the grades are issued once per testing window, but Dustin told the Board for Public Accountancy in April that the AICPA is working on issuing results on a continuous basis, which could reduce the turnaround time to one month.

Dustin is also trying to work out other issues. For instance, candidates have complained of having to sit for exams in test centers alongside high school students, whose behavior has been reportedly distracting in some instances. Officials are working on an idea to “box out” specific times for candidates to sit for the exam without such distractions. There have also been problems with proctors at testing locations not following protocols for dealing with computer malfunctions. Dustin has also said that there has been much concern about the continuing rise in costs for taking the exam, which some Society members have called an obstacle for candidates to enter the profession.

With many questions hovering over the quality and future of the exam, the NYSSCPA’s Board of Directors in May agreed to establish a task force that will monitor CBT developments. Members will be appointed to the task force at some point after June 1.