- Staff photo by Don Himsel
Stuart Kahl is the Chief Executive Officer of Measured Progress in Dover.
- Staff photo by Don Himsel
Steve Scribner, quality assurance coordinator at Measured Progress, works at the Dover facility Thursday, January 28, 2010.
- Staff photo by Don Himsel
Material produced by Measured Progress is carefully tracked, organized and stored.
Company that produces standardized tests based in NH
DOVER – In a classroom somewhere in America, the kid nervously chewing a No. 2 pencil during a statewide graduation exam probably has no idea that a few of the questions don’t count toward his score.
They’re kind of like questions-in-training: An expert has decided they might be suitable for an exam, and now they’re in the test phase.
For example, if the question stumps 99 out of 100 kids, there’s something wrong with it.
“Field testing” is one of hundreds of controls that go into making and scoring a standardized test. It’s a process that those in the industry emphatically describe as a science, not an art.
They’ve developed procedures for everything from how to make sure a child’s demographics don’t preclude her from understanding a question to what to do if a student throws up on his exam.
“In any other field, this would be the most objective process in the world,” said Stuart Kahl, chief executive officer and cofounder of standardized testing company Measured Progress. “But people always say, ‘This is so subjective.’ … That’s baloney.”
Measured Progress company develops and scores standardized tests for more than 20 states, including New Hampshire. With 450 full-time employees nationwide, it’s based in a large, two-building campus in Dover.
This time of year is quiet for Measured Progress. The forklifts that stack boxes of tests 40 feet high in the warehouse are idle. The conveyor belts that move packets of completed tests through the check-in process are at a halt.
Scores have already been processed for the fall round of testing – New England Common Assessment Program (NECAP) results were delivered to schools last week and released publicly on Thursday – and things will remain this way until the company ramps up for the spring testing season.
In the meantime, the company is concentrating on other projects, including a partnership with the Nashua School District to help the high schools better align their curriculum to testing standards. Nashua is the only district in the state that has this kind of partnership, although the company is in talks with others.
“Nashua is a very forward-thinking district,” said Diane Bailey, national assessment consultant for Measured Progress.
Changing and growing
Measured Progress got its start 27 years ago the way most companies do: with a tiny operation and hopes of creating something bigger. The company, then known as Advanced Systems, was founded in Portsmouth with four employees.
In 2000, Measured Progress transitioned from a for-profit company to a nonprofit.
“We thought we’d make the legal designation match the reality,” Kahl said, laughing, during an interview in his office Thursday.
Today, in addition to the 450 full-time employees, the company brings in as many as 5,000 temporary workers during the busiest testing seasons to help unpack completed student tests, scan them into the system, score them and send the results back to the schools.
Before the recession, the company had a hard time filling some of those jobs, particularly the third-shift scanning positions, said Eric Gilbert, assistant director of operational services. Now, not only are the positions easy to fill, but they also attract people who are overqualified.
“The quality of people coming in is just incredible,” Gilbert said.
Although the training and controls are stringent, many of the temporary employees have a personal reason to make sure the job is done well.
“For most people doing NECAP, it is their kids’ tests coming through,” Gilbert said.
In addition to the main campus in Dover and a shipping warehouse in Rochester, Measured Progress operates facilities in Colorado, Kentucky and New York.
The company writes general standardized tests for 12 states – including New Hampshire, Maine, Rhode Island and Vermont (the NECAP states). And while Measured Progress has a lot of control over what goes into those exams, each state sets parameters and has the final say over tests’ appearance, topics and even the wording of every question.
The company also contracts with eight states to write “alternative tests,” which assess the skills of students with disabilities or those for whom English is a second language.
No Child Left Behind, the federal legislation enacted in 2002 that required states to test students and measure their progress, completely flipped the standardized testing industry on its head, Kahl said.
Suddenly, Measured Progress was churning out tests for seven grade levels instead of three. NCLB requires every student in the country to be tested in grades 3-8 and 11 for math, science, reading and writing proficiency.
The NECAP exam was created specifically to meet the new requirements, replacing an old test that measured just grades 3, 6 and 10.
“Everybody had to handle much greater volume,” Kahl said. “Just working with the existing customers, we had to double our capacity.”
The downside to the legislation, Kahl said, is that many states – not including New Hampshire – dropped the essay portions of the tests, making them strictly multiple choice, in order to save money. Teachers began modeling that format in the classroom to practice for statewide testing and end up seeing less of their students’ written work, he said.
No room for error
Developing a single question is a 12-step process that can take years to complete.
Each question is created by an in-house expert known as an “item writer” and then goes through a series of initial reviews to make sure it has the appropriate level of difficultly and can be understood by students with varying approaches to test taking. If the question doesn’t pass, it goes back to a writer.
Questions that make the cut go to a committee, where they’re reviewed for such things as grade-level appropriateness, before being passed on to the publishing department to proof grammar, punctuation and spelling. Some of that work can be mundane, such as checking questions to make sure they’re numbered or lettered appropriately.
Next, the state gets a chance to review the question. Later, there is a “bias” review designed to make sure questions aren’t culturally offensive and are appropriate to the region where the testing will take place. A question may be worded differently for students in New Hampshire than for those in New Mexico.
The final steps include field testing, statistical analysis of how students responded to the question and final selection.
“It’s a daunting and fascinating process,” said Pat Ross, external relations manager for Measured Progress.
Kahl said the vast majority of the questions that are developed make it through the process.
“I would say our hit rate is pretty high,” he said. “Maybe 90 percent of what’s developed from scratch.”
Back in the warehouse, Gilbert says the same care is given to the process of unpacking and scanning the tests once the schools return them. Every piece of paper in every test booklet needs to be accounted for – including the scrap paper, he said.
“If you manufacture Matchbox cars or flowerpots, it’s no big deal if something falls off the assembly line,” Gilbert said. “Here, you can’t miss on thing. … It could literally be the difference between a kid graduating or not.”
New Hampshire doesn’t require a standardized test in order for students to graduate, but some other states do.
The process is similarly streamlined on the scoring end. For essay and problem-solving math questions that need to be scored manually, Kahl said, employees are given such strict guidelines that there isn’t much room for subjectivity.
After questions are developed, experts set parameters for what a good answer should look like, ranking types of responses from 0-4.
They put together instructions for scorers with guidelines describing the characteristics of an answer that earns a 4 compared to that of a 3 or 2 and so on.
The pay for scorers starts at $11 an hour, and most states require a college degree. Some scorers are former teachers, but not all of them.
“We do get a lot of educators, but we get people from all walks of life,” said Jonathan Nash, assistant director of scoring. “You can actually score something that matches your temperament.”
Not surprisingly, scorers go through a training process that requires them to achieve a certain level of agreement with the expert scorers in order to continue working for Measured Progress. Scorers are also compared to one another in what’s called double-blind scoring.
Rather than grading one test, scorers look at individual questions from a variety of students. They never know the name or background of the student whose test they’re examining.
Although the scoring room is empty right now, filled with blank computer screens, the calm inside these big red buildings in Dover won’t last much longer. Soon, the company will ramp up for the spring round of testing. In New Hampshire, that test is focused on science.
“We’ll be straight out from February through July,” Gilbert said.
Ashley Smith can be reached at 594-6446 or firstname.lastname@example.org.