A NASA-funded survey of
commercial pilots was so flawed
that the results should not be used to assess the safety
of the nation’s air travel system, according to a National Research Council review of the U.S. space agency’s National Aviation Operations Monitoring Service (NAOMS) project
. Between 2001 and the end of 2004, NAOMS contractors surveyed 29,000 pilots in an effort
to statistically track rates of safety-related incidents in the nation’s airspace system and detect trends in those rates over time. In 2005, NASA handed off the survey to the Air Line Pilots Association, which discontinued the telephone survey in favor of less expensive Internet surveys.
NASA caught heat for its close out of the NAOMS project when the Associated Press reported in October 2007 that the agency was withholding survey results
showing U.S. pilots experienced more bird strikes, near mid-air collisions and runway incursions than other government monitoring systems revealed at the time.
The House Science and Technology Committee held a hearing on the NAOMS controversy where then-NASA Administrator Mike Griffin told lawmakers the agency had erred in shutting down the safety survey program without fully analyzing the data and publishing a final report, but insisted there was no attempt to hide the information from the public. He said NASA canceled NAOMS after a 2004 review by the National Academy of Sciences raised concerns about the survey’s reliability.
According to the National Research Council
review released Oct. 28
, NASA and its industry partner, Batelle Memorial Institute, used a number of generally accepted practices, such as computer-assisted telephone interviews and professionally trained interviewers. But numerous flaws in the design and implementation of the study marred the usefulness of its data, the National Research Council
found. For example, many of the survey questions were poorly worded or vague, making them potentially difficult for the pilots to digest during a telephone interview. In addition, the survey was restricted to pilots with specific flight certifications, which led to an over-representation of pilots of wide-body aircraft, and an under-representation of those using small aircraft, the review found.
Griffin made some of the same points during the House Science and Technology Committee’s 2007 hearing, telling lawmakers that if the survey results were to be believed, unruly passengers were forcing four unplanned landings every day, for example.
“That’s not happening. So it causes us to suspect the quality of this data,” he said.
NASA, however, ultimately bowed to congressional pressure to release the survey results by the end of 2007, posting 16,000 pages of hard-to-decipher documents on its Web site on New Year’s Eve.
