Updated 7:35 p.mm. Eastern with Boeing comments.
MOUNTAIN VIEW, Calif. — A NASA safety panel is recommending a review of Boeing’s software verification processes after revealing there was a second software problem during a CST-100 Starliner test flight that could have led to a “catastrophic” failure.
That new software problem, not previously discussed by NASA or Boeing, was discussed during a Feb. 6 meeting of NASA’s Aerospace Safety Advisory Panel that examined the December uncrewed test flight of Starliner that was cut short by a timer error.
That anomaly was discovered during ground testing while the spacecraft was in orbit, panel member Paul Hill said. “While this anomaly was corrected in flight, if it had gone uncorrected, it would have led to erroneous thruster firings and uncontrolled motion during [service module] separation for deorbit, with the potential for a catastrophic spacecraft failure,” he said.
The exact cause of the failure remains under investigation by Boeing and NASA, who are also still examining the timer failure previously reported. Those problems, Hill said, suggested broader issues with how Boeing develops and tests the software used by the spacecraft.
“The panel has a larger concern with the rigor of Boeing’s verification processes,” he said. The panel called for reviews of Boeing’s flight software integration and testing processes. “Further, with confidence at risk for a spacecraft that is intended to carry humans in space, the panel recommends an even broader Boeing assessment of, and corrective actions in, Boeing’s [systems engineering and integration] processes and verification testing.”
The panel added that all those investigations and reviews be completed as “required input for a formal NASA review to determine flight readiness for either another uncrewed flight test or proceeding directly to a crewed test flight.”
In a statement late Feb. 6, Boeing said it accepted the “suggestions” from the panel as well as recommendations from a separate NASA-Boeing independent review team (IRT) investigating the issues with the December Starliner flight. Neither NASA nor Boeing has released any summary of that independent team’s work prior to the statement.
In the statement, Boeing described the new software problem as “a valve mapping software issue, which was diagnosed and fixed in flight.” According to the company, “That error in the software would have resulted in an incorrect thruster separation and disposal burn. What would have resulted from that is unclear.”
Boeing added that the independent review team also identified what it believes to be the root cause of the timer problem and offered corrective actions and recommendations, but did not disclose that cause. The team is also making “significant progress” on resolving communications dropouts that exacerbated problems early in the test flight.
Boeing said Jan. 30 that, while no decision had been made yet about performing a second uncrewed test flight, it was taking a $410 million charge against its earnings in part to cover the costs of a second uncrewed flight. In a Jan. 30 interview, Jim Chilton, senior vice president for space and launch at Boeing Defense and Space, said he expected a decision on flying a second test flight to come by the end of February.
Patricia Sanders, chair of the panel, noted that NASA has decided to proceed with an “organizational safety assessment” with Boeing. NASA announced in 2018 it would conduct such reviews of both Boeing and SpaceX, the other commercial crew company, after SpaceX Chief Executive Elon Musk was seen briefly smoking marijuana during a podcast. NASA, while completing the SpaceX review, deferred the Boeing one, reportedly because of cost issues.
“The review of SpaceX proved to be valuable to both NASA and the company, so it’s a prudent step to execute the same process with the other provider,” she said.
Boeing acknowledged that review in its statement. “Our next task is to build a plan that incorporates IRT recommendations, NASA’s Organizational Safety Assessment (OSA) and any other oversight NASA chooses after considering IRT findings,” Boeing said. “Once NASA approves that plan, we will be able to better estimate timelines for the completion of all tasks. It remains too soon to speculate about next flight dates.”
At the panel meeting, Sanders painted a more optimistic view of SpaceX’s commercial crew work during the meeting. She said SpaceX still has to resolve a number of technical issues, such as the interaction of titanium with nitrogen tetroxide, which was blamed for the explosion of a Crew Dragon spacecraft during preparations for a static fire test of its abort engines in April 2019. But, she added, “the end appears to be in sight” for that work.
“The panel’s assessment of the status of SpaceX is that NASA is at a point where there is not a question of whether they will be flying crew in the near term, but when, and under what risk conditions,” she said.
