WASHINGTON — Boeing will reverify all the software on its CST-100 Starliner commercial crew spacecraft after an ongoing investigation found “numerous” problems in the original development process that allowed at least two major problems to escape detection.
In a call with reporters Feb. 7, NASA and Boeing officials said they had made no decisions about whether a second uncrewed test flight, or Orbital Flight Test (OFT) of the spacecraft will be needed, but that there were significant issues with the spacecraft, in particular how its software was developed, that need to be corrected.
“We do think that the OFT flight had a lot of anomalies,” said NASA Administrator Jim Bridenstine during the call.
Of particular concern is the software on Starliner. One issue, found immediately after separating from its upper stage, was a timer offset that prevented the spacecraft from firing its thrusters as planned to reach orbit. While the spacecraft was able to reach orbit, it consumed more fuel than planned, ruling out a planned International Space Station docking and ending the mission just two days after launch.
John Mulholland, vice president and program manager for the Starliner program at Boeing, said the Starliner software is intended to initialize its mission elapsed timer from the Atlas 5 launch vehicle, but only in the “terminal count” phase of the countdown. The software, he said, lacked that terminal count requirement. “So, it polled an incorrect mission elapsed time from the launch vehicle, which then gave us an 11-hour mismatch,” he said.
The second problem, revealed Feb. 6 at a meeting of the Aerospace Safety Advisory Panel (ASAP), was a “valve mapping error” for the thrusters in the vehicle’s service module. Those thrusters perform a “disposal burn” of the service module after separating from the crew module just before reentry.
Mulholland said the valves were configured for conditions in normal flight for that disposal burn, which, had it not been corrected, could have pushed the service module into the crew module. That could have caused the crew capsule to become unstable, requiring additional thruster firings to reorient itself, or have damaged the capsule’s heat shield.
The second error was detected during the review of the spacecraft software on the ground after the timer problem took place. Mulholland said engineers found the thruster software issue late Dec. 21, with the corrected and reverified code uploaded to the spacecraft around 5 a.m. Eastern Dec. 22, or about three hours before the spacecraft landed at White Sands Missile Range in New Mexico.
“We went hunting immediately after our first software problem, and we found one,” said Jim Chilton, senior vice president of Boeing Space and Launch, of the thruster error. “I don’t think we would have found it if we hadn’t gone looking right after that first one.”
The two software problems are signs of a more fundamental issue, NASA argued. “The real problem is that we had numerous process escapes in the design, development and test cycle for software,” said Doug Loverro, NASA associate administrator for human exploration and operations. “As we go forward, that is what we’re going to be concentrating on.”
The software, Mulholland said, is supposed to go through a “pretty standard” development process where code is written and goes through peer reviews and a series of tests leading up to formal qualification tests. “There are a number of checks along the way that are designed to uncover and correct code errors as early as you can,” he said.
However, there were “breakdowns in multiple areas in that process” discovered by the independent review team, Loverro said. “For each of these two problems that we know about, some of that breakdown was in different spots and some was in the same spot of the process.”
“The process broke down in many areas for each of these things, and that’s one of the reasons why we have to go back and do such a thorough review of all of the software,” he added.
Mulholland said that Boeing planned to review all of the software developed for Starliner, which totals about one million lines of code. “We believe we need to go back and reverify all of the software code,” he said. Boeing will consult with NASA and the independent review team to confirm that plan, but didn’t state how long that reverification process would take.
The overall investigation into the problems encountered with the mission, which also includes communications glitches not related to the software, is still in progress. Bridenstine said that the investigation should be complete by the end of the month.
He suggested the only reason NASA and Boeing held this briefing was that ASAP had been briefed about an interim report on the ongoing investigation, which ASAP then discussed at its Feb. 6 public meeting. “But in the interests of transparency, and some of the things that I saw online yesterday, I wanted to make sure that everybody knew kind of where we were in the investigation,” he said.
Because that investigation is ongoing, he said it was premature to decide whether a second uncrewed test flight will be needed, something Loverro agreed with. “You don’t go ahead and do flight tests to verify that you’ve solved problems. You do flight tests to look at a holistic picture of the system,” Loverro said. The need for another flight test, he said, will only become clear after completing the reviews and fixing the process errors.
That will include a full organizational safety assessment of Boeing, which the ASAP also revealed at its meeting. Part of the reason for that review, Loverro said, was “press reports that we’ve seen from other parts of Boeing,” an apparent reference to problems with its 737 Max airliner, which has been grounded since two crashes blamed on the plane’s new software. “There could possibly be process issues at Boeing, and so we want to understand what the culture is at Boeing that may have led to that.”
“This just continues to show that we need to be vigilant,” said Kathy Lueders, manager of NASA’s commercial crew program, of the overall investigation. “We’ll continue to take the lessons and the items they are bringing up in their very thorough review forward, and continue to get better.”
