WASHINGTON — As the independent review of last December’s test flight of Boeing’s CST-100 Starliner commercial crew vehicle nears completion, the company said it will perform more rigorous testing to catch errors that slipped through on that flight.
In a Feb. 28 briefing at Boeing offices here, John Mulholland, vice president and program manager of Starliner, said the company is continuing an audit of the software on the spacecraft after two significant errors were found during its two-day uncrewed test flight, while making plans to perform more rigorous testing prior to future missions.
“We know we need to improve, particularly with rebuilding trust with our customer, and we pledge our discipline and commitment to doing so,” he said. “We’re going to apply additional rigor to systems engineering and software development.”
That test flight, known as Orbital Flight Test (OFT), was shortened to two days, and a planned docking with the International Space Station cancelled, because a problem with a mission elapsed timer on the spacecraft kept it from performing an orbital insertion burn as planned shortly after separation from the rocket’s upper stage. An investigation found that the spacecraft’s timer was initialized at the wrong time during the launch countdown, causing it to be off by 11 hours.
At a Feb. 6 meeting of NASA’s Aerospace Safety Advisory Panel, members said they had been briefed about a second software problem, called a “valve mapping error,” for the thrusters on the Starliner’s service module. That problem could have caused the service module to collide with the crew module after separation just before reentry, damaging the module and putting a safe landing in jeopardy. Engineers found the problem while Starliner was in orbit and transmitted corrected software to the spacecraft about three hours before landing.
Mulholland said that the timer problem was not corrected in prelaunch testing with United Launch Alliance because the company split testing of the spacecraft into different phases of the mission. For launch, the tests ended immediately after spacecraft separation, and thus did not detect the timer offset. “If we would have run the integrated test with ULA through the first orbital insertion burn timeframe, we would have seen that we would have missed the orbital insertion burn because the timing was corrupt,” he said.
Going forward, he said Boeing will test Starliner operations from launch through docking, and from undocking to landing. That wasn’t done earlier because of the length of such tests: more than a day from launch to docking. “The team thought at the time it was more logical to break these mission phases into chunks and do a lot of testing in those smaller chunks,” he said.
The valve mapping problem involved the use of what Mulholland called a “legacy propulsion controller” on the service module. One mapping, which identified thrusters and valves in software, is needed when the service module is attached to the crew module while another is required for use after separation.
“Unfortunately, that requirement was not picked up” in interface control documents for that propulsion controller, he said. “The only thing that was picked up was the one jet map for the integrated spacecraft and we missed the jet map that was required for the service module after separation.”
That oversight wasn’t caught in testing because the controller itself was not available when the software was tested because it was being used for a service module hotfire test. The emulator used in its place didn’t allow engineers to identify the missing jet mapping.
In the future, Mulholland said they’ll more closely study hardware requirements for testing. “We’re not only going to define exactly what tests have to be performed, but we’re going to require that define exactly what the hardware configuration needs to be in the lab,” he said.
At a Feb. 7 briefing, Boeing announced it would review all the Starliner software, accounting for about one million lines of code. Mulholland said that audit had completed all of the “high” and “medium” items in terms of complexity of their logic, and most of the low-complexity ones. That audit found a few gaps in testing that engineers are now following up on, but no evidence of additional software anomalies.
Asked why some testing was overlooked, Mulholland said it wasn’t a matter of cost-cutting or otherwise taking deliberate shortcuts. “They did an abundance of testing and in certain areas we obviously have gaps to go fill,” he said of the Starliner team.
Other aspects of Starliner performed well during the abbreviated test flight with only minor technical issues. Engineers are still investigating a communications problem that hindered initial efforts to recover the spacecraft after the timing problem kept it from performing its orbit insertion burn. That problem took place 37 times during the mission, all but one over the same area, northern Europe and Russia, he said, with the sole exception a known case where the antenna falsely thought it was locked.
Mulholland said it’s not clear yet if the problem is caused by interference unique to that area or if the specific selection of antennas for communicating with relay satellites made it more susceptible to ordinary interference. “We really need to look a little bit deeper into that before we give any final results,” he said.
The investigation into the communications problem is unlikely to be completed before an independent review team presents its results. While the briefing was in progress, NASA announced another media briefing was scheduled for March 6 to discuss the findings from the rest of that independent team’s work, with both NASA and Boeing personnel scheduled to participate.
Mulholland declined to speculate on whether a second OFT mission should be flown before performing a crewed Starliner flight, or when that flight would take place. “The timeframe between now and the next flight is going to be determined by us methodically working our way through that audit process,” he said, including any problems it might yet uncover. “It’s a little too early” to set that schedule, he argued.
A decision on whether the next flight will be uncrewed or crewed, he added, will ultimately be made by NASA and not Boeing. “NASA is doing the evaluation of that now, and it’s their decision on which flight will be next.”
Congress will be closely watching that decision and other aspects of NASA’s overall commercial crew program. “It is absolutely something we’re following and concerned about,” said Rep. Kendra Horn (D-Okla.), chair of the House space subcommittee, in a Feb. 28 interview prior to the Boeing briefing.
“We’re looking at something that has slipped. There have been problems with both of the contractors,” she said, a reference to SpaceX, the other commercial crew provider, which suffered the loss of a Crew Dragon spacecraft last April during testing for a planned in-flight abort test. “That’s why it’s important that NASA have that ability to direct and oversee and have access at all parts of this process.”
