The joint NASA and Boeing Independent Review Team formed following the anomalies during the company’s uncrewed Orbital Flight Test as a part of the agency’s Commercial Crew Program has completed its initial investigation.
The team was tasked with reviewing three primary anomalies experienced during the mission: two software coding errors and unanticipated loss of space-to-ground communication capability. During the investigation, the team identified several technical and organizational issues related to Boeing’s work. Separate from the independent team, NASA reviewed its role in the flight test and identified several areas where the agency can improve its level of participation and involvement into company’s processes.
While the review team, NASA and Boeing have made significant progress during the last month, more work will be required to inform the agency’s decision of whether Boeing will need to perform another uncrewed test flight of the Starliner system. NASA will determine if a repeat of the flight will be needed after Boeing has presented its detailed resolution and rework plan and NASA has independently assessed the thoroughness of that plan.
NASA also will perform an evaluation of the workplace culture of Boeing ahead of crewed test flights through an Organizational Safety Assessment (OSA). The goal of the OSA is to provide a comprehensive safety assessment through individual employee interviews with a sampling from a cross-section of personnel, including senior managers, mid-level management and supervision, and engineers and technicians at various sites.
Further, NASA will designate the anomalies experienced during the mission as a high visibility close call. As there were no injuries during the flight, this close call designation is where the potential for a significant mishap could have occurred and should be investigated to understand the risk exposure and the root cause(s) that placed equipment or individuals at risk. Since 2004, the year NASA updated this procedural requirement, NASA has designated about 24 high visibility close calls. For example, in July 2013, astronaut Luca Parmitano discovered a leak in his spacesuit that could have resulted in asphyxiation; as a result, that incident also was given the same designation.
Description of the three primary anomalies:
– Mission Elapsed Timer (MET): Following spacecraft separation with the Atlas V launch vehicle, Boeing’s CST-100 Starliner is programmed to execute a few maneuvers tied to the mission timer. Because of an error in the coding, the Starliner synced its clock with the rocket before the terminal count had begun, which is when the rocket sets the correct time for a designated T-0. This led to the spacecraft thinking it was at a different point in the mission following separation, and it did not conduct the correct maneuvers.
– Service Module Disposal Burn: Following the MET anomaly, Boeing and NASA reviewed other phases of flight where software coding could impact mission success. This review resulted in the team discovering and correcting a software issue during Starliner’s crew and service module separation sequence. The correction ensured a successful separation and disposal of the service module.
– Space-to-Ground Communication (S/G): An Intermittent S/G forward link issue impeded the flight control team’s ability to command and control Starliner during the mission and could impede reliable voice communication with crew during a flight with astronauts.
What the Review Team Found and Recommends
The review team’s analysis identified 61 corrective and preventative actions to address the two software anomalies; those actions are organized into four categories to help manage and execute the scope of the work. Below are the four categories and examples of the resulting actions that Boeing has already begun working on:
– Perform code modifications: Boeing will review and correct the coding for the mission elapsed timer and service module disposal burn.
– Improve focused systems engineering: Boeing will strengthen its review process including better peer and control board reviews, and improve its software process training.
– Improve software testing: Boeing will increase the fidelity in the testing of its software during all phases of flight. This includes improved end-to-end testing with the simulations, or emulators, similar enough to the actual flight system to adequately uncover issues.
– Ensure product integrity: Boeing will check its software coding as hardware design changes are implemented into its system design.
Boeing already has accepted the full action list as defined by the review team and is in the process of refining its implementation schedule and incorporating this work into its plans with multiple actions already underway. As work continues, NASA and Boeing have asked the joint review team to track their progress and execution of each action.
The review team also is continuing its investigation of the intermittent space-to-ground forward link issue that impeded the flight control team’s ability to command and control the spacecraft. The team has identified the technical root cause as radiofrequency interference with the communications system. While the team has recommended specific hardware improvements already in work by the company, the full assessment and resulting recommendations will continue through March.
In addition to the technical issues described above, the review team identified organizational issues that contributed to the anomalies. In response, Boeing plans to institutionalize improvements in its engineering board authority, operational testing practices for both hardware and software, and the standardization problem review and approval processes.
NASA’s Internal Review and Forward Work
Concurrent with the independent review team, NASA performed an in-depth assessment of its role and identified multiple actions the agency will take to complement the actions planned by the Boeing Starliner team.
NASA has developed a comprehensive plan to ensure the agency has full coverage of critical Boeing software improvements. This plan also includes reassessing all hazard report verifications of software controls, re-opening hazard reports as necessary, reviewing software verification plans, and reviewing the adequacy of the test environments and audits of scripts used in testing. NASA also will co-locate personnel with the Boeing software team, increase support to the Boeing Software Change Control Board and the problem resolution process. NASA also plans to perform additional flight software audits.
In addition, NASA will improve its software independent verification and validation performance and overall NASA insight into this area. NASA also plans to address areas where additional NASA “safety nets” may be beneficial for all providers.
NASA also will take several actions to improve the overall system integration of Starliner, including revisiting all hazard causes related to system interfaces to ensure hazards are fully defined, well-controlled, and properly verified; and reviewing existing Interface Control Documents to ensure NASA understands where the definitive data sources are for subsystem interfaces.
