Statement of Allen Li, Director, Acquisition and Sourcing Management, General Accounting Office
Mr. Chairman and Members of the Subcommittee:
I am pleased to be here today to discuss our work, requested by this Subcommittee, on the National Aeronautics and Space Administration’s (NASA) X-33 and X-34 programs. As you know, the
purpose of these efforts was to significantly reduce the cost of access to space by partnering with private industry to develop and demonstrate technologies needed for future reusable launch
vehicles reaching orbit in one stage (single-stage-to-orbit). In essence, these are vehicles whose components-either all or in part-can be utilized on subsequent flights. Both programs were
recently terminated because of significant cost increases caused by problems developing the necessary technologies and flight demonstration vehicles. NASA is now focusing instead on its new
Space Launch Initiative. This is a broader effort to develop the next generation of reusable launch vehicles, referred to as the Second Generation Reusable Launch Vehicle Program (2nd
Generation Program). Today, I will discuss the primary factors that contributed to the difficulties experienced by the X-33 and X-34 programs and the steps needed to avoid repeating those
problems within the 2nd Generation Program.
In brief, the X-33 and X-34 programs experienced difficulties achieving their goals primarily because NASA did not develop realistic cost estimates, timely acquisition and risk management
plans, and adequate and realistic performance goals. In particular, neither program fully assessed the costs associated with developing new, unproven technologies; provided for the financial
reserves needed to deal with technical risks and accommodate normal development delays; developed plans to quantify and mitigate the risks to NASA; or established performance targets
showing a clear path leading to an operational reusable launch vehicle. Underlying these difficulties were problems with the agreements and contracts that established the relationship between
NASA and its industry partners and eventual erosion of commercial prospects for the development of new reusable launch vehicles.
Currently, NASA is in the process of taking steps in the 2nd Generation Program to help avoid problems like those encountered in the X-33 and X-34 programs. While it is too early to tell if
these measures will be sufficient, our review of the two programs has shown that three critical areas need to be addressed. First, the technical complexity involved requires that realistic cost
estimates and risk mitigation plans are developed and the projects are funded accordingly. Second, because the 2nd Generation Program will involve numerous interrelated, complex efforts to
develop technology, NASA needs to ensure that all these efforts move forward in a coordinated manner and that open communication is fostered at all levels. Third, performance measures need
to be implemented and periodically validated to ensure that the rationale for developing specific technology applications merits continued support.
Background
The X-33 and X-34 programs were part of an effort that began in 1994-known as the Reusable Launch Vehicle Technology/Demonstrator Program (Reusable Launch Vehicle Program)-to
pave the way to full-scale, commercially-developed, reusable launch vehicles reaching orbit in one stage. In embarking on the Reusable Launch Vehicle Program, NASA sought to significantly
reduce the cost of developing, producing and operating launch vehicles. NASA’s goal was to reduce payload launch costs from $10,000 per pound on the space shuttle to $1,000 per pound.
It planned to do so, in part, by finding “new ways of doing business” such as using innovative design methods, streamlined acquisition procedures, and creating industry-led partnerships
with cost sharing to manage the development of advanced technology demonstration vehicles. The vehicles were seen as the “stepping stones” in what NASA described as an incremental
flight demonstration program. The strategy was to force technologies from the laboratory into the operating environment.
The X-34 Project started in 1995 as a cooperative agreement between NASA and Orbital Sciences Corporation[1] (Orbital). The project was to demonstrate streamlined management and
procurement, industry cost sharing and lead management, and the economics of reusability. However, the industry team, led by Orbital, withdrew from the agreement in less than 1 year, for a
number of reasons including changes in the projected profitability of the venture. NASA subsequently started a new X-34 program with a smaller vehicle design. It was intended only as a
flight demonstration vehicle to test some of the key features of reusable launch vehicle operations, such as quick turn-around times between launches.
Under the new program, NASA again selected Orbital as its contractor in August 1996, awarding it a fixed price, $49.5 million contract. Under the new contract, Orbital was given lead
responsibility for vehicle design, fabrication, integration, and initial flight testing for powered flight of the X-34 test vehicle. The contract also provided for two options, which were later
exercised, totaling about $17 million for 25 additional experimental flights and, according to a project official, other tasks, including defining how the flight tests would be undertaken. Under
the new effort, NASA’s Marshall Space Flight Center was to develop the engine for the X-34 as part of its Low Cost Booster Technology Project. The initial budget for this development was
about $18.9 million.
In July 1996, NASA and Lockheed Martin Corporation and its industry partners[2] entered into a cooperative agreement for the design, development, and flight-testing of the X-33. [3] The
X-33 was to be an unmanned technology demonstrator. It would take off vertically like a rocket, reaching an altitude of up to 60 miles and speeds to about Mach 13 (13 times the speed of
sound), and land horizontally like an airplane. The X-33 would flight test a range of technologies needed for future launch vehicles, such as thermal protection systems, advanced engine
design and lightweight fuel tanks made of composite materials. The vehicle would not actually achieve orbit, but based on the results of demonstrating the new technologies, NASA envisioned
being in a better position to make a decision on the feasibility and affordability of building a full-scale system. Under the initial terms of the cooperative agreement, NASA’s contribution was
fixed at $912.4 million and its industry partners’ initial contribution was $211.6 million. In view of the potential commercial viability of the launch vehicle and its technologies, the industry
partners also agreed to finance any additional costs.[4] During a test in November 1999, one of the fuel tanks failed due to separation of the composite surface. Following the investigation,
NASA and Lockheed Martin agreed to replace the composite tanks with aluminum tanks.
In February 2001, NASA announced it would not provide any additional funding for the X-33 or X-34 programs under its new Space Launch Initiative. The Space Launch Initiative is
intended to be a more comprehensive, long-range plan to reduce high payload launch costs. NASA’s goal is still to reduce payload launch cost to $1,000 per pound to low Earth orbit but it is
not limited to single-stage-to-orbit concepts. Specifically, the 2nd Generation Program’s objective is to substantially reduce the technical, programmatic, and business risks associated with
developing reusable space transportation systems that are safe, reliable and affordable.
NASA has budgeted about $900 million for the SLI initial effort and, in May 2001, it awarded the first contracts to 22 large and small companies for space transportation system design
requirements, technology risk reduction, and flight demonstration. In subsequent procurements in mid-fiscal year 2003, NASA plans to select at least two competing reusable launch system
designs. The following 2.5 to 3.5 years (through fiscal years 2005 or 2006) will be spent finalizing the preliminary designs of the selected space transportation systems, and maturing the
specific technologies associated with those high-risk, high-priority items needed to develop the selected launch systems.
Factors
Contributing to X-33
and X-34 Program
Difficulties
Undertaking ambitious, technically challenging efforts like the X-33 and X-34 programs-which involve multiple contractors and technologies that have yet to be developed and
proven-requires careful oversight and management. Importantly, accurate and reliable cost estimates need to be developed, technical and program risks need to be anticipated and mitigated,
sound configuration controls need to be in place, and performance needs to be closely monitored. Such undertakings also require a high level of communication and coordination. Not
carefully implementing such project management tools and activities is a recipe for failure. Without realistically estimating costs and risks, and providing the reserves needed to mitigate those
risks, management may not be in a position to effectively deal with the technical problems that cutting-edge projects invariably face.
In fact, we found that NASA did not successfully implement and adhere to a number of critical project management tools and activities. Specifically:
success-oriented schedule that did not allow for major delays. Nevertheless, in September 1999, NASA’s Office of the Inspector General (OIG) reported[5] that NASA’s cost estimate did not
include a risk analysis to quantify technical and schedule uncertainties. Instead, the cost estimate assumed that needed technology would be available on schedule and as planned. According to
the OIG, a risk analysis would have alerted NASA decision-makers to the probability of cost overruns in the program. Since NASA’s contribution to the program was fixed-with Lockheed
Martin and its industry partners responsible for costs exceeding the initial $1.1 billion-X-33 program management concluded that there was no risk of additional government financial
contributions due to cost overruns. They also believed that the projected growth in the launch market and the advantages of a commercial reusable launch vehicle would provide the necessary
incentive to sustain industry contributions.
document risks associated with cost, resource, schedule, and technical aspects of a project and determine the procedures that will be used to manage those risks. In doing so, they help ensure
that a system will meet performance requirements and be delivered on schedule and within budget. A risk management plan for the X-34 was not developed until the program was restructured
in June 2000. Although Lockheed Martin developed a plan to manage technical risks as part of its 1996 cooperative agreement for the X-33, NASA did not develop its own risk management
plan for unique NASA risks until February 2000. The NASA Administrator and the NASA Advisory Council have both commented on the need for risk plans when NASA uses partnering
arrangements such as a cooperative agreement. Furthermore, we found that NASA’s risk mitigation plan for the X-33 program provided no mechanisms for ensuring the completion of the
program if significant cost growth occurred and/or the business case motivating industry participation weakened substantially.
technical, schedule, and cost commitments, and overall acquisition strategy. The program plan addresses these issues as well but also defines the effort’s management structure as well as
program resources, data management, risk management, test and verification, and planned program reviews. Such plans, when done with rigor, would help NASA to define realistic time
frames, identify responsibility for key tasks and deliverables and provide a yardstick by which to measure the progress of the effort.
Lockheed Martin began the design and development of a flight demonstration vehicle. Configuration management plans define the process to be used for defining the functional and physical
characteristics of a product and systematically controlling changes in the design. As such, they enable organizations to establish and maintain the integrity of a product throughout its lifecycle
and prevent the production and use of inconsistent product versions. By the time the plan was implemented, hardware for the demonstration vehicle was already being fabricated.
tank required high levels of communication, and that such communication did not occur in this case.[6] A NASA official told us that some NASA and Lockheed personnel, who had
experience with composite materials and the phenomena identified as one of the probable causes for the tank’s failure, expressed concerns about the tank design. However, because of the
industry-led nature of the cooperative agreement, Lockheed Martin was not required to react to such concerns and did not request additional assistance from NASA.
major programs. Doing so enables agencies to gauge the progress of programs like the X-33 and X-34 and in turn to take quick action when performance goals are not being met. For
example, we reported in August 1999 that NASA’s Fiscal Year 2000 Performance Plan did not include performance targets that established a clear path leading to a reusable launch vehicle
and recommended such targets be established.
Without relying on these important project management tools up front, NASA encountered numerous problems on both the X-33 and X-34 programs. Compounding these difficulties was a
decrease in the projected commercial launch market, which in turn lessened the incentive of NASA’s X-33 industry partners to continue their investments.
In particular, technical problems in developing the X-33’s composite fuel tanks, aerospike engines, heat shield, and avionics system resulted in significant schedule delays and cost overruns.
After two program reviews in 1998 and 1999, the industry partners added a total of $145.6 million to the cooperative agreement to pay for cost overruns and establish a reserve to deal with
future technical problems and schedule delays. However, NASA officials stated that they did not independently develop their own cost estimates for these program events to determine whether
the additional funds provided by industry would be sufficient to complete the program. Also, these technical problems resulted in the planned first flight being delayed until October 2003,
about 4.5 years after the original March 1999 first flight date.
After the composite fuel tank failed during testing in November 1999, according to NASA officials, Lockheed Martin opted not to go forward with the X-33 Program without additional
NASA financial support. Lockheed Martin initially proposed adding $95 million of its own funds to develop a new aluminum tank for the hydrogen fuel, but also requested about $200
million from NASA to help complete the program. Such contributions would have increased the value of the cooperative agreement to about $1.6 billion or about 45 percent (about $500
million) more than the $1.1 billion initial cooperative agreement funding. NASA did not have the reserves available to cover such an increase. The agency did, however, allow Lockheed Martin
to compete, in its 2nd Generation Program solicitation, for the additional funds Lockheed Martin believed it needed to complete the program.
Similarly, NASA started the X-34 Project, and the related NASA engine development project, with limited government funding, an accelerated development schedule, and insufficient reserves
to reduce development risks and ensure a successful test program. Based on a NASA X-34 restructure plan in June 2000, we estimate that NASA’s total funding requirements for the X-34
would have increased to about $348 million-a 307-percent ($263 million) increase from the estimated $86 million budgeted for the vehicle and engine development projects in 1996. Also,
since 1996, the projected first powered flight had slipped about 4 years from September 1998 to October 2002 due to the cumulative effect of added risk mitigation tasks, vehicle and engine
development problems, and testing delays.
Most of the cost increase (about $213 million) was for NASA-directed risk mitigation tasks initiated after both projects started. For example, in response to several project technical reviews
and internal assessments of other NASA programs,[7] the agency developed a restructure plan for the X-34 Project in June 2000. This plan included consolidating the vehicle and engine
projects under one NASA manager. The project would be managed with the NASA project manager having the final decision-making authority; Orbital would be relegated to a more
traditional subordinate contractor role. Under the plan, the contract with Orbital would also be rescoped to include only unpowered flights; Orbital would have to compete for 2nd Generation
Program funding for all the powered flight tests. The plan’s additional risk mitigation activities would have increased the X-34 Project’s funding requirements by an additional $139 million,
which included about $45 million for additional engine testing and hardware; $33 million for an avionics redesign; $42 million for additional project management support and personnel; and
$18 million to create a contingency reserve for future risk mitigation efforts.
Incorporating
Lessons Learned in
2nd Generation
Program
NASA is revising its acquisition and management approach for the 2nd Generation Program. Projects funded under the program will be NASA-led rather than industry-led. NASA also plans
to increase the level of insight into the program’s projects, for example, by providing more formal reviews and varying levels of project documentation from contractors depending on the risk
involved and the contract value. NASA also required that all proposals submitted in response to its research announcement be accompanied by certifiable cost and pricing data. Finally, NASA
discouraged the use of cooperative agreements since these agreements did not prove to be effective contract vehicles for research and development efforts where large investments are required.
While it is too early to tell if the agency measures aimed at avoiding the problems experienced in the X-33 and X-34 programs will be sufficient, these experiences show that three critical areas
need to be addressed. These relate to (1) adequate project funding and cost risk provisions, (2) the effective and efficient coordination and communication required by many individual but
related efforts, and (3) periodically revalidating underlying assumptions by measuring progress toward achieving a new safe, affordable space transportation system that meets NASA’s
requirements.
First, the technical complexity of the 2nd Generation Program requires that NASA develop realistic cost estimates and risk mitigation plans, and accordingly set aside enough funds to cover
the program’s many projects. NASA plans to invest substantially more funds in the 2nd Generation Program than it did in the previous Reusable Launch Vehicle Program, and plans to
provide reserves for mitigating program risk. For example, the agency plans to spend about $3.3 billion over 6 years to define system requirements for competing space transportation
systems and related risk reduction activities. Most of this amount, about $3.1 billion, is for risk-reduction activities, such as the development of new lightweight composite structures, durable
thermal protection systems, and new high performance engine components.
NASA officials told us that an important way they plan to mitigate risk is by ensuring adequate management reserves in the 15- to 20-percent range, or higher if needed. They also
acknowledged the need for adequate program cost estimates on which to base reserve requirements. However, we are still concerned about the timely preparation of cost estimates. The 2nd
Generation deputy program manager stated that, based on the scope of the first contracts awarded, the program office planned to update their cost estimate this summer before NASA
conducted a separate, independent technical review and cost estimate in September 2001. Thus, neither of these important analyses were completed prior to the first contract awards. We
believe that until the program office completes it own updated cost estimate and NASA conducts an independent cost and technical review, a credible estimate of total program costs and the
adequacy of planned reserves will not be available. Also, NASA is still in the process of developing the documentation required for the program, including a risk mitigation plan. NASA policy
requires that key program documentation be finalized and approved prior to implementing a program.
Second, NASA will face coordination and communication challenges in managing the 2nd Generation Program. As noted earlier, NASA recently awarded initial contracts for systems
engineering and various risk reduction efforts to 22 different contractors. Yet to successfully carry out the program NASA must, early on, have coordinated insight into all of the space
transportation architectures[8] being proposed by these contractors and their related risk reduction activities. Clearly, this will be a significant challenge. The contractors proposing overall
architecture designs must be aware of all the related risk reduction development activities affecting their respective designs. It may also prove difficult for contractors proposing space
transportation system designs to coordinate work with other contractors without a prime contractor to subcontractor relationship. NASA’s own Aerospace Technology Advisory Committee,
made up of outside experts, has also expressed serious concerns about the difficulty of integrating these efforts effectively.
The need for improvement in coordination and communications in all NASA programs has been noted in the past and is not unique to the X-33 and X-34 programs. We and other NASA
investigative teams have found and noted similar problems with other NASA programs such as the Propulsion Module for the International Space Station, and several other projects including
the two failed Mars missions. NASA’s Space Launch Initiative Program would benefit from lessons learned from past mishaps. At the request of the House Science Committee, we are
undertaking a review of NASA’s lessons learned process and procedures. The principal objectives of this review are to determine (1) how NASA captures and disseminates lessons learned
and (2) if NASA is effectively applying lessons learned toward current programs and projects. We will report the results of our evaluation in December of this year.
The third challenge is establishing performance measures that can accurately gauge the progress being made by NASA and its contractors. NASA officials told us that they plan to
periodically reassess the assumptions underlying key program objectives to ensure that the rationale for developing specific technology applications merits continued support. They also told
us that they were in the process of establishing such metrics to measure performance. Ensuring that the results from the 2nd Generation Program will support a future decision to develop
reusable launch vehicles also deserves attention in NASA’s annual Performance Plan. The plan would be strengthened by recognizing the importance of clearly defined indicators which
demonstrate that NASA is (1) on a path leading to an operational reusable launch vehicle and (2) making progress toward its objective of significantly reducing launch costs, and increasing
safety and reliability compared to existing systems. Affected NASA Enterprise and Center performance plans would also be strengthened with the development of related metrics.
Mr. Chairman, this concludes my statement. I would be happy to answer any questions you or other Members of the Subcommittee may have.
Objectives,
Scope, And
Methodology
We interviewed officials at NASA headquarters in Washington D.C., NASA’s Marshall Space Flight Center, Huntsville, Alabama, and at the NASA X-33 program office at Palmdale,
California to (1) determine the factors that contributed to the difficulties experienced in the X-33 and X-34 programs, and (2) to identify steps that need to be taken to avoid repeating those
problems within the Space Launch Initiative framework. We also talked to representatives of NASA’s Independent Program Assessment Office located at the Langley Research Center,
Hampton, Virginia and the OIG located at NASA headquarters and Marshall Space Flight Center. At these various locations we obtained and analyzed key program, contractual and
procurement documentation for the X-33, X-34 and 2nd Generation programs. Further, we reviewed reports issued by the NASA’s OIG and Independent Program Assessment Office
pertaining to the management and execution of the X-33 and X-34 programs, and NASA Advisory Council minutes regarding NASA’s efforts to develop reusable launch vehicles. In addition,
we reviewed other NASA internal reports documenting management issues associated with program formulation and implementation of other NASA programs. We also reviewed applicable
NASA policy regarding how NASA expects its programs and projects to be implemented and managed.
We conducted our review from August 2000 to June 2001 in accordance with generally accepted government auditing standards.
For information about this testimony, please contact Allen Li at (202) 512-4841. Contributors to this testimony included Jerry Herley, Noel Lance, and Carlos Garcia.
[1]Rockwell International was an industry partner with Orbital Science Corporation in the development of the commercial reusable vehicle.
[2]Lockheed Martin made agreements with Allied Signal Aerospace, B.F. Goodrich Aerospace, Boeing-Rocketdyne Division, and Sverdrup Corporation for the X-33 Program.
[3]NASA’s use of a cooperative agreement allowed the industry partners, and NASA, to withdraw from the agreement without penalty at any time.
[4]We reported on X-33 costs in Space Transportation: Status of the X-33 Reusable Launch Vehicle Program (GAO/NSIAD?99?176, Aug. 11, 1999).
[5]Audit Report: X-33 Cost Estimating Processes, NASA Office of Inspector General, IG-99-052, Sept. 24,1999.
[6]Final Report of the X-33 Liquid Hydrogen Tank Test Investigation Team. National Aeronautics and Space Administration, George C. Marshall Space Flight Center, Huntsville, Alabama
35812, May 2000.
[7]These assessments include NASA internal reports on failures in the Mars Program, Shuttle wiring problems and an assessment of NASA’s approach to executing “Faster, Better,
Cheaper.”
[8]A space transportation architecture is defined as an Earth-to-orbit launch vehicle, on-orbit transfer vehicles and upper stages, mission planning, ground and flight operations, and support
infrastructure.
