This testimony discusses the National Aeronautics and Space
Administration’s (NASA) X-33 and X-34 reusable launch vehicle
programs. The X-33 and X-34 programs experienced difficulties
achieving their goals primarily because NASA did not develop
realistic cost estimates, timely acquisition and risk management
plans, and adequate and realistic performance goals. In
particular, neither program fully (1) assessed the costs
associated with developing new, unproven technologies, (2)
provided for the financial reserves needed to deal with technical
risks and accommodate normal development delays, (3) developed
plans to quantify and mitigate the risks to NASA, or (4)
established performance targets showing a clear path leading to
an operational reusable launch vehicle. As a result, both
programs were terminated. Currently, NASA is in the process of
taking steps in the Second Generation Reusable Launch Vehicle
Program to help avoid problems like those encountered in the X-33
and X-34 programs.
GAO-01-826T
Testimony Before the Committee on Science, Subcommittee on Space and
Aeronautics, House of Representatives
United States General Accounting Office
GAO For Release on Delivery Expected at 2: 00 p. m. EDT Wednesday June 20,
2001 SPACE
TRANSPORTATION Critical Areas NASA Needs to Address in Managing Its Reusable
Launch Vehicle Program
Statement of Allen Li, Director, Acquisition and Sourcing Management
GAO-01-826T
Page 1 GAO-01-826T
Mr. Chairman and Members of the Subcommittee: I am pleased to be here today
to discuss our work, requested by this Subcommittee, on the National
Aeronautics and Space Administration’s (NASA) X-33 and X-34 programs. As
you know, the purpose of these efforts was to significantly reduce the cost
of access to space by partnering with private industry to develop and
demonstrate technologies needed for future reusable launch vehicles reaching
orbit in one stage (single-stage-to-orbit). In essence, these are
vehicles whose components-either all or in part-can be utilized on
subsequent flights. Both programs were recently terminated because of
significant cost increases caused by problems developing the necessary
technologies and flight demonstration vehicles. NASA is now focusing instead
on its new Space Launch Initiative. This is a broader effort to develop the
next generation of reusable launch vehicles, referred to as the Second
Generation Reusable Launch Vehicle Program (2nd Generation Program). Today,
I will discuss the primary factors that contributed to the difficulties
experienced by the X-33 and X34 programs and the steps needed to avoid
repeating those problems within the 2nd Generation Program.
In brief, the X-33 and X-34 programs experienced difficulties achieving
their goals primarily because NASA did not develop realistic cost estimates,
timely acquisition and risk management plans, and adequate and realistic
performance goals. In particular, neither program fully assessed the costs
associated with developing new, unproven technologies; provided for the
financial reserves needed to deal with technical risks and accommodate
normal development delays; developed plans to quantify and mitigate the
risks to NASA; or established performance targets showing a clear path
leading to an operational reusable launch vehicle. Underlying these
difficulties were problems with the agreements and contracts that
established the relationship between NASA and its industry partners and
eventual erosion of commercial prospects for the development of new reusable
launch vehicles.
Currently, NASA is in the process of taking steps in the 2nd Generation
Program to help avoid problems like those encountered in the X-33 and X34
programs. While it is too early to tell if these measures will be
sufficient, our review of the two programs has shown that three critical
areas need to be addressed. First, the technical complexity involved
requires that realistic cost estimates and risk mitigation plans are
developed and the projects are funded accordingly. Second, because the 2nd
Generation Program will involve numerous interrelated, complex efforts to
develop technology, NASA needs to ensure that all these efforts
Page 2 GAO-01-826T
move forward in a coordinated manner and that open communication is fostered
at all levels. Third, performance measures need to be implemented and
periodically validated to ensure that the rationale for developing specific
technology applications merits continued support.
The X-33 and X-34 programs were part of an effort that began in 1994-
known as the Reusable Launch Vehicle Technology/ Demonstrator Program
(Reusable Launch Vehicle Program)-to pave the way to fullscale,
commercially-developed, reusable launch vehicles reaching orbit in one
stage. In embarking on the Reusable Launch Vehicle Program, NASA sought to
significantly reduce the cost of developing, producing and operating launch
vehicles. NASA’s goal was to reduce payload launch costs from $10,000 per
pound on the space shuttle to $1,000 per pound. It planned to do so, in
part, by finding “new ways of doing business” such as using innovative
design methods, streamlined acquisition procedures, and creating industry-
led partnerships with cost sharing to manage the development of advanced
technology demonstration vehicles. The vehicles were seen as the ‘stepping
stones? in what NASA described as an incremental flight demonstration
program. The strategy was to force technologies from the laboratory into the
operating environment.
The X-34 Project started in 1995 as a cooperative agreement between NASA
and Orbital Sciences Corporation 1 (Orbital). The project was to demonstrate
streamlined management and procurement, industry cost sharing and lead
management, and the economics of reusability. However, the industry team
withdrew from the agreement in less than 1 year, for a number of reasons
including changes in the projected profitability of the venture. NASA
subsequently started a new X-34 program with a smaller vehicle design. It
was intended only as a flight demonstration vehicle to test some of the key
features of reusable launch vehicle operations, such as quick turn-around
times between launches.
Under the new program, NASA again selected Orbital as its contractor in
August 1996, awarding it a fixed price, $49.5 million contract. Under the
new contract, Orbital was given lead responsibility for vehicle design,
fabrication, integration, and initial flight testing for powered flight of
the X-34 test vehicle. The contract also provided for two options, which
were
1 Rockwell International was an industry partner with Orbital Science
Corporation in the development of the commercial reusable vehicle.
Background
Page 3 GAO-01-826T
later exercised, totaling about $17 million for 25 additional experimental
flights and, according to a project official, other tasks, including
defining how the flight tests would be undertaken. Under the new effort,
NASA’s Marshall Space Flight Center was to develop the engine for the X-34
as part of its Low Cost Booster Technology Project. The initial budget for
this development was about $18.9 million.
In July 1996, NASA and Lockheed Martin Corporation and its industry partners
2 entered into a cooperative agreement for the design, development, and
flight-testing of the X-33. 3 The X-33 was to be an unmanned technology
demonstrator. It would take off vertically like a rocket, reaching an
altitude of up to 60 miles and speeds to about Mach 13 (13 times the speed
of sound), and land horizontally like an airplane. The X-33 would flight
test a range of technologies needed for future launch vehicles, such as
thermal protection systems, advanced engine design and lightweight fuel
tanks made of composite materials. The vehicle would not actually achieve
orbit, but based on the results of demonstrating the new technologies, NASA
envisioned being in a better position to make a decision on the feasibility
and affordability of building a full-scale system. Under the initial terms
of the cooperative agreement, NASA’s contribution was fixed at $912.4
million and its industry partners? initial contribution was $211.6 million.
In view of the potential commercial viability of the launch vehicle and its
technologies, the industry partners also agreed to finance any additional
costs. 4 During a test in November 1999, one of the fuel tanks failed due to
separation of the composite surface. Following the investigation, NASA and
Lockheed Martin agreed to replace the composite tanks with aluminum tanks.
In February 2001, NASA announced it would not provide any additional funding
for the X-33 or X-34 programs under its new Space Launch Initiative. The
Space Launch Initiative is intended to be a more comprehensive, long-range
plan to reduce high payload launch costs. NASA’s goal is still to reduce
payload launch cost to $1,000 per pound to low Earth orbit but it is not
limited to single-stage-to-orbit concepts.
2 Lockheed Martin made agreements with Allied Signal Aerospace, B. F.
Goodrich Aerospace, Boeing-Rocketdyne Division, and Sverdrup Corporation
for the X-33 Program.3 NASA’s use of a cooperative agreement allowed the
industry partners, and NASA, to withdraw from the agreement without penalty
at any time.4 We reported on X-33 costs in Space Transportation: Status of
the X-33 Reusable Launch Vehicle Program (GAO/ NSIAD-99-176, Aug. 11,
1999).
Page 4 GAO-01-826T
Specifically, the 2nd Generation Program’s objective is to substantially
reduce the technical, programmatic, and business risks associated with
developing reusable space transportation systems that are safe, reliable and
affordable.
NASA has budgeted about $900 million for the SLI initial effort and, in May
2001, it awarded initial contracts to 22 large and small companies for space
transportation system design requirements, technology risk reduction, and
flight demonstration. In subsequent procurements in midfiscal year 2003,
NASA plans to select at least two competing reusable launch system designs.
The following 2.5 to 3.5 years (through fiscal years 2005 or 2006) will be
spent finalizing the preliminary designs of the selected space
transportation systems, and maturing the specific technologies associated
with those high-risk, high-priority items needed to develop the selected
launch systems.
Undertaking ambitious, technically challenging efforts like the X-33 and
X-34 programs-which involve multiple contractors and technologies that have
yet to be developed and proven-requires careful oversight and management.
Importantly, accurate and reliable cost estimates need to be developed,
technical and program risks need to be anticipated and mitigated, sound
configuration controls need to be in place, and performance needs to be
closely monitored. Such undertakings also require a high level of
communication and coordination. Not carefully implementing such project
management tools and activities is a recipe for failure. Without
realistically estimating costs and risks, and providing the reserves needed
to mitigate those risks, management may not be in a position to effectively
deal with the technical problems that cutting-edge projects invariably
face.
In fact, we found that NASA did not successfully implement and adhere to a
number of critical project management tools and activities. Specifically:
NASA did not develop realistic cost estimates in the early stages of the
X33 program. From its inception, NASA officials considered the program to be
high risk, with a success-oriented schedule that did not allow for major
delays. Nevertheless, in September 1999, NASA’s Office of the Inspector
General (OIG) reported 5 that NASA’s cost estimate did not include a risk
5 Audit Report: X-33 Cost Estimating Processes, NASA Office of Inspector
General, IG-99052, Factors Contributing to X-33 and X-34 Program Difficulties
Page 5 GAO-01-826T
analysis to quantify technical and schedule uncertainties. Instead, the cost
estimate assumed that needed technology would be available on schedule and
as planned. According to the OIG, a risk analysis would have alerted NASA
decision-makers to the probability of cost overruns in the program. Since
NASA’s contribution to the program was fixed-with Lockheed Martin and its
industry partners responsible for costs exceeding the initial $1.1 billion-
X-33 program management concluded that there was no risk of additional
government financial contributions due to cost overruns. They also believed
that the projected growth in the launch market and the advantages of a
commercial reusable launch vehicle would provide the necessary incentive to
sustain industry contributions.
NASA did not prepare risk management plans for both the X-33 and X-34
programs until several years after the projects were implemented. Risk
management plans identify, assess, and document risks associated with cost,
resource, schedule, and technical aspects of a project and determine the
procedures that will be used to manage those risks. In doing so, they help
ensure that a system will meet performance requirements and be delivered on
schedule and within budget. A risk management plan for the X-34 was not
developed until the program was restructured in June 2000. Although Lockheed
Martin developed a plan to manage technical risks as part of its 1996
cooperative agreement for the X-33, NASA did not develop its own risk
management plan for unique NASA risks until February 2000. The NASA
Administrator and the NASA Advisory Council have both commented on the need
for risk plans when NASA users partnering arrangements such as a cooperative
agreement. Furthermore, we found that NASA’s risk mitigation plan for the X-
33 program provided no mechanisms for ensuring the completion of the program
if significant cost growth occurred and/ or the business case motivating
industry participation weakened substantially.
Against its own policy, NASA did not prepare program commitment agreements
or program plans at the onset for either program. The commitment agreement
lays out the program’s technical, schedule, and cost commitments, and
overall acquisition strategy. The program plan addresses these issues as
well but also defines the effort’s management structure as well as program
resources, data management, risk management, test and verification, and
planned program reviews. Such plans would help NASA to define realistic time
frames, identify
Sept. 24, 1999.
Page 6 GAO-01-826T
responsibility for key tasks and deliverables and provide a yardstick by
which to measure the progress of the effort.
According to the OIG, NASA did not complete a configuration management
plan for the X-33 until May 1998-about 2 years after NASA awarded the
cooperative agreement and Lockheed Martin began the design and development
of a flight demonstration vehicle. Configuration management plans define the
process to be used for defining the functional and physical characteristics
of a product and systematically controlling changes in the design. As such,
they enable organizations to establish and maintain the integrity of a
product throughout its lifecycle and prevent the production and use of
inconsistent product versions. By the time the plan was implemented,
hardware for the demonstration vehicle was already being fabricated.
Communications and coordination were not effectively facilitated. In a
report following the failure of the X-33’s composite fuel tank, the
investigation team reported that the design of the tank required high levels
of communication, and that such communication did not occur in this case. 6
A NASA official told us that some NASA and Lockheed personnel, who had
experience with composite materials and the phenomena identified as one of
the probable causes for the tank’s failure, expressed concerns about the
tank design. However, because of the industry-led nature of the cooperative
agreement, Lockheed Martin was not required to react to such concerns and
did not request additional assistance from NASA.
The Government Performance and Results Act of 1993 requires federal
agencies to prepare annual performance plans to establish measurable
objectives and performance targets for major programs. Doing so enables
agencies to gauge the progress of programs like the X-33 and X-34 and in
turn to take quick action when performance goals are not being met. For
example, we reported in August 1999 that NASA’s Fiscal Year 2000 Performance
Plan did not include performance targets that established a clear path
leading to a reusable launch vehicle and recommended such targets be
established.
6 Final Report of the X-33 Liquid Hydrogen Tank Test Investigation Team.
National Aeronautics and Space Administration, George C. Marshall Space
Flight Center, Huntsville, Alabama 35812, May
2000.
Page 7 GAO-01-826T
Without relying on these important project management tools up front, NASA
encountered numerous problems on both the X-33 and X-34 programs.
Compounding these difficulties was a decrease in the projected commercial
launch market, which in turn lessened the incentive of NASA’s X-33 industry
partners to continue their investments.
In particular, technical problems in developing the X-33’s composite fuel
tanks, aerospike engines, heat shield, and avionics system resulted in
significant schedule delays and cost overruns. After two program reviews in
1998 and 1999, the industry partners added a total of $145.6 million to the
cooperative agreement to pay for cost overruns and establish a reserve to
deal with future technical problems and schedule delays. However, NASA
officials stated that they did not independently develop their own cost
estimates for these program events to determine whether the additional funds
provided by industry would be sufficient to complete the program. Also,
these technical problems resulted in the planned first flight being delayed
until October 2003, about 4.5 years after the original March 1999 first
flight date.
After the composite fuel tank failed during testing in November 1999,
according to NASA officials, Lockheed Martin opted not to go forward with
the X-33 Program without additional NASA financial support. Lockheed Martin
initially proposed adding $95 million of its own funds to develop a new
aluminum tank for the hydrogen fuel, but also requested about $200 million
from NASA to help complete the program. Such contributions would have
increased the value of the cooperative agreement to about $1.6 billion or
about 45 percent (about $500 million) more than the $1.1 billion initial
cooperative agreement funding. NASA did not have the reserves available to
cover such an increase. The agency did, however, allow Lockheed Martin to
compete, in its 2nd Generation Program solicitation for the additional
funds Lockheed Martin believed it needed to complete the program.
Similarly, NASA started the X-34 Project, and the related NASA engine
development project, with limited government funding, an accelerated
development schedule, and insufficient reserves to reduce development risks
and ensure a successful test program. Based on a NASA X-34 restructure plan
in June 2000, we estimate that NASA’s total funding requirements for the X-
34 would have increased to about $348 million-a 307-percent ($ 263
million) increase from the estimated $86 million budgeted for the vehicle
and engine development projects in 1996. Also, since 1996, the projected
first powered flight had slipped about 4 years
Page 8 GAO-01-826T
from September 1998 to October 2002 due to the cumulative effect of added
risk mitigation tasks, vehicle and engine development problems, and testing
delays.
Most of the cost increase (about $213 million) was for NASA-directed risk
mitigation tasks initiated after both projects started. For example, in
response to several project technical reviews and internal assessments of
other NASA programs, 7 the agency developed a restructure plan for the X34
project in June 2000. This plan included consolidating the vehicle and
engine projects under one NASA manager. The project would be managed with
the NASA project manager having the final decision-making authority;
Orbital would be relegated to a more traditional subordinate contractor
role. Under the plan, the contract with Orbital would also be rescoped to
include only unpowered flights; Orbital would have to compete for 2nd
Generation Program funding for all the powered flight tests. The plan’s
additional risk mitigation activities would have increased the X-34
project’s funding requirements by an additional $139 million, which included
about $45 million for additional engine testing and hardware; $33 million
for an avionics redesign; $42 million for additional project management
support and personnel; and $18 million to create a contingency reserve for
future risk mitigation efforts.
NASA is revising its acquisition and management approach for the 2nd
Generation Program. Projects funded under the program will be NASA-led
rather than industry-led. NASA also plans to increase the level of insight
into the program’s projects, for example, by providing more formal reviews
and varying levels of project documentation from contractors depending on
the risk involved and the contract value. NASA also required that all
proposals submitted in response to its research announcement be accompanied
by certifiable cost and pricing data. Finally, NASA discouraged the use of
cooperative agreements since these agreements did not prove to be effective
contract vehicles for research and development efforts where large
investments are required.
While it is too early to tell if the agency measures aimed at avoiding the
problems experienced in the X-33 and X-34 programs will be sufficient,
7 These assessments include NASA internal reports on failures in the Mars
Program, Shuttle wiring problems and an assessment of NASA’s approach to
executing “Faster, Better, Cheaper.” Incorporating Lessons Learned in 2nd Generation Program
Page 9 GAO-01-826T
these experiences show that three critical areas need to be addressed. These
relate to (1) adequate project funding and cost risk provisions, (2) the
effective and efficient coordination and communication required by many
individual but related efforts, and (3) periodically revalidating underlying
assumptions by measuring progress toward achieving a new safe, affordable
space transportation system that meets NASA’s requirements.
First, the technical complexity of the 2nd Generation Program requires that
NASA develop realistic cost estimates and risk mitigation plans, and
accordingly set aside enough funds to cover the program’s many projects.
NASA plans to invest substantially more funds in the 2nd Generation Program
than it did in the previous Reusable Launch Vehicle Program, and plans to
provide reserves for mitigating program risk. For example, the agency plans
to spend about $3.3 billion over 6 years to define system requirements for
competing space transportation systems and related risk reduction
activities. Most of this amount, about $3.1 billion, is for riskreduction
activities, such as the development of new lightweight composite structures,
durable thermal protection systems, and new high performance engine
components.
NASA officials told us that an important way they plan to mitigate risk is
by ensuring adequate management reserves in the 15-to 20-percent range, or
higher if needed. They also acknowledged the need for adequate program cost
estimates on which to base reserve requirements. However, we are still
concerned about the timely preparation of cost estimates. The 2nd
Generation deputy program manager stated that, based on the scope of the
first contracts awarded, the program office planned to update their cost
estimate this summer before NASA conducted a separate, independent technical
review and cost estimate in September 2001. Thus, neither of these important
analyses were completed prior to the first contract awards. We believe that
until the program office completes it own updated cost estimate and NASA
conducts an independent cost and technical review, a credible estimate of
total program costs and the adequacy of planned reserves will not be
available. Also, NASA is still in the process of developing the
documentation required for the program, including a risk mitigation plan.
NASA policy requires that key program documentation be finalized and
approved prior to implementing a program.
Second, NASA will face coordination and communication challenges in managing
the 2nd Generation Program. As noted earlier, NASA recently awarded initial
contracts for systems engineering and various risk
Page 10 GAO-01-826T
reduction efforts to 22 different contractors. Yet to successfully carry out
the program NASA must, early on, have coordinated insight into all of the
space transportation architectures 8 being proposed by these contractors and
their related risk reduction activities. Clearly, this will be a significant
challenge. The contractors proposing overall architecture designs must be
aware of all the related risk reduction development activities affecting
their respective designs. It may also prove difficult for contractors
proposing space transportation system designs to coordinate work with other
contractors without a prime contractor to subcontractor relationship. NASA’s
own Aerospace Technology Advisory Committee, made up of outside experts, has
also expressed serious concerns about the difficulty of integrating these
efforts effectively.
The need for improvement in coordination and communications in all NASA
programs has been noted in the past and is not unique to the X-33 and X-34
programs. We and other NASA investigative teams have found and noted similar
problems with other NASA programs such as the Propulsion Module for the
International Space Station, and several other projects including the two
failed Mars missions. NASA’s Space Launch Initiative Program would benefit
from lessons learned from past mishaps. At the request of the House Science
Committee, we are undertaking a review of NASA’s lessons learned process and
procedures. The principal objectives of this review are to determine (1) how
NASA captures and disseminates lessons learned and (2) if NASA is
effectively applying lessons learned toward current programs and projects.
We will report the results of our evaluation in December of this year.
The third challenge is establishing performance measures that can accurately
gauge the progress being made by NASA and its contractors. NASA officials
told us that they plan to periodically reassess the assumptions underlying
key program objectives to ensure that the rationale for developing specific
technology applications merits continued support. They also told us that
they were in the process of establishing such metrics to measure
performance. Ensuring that the results from the 2nd Generation Program will
support a future decision to develop reusable launch vehicles also deserves
attention in NASA’s annual Performance Plan. The plan would be strengthened
by recognizing the importance of
8 A space transportation architecture is defined as an Earth-to-orbit
launch vehicle, on-orbit transfer vehicles and upper stages, mission
planning, ground and flight operations, and support infrastructure.
Page 11 GAO-01-826T
clearly defined indicators which demonstrate that NASA is (1) on a path
leading to an operational reusable launch vehicle and (2) making progress
toward its objective of significantly reducing launch costs, and increasing
safety and reliability compared to existing systems. Affected NASA
Enterprise and Center performance plans would also be strengthened with the
development of related metrics.
Mr. Chairman, this concludes my statement. I would be happy to answer any
questions you or other Members of the Subcommittee may have.
We interviewed officials at NASA headquarters in Washington D. C., NASA’s
Marshall Space Flight Center, Huntsville, Alabama, and at the NASA X-33
program office at Palmdale, California to (1) determine the primary program
management factors that contributed to the difficulties experienced in the
X-33 and X-34 programs, and (2) to identify steps that need to be taken to
avoid repeating those problems within the Space Launch Initiative framework.
We also talked to representatives of NASA’s Independent Program Assessment
Office located at the Langley Research Center, Hampton, Virginia and the OIG
located at NASA headquarters and Marshall Space Flight Center. At these
various locations we obtained and analyzed key program, contractual and
procurement documentation for the X-33, X-34 and 2nd Generation programs.
Further, we reviewed reports issued by the NASA’s OIG and Independent
Program Assessment Office pertaining to the management and execution of the
X-33 and X-34 programs, and NASA Advisory Council minutes regarding NASA’s
efforts to develop reusable launch vehicles. In addition, we reviewed other
NASA internal reports documenting management issues associated with program
formulation and implementation of other NASA programs. We also reviewed
applicable NASA policy regarding how NASA expects its programs and projects
to be implemented and managed.
We conducted our review from August 2000 to June 2001 in accordance with
generally accepted government auditing standards. Objectives, Scope,
And Methodology
Page 12 GAO-01-826T
For information about this testimony, please contact Allen Li at (202)
5124841. Contributors to this testimony included Jerry Herley, Noel Lance,
and Carlos Garcia.
