Early Design

[120] The Space Shuttle's Solid Rocket Booster problem began with the faulty design of its joint and increased as both NASA and contractor management first failed to recognize it as a problem, then failed to fix it and finally treated it as an acceptable flight risk.

Morton Thiokol, Inc., the contractor, did not accept the implication of tests early in the program that the design had a serious and unanticipated flaw.¹ NASA did not accept the judgment of its engineers that the design was unacceptable, and as the joint problems grew in number and severity NASA minimized them in management briefings and reports. ² Thiokol's stated position was that "the condition is not desirable but is acceptable." ³

Neither Thiokol nor NASA expected the rubber O-rings sealing the joints to be touched by hot gases of motor ignition, much less to be partially burned. However, as tests and then flights confirmed damage to the sealing rings, the reaction by both NASA and Thiokol was to increase the amount of damage considered "acceptable." At no time did management either recommend a redesign of the joint or call for the Shuttle's grounding until the problem was solved.

Thiokol was selected to receive the NASA contract to design and build the Solid Rocket Boosters on November 20, 1973.⁴ The booster was the largest Solid Rocket Motor ever produced in the United States; it was also the first solid motor program managed by NASA's Marshall Space Flight Center in Huntsville, Alabama.

Costs were the primary concern of NASA's selection board, particularly those incurred early in the program.

Thiokol's three competitors were Aerojet Solid Propulsion Co., Lockheed Propulsion Co. and United Technologies. The Source Evaluation Board on the proposals rated Thiokol fourth under the design, development and verification factor, second under the manufacturing, refurbishment and product support factor and first under the management factor.⁵

Thiokol received the second highest overall Mission Suitability score, tied with United Technologies.⁶

In a December 12, 1973, report, NASA selection officials said Thiokol's "cost advantages were substantial and consistent throughout all areas evaluated." ⁷ They also singled out Thiokol's joint design for special mention.

"The Thiokol motor case joints utilized dual O-rings and test ports between seals, enabling a simple leak check without pressurizing the entire motor," the officials' report said. "This innovative design feature increased reliability and decreased operations at the launch site, indicating good attention to low cost (design, development, testing and engineering) and production." ⁸

"We noted that the [NASA Source Selection] board's analysis of cost factors indicated that Thiokol could do a more economical job than any of the other proposers in both the development and the production phases of the program; and that, accordingly, the cost per flight to be expected from a Thiokol-built motor would be the lowest," the officials said. "We, therefore, concluded that any selection other than Thiokol would give rise to an additional cost of appreciable size." ⁹

The Selection officials said they "found no other [121] factors bearing upon the selection that ranked in weight with the foregoing."

Cost consideration overrode any other- objections, they decided. We concluded that the main criticisms of the Thiokol proposal in the Mission Suitability evaluation were technical in nature, were readily correctable, and the costs to correct did not negate the sizable Thiokol cost advantage," the selection officials concluded.

The cost-plus-award-fee contract, estimated to be worth $800 million, was awarded to Thiokol.

The design of the Shuttle Solid Rocket Booster was primarily based on the Air Force's Titan III solid rocket, one of the most reliable ever produced. Thiokol hoped to reduce new design problems, speed up the development program and cut costs by borrowing from the Titan design. In Thiokol's Solid Rocket Motor proposal, the rocket fuel is contained in four- forged steel cases which are stacked one on top of the other. The casings were connected by a circumferential tang and clevis, as were the Titans.¹⁰

Despite their many similarities, the Thiokol Solid Rocket Booster and the Titan motors had some significant design differences. For example, the joints of the Titan were designed so that the insulation of one case fits tightly against the insulation of the adjacent case to form a more gastight fit than the Thiokol design. One O-ring bore seal was used in each Titan joint to stop any hot gas pressure that might pass by the insulation overlap, ¹¹ but in the Titan design the O-ring was able but not intended to take the brunt of the combustion pressure. In contrast, the Thiokol O-rings were designed to take the brunt of the combustion pressure, with no other gas barriers present except an insulating putty. Also, the Solid Rocket Motor joint had two O-rings, the second to provide a backup in case the primary seal failed.

Asbestos-filled putty was used in the Solid Rocket Motor to pack the space between the two case segments to prevent O-ring damage from the heat of combustion gases. ¹² Thiokol believed the putty was plastic, so when acted on by the combustion pressure at the motor's ignition the putty flow towards the O-ring would compress the air in the gap between the putty and the primary O-ring.¹³ The compressed air, in turn, would.....

[122] ...cause the primary O-ring to extrude into the gap between the clevis and the tang, behind the primary O-ring groove, thereby sealing the opening. If the primary O-ring did not seal, the intent was that the secondary would pressurize and seal the joint by extruding into the gap behind its groove.¹⁴

Another difference in the Solid Rocket Motor and the Titan was that the tang portion of the Thiokol joint was longer in order to accommodate two O-rings instead of one. It was more susceptible to bending under combustion pressure than the Titan joint, as post-design tests and later flight experience demonstrated.¹⁵

The initial Thiokol design proposal was changed before the production motors were manufactured. Originally, the joint seal design incorporated both a face seal and a bore seal.¹⁶ (Figure 1.) However, the motor that was eventually used had double bore O-rings. The original bore seal/face seal design was chosen because it was anticipated that it "provides [better] redundance over a double bore ring seal since each is controlled by different manufacturing tolerances, and each responds differently during joint assembly. " ¹⁷ Because the early design incorporated tolerances similar to the Titan and it also incorporated a face seal, Thiokol believed it possessed "complete, redundant seal capability." ¹⁸

Nevertheless, as the Solid Rocket Motor program progressed, Thiokol-with NASA's concurrence-dropped the face/bore seal design for one using a double bore seal (Figure 1). NASA engineers at Marshall said the original design would have required tapered pins to maintain necessary tolerances and assure enough"squeeze" on the face-sealing O-ring.¹⁹ However, design analysis determined that motor ignition would create tension loads on the joint sufficient to cause the tapered pins to pop out. Solving that would have meant designing some type of pin-retainers. Moreover, the rocket assembly was much easier with the dual bore seals. Because inspections and tests had to be conducted on the Solid Rocket Motor stack, horizontal assembly was required. Thiokol engineer, Howard McIntosh, described this in a Commission interview on April 2, 1986:

"We were concerned very much about the horizontal assembly that we had to do to do the static tests. The Titan had always been assembled vertically, and so there had never been a larger rocket motor to our knowledge that was assembled (horizontally)".²⁰

Because of the extremely tight tolerances in the joints caused by horizontal assembly, McIntosh noted, "We . . . put the bore seals in there, and we opened the tolerance in the gaps slightly to accommodate that."²¹ To tighten the joint's fit and to increase the squeeze in the O-rings to compensate for the larger tolerances, Thiokol subsequently put thin metal shims between the outer walls of the tang and clevis.

Another significant feature of the Thiokol design was a vent, or port, on the side of the motor case used after assembly to check the sealing of the O-rings. As will be noted later, this leak check eventually became a significant aspect of the O-ring erosion phenomenon.²²

The manufacture of the O-rings themselves constituted another difference between the Titan and the Thiokol Solid Rocket Motor. While both O-rings were Viton rubber, the Titan O-rings were molded in one piece. The Solid Rocket Motor O-rings were made from sections of rubber O-ring material glued together. The specifications allowed five such joints, a number chosen arbitrarily, and the vendor routinely made repairs of voids and inclusions after getting the material supplies. Only surface inspections were performed by Thiokol and by the manufacturer.

Finally, unlike the Titan, the Thiokol Solid Rocket Motor was designed for multiple firings. To reduce program costs, each Thiokol motor case for the Shuttle was to be recovered after flight and reused up to 20 times.²³

Early Tests

Thiokol began testing the Solid Rocket Motor in the mid-1970's. One of the early important tests was a 1977 "hydroburst test."²⁴

Its purpose was to test the strength of the steel cases by simulating a motor firing. The case was pressurized with water to about one and one-half times the pressure of an ignited motor (about 1,500 pounds per square inch) to make certain the case had adequate structural margin.²⁵ Also, to measure the pressure between the O-rings, engineers attached instruments to the leak test port at a segment joint. Although the test was successful in that it demonstrated the case met strength requirements, test measurements showed that, contrary to design expectations, the joint [123] tang and inside clevis bent away from each other instead of toward each other and by doing so reduced-instead of increased-pressure on the 0-ring in the milliseconds after ignition.²⁶ This phenomenon was called "joint rotation." Testifying before the Commission, Arnold Thompson, Thiokol's supervisor of structures, said,

"We discovered that the joint was opening rather than closing as our original analysis had indicated, and in fact it was quite a bit. I think it was up to 52 onethousandths of an inch at that time, to the primary O-ring."²⁷

Thiokol reported these initial test findings to the NASA program office at Marshall. Thiokol engineers did not believe the test results really proved that "joint rotation" would cause significant problems,²⁸ and scheduled no additional tests for the specific purpose of confirming or disproving the joint gap behavior.

Design Objections

Reaction from Marshall to the early Solid Rocket Motor test results was rapid and totally opposite of Thiokol's. In a September 2, 1977 memorandum, Glenn Eudy, Marshall's Chief Engineer of the Solid Rocket Motor Division, informed Alex McCool, Director of the Structures and Propulsion Laboratory, that the assembly of a developmental motor provided early indications that the Thiokol design:

"Allowed O-ring clearance.... Some people believe this design deficiency must be corrected by some method such as shimming and perhaps design modification to the case joint for hardware which has not been final machined.... I personally believe that our first choice should be to correct the design in a way that eliminates the possibility of O-ring clearance.... Since this is a very critical SRM issue, it is requested that the assignment results be compiled in such a manner as to permit review at the S&E Director's level as well as project manager."

After seeing the data from the September 1977 hydroburst test, Marshall engineer Leon Ray submitted a report entitled "Solid Rocket Motor Joint Leakage Study" dated October 21, 1977. It characterizes "no change" in the Thiokol design as "unacceptable"-"tang can move outboard and cause excessive joint clearance resulting in seal leakage. Eccentric tang/clevis interface can cause O-ring extrusion when case is pressurized." Ray recommended a "redesign of the tang and reduce tolerance on the clevis" as the "best option for a long-term fix." ²⁹

After Ray's 1977 report, John Q. Miller, chief of the Solid Rocket Motor branch at Marshall, signed and sent a memorandum on January 9, 1978 to his superior, Glenn Eudy, describing the problems evident in the Solid Rocket Motor joint seal. "We see no valid reason for not designing to accepted standards," the memo said, and it emphasized that proper sealing of the joint by use of shims to create necessary O-ring pressure was "mandatory to prevent hot gas leaks and resulting catastrophic failure."³⁰

One year later, not having received a response to his 1978 memo, Miller signed and forwarded a second memo strenuously objecting to Thiokol's Solid Rocket Motor joint seal design. This memo, dated January 19, 1979, opened with: "We find the Thiokol position regarding design adequacy of the clevis joint to be completely unacceptable...." ³¹ The memorandum made three principal objections to Thiokol's joint design. The first was the "large sealing surface gap created by extensive tang/clevis relative movement." The memo said this movement, the so-called"joint rotation," caused the primary O-ring to extrude into the gap, "forcing the seal to function in a way which violates industry and government O-ring application practices." ³² Moreover, joint rotation allowed the secondary O-ring to "become completely disengaged from its sealing surface on the tang." Finally, the memorandum noted that although Thiokol's contract required all high pressure case seals to be verifiable, "the clevis joint secondary O-ring seal has been verified by tests to be unsatisfactory."³³ A copy of the second memorandum was sent to George Hardy, then Solid Rocket Booster project manager at Marshall. Thiokol apparently did not receive copies of either Miller memorandum, and no reply from Eudy to Miller has been found.

The Commission has learned that Leon Ray actually authored the Miller memos to Eudy, although Miller signed them and concurred in the objections raised.³⁴ During February, 1979, Ray also reported on a visit he made to two O-ring manufacturers-the Precision Rubber Products Corporation at Lebanon Tennessee, and the Parker Seal Co. at Lexington, Kentucky.³⁵ Eudy [124] accompanied Ray on the Precision visit. The purpose of the trips was to give the manufacturers the data on the O-ring experiences at Thiokol and to "seek opinions regarding potential risks involved," Ray wrote in a February 9, 1979, memo describing the visit. Officials at Precision did "voice concern for the design, stating that the Solid Rocket Motor O-ring extrusion gap was larger than that covered by their experience," Ray reported. "Their first thought was that the O-ring was being asked to perform beyond its intended design and that a different type of seal should be considered," Ray added.³⁶

During the Commission hearing on May 2, 1986, Ray was asked why the 1978 and 1979 memoranda were written:

 

Mr. Ray: The reason they were written was as a result of test data that we had, and I have to go back to, I guess, a little bit further back in time than these memos. When the joint was first designed, the analysis produced by Thiokol says the joint would close, the extrusion gap would actually close.

We had quite a debate about that until we did a test on the first couple of segments that we received from the manufacturer, which in fact showed that the joint did open. Later on we did some tests with the structural test article, and this is mentioned in the memo as STA-1 [Structural Test Article].

At that time, we really nailed it down. We got some very accurate numbers on joint rotation, and we know for a fact that during these tests that, just what the memo says, the joint rotated. The primary O-ring was extruded up into the joint. The secondary O-ring did in fact detach from the seat.³⁷

No records show Thiokol was informed of the visits, and the O-ring design was not changed.

Thiokol's phase 1 certification review on March 23, 1979, mentioned leak check failures, and forces during case joint assembly that resulted in clevis O-ring grooves not conforming with tang sealing surfaces. However, this was not listed as a problem or a failure.³⁸

Verification and Certification Committee

While Ray was warning of problems with joint rotation, static motor tests in July 1978 and April 1980 again were demonstrating that inner tang/clevis relative movement was greater than originally predicted.³⁹ Thiokol continued to question the validity of these joint rotation measurements and their effect on the availability of the secondary O-ring.

In 1980, NASA empanelled a Space Shuttle Verification/Certification Committee to study the flight worthiness of the entire Shuttle system. A subdivision of that group, the Propulsion Committee, met with NASA Solid Rocket Motor program personnel and raised several concerns about the joint design.⁴⁰ The Committee pointed out that the booster's leak test pressurized the primary O-ring in the wrong direction so that the motor ignition would have to move the ring across its groove before it sealed. The Committee added that the effect of the insulation putty was not certain. Redundancy of the O-rings was also listed as a verification concern. The same report, however, said "the Committee understands from a telecon that the primary purpose of the second O-ring is to test the primary and that redundancy is not a requirement." George Hardy testified that the Committee's statement conflicted with his understanding:

"The discussion there or the reference there to a telecon-and I don't know who that was with-that implies there was no intent for the joint to be redundant is totally foreign to me. I don't know where they would have gotten that information because that was the design requirement for the joint." ⁴¹

In May 1980, the Verification/Certification Committee recommended that NASA conduct full-scale tests to verify the field joint integrity, including firing motors at a mean bulk propellant temperature range of 40-90 degrees Fahrenheit. The panel also asked NASA to:

"Perform case burst test with one O-ring removed. During the burst test for final verification of the motor case safety factor, one of the two O-rings failed by extrusion and leaked. The analysis used for additional verification did not include further gap openings caused by joint deflection at pressurization or any deflections caused by bending loads. The panel considers the above to be inadequate to provide operational program reliability, and marginal to provide adequate [125] safety factor confidence on [Shuttle flight] one." ⁴²

The NASA program response to these issues was included in the final Committee report in September 1980. It said that the original hydroburst tests and the lightweight case tests, being conducted at the time, satisfied the intent of the Committee's recommendations. Moreover, the response stated: "NASA specialists have reviewed the field joint design, updated with larger O-rings and thicker shims and found the safety factors to be adequate for the current design. Re-analysis of the joint with larger O-rings and thicker shims is being accomplished as part of the lightweight case program.... The joint has been sufficiently verified with the testing accomplished to date (joint lab tests, structural test article, and seven static firings and the two case configuration burst tests) and currently scheduled for lightweight case program."⁴³

Criticality Classification and Changes

The Solid Rocket Motor certification was deemed satisfactory by the Propulsion Committee of the Verification/Certification Group on September 15, 1980. Shortly thereafter, on November 24, 1980, the Solid Rocket Booster joint was classified on the Solid Rocket Booster Critical Items List as criticality category 1 R. NASA defines "Criticality 1R" as any subsystem of the Shuttle that contains "redundant hardware, total element failure of which could cause loss of life or vehicle."⁴⁴ The use of "R", representing redundancy, meant that NASA believed the secondary O-ring would pressurize and seal if the primary O-ring did not. Nonetheless, the 1980 Critical Items List (CIL) states:

"Redundancy of the secondary field joint seal cannot be verified after motor case pressure reaches approximately 40 percent of maximum expected operating pressure. It is known that joint rotation occurring at this pressure level with a resulting enlarged extrusion gap causes the secondary O-ring to lose compression as a seal. It is not known if the secondary O-ring would successfully reseal if the primary O-ring should fail after motor case pressure reaches or exceeds 40 percent of maximum expected operating pressure."

When asked about the text of the 1980 Criticality 1R classification, Arnold Aldrich, NASA Manager of the National Space Transportation System, said,

"The way that . . . language [reads], I would call it [criticality] 1."⁴⁵

Notwithstanding this apparent contradiction in the classification 1R and the questionable status of the secondary described in the text of the CIL, the joint carried a 1 R classification from November 1980 through the flight of STS-5 (November 1982).

The Space Shuttle first flew on April 12-14, 1981. After the second flight, STS-2, in November 1981, inspection revealed the first in-flight erosion of the primary O-ring.⁴⁶ It occurred in the right Solid Rocket Booster's aft field joint and was caused by hot motor gases.⁴⁷ The damage to the ring proved to be the worst ever found on a primary O-ring in a field joint on any recovered Solid Rocket Booster.⁴⁸ Post-flight examination found an erosion depth of .053 inches on the primary O-ring; nonetheless, the anomaly was not reported in the Level I Flight Readiness Review for STS-3 held on March 9, 1982. Furthermore, in 1982 the STS-2 O-ring erosion was not reported on the Marshall problem assessment system and given a tracking number as were other flight anomalies.⁴⁹

In mid- 1982, two significant developments took place. Because Thiokol believed blow holes in the insulating putty were a cause of the erosion on STS-2, ⁵⁰ they began tests of the method of putty layup and the effect of the assembly of the rocket stages on the integrity of the putty. The manufacturer of the original putty, Fuller-O'Brien, discontinued the product and a new putty, from the Randolph Products Company, was tested and selected in May 1982.⁵¹ The new Randolph putty was eventually substituted for the old putty in the summer of 1983, for the STS-8 Solid Rocket Motor flow.⁵²

A second major event regarding the joint seal occurred in the summer of 1982. As noted before, in 1977-78, Leon Ray had concluded that joint rotation caused the loss of the secondary O-ring as a backup seal. Because of May 1982 high pressure O-ring tests and tests of the new lightweight motor case, Marshall management [126] finally accepted the conclusion that the secondary O-ring was no longer functional after the joints rotated when the Solid Rocket Motor reached 40 percent of its maximum expected operating pressure. It obviously followed that the dual O-rings were not a completely redundant system, so the Criticality 1R had to be changed to Criticality 1.⁵³ This was done at Marshall on December 17, 1982. The revised Critical Items List read (See pages 157 and 158):

"Criticality Category 1.

"Failure Mode and Causes: Leakage at case assembly joints due to redundant O-ring seal failures or primary seal and leak check port O-ring failure.

"Note. Leakage of the primary O-ring seal is classified as a single-failure point due to possibility of loss of sealing at the secondary O-ring because of joint rotation after motor pressurization.

"Failure Effect Summary: Actual Loss- Loss of mission, vehicle and crew due to metal erosion, burn through, and probable case burst resulting in fire and deflagration. .

"Rationale for Retention:

"The Solid Rocket Motor case joint design is common in the lightweight and regular weight cases having identical dimensions. The joint concept is basically the same as the single O-ring joint successfully employed on the Titan III Solid Rocket Motor.... On the Shuttle Solid Rocket Motor, the secondary O-ring was designed to provide redundancy and to permit a leak check, ensuring proper installation of the O-rings. Full redundancy exists at the moment of initial pressurization. However, test data shows that a phenomenon called joint rotation occurs as the pressure rises, opening up the O-ring extrusion gap and permitting the energized ring to protrude into the gap. This condition has been shown by test to be well within that required for safe primary O-ring sealing. This gap may, however, in some cases, increase sufficiently to cause the unenergized secondary O-ring to lose compression, raising question as to its ability to energize and seal if called upon to do so by primary seal failure. Since, under this latter condition only the single O-ring is sealing, a rationale for retention is provided for the simplex mode where only one O-ring is acting" [emphasis added] . ⁵⁴

The retention rationale for the "simplex" or single O-ring seal was written on December 1, 1982, by Howard McIntosh, a Thiokol engineer.⁵⁵ This document gave the justification for flight with the single functional O-ring. It reported that tests showed the Thiokol design should be retained, citing the Titan history, the leak and hydroburst tests, and static motor firings as justification. However, it also contained the following rationale which appeared to conflict with the Criticality 1 classification that the secondary O-ring was not redundant:

"Initial information generated in a lightweight cylinder-to-cylinder proof test shows a total movement of only .030 inch at pounds per square inch, gauge pressure in the center joint. This . . . indicates that the tang-to-clevis movement will not unseat the secondary O-ring at operating pressures."⁵⁶

Testimony in hearings and statements given in Commission interviews support the view that NASA management and Thiokol still considered the joint to be a redundant seal even after the change from Criticality 1R to 1. For example, McIntosh's interview states:

 

Question: [After the Criticality I classification], what did you think it would take to make [the joint seal] 1R?

Mr. McIntosh: I thought it was already 1R. I thought that after those tests that would have been enough to do it.

Question: Well, you knew it was 1 but you were hoping for 1R?

Mr McIntosh: Yeah, I was hoping for 1R, and I thought this test data would do it, but it didn't.⁵⁷

At the time (in 1982-83), the redundancy of the secondary O-ring was analyzed in terms of joint or hardware geometry, with no consideration being given to the resiliency of the ring as affected by temperatures.⁵⁸ Moreover, Marshall engineers like Ray and Miller disagreed with Thiokol's calculations on the measurement of joint opening.⁵⁹ That engineering debate eventually went to a "referee" for testing which was not concluded until after the 51-L accident.

[127] Notwithstanding the view of some of Marshall engineers that the secondary ring was not redundant, even at the time of the Criticality revision, Marshall Solid Rocket Motor program management appeared to believe the seal was redundant in all but exceptional cases. Dr. Judson Lovingood told the Commission:

" . . . [T]here are two conditions you have to have before you don't have redundancy. One of them is what I call a spatial condition which says that the dimensional tolerances have to be such that you get a bad stackup, you don't have proper squeeze, etc. On the O-ring so that when you get joint rotation, you will lift the metal surfaces off the O-ring. All right, that's the one condition, and that is a worst case condition involving dimensional tolerances.

"The other condition is a temporal condition which says that you have to be past a point of joint rotation, and of course, that relates back to what I just said.

"So first of all, if you don't have this bad stackup, then you have full redundancy. Now, secondly, if you do have the bad stackup, you had redundancy during the ignition transient up to the 170 millisecond point, whatever it is, but that is the way I understand the [Critical Items List]."⁶⁰

George Hardy and Lawrence Mulloy shared Lovingood's view that the secondary seal was redundant in all but situations of worst case tolerances.⁶¹ However, there is no mention of this caveat in the Critical Items List itself, nor does it appear in the subsequent "waiver" of the Criticality 1 status granted by NASA Levels I and II in March, 1983.⁶² This waiver was approved to avoid the obligations imposed on the Shuttle Program by Paragraph 2.8 of the Space Shuttle Program Requirements Document, Level I, dated June 30, 1977. That paragraph states:

"The redundancy requirements for all flight vehicle subsystems (except primary structure, thermal protection system, and pressure vessels) shall be established on an individual subsystems basis, but shall not be less than fail-safe. 'Fail-safe' is defined as the ability to sustain a failure and retain the capability to successfully terminate the mission. Redundant systems shall be designed so that their operational status can be verified during ground turnaround and to the maximum extent possible while in flight." ⁶³

Glynn Lunney, the former manager of the STS Program (Level II at JSC) described the Criticality 1 change and resulting waiver to the Commission on May 2:

Mr. Lunney: Well, the approval of the waiver in March of 83, at the time I was involved in that. I was operating on the assumption that there really would be redundancy most of the time except when the secondary O-ring had a set of dimensional tolerances add up, and in that extreme case there would not be a secondary seal.

So I was dealing with what I thought was a case where there were two seals unless the dimensional tolerances were such that there might only be one seal in certain cases.

Chairman Rogers: Now, to me, if you will excuse the expression, that sounds almost contradictory, what you just said. What you first said was you came to the conclusion that you could only rely on the primary seal and therefore you removed the R.

Mr. Lunney: Yes, sir.

Chairman Rogers: And now you're saying, if I understand it, that experience showed that there was redundancy after all.

Mr. Lunney: No, I don't know of any experience showing that. What I'm saying is that the removal of the R is an indicator that under all circumstances we did not have redundancy. There were a certain number of cases under which we would not have redundancy of the secondary O-ring.

Recognizing that, even though there were a lot of cases where we expected we would have redundancy we changed the criticality designation.

Chairman Rogers: It was saying to everybody else you can't necessarily rely on the primary seal, and if the primary seal fails, as you've said here, there may be loss of vehicle, mission and crew.

Mr. Lunney: I would adjust that to only say you cannot rely on the secondary O-ring [128] but we would expect the primary O-ring to always be there.⁶⁴

The criticality waiver was processed outside the formal NASA Program Requirements Control Board, however, representatives of that group "signed off" on the document.⁶⁵ It was forwarded to Level I and approved by Associate Administrator for Space Flight (Technical), L. Michael Weeks on March 28, 1983. Weeks told the Commission he signed the waiver because of the Certification/Verification Review of the Propulsion Committee in 1980. Weeks explained, "We felt at the time-all of the people in the program I think felt that this Solid Rocket Motor in particular or the Solid Rocket Booster was probably one of the least worrisome things we had in the program." ⁶⁶ The waiver was signed less than one week prior to the launch of STS-6 on April 4. According to interviews of Arnold Aldrich and of Richard Kohrs, the latter having been involved with the waiver review at Johnson Level II, the waiver was approved so that STS-6 could fly.⁶⁷ However, Weeks denied any connection between the Level I waiver approval and the flight of STS-6.⁶⁸

Although some Thiokol engineers and officials claimed that they had no notice of the Criticality change and waiver in December, 1982 and in March, 1983, from the approval signatures (including Thiokol's Operations Manager at Marshall, Maurice Parker) and the distribution of the Criticality and Waiver documents, apparently Thiokol officials were sent copies and were involved in the criticality reclassification. ⁶⁹ Nonetheless, the Commission has also determined that several documents tracking the O-ring erosion at Thiokol and Marshall refer to the Solid Rocket Motor field joint seal as Criticality 1-R, long after the status was changed to Criticality 1. ⁷⁰

STS 41-B O-Ring Erosion

As Figure 2 shows,⁷¹ prior to STS 41-B, the O-ring erosion/blow-by problem was infrequent, occurring on a field joint of STS-2 (November, 1981), nozzles of STS-6 (April, 1983) and a nozzle of QM-4 (March, 1983), a qualification test motor fired by Thiokol.⁷² However, when STS 41-B flew on February 3, 1984, the left Solid Rocket Booster forward field joint and the right nozzle joint primary O-rings both suffered erosion damage. Thiokol engineers reacted to this discovery by filing a problem report on the O-ring erosion found on STS 41 -B. Thiokol presented a series of charts to the Marshall Solid Rocket Booster Engineering Office about the 41-B O-ring erosion. Thiokol told Marshall that recent joint rotation measurements in tests indicated the secondary O-ring will not unseat, providing confidence that the secondary was an adequate backup. Keith Coates described his view about Thiokol's data in a February 29, 1984 memorandum to George Hardy:

"We have two problems with their rationale. The effect of 0.065 inch erosion on O-ring sealing capability is not addressed. We have asked Thiokol to provide their data to justify their confidence in the degraded O-ring. The second concern is the amount of joint rotation. L. Ray does not agree with Thiokol numbers, and he has action to discuss his concern with R. Boisjoly (Thiokol) and reach agreement.

"Thiokol definition of their plans on resolution of the problem is very weak."

The erosion problem was identified and tracked by the Marshall Problem Assessment System as Marshall Record A07934 and by Thiokol as Thiokol Contractor Record DR4-5/30, "Slight char condition on primary O-ring seal in forward field joint on SRM A57 of STS-11 flight, Mission 41B." ⁷³ The Marshall Problem Assessment System Report states:

"Remedial action-none required; problem occurred during flight. The primary O-ring seal in the forward field joint exhibited a charred area approximately 1 inch long .03-.050 inches deep and .100 inches wide. This was discovered during post-flight segment disassembly at KSC."

A March 8, 1984 entry on the same report continues:

"Possibility exists for some O-ring erosion on future flights. Analysis indicates max erosion possible is .090 inches according to Flight Readiness Review findings for STS-13. Laboratory test shows sealing integrity at 3,000 psi using an O-ring with simulated erosion depth of .095 inches Therefore, this is not a constraint to future launches." ⁷⁴

Dash (-) denotes no anomaly.

NA denotes not applicable.

NOTE: A list of the sequence of launches (1-25), identified by STS mission designation, is provided on pages 4 thru 6.

¹ On STS-6, both nozzles had a hot gas path detected in the putty with an indication of heat on the primary O-ring.

² On STS-9, one of the right Solid Rocket Booster field joints was pressurized at 200 psi after a destack.

³ On STS 41-C, left aft field had a hot gas path detected in the putty with an indication of heat on the primary O-ring.

⁴ On a center field joint of STS 51-C, soot was blown by the primary and there was a heat effect on the secondary.

⁵ On STS 51-G, right nozzle has erosion in two places on the primary O-ring.

⁶ On STS 51-F, right nozzle had hot gas path detected in putty with an indication of heat on the primary O-ring.

⁷ On STS 51-I, left nozzle had erosion in two places on the primary O-ring.

[132] This last entry is also a summary of the briefing given by Thiokol to Lawrence Mulloy about the 41-B erosion at the Level III Flight Readiness Review for STS 41-C held at Marshall on March 8, 1984. At that same briefing, the Chief Engineer for United Space Boosters, George Morefield, raised prior Titan experience with O-ring problems. He explained in a memorandum to Mulloy the following day:

"I alluded to the Titan III SRM history which is quite similar to the current STS Solid Rocket Motor experience. Post-fire inspection of Titan Solid Rocket Motor static test motors showed that pressurization of the single O-rings in the pressure vessel routinely occurred via a single break-down path across the joint putty. There was also evidence that some O-rings never see pressure in the Titan motor. The segment -to-segment case insulation design results in a compression butt joint which apparently is often sufficient to withstand Pc, ....

"Your review showed that there was sufficient margin of O-ring remaining to do the job. I'm sure you have considered that if it does burn through, the secondary O-ring will then be similarly pressurized through a single port. So, some concern remains.

"I recommend that you set up a panel to study the use of putty and consider some alternatives:

"1) Is putty needed at all?

"2) If the tradition can't be broken, can the putty be applied with multiple (6 or 8) pressurization paths built in?

"I think that the primary seal should be allowed to work in its classical design mode. Both the Titan and STS Solid Rocket Motors have been designed for this not to happen. Titan has flown over a thousand pressure joints with no failure. My opinion is that the potential for failure of the joint is higher for the STS Solid Rocket Motor, especially when occasionally the secondary seal may not be totally effective." ⁷⁵

When the 41-B erosion was taken to the Level I Flight Readiness Review for 41-C on March 30, 1984, it was briefed as a"technical issue". A recommendation to fly 41-C was approved by Level I "accepting the possibility of some O-ring erosion due to the hot gas impingement." ⁷⁶ The rationale for acceptance was the same as that given at the Level III Flight Readiness Review and entered into the Marshall problem assessment report. An outgrowth of this review was an April 5, 1984, directive from NASA Deputy Administrator Dr. Hans Mark to Lawrence Mulloy at Marshall. This "Programmatic Action Item" was signed by Weeks and asked Mulloy to conduct a "formal review of' the Solid Rocket Motor case-to-case and case-to-nozzle joint sealing procedures to ensure satisfactory consistent closeouts." ⁷⁷ This action item had been preceded by a letter written from NASA Associate Administrator for Space Flight General Abrahamson to Marshall Center Director Lucas.⁷⁸ That letter, sent January 18, 1984, requested that Marshall develop a plan of action to make improvement in NASA's ability to design, manufacture and fly Solid Rocket Motors. Abrahamson pointed out that NASA was flying motors where basic design and test results were not well understood. The letter addressed the overall general Solid Rocket Motor design but did not specifically mention O-ring erosion.

After Mulloy received the April 5, 1984 STS 41-C action item on the O-rings, he had Lawrence Wear for-ward a letter- to Thiokol which asked for a formal review of' the booster field joint and nozzle joint sealing procedures. Thiokol was to identify the cause of the erosion, determine whether it was acceptable. define necessary changes, and reevaluate the putty then in use. The Wear letter also requested small motor tests reflecting joint dynamics as well as analysis of the booster assembly process.⁷⁹

Thiokol replied to the Marshall STS 41-C action item on May 4, 1984, with a program plan entitled "Protection of' SRM Primary Motor Seals." The plan was prepared by Brian Russell, then Thiokol's Manager of Systems Engineering. It outlined a systematic program to isolate the 0-ring erosion and charring problem and to eliminate damage to the joint seals. ⁸⁰ Proposed areas of inquiry included the leak check pressures, assembly loads, case eccentricity and putty layup. The Thiokol response in May 1984 was merely a proposal. The actual final response to the directive from Marshall was not completed until the August 19, 1985 briefing on the Solid Rocket Motor seal held at NASA headquarters some 15 months later. ⁸¹

Leak Check and Putty

In addition to the action item from NASA Headquarters, another result of the 41-B erosion was a warning written by John Q. Miller, Marshall chief of the solid motor branch, to George Hardy, through Keith Coates.⁸² Miller was worried about the two charred rings on 41-B and the "missing putty" found when the Solid Rocket Boosters were recovered and disassembled. He specifically identified the putty's sensitivity to humidity and temperature as potential sources of problems. "The thermal design of the [Solid Rocket Motor] joints depends on thermal protection of the O-ring by the [putty]," Miller said. Failure of the putty to "provide a thermal barrier can lead to burning both O-rings and subsequent catastrophic failure." The memorandum also said that "the O-ring leak check procedure and its potential effect on the (putty) installation and possible displacement is also an urgent concern which requires expedition of previously identified full scale tests."

From the beginning, Thiokol had suspected the putty was a contributing factor in O-ring erosion, even after STS-2.⁸³ In April 1983, Thiokol reported on tests conducted to study the behavior of the joint putty. One conclusion of the report was that the STS-2 erosion was probably caused by blow holes in the putty, which allowed a jet of hot gas to focus on a point on the primary O-ring. Thiokol discovered the focused jet ate away or "impinged" on portions of the O-ring. Thiokol calculated that the maximum possible impingement erosion was .090 inch, and that lab test proved that an O-ring would seal at 3,000 psi when erosion of .095 inches was simulated. This "safety margin" was the basis for approving Shuttle flights while accepting the possibility of O-ring erosion. ⁸⁴

Shortly after Miller's routing slip to Hardy about the "urgent concern" of the missing putty on 41-B, at Thiokol, Brian Russell authored a letter to Robert Ebeling which analyzed the erosion history and the test data. Russell's April 9, 1984 conclusion was that the putty itself and its layup were not at fault but that the higher stabilization pressure adopted in leak check procedures, first implemented in one field joint on STS-9, may increase the chances of O-ring erosion. The conclusion by Miller and Russell was that the air pressure forced through the joint during the O-ring leak check was creating more putty blow holes, allowing more focused jets on the primary O-ring, thereby increasing the frequency of erosion.⁸⁵

This hypothesis that O-ring erosion is related to putty blow holes is substantiated by the leak check history (Figure 3). Prior to January, 1984, and STS 41-B, when the leak check pressure was [134] 50 or 100 psi, only one field joint O-ring anomaly had been found during the first nine flights. However, when the leak check stabilization pressure was officially boosted to 200 psi for STS 41-B, over half the Shuttle missions experienced field joint O-ring blow-by or erosion of some kind. ⁸⁶

Moreover, the nozzle O-ring history of problems is similar. The nozzle joint leak check was changed from 50 psi to 100 psi before STS-9 launched in November 1983. After this change, the incidence of O-ring anomalies in the nozzle joint increased from 12 percent to 56 percent of all Shuttle flights. The nozzle pressure was increased to 200 psi for mission 51-D in April, 1985, and 51-G in June, 1985, and all subsequent missions. Following the implementation of the 200 psi check on the nozzle, 88 percent of all flights experienced erosion or blow-by. ⁸⁷

Both Thiokol and NASA witnesses agreed that they were aware that the increase in blow holes in the putty could contribute to O-ring erosion. The Commission testimony of May 2, 1986, reads:

Dr. Walker: The analysis that some of our staff has done suggests that after you increase the test pressure to 200 pounds, the incidence of blow-by and erosion actually increased.

Mr. Russell: We realized that.

Lawrence Mulloy was also questioned above the blow holes in the putty:

Dr. Walker: Do you agree that the primary cause of the erosion is the blow holes in the putty?

Mr. Mulloy: I believe it is. Yes.

Dr. Walker: And so your leak check procedure created blow holes in the putty?

Mr. Mulloy: That is one cause of blow holes in the putty.

Dr. Walker: But in other words, your leak check procedure could indeed cause what was your primary problem. Didn't that concern you?

Mr. Mulloy: Yes, sir. ⁸⁸

Notwithstanding the knowledge that putty blow holes caused erosion and that higher pressure in the leak check caused more blow holes, Thiokol recommended and NASA accepted the increased pressure to ensure that the joint actually passed the integrity tests.⁸⁹

The documentary evidence produced by NASA and Thiokol demonstrates that Marshall was very concerned about the putty erosion/blow hole problem after STS 41-B. In addition to John Miller's routing slip about putty on STS 41-B discussed above, there is a report of a June 7, 1984, telephone conference between Messrs. Thompson, Coates and Ray (Marshall) and Messrs. Sayer, Boisjoly, Russell and Parker (Thiokol), among others.⁹⁰ Marshall told Thiokol that NASA was very concerned about the O-ring erosion problem and that design changes were necessary, including possible putty changes. The Thiokol engineers discussed Marshall's suggestions after the telephone conference, but decided they could not agree a change was mandatory. A follow-up telephone conference was held between Ben Powers of Marshall and Lawrence Sayer of Thiokol on July 2. Powers told Saver that NASA would not accept the removal of the putty from the joint and that everyone expected the tests to show that gas jets would damage an O-ring. However, Powers expressly stated that Marshall would not accept Thiokol's opinion that no further tests were necessary.

In mid-1984, the early tests after NASA's action item for 41-C led Thiokol to the conclusion that O-ring erosion was a function of the putty blow hole size and the amount of free volume between the putty orifice and the O-ring. The damage to the O-ring was judged to be worse when the blow hole was smaller and the free volume was larger.⁹¹

While Thiokol did establish plans for putty tests to determine how it was affected by the leak check in response to the 41-C action item, their progress in completing the tests was slow. The action item was supposed to be completed by May 30, 1984, but as late as March 6, 1985, there are Marshall internal memos that complain that Thiokol had not taken any action on Marshall's December 1983 directive to provide data on putty behavior as affected by the joint leak check stabilization pressure.⁹²

STS 51-C and Cold Temperature

On January 24, 1985, STS 51-C was launched. The temperature of the O-rings at launch was 53....

[135] Figure 4.

NASA Official	Position	Description of Awareness of O-Ring Problems
.
John Young	Chief, Astronaut Office	"The secret seal, which no one that we know knew about." 93
Milton Silveira	Chief Engineer	". . .If I had known . . . I'm sure in the '82 time period when we first came to that conclusion [that the seal was not redundant], I would have insisted that we get busy right now on a design change and also look for any temporary fix we could do to improve the operation of the seal. " 94
.
James Beggs	(Former) NASA Administrator	"I had no specific concerns with the joint, the O-rings or the putty...." 95
.
Arnold Aldrich	Manager, National Space Transportation System	None were aware of Thiokol's concern about negative effect of cold temperature on O-ring performance, nor were they informed of the same concern raised after STS 51-C. 96
Jesse Moore	(Former) Associate Administrator for Space Flight
Richard Smith	Director, Kennedy Space Center
James A. Thomas	Deputy Director, Kennedy Launch and Landing Operations

....degrees, the coldest to that date. O-ring erosion occurred in both solid boosters. The right and left nozzle joint showed evidence of blow-by between the primary and secondary O-rings. The primary O-ring in the left booster's forward field joint was eroded and had blow-by, or soot behind the ring.⁹⁷ The right booster's damage was in the center field joint-the first time that field joint seal was damaged. Both its primary and secondary O-rings were affected by heat, and the primary ring also had evidence of blow-by of soot behind it. This was also the first flight where a secondary O-ring showed the effect of heat.

STS 51-C was the second example of O-ring damage in flight where there was evidence of blow-by erosion as well as impingement erosion. As noted previously, impingement erosion occurs where the O-ring has already sealed and a focused jet of hot gas strikes the surface of the ring and removes a portion of it. Blow-by erosion happens when the O-ring has not yet sealed the joint gap and the edge of the ring erodes as the hot gas flows around it.

Roger Boisjoly described the blow-by erosion seen in 51-C:

"SRM 15 [STS 51-C] actually increased [our] concern because that was the first time we had actually penetrated a primary,, O-ring on a field joint with hot gas, and we had a witness of that event because the grease between the O-rings was blackened just like coal . . . and that was so much more significant than had ever been seen before on any blow-by on any joint . . . the fact was that now you introduced another phenomenon. You have impingement erosion and bypass erosion, and the O-ring material gets removed from the cross section of the O-ring much, much faster when you have bypass erosion or blow-by." ⁹⁸

[136] Boisjoly also said blow-by erosion was where the primary O-ring "at the beginning of the transient cycle . . . is still being attacked by hot gas, and it is eroding at the same time it is trying to seal, and it is a race between, will it erode more than the time allowed to have it seal." He described the blow-by on 51-C as "over 100 degrees of arc, and the blow-by was absolutely jet black. It was totally intermixed in a homogeneous mixture in the grease." When the blow-by material was chemically analyzed, Boisjoly said, "we found the products of putty in it, we found the products of O-ring in it."⁹⁹

On the Marshall problem assessment report that was started to track field joint erosion after STS 41-B, the STS 51 -C O-ring anomaly was described as "O-ring burns were as bad or worse than previously experienced . . . Design changes are pending test results." ¹⁰⁰ The changes being considered included modifying the O-rings and adding grease around the O-rings to fill the void left by putty blow holes.

On January 31, 1985, Marshall Solid Rocket Booster Project Manager Mulloy sent an urgent message to Lawrence Wear with the stated subject: "51-C O-Ring Erosion Re: 51-E FRR." The message ordered that the Flight Readiness Review for the upcoming flight:

"Should recap all incidents of O-ring erosion, whether nozzle or case joint, and all incidents where there is evidence of flow past the primary O-ring. Also, the rationale used for accepting the condition on the nozzle O-ring. Also, the most probable scenario and limiting mechanism for flow past the primary on the 51 -C case joints. If [Thiokol] does not have all this for today I would like to see the logic on a chart with blanks [to be filled in ] . " ¹⁰¹

On February 8, 1985, Thiokol presented its most detailed analysis to date of the erosion problems to the Solid Rocket Motor project office at Marshall for what was then called Shuttle mission 51-E, but later changed to 51-D. Thiokol included a report on damage incurred by the O-rings during flight 51-C at the left forward and right center field joints. The right center joint had hot gas past the primary O-ring. Thiokol said that caused a concern that the gas seal could be lost, but its resolution was "accept risk." ¹⁰²

Thiokol presented test results showing "maximum expected erosion" and "maximum erosion experienced" for both primary and secondary O-rings for- the field and nozzle joints. Accepting damage to the primary O-ring was being justified, in part, based on an assumption of the secondary O-ring working even with erosion. However, the Criticality classification indicated the primary seal was a "single point failure." During this flight readiness assessment at Marshall, for the first time Thiokol mentioned temperature as a factor in O-ring erosion and blow-by. Thiokol said in its conclusions that "low temperature enhanced probability of blow-by-[flight] 51 -C experienced worst case temperature change in Florida history." Thiokol concluded that while the next Shuttle flight "could exhibit same behavior," nonetheless "the condition is not desirable but is acceptable." ¹⁰³

At the Level I Flight Readiness Review conducted on February 21, there was no detailed analysis of O-ring problems presented or any reference made to low temperature effects. Instead, a single reference indicated the O-ring erosion and blow-by experienced was "acceptable" because of 'limited exposure time and redundancy."

STS 51-B and the Launch Constraint

Joint seal problems occurred in each of the next four Shuttle flights. Flight 51-D, launched April 12, 1985 had nozzle O-ring erosion and blow-by on an igniter joint. STS 51-B, launched 17 days later, experienced both nozzle O-ring erosion and blow-by as did 51-G, which flew on the following June 17. STS 51-F, launched duly 29, 1985 had nozzle O-ring blow-by.¹⁰⁴

In response to the apparent negative effect of cold leading to the extensive O-ring problems on flight 51 -C in January, Thiokol conducted some O-ring resiliency tests in early 1985. ¹⁰⁵ The tests were conducted to quantify the seal timing function of the secondary O-ring and the effect of joint rotation on its ability to back up the primary ring. The key variable was temperature. The June 3 test report, which was described in an August 9, 1985 letter from Brian Russell at Thiokol to Jim Thomas at Marshall, showed:

"Bench test data indicates that the O-ring resiliency (its capability to follow the metal) is a function of temperature and rate of case expansion. [Thiokol] measured the force of the O-ring against Instron platens, which [137] simulated the nominal squeeze on the O-ring and approximated the case expansion distance and rate.

"At 100°F, the O-ring maintained contact. At 75°F the O-ring lost contact for 2.4 seconds. At 50°F, the O-ring did not reestablish contact in ten minutes at which time the test was terminated." ¹⁰⁶

On June 25, 1985, the left nozzle joint of STS 51-B (launched April 29) was disassembled and inspected after it had been shipped back to Thiokol. What Thiokol found was alarming. The primary O-ring seal had been compromised because it eroded .171 inches and it did not seal. The secondary O-ring did seal, but it had eroded .032 inches. Lawrence Mulloy described the 51-B problem as follows:

"This erosion of a secondary O-ring was a new and significant event . . . that we certainly did not understand. Everything up to that point had been the primary O-ring, even though it had experienced some erosion does seal. What we had evidence of was that here was a case where the primary O-ring was violated and the secondary O-ring was eroded, and that was considered to be a more serious observation than previously observed . . .¹⁰⁷

"What we saw [in 51-B], it was evident that the primary ring never sealed at all, and we saw erosion all the way around that O-ring, and that is where the .171 came from, and that was not in the model that predicated a maximum of .090, the maximum of .090 is the maximum erosion that can occur if the primary O-ring seals.

"But in this case, the primary O-ring did not seal; therefore, you had another volume to fill, and the flow was longer and it was blow-by and you got more erosion." ¹⁰⁸

Upon receiving the report of the 51-B primary ring failure, Solid Rocket Booster Project Manager Mulloy and the Marshall Problem Assessment Committee placed a "launch constraint" on the Shuttle system. ¹⁰⁹ A 1980 Marshall letter which references "Assigning Launch Constraints on Open Problems Submitted to MSFC PAS" defines launch constraint as:

"All open problems coded Criticality 1, 1R, 2, or 2R will be considered launch constraints until resolved (recurrence control established and its implementation effectivity determined) or sufficient rationale, i.e., different configuration, etc., exists to conclude that this problem will not occur- on the flight vehicle during pre-launch, launch, or flight." ¹¹⁰

Lawrence Mulloy told the Commission that the launch constraint was "put on after we saw the secondary O-ring erosion on the [51-B] nozzle." "Based on the amount of charring," the problem report listing the constraint said, "the erosion paths on the primary O-ring and what is understood about the erosion phenomenon, it is believed that the primary O-ring [of the joint] never sealed." ¹¹¹ The constraint applied to STS 51-F and all flights subsequent, including STS 51-L. Although one Marshall document says that the constraint applied to all O-ring anomalies, ¹¹² no similar launch constraint was noted on the Marshall Problem Assessment Report that started tracking the field joint erosion after STS 41-B. Thiokol officials who testified before the Commission all claimed they were not aware of the July 1985 launch constraint; ¹¹³ however, Thiokol letters referenced Marshall Record number A09288, the report that expressly identified the constraint. ¹¹⁴

After the launch constraint was imposed, Project Manager Mulloy waived it for each Shuttle flight after July 10, 1985. Mr. Mulloy and Mr. Lawrence Wear outlined the procedure in the following manner:

Chairman Rogers: To you, what does a constraint mean, then?

Mr. Mulloy: A launch constraint means that we have to address the observations, sec if we have seen anything on the previous flight that changes our previous rationale and address that at the Flight Readiness Review.

Chairman Rogers: When you say»address it," I always get confused by the word. Do you mean think about it? Is that what you mean?

Mr. Mulloy: No, sir. I mean present the data as