CHAIRMAN ROGERS: I call the Commission to order.
Mr. Mulloy, Mr. Wear, we have been considering the history of the joint that failed, and in connection with that we have been considering the history of it going back several years. The purpose for the meeting this morning is to discuss some of the documents that relate to the history of it and give all of you who were involved in its development and the experience that you have had with it the opportunity to refer to some of the documents that have been provided to us.
[1506] This will be on the record. I assume-has Mr. Wear been sworn?
MR. WEAR: I have not.
CHAIRMAN ROGERS: Well, let me finish. Then we will do that. We decided to have this in executive session because we want it to be informal, and secondly, we want to deal with several documents. In an exploratory kind of discussion of this kind, it isn't really suitable for public session, but we will make the testimony public at some time later on, because we may use some of it in our final report, and we thought that we would have one of our staff members refer to the documents, Mr. Kehrli, so that you can identify the
documents, so that we are all talking about the same documents, and then he will ask some questions about those.
It is our intention to end the hearings of the Commission today. We do not plan to have any further hearings, so this will be the last hearing. If we might swear Mr. Wear.
(Witnesses sworn.)
CHAIRMAN ROGERS:
Randy, do you want to start, please?
MR. KEHRLI: Thank you, Mr. Chairman. Gentlemen, in front of you you have a book that contains the documents that we would like you to refer to today. I am going to refer to them specifically and read portions of them into the record, and you might want to follow along while I do that, before the Commissioners begin asking you questions.
That booklet is divided into two sections. The first one is a major tab, O-ring History, and the attachments run, the tab attachments run from Number 1 through Number 35. The second major tab, the second division is Launch Constraint, in the back, and those attachments run from 1 through 4.
I would like to direct your attention first of all to the Launch Constraint Attachments 1 through 4.
During the course of the investigation, either as a result of interviews or specific requests by the Commission to NASA, we received some of these documents, as the Chairman indicated to you.
The first document I would like to refer to is Launch Constraint Attachment Number 1, which is a document dated September 15, 1980, to distribution from Mr. Lindstrom, and the subject of this memorandum is signing launch constraints on open problems submitted to Marshall PAS. This is in the back of the book, in back of the Launch Constraint section, Attachment Number 1. [Ref. 5/2-1]
The memorandum reads, "The following guidelines have been established to aid in making constraint decisions on open problems and are limited to recurrence control determination only. In accordance with practices established on past programs, remedial actions, for example, removal and replacement of defective hardware, et cetera; for correcting discrepancies on the vehicle to be launched are considered launch constraints and are tracked by the KSC system."
This is an excerpt from the document. I haven't read it in its entirety. It is Paragraph A. Subparagraph 1 reads, "All open problems coded Criticality 1, 1R, 2, or 2R will be considered launch
constraints until resolved, which is recurrence control established and its implementation effectively determined, or sufficient rationale; in other words, different configuration, et cetera, exists [1507] to conclude that this problem will not occur on the flight vehicle during prelaunch, launch, or flight."
Now, I would like to direct your attention to Attachment Number 2 of the Launch Constraint section, and this attachment is a Marshall problem assessment system. This document was in fact obtained from Marshall. It is dated February 26th, 1986, and it is my understanding, and of course you are free to correct me on any of this, that this is a chronological history of entries from the Marshall problem assessment system.
The relevant items on this document that we would like to have you address include the fact that the document indicates that a launch constraint was assigned to STS-51F, 511, 51J, 61A, 61B, and 61C. The date, I believe that is R-e-c-which apparently means recorded or received-over on the right is July 10, 1985. That matches with the first entry on the document, which is dated July 11th, 1985, and it indicates that post-the entry on the 11th indicates that post-flight inspection, SRM-16A revealed a gas path through the vacuum putty at 54 degrees, and it goes on to describe essentially the
erosion problem in the nozzle joint, I believe, in 51-STS-51B, which was inspected on June 25th at Thiokol, in 1985.
There is an indication that the constraint closure recorded was December 18, 1985, in the document under the date recorded, and without belaboring the further entries on the document, it indicates that the problem was "not considered a constraint to 51F; 100 psi leak check is performed, which confirms seating of the secondary O-ring. The nozzle O-rings have been shown to survive erosion, gaps of 125 mils in hot subscale test," and it continues to discuss the math model, as you can read.
Additionally, there are individual entries later on in that document which indicate that the constraint was lifted for subsequent flights that were listed previously, 511, 51J, 61A, 61B, and 61C.
Finally, there was a handwritten notation on the last page of the document that says: "constraint can be lifted by project manager," and this notation was on the document when we received it. Additionally, the last entry on this problem assessment report dated 1/23/86 indicates that there was a resolution: "In the SRM, field and nozzle joints have experienced erosion of the primary O-rings during several missions and static
tests as determined by post-flight inspection," and it continues to go on through that entry and give the rationale for closure of the problem.
The last paragraph reads: "This problem is considered closed based on MTI report TWR14359, Revision A, improvements in Space Shuttle SRM motor seals dated 8/30/85, and MTI letter E150BGR86144, rationale for closure of the O-ring erosion problems."
The last important information on this document is the-at the bottom of the first page there is an indication of assignee and approval, and the names there are R. McIntosh, D. Newman, L. Wear, J. Fletcher, and there is an indication of the status of the document: "PAC review is complete." The status is still marked open, Code M, I believe.
Finally, or additionally, the next attachment, Attachment Number 3, is a similar problem assessment which was initiated on February 17th, 1984, and summarized in this document. This is a problem assessment begun after the erosion, the O-ring erosion problem on Flight 41B, and it again tracks that problem up until the time the problem was closed, which is the same date as the nozzle, the previous nozzle problem, 1/23/86. [Ref. 5/2-3]
[1508] Again, the same names are on the bottom. The difference, the key difference on this document is under
"launch constraints" it says "none," and there is no indication on the document that a launch constraint was attached to this particular problem assessment report on the field joints.
Again, I won't go through the various entries. You gentlemen are probably familiar with them, and the Commission has seen them. The closure entry is the same on this document as it was on the nozzle, the nozzle document described previously.
CHAIRMAN ROGERS: May I ask, how did this first document you referred to come to our attention?
MR. KEHRLI: We first discovered through an interview conducted by one of the investigators at Thiokol that there was a monthly problem report that Thiokol filed internally which tracked the various problems that arose during and after a particular flight, and tracked the problem until it was worked and/or closed.
We discovered in one of those entries a reference to a specific Marshall problem assessment report, and it indicated that there had been a tracking number, a problem number assigned to the 41B erosion problem, and also to 51B, and additionally DM7, I also believe, had a separate report. Then upon talking to George Hardy and, I believe, Mr. Mulloy,
Marshall searched their records and found these documents which you have in front of you, the Marshall assessment report. So these are in fact Marshall documents.
CHAIRMAN ROGERS: And that was after our public hearings, so that was how we found this document?
MR. KEHRLI: Yes, this was after the public hearings, and we found these in the course of interviews.
CHAIRMAN ROGERS: Just to be sure the record is complete, this J. Fletcher is a separate J. Fletcher from the Fletcher that has been nominated and affirmed, I guess, as administrator.
MR. KEHRLI: Yes.
CHAIRMAN ROGERS: Well, I think what the Commission would like to hear from you, Mr. Wear and Mr. Mulloy, is an explanation of this, first, why we didn't know about it, and secondly, what gave rise to the launch constraint, how they were handled, who made the decision to waive on all of these flights. I guess these were the flights that preceded 51L.
MR. MULLOY: Yes, sir.
CHAIRMAN ROGERS: And why it was closed out just before 51L.
MR. MULLOY: Okay, sir. I will start to address that, and then Mr. Wear, I think, can elaborate.
The problem assessment system was put in place to provide visibility throughout the shuttle system for the types of problems that do occur, not just in flight, but also in qualification tests, and in failure of hardware that is back for refurbishment at a vendor or whatever. And it is a closed loop tracking system that lists the anomaly.
[1509] Now, in accordance with the memo there from Bob Lindstrom in September of 1980, the procedure was established that in all flight readiness reviews, all open problems would be flagged at the flight readiness review, and it is our quality organization that does that, and that requires them when the problem is not completely closed out, it required dealing with it in the flight readiness review process, as I testified in both the previous private and public hearings, that these anomalies were covered in the flight readiness review.
There was a cross-check by the quality people to assure that the problem had been discussed and resolved, a rationale for making the next flight, in view of the fact that the problem had not been completely eliminated in making the flight with the acknowledged chance that there would be a recurrence of the type of thing that had been observed in the initial problem report, so on each flight up to, from the time that the constraint was shown, it required a signature by the project office, which is Mr. Wear's office for the solid rocket motor, that that had been addressed and closed in the flight readiness review, and it's required, his initial, essentially on the problem assessment, and where you see on the problem assessment there, I believe there is a JWT initial in there for each one of those. That JWT is Jim Thomas, who works in Mr. Wear's office, and is director of his engineering branch in the project office.
Now, the entry that is shown in there that the problem was closed prior to 51L is in error. What happened there was, one of your documents here which we did not discuss is the letter from Mr. McDonald to Mr. Wear which proposed that this problem be dropped from the problem assessment system and no longer be tracked for the reasons stated in Mr. McDonald's letter.
That letter was in the review cycle. The
letter, I believe, was dated 10 December 1985. It came into the center, it was in the review cycle. After Mr. Wear brought this letter to my attention, my reaction was, we are not going to drop this from the problem assessment system because the problem is not resolved and it has to be dealt with on a flight-by-flight basis.
Since that was going through the review cycle, the people who run this problem assessment system erroneously entered a closure for the problem on the basis of this submittal from Thiokol. Having done that then for the 51L review, this did not come up in the flight readiness review as an open launch constraint, so you won't find a project signature because the PAS system showed the problem was closed, and that was an error.
CHAIRMAN ROGERS: Who made the error? Do you know?
MR. MULLOY: The people who do the problem assessment system.
MR. WEAR: Mr. Fletcher, and he reports within our quality organization at the flight readiness review, at the incremental flight readiness reviews, as I think have been described to you before. There is one from Thiokol to me, and there is one from my group to Larry, and then Larry, of course, does one with the Shuttle
project office, and so forth, on up the line. At my review and at Larry's review, there is a heads up given to the quality representative at that board for what problems the system has open, and they cross-check to make sure that we address that problem in the readiness review.
On this particular occasion, there was no heads up given because their PAS system considered that action closed. That is unfortunate.
[1510] CHAIRMAN ROGERS: Let's go back just a bit, because I think it is helpful to me if you you use words that I understand a little bit. What caused the constraint to be put on in the first place?
MR. MULLOY: The constraint was put on after we saw the secondary O-ring erosion on the nozzle, I believe.
CHAIRMAN ROGERS: Who decided that?
MR. MULLOY: I decided that, that that would be addressed, until that problem was resolved, it would be considered a launch constraint, and addressed at flight readiness reviews to assure that we were staying within our test experience base.
CHAIRMAN ROGERS: And was that-what is meant by the problem description?
MR. MULLOY: Which document, sir?
MR. KEHRLI: That is Number 2.
MR. MULLOY: Yes, reference similar reports. Yes, sir.
DR. RIDE: Why didn't you put a launch constraint on the field joint at the same time?
MR. MULLOY: I think at that point, and I will react to that question in real time, because I haven't really thought about it, but I think the logic was that we had been observing the field joint, the field and nozzle joint primary O-ring erosion. This erosion of a secondary O-ring was a new and significant event, very new and significant event that we certainly did not understand. Everything up to that point had been that the primary O-ring, even though it had experienced some erosion, does seal. What we had evidence of was that here was a case where the primary O-ring was violated and the secondary O-ring was eroded, and that was considered to be a more serious observation than previously observed.
DR. RIDE: Correct me if I am wrong, but weren't you basing most of your decisions on the field joint on analysis of what was the maximum, what you believed to be the maximum possible erosion, and you had that analysis for the field joint and for the nozzle joint. When you saw the complete erosion of the primary
O-ring on the nozzle joint, that showed you that your analysis on the nozzle joint wasn't any good, I would think. That would indicate to you that your analysis on the field joint wasn't very good, either, or at least should be suspect.
MR. MULLOY: The conclusion, rightly or wrongly, for the cause of the secondary O-ring erosion on the nozzle joint, it was concluded from test data we had that 100 psi pressurization leak check, that the putty could mask a primary O-ring that was not sealing. The conclusion was-and that one was done at 100 psi. The conclusion was that in order to get that type of erosion that we saw on the primary O-ring, that that O-ring never sealed, and therefore the conclusion was that it never was capable of sealing.
The leak check on subsequent nozzles, all subsequent nozzles was run at 200 psi, which the test data indicated would always blow through the putty, and in always blowing through the putty we were guaranteed that we had a primary O-ring seal that was capable of sealing, and then we further did, and we already had that on the field joints at that time.
DR. RIDE: The 200 psi check on the field joints were started about a year earlier. Is that right?
MR. MULLOY: I don't recall specifically where. I don't recall the dates.
MR. WEAR: I believe it was-I think there was a separation of about three flows, I believe.
[1511] Mr. MULLOY: So what concerned us about the nozzle joint was that we had, as you have stated, Dr. Ride, we had to go back and then look at the analysis for a violated primary O-ring, so the rationale for the proceeding with flight, having looked at the occurrence on the nozzle joint, another analysis was run which included violation of the primary O-ring to determine what is the maximum erosion that could occur on the secondary O-ring.
That analysis matched very well with the observations that we had from the flight that that occurred on, and that was corroborated by test, and that was the rationale for proceeding with the next flight, and whether or not it was treated as launch constraint in the past, it was treated as an issue to be discussed in each one of the flight readiness reviews on the field joints as well as the nozzle joints.
DR. WALKER: Were you at all concerned about violations of the integrity of the putty, which was really the first line of defense in this joint? In fact, apparently you were deliberately trying to violate the putty by going to 200 pounds per square inch.
MR. MULLOY: Yes, sir. We were concerned with
violation of the putty, and as has been laid out during this investigation, there was a program under way to find something as an alternative to the putty. We were not deliberately trying to violate the putty. What we wanted to be sure was that we got a leak check on the primary O-ring, and the only way you could be sure you got a leak check on the primary O-ring was to put a pressure between the two rings that would, if the primary did leak, that it would blow through the putty.
DR. WALKER: Well, all right. Then if you found that-well, if you were going to violate the putty, there was no procedure you had which could determine that. You could violate the putty in your test, and yet you would never know that, and yet that was an integral part of your protection, and no actions were instituted to address that problem, as I understand it.
MR. MULLOY: Oh, yes, sir, that was the whole test program that was laid out, to find an alternative to the putty.
DR. KEEL: Mr. Mulloy, didn't it occur to you that if the putty could be masking a leak check, then the putty could sure be masking pressurization actuation of the primary O-ring, and hence your whole analysis about how that primary O-ring sealed and the time it
takes to seal could be just as suspect?
MR. MULLOY: No, sir, it did not. The pressurization is an entirely different direction.
DR. KEEL: Of course, the putty is still a barrier in both instances.
MR. MULLOY: Yes, sir. And the putty has a substantially different pressure carrying capability, depending upon which side you pressurize it from.
DR. KEEL: That was your assumption?
MR. MULLOY: That is a fact.
DR. KEEL: Do you still think the putty doesn't have anything to do with the pressurization of the primary O-ring?
MR. WEAR: You are speaking of today?
DR. KEEL: Yes.
MR, WEAR: Well, today there has been quite an extensive testing conducted.
DR. KEEL: What is your answer?
MR. WEAR: Well, I think that the testing that has been done has demonstrated that the time delay factor to the putty, as I understand, and I haven't been that close to the investigation per se
[1512] DR. KEEL: So regardless of which side you pressurize, it has an effect of delay.
MR. MULLOY: The putty is highly variable, and
we understand that its ability to sustain or not sustain pressure is unpredictable.
DR. WALKER: But you were not aware of that earlier. Is that correct?
MR. MULLOY: Yes, sir, we were aware of it, because sometimes we saw paths through the putty, evidence of paths through the putty. Sometimes we saw evidence of paths through the putty and O-ring erosion. Sometimes we saw no paths through the putty and no evidence of hot gas past the putty, and so we knew that it was performing in that variable manner.
DR. WALKER: The evidence of paths through the putty that you speak of, was that evidence directly looking at the putty after demating, or was that by implication because you saw O-ring erosion?
MR. MULLOY: Both. It was looking at the putty and at the joint, looking for evidence of soot between the putty and the primary O-ring, and the distribution of that soot, and the evidence of the path, hot gas path through the putty.
DR. WALKER: So you knew that in most cases pressurization was occurring by leaks through the putty?
MR. MULLOY: I don't believe it is in most cases. No, sir, because we had a very limited number of observations of the 170 some odd joints. There is a
very limited number of observations where you have pressurization by breakdown of the putty. The rest of them, there is no evidence of paths through the putty.
DR. WALKER: But there was a memo from the Titan program suggesting that pressurization, at least in the case of Titan, was occurring primarily through blow holes in the putty. Did you receive that memo?
MR. MULLOY: Yes, sir. You are speaking of, I think it was on your list for about a March, 1984, memo that I got from George Morefield. Yes. [Ref. 5/2-5]
DR. WALKER: And I think that was passed on to you from the chief engineer's office at headquarters. Is that correct?
MR. MULLOY: No, sir, that was written directly to me from Mr. Morefield at my request, observing this problem, asking CSD and Hercules to look at this problem and give me their observations about what they thought was happening and what could be done to rectify it, and so that is what we were working on.
DR. WALKER: So did you accept their view that pressurization in the shuttle was occurring primarily by blow holes, or did you think that the situation was different in your booster than the Titan?
MR. MULLOY: Yes, sir, I thought it was different based upon the evidence that we had that we
had a lot of joints that didn't have any blow holes in them, and evidence that-no evidence of any kind of a hot gas past the putty. And that is an established fact, I believe.
CHAIRMAN ROGERS: Assuming that you were advised as you were by Thiokol that they opposed the launch on the 27th because of weather, would you have reacted differently?
[1513] MR. MULLOY: No, sir. Frankly, I was not aware that this erroneous entry had entered in the PAS because it did not come up.
CHAIRMAN ROGERS: To you, what does a constraint mean, then?
MR. MULLOY: A launch constraint means that we have to address the observations, see if we have seen anything on the previous flight that changes our previous rationale, and address that at the Flight Readiness Review.
CHAIRMAN ROGERS: When you say "address it," I always get confused by the word. Do you mean think about it? Is that what you mean?
MR. MULLOY: No, sir. I mean present the data as to whether or not what we have seen in our most recent observation, which may not be the last flight, it may be the flight before that, is within our experience base and whether or not the previous analysis and tests that previously concluded that was an acceptable situation is still valid, based upon later observations.
CHAIRMAN ROGERS: Okay. So in this case, because you didn't know that the constraint had been closed due to an error, if the constraints were still on, if no error would have been made, you would still have reacted the same way?
MR. MULLOY: Reacted to what, sir?
CHAIRMAN ROGERS: Reacted as you did on the telecon.
MR. MULLOY: Yes, sir.
CHAIRMAN ROGERS: And so each time one of these flights took off that you knew there was a constraint on, you had to make a decision to waive it, what went through your mind?
MR. MULLOY: Okay. What went through my mind is, we looked at the most recent observation of recovered hardware. We compared what we were seeing to our previous successful experience with the joint, realizing that we were having some O-ring erosion, looking if we were seeing anything that changed the previous rationale. That is what led to opening the problem report, when we saw that we violated the primary O-ring. That was something that was different and therefore required additional analysis and test and until that was done.
CHAIRMAN ROGERS: But what did you do about
it, though? It seems to me in that case, when you say that you addressed it, no change was made in it. All you did on these waivers was to waive it. You just apparently-there is no indication.
What did you do? There is nothing in this chart that suggests that you corrected the joint.
Each time there was further experience, further erosion, and we don't see any examples of correction or effective action taken or anything of that kind.
MR. MULLOY: I guess I would have to take issue with that, sir. On page 2- -
CHAIRMAN ROGERS: That is why we want you here.
MR. MULLOY: On page 2 of 3 of document No. 2, if you look at-and I will just pick one and I think there's a rationale here. It says this problem is not considered a constraint to 510-F; a 100 psi leak check is performed which confirms seating of the secondary O-ring. Nozzle O-rings have been shown to survive erosion depths of 125 mils in hot subscale test. The math model predicts maximum erosion depth for the secondary O-ring to be 75 mils in the worst case condition. [Ref. 5/2-2]
The erosion on the secondary O-ring was 32
mils, well below the demonstrated sealing capability.
[1514] What was done was analysis and test to substantiate that rationale for accepting that condition for that flight. It is not just a matter of nothing was done.
DR. WALKER: You are relying on the secondary O-ring.
MR. MULLOY: No, sir. What we were saying was if we got a repeat, we didn't think we would get a repeat of the condition because we had gone to 200 psi and it was concluded that, we knew the putty could mask 100 psi leak check and we could have had a bad O-ring primary to begin with on that one.
We were sure on the 51-F flight because of the 200 psi leak check that we had a good primary O-ring was a substantial part of the rationale. The second then was if the primary O-ring was violated, the maximum erosion that could occur on the secondary O-ring was only 75 mils, which tests had shown could sustain 125 mils.
Now, rightly or wrongly, that was the rationale.
DR. WALKER: So you then were relying on the secondary O-ring in that case.
MR. MULLOY: We were relying on the
redundancy, yes. We showed that we had redundancy that, should the primary O-ring fail, the secondary would function.
DR. WALKER: Well, I guess we are going to address the case of the criticality later, and so I think I will save those questions.
MR. RUMMEL: Mr. Mulloy, I wonder how detailed your investigations were when you applied the experience of one flight to a subsequent flight.
For example, did you consider the differences in dimensional tolerances and dimensions of the various joints-out of roundness, they varied in diameter, they varied in numbers of ways and so when you evaluated a specific flight, did you look into where the previous flight or flights stood in this regard and then analyzed what would be applicable on a specific flight under consideration?
MR. MULLOY: Out of roundness? No, sir. We gave no specific consideration to a variation in gap that could occur due to the out-of-round condition.
What we did do was look at the dimensional tolerances for the specific flight, the tang dimension and the clevis dimension, to assure that we had the minimum O-ring squeeze that was specified and assure that that was within our experience base.
We did not go in and look at what would be the effect of out-of-roundness and possibly a higher squeeze on O-rings in a local area of the joint.
That is a revelation, I think, that has come from the investigation of the 51-L failure.
DR. FEYNMAN: Mr. Mulloy, when you use a math model, do you have any idea of how accurate it is?
MR. MULLOY: We did not just use the math model. What we did was build a math model that was correlated to test. There was a test fixture that was built to empirically determine the maximum erosion that could occur while filling the annulus between the putty and the primary O-ring and the annulus between the primary and the secondary O-ring.
Then Thiokol's, Dr. Salita's math model was shown to correlate very well with that, and I guess I can't put a percentage accuracy on that. But the fact that the math model correlated pretty well with the test results gave us some confidence in that and the fact that the test demonstrated that there was a significant margin that was tolerable in terms of the amount of erosion, given the dimensional tolerance.
[1515] DR. FEYNMAN: I think that the math model determined how the constants were determined and a line was put through the previous data on a somewhat similar
material. And the line that was put through deviated. It doesn't always give the same answer. You took an average rather than the maximum, so that there were factors of 2 above and factors of 2 below on the original data. If you would have known that, you could have appreciated that what this thing predicted could easily be a factor of 2 below the right answer, because in fact it didn't even fit with the data on which it was constructed.
You weren't aware of that?
MR. MULLOY: No, sir. I was not aware of that.
DR. KEEL: Well, Mr. Mulloy, what was your previous conservative estimate of maximum erosion on the primary O-ring for a nozzle joint? Do you remember that? You briefed it at several Flight Readiness Reviews.
(Pause.)
DR. KEEL: Going back to the record, it is 90/1000, based upon what you've characterized as a conservative estimate, by which you presumably meant worst case. Is that correct?
MR. MULLOY: That is correct.
DR. KEEL: And what erosion was there on 51-B that caused the launch constraints?
MR. MULLOY: On the primary O-ring erosion it was 171/1000.
DR. KEEL: So 171 compared to the previous worst case analysis prediction of 90.
MR. MULLOY: Yes, sir. And what was different-that again is why we were concerned about this and established the understanding of this as a launch constraint.
What we observed was we were seeing a different type of erosion on this primary O-ring. And that different type-what we had been seeing previously was
DR. KEEL: By different type, you mean worse erosion than predicted?
MR. MULLOY: No, sir. I mean a different type. The erosion that we had been seeing was due to the hot gas direct jet impingement on the surface of the primary ring as it seated.
What we saw here, it was evident that the primary ring never sealed at all, and we saw erosion all the way around that O-ring, and that is where the 171/1000 came from, and that was not in the model that predicted a maximum of 90/1000. The maximum of 90/1000 is the maximum erosion that can occur if the primary O-ring seals.
But in this case, the primary O-ring did not seal; therefore, you had another volume to fill, and the flow was longer and it was blow-by and you got more erosion.
DR. KEEL: But I think that the bottom line of all that is that perhaps your analysis didn't consider all the physical phenomena, including how that primary O-ring seals and how long it takes to seal, and if it doesn't seal, what erosion should take place then.
MR. MULLOY: It did consider how long it takes to seal and how long it takes to fill that volume. What it didn't consider was that the primary O-ring did not seal and you had hot gas impingement for the additional period of time that it takes to fill the volume during the primary and secondary. It did not consider that.
DR. KEEL: And as a consequence, it was off by a factor of almost 2.
MR. MULLOY: No, sir. It was correct for the mode that it was analyzed for. It was not correct for an O-ring that was not sealing. It is absolutely correct and has been demonstrated to be correct for a primary O-ring that seals.
[1516] DR. KEEL: It's hard, I think, certainly for me and I think the rest of the Commission, to understand how you can say it's absolutely correct. I don't know
of any models you've done since then or tests you've done since then that can correlate any more accurately than Dr. Feynman has indicated with your erosion models, an analytic empirical model. And it has to have uncertainty about it.
MR. MULLOY: Absolutely. That is a very unfortunate adjective. I withdraw that. Nothing is absolute.
DR. KEEL: I would have thought when you had this experience that you wouldn't have immediately gone back to your analytic methods as the basis of your confidence for lifting these launch constraints because certainly this case of erosion was certainly worse than what you would hypothesize from the previous worst case.
MR. MULLOY: Yes, sir. And that is why we ran additional tests and expanded the analytical model to account for this mode where the primary O-ring did not seal.
DR. KEEL: And you expanded the analytical model and what was the worst case erosion that that model then predicted?
MR. MULLOY: I believe I said 75 mils on the secondary O-ring if you had a primary O-ring that was violated and never sealed. And that is what the problem assessment system report shows on page 2 of 3.
DR. RIDE: Did you consider that acceptable?
MR. MULLOY: Yes.
DR. RIDE: 75 mil erosion of the secondary O-ring.
MR. MULLOY: Yes.
DR. COVERT: Mr. Mulloy, at the time this joint was conceived of, did you envision that the O-rings would be eroded to this extent?
MR. MULLOY: I'm sorry, Dr. Covert?
DR. COVERT: At the time that the joint was designed, was it designed with the intent in mind that the O-rings would be eroded to this extent?
MR. MULLOY: No.
DR. COVERT: So in some way, then, the acceptance of this erosion as a fact of life represented a departure from margins of safety that you originally had in mind at the time you were designing it?
MR. MULLOY: Yes, sir. It was treated as an anomaly.
CHAIRMAN ROGERS: Going back just for a moment to the criticality I statement, one of the things that has troubled me from the start and still troubles me is, if I understand English at all, this says that the leakage of the primary O-ring seal is classified as a single failure point due to the possibility of loss of the sealing at the secondary O-ring, because of joint
rotation after motor pressurization. [Ref. 5/2-6]
MR. MULLOY: Yes, sir.
CHAIRMAN ROGERS: That to me says that if the primary O-ring fails, then there is apt to be a loss of mission, vehicle and crew. That is what it says.
Then you have experiences of one kind or another over a period of time with a seal which certainly causes a lot of discussion and a lot of concern.
[1517] At one point in the same document it says, based on the amount of charring and erosion paths through the primary O-ring, and what is understood about the erosion phenomenon, it is believed that the primary O-ring SMR-61A never sealed or never seated.
Now, why at that point, wouldn't you all have said we were lucky not to have a loss of mission and crew at that point because criticality I says if that primary O-ring fails, we will lose everything? Why wasn't that a cause for concern on the part of the whole NASA organization?
MR. MULLOY: It was cause for concern, sir.
CHAIRMAN ROGERS: Who did you tell about this?
MR. MULLOY: Everyone, sir.
CHAIRMAN ROGERS: They all knew about it?
MR. MULLOY: Yes, sir.
CHAIRMAN ROGERS: And they all knew about it at the time of 51-L?
MR. MULLOY: Yes, sir. You will find in the Flight Readiness Review record that went all the way to the L minus one review.
CHAIRMAN ROGERS: That's why I'm saying we want to talk to you this morning. That suggests that was pretty well glossed over; that they didn't really realize it. But in any event
DR. SUTTER: Well, could I ask a quick question? I'm confused about this thing being signed off and that was a mistake, apparently. But is it still on the books as being signed off.? When was that mistake discovered?
MR. WEAR: It is in the books today, just like you see it here.
DR. SUTTER: So then a lot of
MR. WEAR: We haven't gone back and fixed the books.
DR. SUTTER: A lot of people must read these constraints, and a lot of people could read it as saying, hey, that's signed off-, don't worry about it any more. Who reads this? It's in the books.
MR. WEAR: Let me explain how this occurs. There is what is called a Problem Review Board Meeting.
that is held within each project. There's one for each of the projects at Marshall as a result of Bob Lindstrom's letter that was read earlier that set this out, where they go over these items.
The reason you see Jim Thomas there, he's the alternate Chairman of that Board for me. That is discussed at that time and that is where these are recognized as being within our data base or perhaps not within our data base. And it is covered in the Flight Readiness Review and that is the process.
DR. SUTTER: So at the Flight Readiness Review, the people read this and understand it.
MR. WEAR: He or I in doing this activity know it's in the Flight Readiness Review or it's not, as the case may be. Plus the other participants there know that. And therefore it is listed as a result of being presented and discussed in the Flight Readiness Review.
And it would be within our data base or not.
CHAIRMAN ROGERS: What do you mean by data base, because Thiokol keeps saying that experience was not within your data base on Flight 51-L, and you all say that it was within your data base.
Which is right?
MR. WEAR: The Thiokol that I addressed, sir, says it is within their data base.
DR. KEEL: I think the Chairman's point, though, is with respect to the night of the telecon, Thiokol was arguing just what you are arguing now with respect to erosion. They were arguing that we want to stay within our data base and we want to go with an O-ring temperature not any lower than 53 degrees, just like you're arguing now that it's okay to fly because we are within our data base, implying you wouldn't fly outside of your data base.
CHAIRMAN ROGERS: What do you mean by data base?
MR. WEAR: The data base to me, sir, is the previous test and flight experience that we have, as supported.
CHAIRMAN ROGERS: You've never had experience in this cold weather so when you're talking about your data base on Flight 51-L you didn't have any. It seems to me it is used as sort of a slick way of just getting over the problem, saying it's within our data base. But you hadn't had any experience of that kind before and so you can't say it was within your data base.
The engineers at Thiokol were saying it is not. We have never had that experience. We warn you: Don't do it. We don't know what's going to happen.
And we keep hearing from NASA, it was within
our data base.
MR. WEAR: Well, going to that particular evening, which I think what you are referring to is the January 27th evening, on that particular evening in my process, in my mind process, we had faced a cold launch the year before, in which we had had some erosion.
CHAIRMAN ROGERS: The worst experience you had.
MR. WEAR: Which they had addressed and they, the Thiokol that you're referring to, they had said that that condition, that cold condition on that particular occasion the year before had-I forget the exact words, but in effect had aggravated the situation, but that it was acceptable and would perform. That was the conclusion.
CHAIRMAN ROGERS: But you agree, though, it was not within your data base, don't you?
MR. WEAR: That experience of the prior year was within my data base; yes, sir.
CHAIRMAN ROGERS: Could I say on that one, though, you had your worst result on that one.
MR. WEAR: Yes.
CHAIRMAN ROGERS: So you can say well, we almost had an accident, but we didn't quite. Therefore, it is within our data base. And then you get to another day when it is colder and you still argue it is within
your data base.
Thiokol said it is not in your data base because you had never tried it in this cold weather.
Now, what I'm asking is how do you explain that controversy? How do you explain that conflict?
MR. WEAR: If I may continue, their engineering organization and my engineering organization had agreed the prior year that that experience was-could be acceptable on the next launch; that if the same condition occurred, that that was the findings that were made at that time, if that same condition existed it could be accepted.
And that is the same engineering organization that was talking on January 27th, so their conclusions and report to us on the previous year was that at those conditions and what we [1519] observed on that particular launch, that if that condition-meaning that type of-that entire condition, that weather condition, whatever-occurred again, that that was acceptable.
MR. HOTZ: But you didn't have those conditions.
CHAIRMAN ROGERS: Let me finish. That was within your data base, you argue. But what I'm saying, you've got a new condition now which was not within your data base.
Now, how do you keep relying on it was within your data base? The Thiokol people said it's not. The engineers said it's not. We've never tried it at this cold a temperature.
How can you keep saying that it was within your data base? It wasn't. It exceeded your data base. You never had that kind of experience before.
MR. WEAR: Well, on that particular evening I heard what the Thiokol engineering people stated. I also know that they then counciled and they came back and they stated that the conditions could be accepted. And so I have to conclude some engineering people must have changed their minds.
CHAIRMAN ROGERS: I'm not taking issue with that. I'm taking issue with the slickness of the words "within our data base," as if that excuses everything.
What I'm pointing out is I don't think you ever had a data base of this kind in Flight 51-L.
MR. WEAR: That's true.
CHAIRMAN ROGERS: Well, that's really all I was pointing out.
DR. FEYNMAN: Can I ask something, too? I would like to understand this idea of within the data base. For instance, 51-C had many seals on which there was no erosion at all. Which seals should we take?
Why is it-suppose we had an accident? You see, we have an accident on five of the six seals. And maybe it wasn't five, but anyway, most of them-there's no erosion, statistically.
Accidentally, it could happen that there was no erosion on any of the seals, only a little bit, and then the next flight a lot of erosion. In other words, I don't understand how the logic works, that because something just made it, that the next time it wouldn't be a bigger variation.
Could you explain to me, therefore, why when you have a successful flight which is successful in the sense that the entire flight takes place, but which is unsuccessful in the sense that you get effects that you didn't expect, that you consider that the next time it isn't going to be accidentally a little bigger?
MR. MULLOY: Yes, sir. That is what we addressed with trying to determine when we would see a 10/1000 erosion or 20/1000 erosion or a 30/1000 erosion. That is why the test setup was made, to determine how much erosion can you physically get.
And that is where the 90 mil calculation came from, because that erosion is limited by the amount of time that hot gas can impinge on the primary O-ring, on the assumption that the primary O-ring seals.
So the logic was that there was a large margin against what were actually observing versus what the theoretically possible erosion was, and then there was a large margin between the theoretically possible erosion and that which you could sustain as demonstrated by tests.
Now, that was the logic.
DR. FEYNMAN: How many tests were made that would show that you really could sustain 90 mils of erosion?
[1520] MR. MULLOY: I can't recall. There were several hot gas tests made, maybe ten, with 125/1000 erosion on the primary O-ring seal.
CHAIRMAN ROGERS: How did the results of those tests compare with the results of the tests that have been done lately?
MR. MULLOY: They are correlatable in terms of the erosion. There are no differences, I don't believe. I haven't seen all of the test data.
DR. FEYNMAN: This isn't quite right, sir. You said there were tests with 125 mils erosion.
MR. MULLOY: Yes, sir.
DR. FEYNMAN: That isn't quite the way I remember it. I think the tests were made by cutting the ring away.
MR. MULLOY: There were two tests run. One of them was a hot gas impingement that sustained up -to 125 and still sealed.
DR. FEYNMAN: How did you make the test with 125?
MR. MULLOY: By allowing the hot gas to impinge for a longer period of time and having it impinge longer. And then there were cold gas tests made where the O-ring was-simulated erosion was put in there, and the maximum demonstrated there I think was 95/1000 as I recall.
But that was the logic. And I guess anyone can question that logic.
DR. SUTTER: These were all the five-inch?
MR. MULLOY: Yes, sir; with a full-scale gland and O-ring, but a short length. Well, a 10-inch diameter, I guess.
MR. HOTZ: Mr. Mulloy, I would like to try to understand this in somewhat simpler terms than you people are used to using.
Is it correct to state that when you originally designed this joint and looked at it, that you did not anticipate erosion of any of the O-rings during flight?
MR. MULLOY: That is my understanding. I entered this program in November of 1982 and I wasn't there on the original design of the joint, but when I took over the program there was no O-ring erosion anticipated.
MR. HOTZ: So that when you did run into signs of O-ring erosion, this was a bad sign.
MR. MULLOY: Yes, sir.
MR. HOTZ: You didn't like it?
MR. MULLOY: No, sir.
MR. HOTZ: So then you decided to introduce a standard based on the measurement or the possibility of the limits of O-ring erosion. And as those limits, as the experience went up, your criteria for safe flight went up too.
In other words, when you experienced more than maximum anticipated O-ring erosion, you waived the flight and said well, it's possible to tolerate that. We still have a margin left.
MR. MULLOY: Are you speaking of the case where we did not have a primary seal?
MR. HOTZ: Yes.
MR. MULLOY: Yes, sir. That is correct. There was another piece of logic that went into that though; that that flight, the next flight, we were
positive we did have a good primary seal because we increased the leak check pressure to 200 psi.
[1521] MR. HOTZ: But getting back to that, you mentioned that the ability of the putty to sustain pressure was pretty unpredictable.
MR. MULLOY: Yes. sir.
MR. HOTZ: And wasn't that a cause for concern that you had an unpredictable element in your equation?
MR. MULLOY: Yes, sir. And what was done there was to look for some alternative to the putty, such as MBR rubber strips or carbon fiber or carbon wool or steel wool or something that would allow pressure to go directly to the primary seal.
MR. HOTZ: But to continue flying in the meantime?
MR. MULLOY: Yes, sir.
MR. HOTZ: Then you finally, you're talking about these margins of safety, and I wonder if you could express in either percentages or actual measurement terms-you have used the term «wide margin". I wonder if you could give us a quantitative measurement as to what you consider a wide margin?
MR. MULLOY: Yes, sir. Well, as I said we had demonstrated that we could stand 125 thousandths of erosion and still seat. The maximum erosion that we had seen in the case joint was on SDS-2, which was 53 thousandths, so that is a factor of two and a half.
MR. HOTZ: But it is still based on a very narrow physical measurement between 125 thousandths.
DR. KEEL: Could I clarify something here, Bob? I think, Larry, if you go back and look at your Flight Readiness Reviews, that you were relying on smaller margins than that.
You were arguing in the Flight Readiness Reviews where you briefed the problems of primary O-ring erosion that for the worst case for the field joint also that it would be 90 thousandths.
MR. MULLOY: That is correct.
DR. KEEL: At that point you were pointing out that's okay, because you can seal at 95, not at 125 but at 95. It wasn't until later on during the process that
you determined you could seal at 125.
MR. MULLOY: That is when we got the hot gas test data.
DR. KEEL: So that's a five percent margin, roughly, five and a half.
MR. MULLOY: On the 90 to 95 on a max predictable, yes.
MR. HOTZ: Just one more question, sir. With all of this experience base, wasn't there any time in this history of the flight that you or anybody else connected with the solid rocket booster said we are getting a lot of anomalies here. We are getting thin that are outside our original predictions, and shouldn't we take a look at it and stop flying until we've fixed it or have a better feel for what is actually happening in the joint?
MR. MULLOY: Only the first part of that, that we are seeing something here that needs to be corrected. We continually emphasized to the contractor that we need to put more emphasis on resolving this problem. We did not recommend that we stop flying for the logic that was presented in the Flight Readiness Reviews and which I reiterated here today.
MR. HOTZ: Thank you.
CHAIRMAN ROGERS: Was consideration
given, as far as you know, to stop flying and fix the joint? Was there any discussions about that? Maybe we should stop flying for a while?
MR. MULLOY: Not to me, no sir. I have become aware of that in the course of some of the testimony since the 51-L accident that there are some memos that are internal memos to that effect.
MR. SUTTER: I still am confused with the answers I got to the question about the statements as the problem was closed. The question that's going on right now is how deep was the concern over the joint.
The statement here that says this problem is closed is against the whole NASA philosophy of really documenting, controlling and having checks and balances. When I read this statement that it was a mistake but it's left on the book, that gives me the impression that NASA did not believe this was a very big problem. [Ref. 5/2-3]
DR. RIDE: How serious do you consider a launch constraint?
MR. SUTTER: Do you have any comments on that?
MR. MULLOY: The comment I have, it's very unfortunate that that was erroneously entered. I had no intention of closing that problem, because I considered this to be a very serious problem.
MR. SUTTER: But did you know this?
MR. MULLOY: No, sir.
VICE CHAIRMAN ARMSTRONG: Am I right that you didn't close it because you didn't want to change anything after the accident? Isn't that right?
MR. SUTTER: But when was this signed off? This was signed off on 12/18/85.
MR. WEAR: This entry was made by this PAS system organization based upon the entry that they got from the contractor where he recommended its closure and they did it.
MR. SUTTER: I'm really confused, because the contractor was the guy that said don't launch, yet he wrote you a piece of paper saying close this item out? If you're going to depend on the paperwork as to controlling the operation-and I've heard this now for three months, all of these documents that, my God, we really controlled and we followed this to the point where we can't get into trouble, yet here's a piece of paper that says there's no problem.
How the-nobody reads this, I guess.
MR. WEAR: No, I think it is read.
MR. SUTTER: Do we have a copy of the letter?
MR. WEAR: Yes, sir, it's in the book here.
DR. KEEL: Mr. Chairman, we're going to ask the contractors exactly that question.
CHAIRMAN ROGERS: But I think we have to follow up on Joe's point. You mean just because somebody wrote a letter, the whole thing was closed out and you didn't know? Nobody knew anything about it? After all of this history of it, one letter would close it out?
MR. MULLOY: That was a failure of the human being within the system.
CHAIRMAN ROGERS: It was a little more than that. It's a failure of the whole system if one letter and one human being can close out a constraint that has been concerning you for many years.
DR. KEEL: Can I ask for one clarification before you go away from this, Mr. Wear? You're listed here as the project representative under the approval line?
[1523] MR. WEAR: That's right.
DR. KEEL: Does that mean you did approve this, or you didn't approve it?
MR. WEAR: No, I did not approve this. It was not brought forward for us to consider.
DR. KEEL: So even though your name is on here as approval, you didn't approve it?
MR. WEAR: That's right. I think if you will notice, there is not an entry here that says so. On the others, I think you will notice over here that there is
an entry of initial, either JWT or LOW on the others as having lifted that for that particular flight. I think most of them them put an L after it or something that refers to it's lifted. In their nomenclature there's an L entered for that purpose.
DR. KEEL: So as far as you're concerned you were still operating as if this was-remained a launch constraint?
MR. MULLOY: Yes, sir.
MR. WEAR: Yes, sir.
DR. KEEL: But may I follow up? That is correct, but as a matter of practicality even though this was a launch constraint it was being waived for each launch?
MR. MULLOY: That is correct, on the basis of the presentation at the Flight Readiness Review.
DR. KEEL: So you in effect waived this for 51-L?
MR. MULLOY: That is correct.
DR. KEEL: But it doesn't show up on your summary?
MR. MULLOY: No because the man assumed when the closure came in from Thiokol that this was going to close the problem, and that requires project concurrence, Mr. Wear's concurrence, which Mr. Wear and
I had discussed it and that was not going to happen.
DR. KEEL: As far as you were concerned, though, you still considered it a constraint in spite of this document?
MR. MULLOY: Yes, sir.
DR. KEEL: So the night of the telecon, for example, when you were arguing about whether the primary O-ring would seal, you still considered the fact that there was a launch constraint on the primary O-ring for 51-L?
MR. MULLOY: Yes, sir.
DR. KEEL: Hence, in effect, the fact that in a strict sense there was no launch commit criteria with respect to temperature, there certainly was a launch constraint with respect to the primary O-ring sealine.
MR. MULLOY: Launch constraint relative to an understanding of our previous history to go into the next flight.
DR. KEEL: Well, it's primary O-ring, isn't that what the launch constraint is on?
MR. MULLOY: Yes, it is related to the primary O-ring on the field joint.
DR. KEEL: And the reason you could have so much erosion is because it didn't seal on 51-B?
MR. MULLOY: On the nozzle joint, that is correct.
DR. KEEL: That's right, and they were arguing on the field joint. They had concern about it sealing or having time to seal.
[1524] MR. MULLOY: Yes, that was the presentation. The concern for the time that it would take for the primary O-ring to seal.
DR. KEEL: So in that sense, all that argument was in the context of there being a launch constraint.
MR. MULLOY: I didn't think of it in those terms at that time.
MR. ACHESON: But I take it it would have made no difference whether it was formally closed out or not because, as I see the way the system had worked, a flag raised on 51-L-I would have thought the determination of the problem was contained by the fact that the leak check had been changed to 200 psi from 100, and you could go on as before with that change in the procedure.
Is that not the way it would have been resolved?
MR. MULLOY: Yes, it was contained by the fact that with the 200 psi leak check we were sure that we had a good primary O-ring.
MR. ACHESON: So it really didn't make any difference whether it was formally closed or not as to what would have happened.
MR. MULLOY: I would like to go back to the point Mr. Wear made, because I think it is important in addressing your concern, Mr. Sutter, and it's a very important point that Mr. Wear made, is that you do not see the accepted- closure on that last entry. That is an entry that the guy who makes entries into the PAS system made, and you do not see project signature concurring in that.
So in essence, it is still open. It is open until Larry Wear concurs that it's closed.
MR. SUTTER: Do the people who read these look for all the signatures?
MR. MULLOY: No, sir. I agree, it is unfortunate that that error was made by the gentleman who make the entries into the PAS system on the basis of a submittal from the contractor that this was no longer a problem of significance to carry in the problem assessment system.
MR. KEHRLI: Mr. Chairman, there are some documents, four documents, that we haven't looked at that address these specific questions that have been asked for the last three or four minutes that I would like to address the witness' attention to and the Commission's on the chronology of this closure.
The first one is the launch constraint
document number 4, which is a letter dated December 10, 1985 to Mr. Wear from Mr. McDonald of Thiokol, which references the particular problem that we've been discussing and suggests that or requests that the subject critical problem be closed. [Ref. 5/2-4]
Additionally, the next document that follows that page is the letter from Thiokol to Mr. Jack Fletcher of Rockwell, and it indicates that there is further information that's being provided with regard to the closure of critical problems. [Ref. 6/2-7]
Back on O-ring history document number 34, there is a letter from Mr. Wear to Morton Thiokol; subject, SRM Problem Review Board. The date of that letter is December 24, 1985.
It reads: "During a recent review of the SRM Problem Review Board open problem list, I found that we had 20 open problems, 11 opened during the past six months, 13 opened over six months; one three years old, two two years old, and one closed during the past six months. As you can see, our closure record is very poor. You are requested to initiate the required effort to assure more timely closures, and the MTI personnel shall directly coordinate with the S&E personnel the contents of the closure reports." The letter is signed by Mr. Wear. [Ref. 5/2-8]
The final document is attachment number 30 to
the O-ring history. This is, again, part of the Marshall package that was received when the launch constraint document that we've been talking about, the problem assessment documents were received. [Ref. 5/2-9]
MR. WEAR: What's the number, please?
MR. KEHRLI: Number 30 in the O-ring history. It gives a chronology. As you see, it notes on the entry of 51-B there is an asterisk next to the Marshall tracking number A09288, which indicates that "this problem contains secondary O-ring erosion of the nozzle joint and constrains launch anomalies."
Then down at the bottom, on STS-61-C, by the date, 1/12/86, there is an indication of erosion or an O-ring anomaly problem on the field and nozzle-to-case which was not reported and not given a problem tracking number in January of 1986.
CHAIRMAN ROGERS: What do you conclude by that?
MR. KEHRLI: Well, one, I am wondering if the closure of the item is the reason that the January 12, 1986 erosion or O-ring anomaly was not reported or given a tracking number.
MR. WEAR: I'm afraid I didn't follow your question there.
MR. KEHRLI: Well, on document number 30, the field and nozzle joint erosion problem, on STS-61-C on the
primary ring, unlike the other problems-except for STS-2-it has an indication that it was not reported.
Is that because the problem had been closed as a result of those previous letters we just referred to, the Thiokol request to close it and your letter indicating that you want open items closed?
MR. WEAR: I'm afraid I can't tell you why this one was not reported, looking at this right here, now.
As far as my letter is concerned, the thrust of my letter was we had problems. They are within this tracking system. It's more than the normal deviation, the deviation records. What the thrust of my letter is is that they are not making adequate progress to satisfy me in resolving problems. We were continuing to have the problem but we weren't making sufficient progress to suit me in resolving them, that they were hanging on for too long and we weren't closing them out.
MR. MULLOY: Let me take a try at answering this question as to why it was not reported. I see this was revised 3/19/86. This would be speculation, but I didn't get the information on the inspection of 61-C until after the Flight Readiness Review for 51-L. That came in-I believe we got those joints demated just two or three days before the L-1, and I reported this in the L-1 review.
Given that timing and given the events of 51-L, I just-and perhaps you could address this question to Thiokol, but I suspect they haven't written up this four mils of erosion that occurred on 61-C and put it into the system since the 51-L accident.
There is no reason why it wouldn't be reported because there was four mils of erosion.
DR. KEEL: Can I ask one followup question with respect to these memos and the documents that Randy has referred to here? Based on our staff interviews with Mr. Thomas from Marshall, he indicated that he had told Thiokol on the telecon to in fact close out all issues that were over six months old, and Mr. Russell at Thiokol confirmed that.
[1526] Was that your understanding, also, that engineering at Marshall was saying close out the problems over six months old?
MR. WEAR: I'm afraid I don't know the context of that discussion. He and I had had conversations.
DR. KEEL: Well, it's the same context here, I assume.
MR. WEAR: Well when you say close them out, I say I'm not sure what the context was of the statement he made. My intent-and I trust it is the same with Jim's because he and I usually communicate well-my
intent was let's get off our duff and work and reach solutions for these problems, not continue to drag them out for six months, five months, et cetera. That is the thrust of my direction to Thiokol.
DR. KEEL: Did ultimately Thiokol recommend, in fact, closing out by taking the joint, SRB joint problem off the problem report list?
MR. WEAR: That's the nature of McDonald's letter back here where he does recommend taking them off, and as I understand the thrust of the letter the way I read it now and the way I read it then was that we are discussing these problems in the Flight Readiness Review. Let's not also put them someplace else, so that in effect he has to report them two places.
As I read his letter, he's saying we are doing the tracking job in the Flight Readiness cycle. Let's take them out of this other tracking system. He's saying let's do it once. I think that's the thrust of his letter, the way I read it.
DR. KEEL: What was Marshall's response to that?
MR. WEAR: My response to that was no.
MR. MULLOY: Mine, too.
MR. WEAR: We will keep it in the system because it a formal check and balance. That is what its
intention was, to be sure if something didn't slip through a crack someplace. That's the way I've always looked at it, was to create a double check by the quality organization to see to it that something wasn't overlooked by the project. Perhaps the project might not even be aware that it could be overlooked, that it had to be faced.
DR. FEYNMAN: Is this answer of yours to McDonald in some document?
MR. WEAR: Pardon?
DR. FEYNMAN: How did you answer Mr. McDonald, in a document?
MR. WEAR: I never have.
DR. FEYNMAN: The answer you've just given us as to how you answered Mr. McDonald, in what document is that where you say you won't close it out?
MR. WEAR: I say I never have answered his memorandum; therefore, the system stands, the Marshall system stands.
DR. FEYNMAN: Except for the mistake.
MR. WEAR: Yes, sir. It makes mistakes, but it stands. Frankly, it wouldn't be within my power to accept his recommendation and take this out anyway, because the establishment of the system is well above me, and so it is not my prerogative.
DR. FEYNMAN: Nobody answered this letter of Mr. McDonald's?
MR. WEAR: No. 1526
[1527] DR. RIDE: How many of the problems in your tracking system carry launch constraints? How many launch constraints have you been waiving flight to flight?
MR. WEAR: I haven't researched it specifically, but I think you would find every one of them having to do with SRM or related to the joint putty, the joint and/or the putty type of affairs that we've discussed here. I don't recall anything else that would be there as a launch constraint.
DR. RIDE: Say on flight 61-C, how many launch constraints did you waive?
MR. WEAR: Offhand I can't tell you.
DR. RIDE: I mean, it must be, and I mean I hope the answer is one. If the answer is more than one, then there is more that we don't know.
MR. WEAR: Well, you've got the record here, and I would just stand on that. I mean, you've got the records from the past system.
CHAIRMAN ROGERS: Well, let's talk about it while you're here,
DR. RIDE: What we've got, I think, is the record of the launch constraints as a result of the
erosion problems on the joint.
MR. KEHRLI: That's correct.
DR. RIDE: I guess my question is are there other problems in the SRM or SRB system that also carry launch constraints that you have been waiving at your FRRs.
Maybe I'm wrong, but I consider a launch constraint to be a big deal that you have to address in a significant, substantial way before every launch, and I would think that if there is more than one you would know that there is more than one.
MR. WEAR: Well, you asked me how many, and that infers that I can pull off the top of my head exactly how many there are, and I can't right here, but we are required
CHAIRMAN ROGERS: Would you be willing to say it's very few?
MR. WEAR: Yes, sir. Absolutely.
CHAIRMAN ROGERS: Would you be willing to say they all relate to the SRB? All the waivers would relate to the solid rocket booster?
MR. WEAR: Do you mean that are going through the whole FRR process, is that what you're saying?
CHAIRMAN ROGERS: Yes.
MR. WEAR: No, because I'm only exposed to the
SRM, per se, so I couldn't answer that.
DR. RIDE: Was there more than this one?
MR. WEAR: In each Flight Readiness Review we addressed problems that occurred on a past flight and/or significant waivers or deviations that occur during the manufacturing process, the manufacturing acceptance process of the hardware. Those are categorized as being within previous data experience or not.
DR. RIDE: I'm trying to understand how you deal with the launch constraint. How important do you think a launch constraint is and how unusual it is in your system?
MR. WEAR: I think a launch constraint is a significant event in our system, and it is one that has to be addressed within the Flight Readiness cycle because I don't have the authority to not do that.
CHAIRMAN ROGERS: The question is how many have you waived?
MR. WEAR: Well, of course each of these joint and putty entries that you've seen, those are there.
[1528] CHAIRMAN ROGERS: Can you think of any others?
MR. WEAR: Yes, sir. I believe I've got one. I think if you check the record I believe it would be categorized as one. On a nozzle, where we had de-bond on a nozzle, I believe it is somewhere in the system and
we had to process it.
CHAIRMAN ROGERS: Is that the only other one you can think of?
MR. WEAR: Well, I know in the history of the program we have addressed several.
CHAIRMAN ROGERS: We're not talking about addressing. We're talking about waivers.
MR. WEAR: There has been a lot of discussion in the Flight Readiness Review process regarding the nozzle itself. I think some of you are aware that we struggled with a severe problem on the nozzle erosion.
CHAIRMAN ROGERS: I'm talking about how many have you waived?
MR. WEAR: Specifically I can't answer that.
CHAIRMAN ROGERS: So probably most of them were in connection with this joint.
MR. WEAR: I think the preponderance of them have to do with this joint and/or the nozzle-to-case joint, and I think it is the preponderance of everything that has been waived is there.
The reason I'm struggling here, I think if I went back and went through the whole record there would be some related to the nozzle because we did have severe nozzle erosion problems about two and a half years ago. We struggled with that problem for a while, and I suspect I would find some there.
VICE CHAIRMAN ARMSTRONG: Is it an easier question if you ask how many waivers were there on 51-L, or is that still a hard question?
MR. WEAR: I just don't recall it, and I don't recall that there were any on 51-L that we haven't discussed.
DR. FEYNMAN: In your letter of the 24th you said there were 20 open problems. These open problems are not all launch constraints?
MR. WEAR: That's right, they are not all launch constraints because the PAS system is concerned with problems other than Crit 1, Crit 1R problems. It is an attempt to keep the record clean for lesser problems as well.
DR. RIDE: That's exactly the point, because you've got the system that records open problems, and you have to have some way of distinguishing unimportant problems from important problems from very important problems, and it seems to me the one that says launch constraint next to it must be the very most important problem.
MR. WEAR: That's right, and that is why it has to be cleared by this PAS system before we can proceed.
DR. RIDE: What I'm trying to understand is
how many problems are in the launch constraint category in your system?
MR. WEAR: I can't give you a precise answer.
CHAIRMAN ROGERS: A followup question which really does surprise me is that I would think that if you have waivers and there weren't many, you would remember them all.
MR. WEAR: When you asked me how many, I would have to go back over history, and historically the only other problem of consequence that I recall we have wrestled with on the SRM has either been related to nozzle erosion and/or these joints.
[1529] DR. WALKER: Have you ever refused to waive a launch constraint because you thought the problem was so serious?
MR. WEAR: No.
DR. WALKER: What would happen if you didn't waive a launch constraint? Let's say you or Mr. Mulloy had looked at this O-ring situation and said, well, I don't think we can waive this because it is too serious. What would happen then?
MR. WEAR: Well, I think immediately there have been some times when we have looked at a problem and either I or someone else associated with my review have not been satisfied with the data that was presented.
On those specific occasions there was some additional work that was done, some additional data was provided, whatever, and then either I or that person was reconciled. But if I had said I refuse to accept this, I refuse to go forward, I refuse to accept this problem and fly with it, that would have gone to Larry and to Larry's board, and it would have had to have been reviewed with them.
DR. WALKER: Let's suppose it now goes to you, Larry, and you're not going to waive it.
MR. MULLOY: The effect of not waiving it the first time, it is presented as a launch delay, and we have had several of them. One of those was after STS-8 nozzle erosion. There were some people who felt that we could proceed with the flight of the next flight, which was STS-9, with the nozzles that we had on there. I did not accept that.
The consequence was we rolled STS-9 back and demated the aft segment and put in an aft segment with a nozzle that had, after understanding, spending some time to understand what the cause of that excess erosion was on STS-8, we changed out a segment.
So the effect of not waiving a constraint when it has first been presented has been launch delay. Up to this point, we have taken the time necessary to do
the analysis and tests to provide sufficient rationale to proceed with the next flight in the face of the open problem.
MR. HOTZ: Then you have refused to do it? On STS-8, you did it?
MR. MULLOY: On STS-9. On STS-9, after the problem was open on the nozzle, until we understood what the probable cause was, and as I say, the effect was a rollback from the pad and about a two-month delay in the launch of that vehicle.
DR. WALKER: So the nozzle was changed out?
MR. MULLOY: Yes, and that is what we would do in any case when there is insufficient rationale to proceed on the schedule that is proposed.
MR. HOTZ: Well, I am confused, Mr. Wear. You said you have never refused to waive a launch constraint, and I understand now that there was one instance when you did. Which is correct?
MR. MULLOY: I said both, sir?
MR. HOTZ: No, Mr. Wear said you never refused.
MR. WEAR: My point is, either the problem was resolved before it went forward, or else we wouldn't have launched, but it would be resolved before we would go forward. That is a case where we hit a problem, and I don't know exactly.
MR. HOTZ: But you did not waive it to go on with STS-9?
MR. MULLOY: As it was configured when it was on the pad. That is correct. And in fact I think that was a two-month launch delay, and that is what we would do in any instance when there has been insufficient time from the time that the problem was observed to do the test and analysis to understand whether you can proceed with the next launch, and as I say, the effect to date has been launch delays, but then ultimately that sufficient data is available to proceed, and then the waiver is before you can fly it. The waiver in the case for STS-9 was that the cause of the excessive erosion on STS-8 had been determined, and that condition did not exist on the nozzle in STS-9, and the rationale as to why the condition did not exist after we changed out the nozzle, and the material change.
CHAIRMAN ROGERS: Do you have ultimate responsibility for waiving the launch constraints?
MR. MULLOY: Yes, sir, I have ultimate responsibility for the launch readiness of the solid rocket boosters.
CHAIRMAN ROGERS: So there was a launch constraint, and you waived it.
MR. MULLOY: Yes, sir, all flights subsequent to.
CHAIRMAN ROGERS: All those you waived on your
own? Who did you consult with in connection with those waivers?
MR. MULLOY: Well, in terms of consulting on the waiver, the waiver is approved by my element managers, and the system that comes to me, the quality people who run the problem assessment system, when they come to my board, will say, you have an open launch constraint that has not been properly closed out. The rationale for closing out that launch constraint is presented by Mr. Wear to me in my flight readiness review, which is that rationale that exists throughout this document that says why it is okay, given this observation to proceed with the flight.
CHAIRMAN ROGERS: Well, I have trouble following this. Could you do it a little slower? There is a launch constraint put on by somebody, some decision.
MR. MULLOY: By me in this case.
CHAIRMAN ROGERS: By you. Okay. Now, who has the authority to waive it?
MR. MULLOY: I do.
CHAIRMAN ROGERS: Okay, you put it on and you take it off.
MR. MULLOY: Yes, sir.
CHAIRMAN ROGERS: Who do you consult with when
you take it off?
MR. MULLOY: Mr. Wear, in this case, who brings me the rationale as to why we can proceed for the next flight in light of the observation that we have seen.
CHAIRMAN ROGERS: So in all of these flights, Mr. Wear, you were involved in waiving it, too?
MR. WEAR: Sure.
CHAIRMAN ROGERS: Now, did you know anything about the telecon before Flight 51L? Were you involved in that?
MR. WEAR: The one on January 27th?
CHAIRMAN ROGERS: Yes.
MR. WEAR: I was there on January 27th. Yes, sir.
[1531] CHAIRMAN ROGERS: And did you point out to everybody there was a launch constraint, and therefore it was a little different than the usual launch? When the Thiokol people said we ought to, because of the weather, we ought to delay this, did that cause you any concern?
MR. WEAR: Did the existence of a launch constraint -
CHAIRMAN ROGERS: The combination of launch constraint and a warning by the manufacturer not to do
it, did that cause you any concern?
MR. WEAR: Let me make sure I answer that in the right context. The joint problem and its understanding had been a matter of concern for some time, and there was no one on that conversation that night to my knowledge-well, there was no one in that conversation that night that was not well aware of the problem and its seriousness.
CHAIRMAN ROGERS: Put aside the word "problem." Was everybody aware of the fact that there was a launch constraint?
MR. WEAR: There was a launch constraint to address the erosion. The problems on prior flights, yes.
CHAIRMAN ROGERS: Now, following along Dr. Ride's comment, we assume or at least I assume that that means there was something a little different or special, more serious, the fact that you had a launch constraint, this was a more serious problem than just an ordinary problem. She pointed out there must be gradations of problems, and this, when you have a launch constraint and it is in a Criticality 1 item, that is pretty serious.
Now, when the manufacturer then said we recommend don't launch to begin with did that cause you
any concern particularly in view of the fact that there was a launch constraint on it?
MR. WEAR: Yes. When the Thiokol engineering people expressed their concern, yes. That caused me some concern, yes.
CHAIRMAN ROGERS: And they suggested a slight delay until the weather was better.
MR. WEAR: That is what the Thiokol engineering people stated that night.