CHAIRMAN ROGERS: I call the Commission to order.
Mr. Mulloy, Mr. Wear, we have been considering the history of the joint that failed, and in connection with that we have been considering the history of it going back several years. The purpose for the meeting this morning is to discuss some of the documents that relate to the history of it and give all of you who were involved in its development and the experience that you have had with it the opportunity to refer to some of the documents that have been provided to us.
[1506] This will be on the record. I assume-has Mr. Wear been sworn?
MR. WEAR: I have not.
CHAIRMAN ROGERS: Well, let me finish. Then we will do that. We decided to have this in executive session because we want it to be informal, and secondly, we want to deal with several documents. In an exploratory kind of discussion of this kind, it isn't really suitable for public session, but we will make the testimony public at some time later on, because we may use some of it in our final report, and we thought that we would have one of our staff members refer to the documents, Mr. Kehrli, so that you can identify the
documents, so that we are all talking about the same documents, and then he will ask some questions about those.
It is our intention to end the hearings of the Commission today. We do not plan to have any further hearings, so this will be the last hearing. If we might swear Mr. Wear.
(Witnesses sworn.)
CHAIRMAN ROGERS:
Randy, do you want to start, please?
MR. KEHRLI: Thank you, Mr. Chairman. Gentlemen, in front of you you have a book that contains the documents that we would like you to refer to today. I am going to refer to them specifically and read portions of them into the record, and you might want to follow along while I do that, before the Commissioners begin asking you questions.
That booklet is divided into two sections. The first one is a major tab, O-ring History, and the attachments run, the tab attachments run from Number 1 through Number 35. The second major tab, the second division is Launch Constraint, in the back, and those attachments run from 1 through 4.
I would like to direct your attention first of all to the Launch Constraint Attachments 1 through 4.
During the course of the investigation, either as a result of interviews or specific requests by the Commission to NASA, we received some of these documents, as the Chairman indicated to you.
The first document I would like to refer to is Launch Constraint Attachment Number 1, which is a document dated September 15, 1980, to distribution from Mr. Lindstrom, and the subject of this memorandum is signing launch constraints on open problems submitted to Marshall PAS. This is in the back of the book, in back of the Launch Constraint section, Attachment Number 1. [Ref. 5/2-1]
The memorandum reads, "The following guidelines have been established to aid in making constraint decisions on open problems and are limited to recurrence control determination only. In accordance with practices established on past programs, remedial actions, for example, removal and replacement of defective hardware, et cetera; for correcting discrepancies on the vehicle to be launched are considered launch constraints and are tracked by the KSC system."
This is an excerpt from the document. I haven't read it in its entirety. It is Paragraph A. Subparagraph 1 reads, "All open problems coded Criticality 1, 1R, 2, or 2R will be considered launch
constraints until resolved, which is recurrence control established and its implementation effectively determined, or sufficient rationale; in other words, different configuration, et cetera, exists [1507] to conclude that this problem will not occur on the flight vehicle during prelaunch, launch, or flight."
Now, I would like to direct your attention to Attachment Number 2 of the Launch Constraint section, and this attachment is a Marshall problem assessment system. This document was in fact obtained from Marshall. It is dated February 26th, 1986, and it is my understanding, and of course you are free to correct me on any of this, that this is a chronological history of entries from the Marshall problem assessment system.
The relevant items on this document that we would like to have you address include the fact that the document indicates that a launch constraint was assigned to STS-51F, 511, 51J, 61A, 61B, and 61C. The date, I believe that is R-e-c-which apparently means recorded or received-over on the right is July 10, 1985. That matches with the first entry on the document, which is dated July 11th, 1985, and it indicates that post-the entry on the 11th indicates that post-flight inspection, SRM-16A revealed a gas path through the vacuum putty at 54 degrees, and it goes on to describe essentially the
erosion problem in the nozzle joint, I believe, in 51-STS-51B, which was inspected on June 25th at Thiokol, in 1985.
There is an indication that the constraint closure recorded was December 18, 1985, in the document under the date recorded, and without belaboring the further entries on the document, it indicates that the problem was "not considered a constraint to 51F; 100 psi leak check is performed, which confirms seating of the secondary O-ring. The nozzle O-rings have been shown to survive erosion, gaps of 125 mils in hot subscale test," and it continues to discuss the math model, as you can read.
Additionally, there are individual entries later on in that document which indicate that the constraint was lifted for subsequent flights that were listed previously, 511, 51J, 61A, 61B, and 61C.
Finally, there was a handwritten notation on the last page of the document that says: "constraint can be lifted by project manager," and this notation was on the document when we received it. Additionally, the last entry on this problem assessment report dated 1/23/86 indicates that there was a resolution: "In the SRM, field and nozzle joints have experienced erosion of the primary O-rings during several missions and static
tests as determined by post-flight inspection," and it continues to go on through that entry and give the rationale for closure of the problem.
The last paragraph reads: "This problem is considered closed based on MTI report TWR14359, Revision A, improvements in Space Shuttle SRM motor seals dated 8/30/85, and MTI letter E150BGR86144, rationale for closure of the O-ring erosion problems."
The last important information on this document is the-at the bottom of the first page there is an indication of assignee and approval, and the names there are R. McIntosh, D. Newman, L. Wear, J. Fletcher, and there is an indication of the status of the document: "PAC review is complete." The status is still marked open, Code M, I believe.
Finally, or additionally, the next attachment, Attachment Number 3, is a similar problem assessment which was initiated on February 17th, 1984, and summarized in this document. This is a problem assessment begun after the erosion, the O-ring erosion problem on Flight 41B, and it again tracks that problem up until the time the problem was closed, which is the same date as the nozzle, the previous nozzle problem, 1/23/86. [Ref. 5/2-3]
[1508] Again, the same names are on the bottom. The difference, the key difference on this document is under
"launch constraints" it says "none," and there is no indication on the document that a launch constraint was attached to this particular problem assessment report on the field joints.
Again, I won't go through the various entries. You gentlemen are probably familiar with them, and the Commission has seen them. The closure entry is the same on this document as it was on the nozzle, the nozzle document described previously.
CHAIRMAN ROGERS: May I ask, how did this first document you referred to come to our attention?
MR. KEHRLI: We first discovered through an interview conducted by one of the investigators at Thiokol that there was a monthly problem report that Thiokol filed internally which tracked the various problems that arose during and after a particular flight, and tracked the problem until it was worked and/or closed.
We discovered in one of those entries a reference to a specific Marshall problem assessment report, and it indicated that there had been a tracking number, a problem number assigned to the 41B erosion problem, and also to 51B, and additionally DM7, I also believe, had a separate report. Then upon talking to George Hardy and, I believe, Mr. Mulloy,
Marshall searched their records and found these documents which you have in front of you, the Marshall assessment report. So these are in fact Marshall documents.
CHAIRMAN ROGERS: And that was after our public hearings, so that was how we found this document?
MR. KEHRLI: Yes, this was after the public hearings, and we found these in the course of interviews.
CHAIRMAN ROGERS: Just to be sure the record is complete, this J. Fletcher is a separate J. Fletcher from the Fletcher that has been nominated and affirmed, I guess, as administrator.
MR. KEHRLI: Yes.
CHAIRMAN ROGERS: Well, I think what the Commission would like to hear from you, Mr. Wear and Mr. Mulloy, is an explanation of this, first, why we didn't know about it, and secondly, what gave rise to the launch constraint, how they were handled, who made the decision to waive on all of these flights. I guess these were the flights that preceded 51L.
MR. MULLOY: Yes, sir.
CHAIRMAN ROGERS: And why it was closed out just before 51L.
MR. MULLOY: Okay, sir. I will start to address that, and then Mr. Wear, I think, can elaborate.
The problem assessment system was put in place to provide visibility throughout the shuttle system for the types of problems that do occur, not just in flight, but also in qualification tests, and in failure of hardware that is back for refurbishment at a vendor or whatever. And it is a closed loop tracking system that lists the anomaly.
[1509] Now, in accordance with the memo there from Bob Lindstrom in September of 1980, the procedure was established that in all flight readiness reviews, all open problems would be flagged at the flight readiness review, and it is our quality organization that does that, and that requires them when the problem is not completely closed out, it required dealing with it in the flight readiness review process, as I testified in both the previous private and public hearings, that these anomalies were covered in the flight readiness review.
There was a cross-check by the quality people to assure that the problem had been discussed and resolved, a rationale for making the next flight, in view of the fact that the problem had not been completely eliminated in making the flight with the acknowledged chance that there would be a recurrence of the type of thing that had been observed in the initial problem report, so on each flight up to, from the time that the constraint was shown, it required a signature by the project office, which is Mr. Wear's office for the solid rocket motor, that that had been addressed and closed in the flight readiness review, and it's required, his initial, essentially on the problem assessment, and where you see on the problem assessment there, I believe there is a JWT initial in there for each one of those. That JWT is Jim Thomas, who works in Mr. Wear's office, and is director of his engineering branch in the project office.
Now, the entry that is shown in there that the problem was closed prior to 51L is in error. What happened there was, one of your documents here which we did not discuss is the letter from Mr. McDonald to Mr. Wear which proposed that this problem be dropped from the problem assessment system and no longer be tracked for the reasons stated in Mr. McDonald's letter.
That letter was in the review cycle. The
letter, I believe, was dated 10 December 1985. It came into the center, it was in the review cycle. After Mr. Wear brought this letter to my attention, my reaction was, we are not going to drop this from the problem assessment system because the problem is not resolved and it has to be dealt with on a flight-by-flight basis.
Since that was going through the review cycle, the people who run this problem assessment system erroneously entered a closure for the problem on the basis of this submittal from Thiokol. Having done that then for the 51L review, this did not come up in the flight readiness review as an open launch constraint, so you won't find a project signature because the PAS system showed the problem was closed, and that was an error.
CHAIRMAN ROGERS: Who made the error? Do you know?
MR. MULLOY: The people who do the problem assessment system.
MR. WEAR: Mr. Fletcher, and he reports within our quality organization at the flight readiness review, at the incremental flight readiness reviews, as I think have been described to you before. There is one from Thiokol to me, and there is one from my group to Larry, and then Larry, of course, does one with the Shuttle
project office, and so forth, on up the line. At my review and at Larry's review, there is a heads up given to the quality representative at that board for what problems the system has open, and they cross-check to make sure that we address that problem in the readiness review.
On this particular occasion, there was no heads up given because their PAS system considered that action closed. That is unfortunate.
[1510] CHAIRMAN ROGERS: Let's go back just a bit, because I think it is helpful to me if you you use words that I understand a little bit. What caused the constraint to be put on in the first place?
MR. MULLOY: The constraint was put on after we saw the secondary O-ring erosion on the nozzle, I believe.
CHAIRMAN ROGERS: Who decided that?
MR. MULLOY: I decided that, that that would be addressed, until that problem was resolved, it would be considered a launch constraint, and addressed at flight readiness reviews to assure that we were staying within our test experience base.
CHAIRMAN ROGERS: And was that-what is meant by the problem description?
MR. MULLOY: Which document, sir?
MR. KEHRLI: That is Number 2.
MR. MULLOY: Yes, reference similar reports. Yes, sir.
DR. RIDE: Why didn't you put a launch constraint on the field joint at the same time?
MR. MULLOY: I think at that point, and I will react to that question in real time, because I haven't really thought about it, but I think the logic was that we had been observing the field joint, the field and nozzle joint primary O-ring erosion. This erosion of a secondary O-ring was a new and significant event, very new and significant event that we certainly did not understand. Everything up to that point had been that the primary O-ring, even though it had experienced some erosion, does seal. What we had evidence of was that here was a case where the primary O-ring was violated and the secondary O-ring was eroded, and that was considered to be a more serious observation than previously observed.
DR. RIDE: Correct me if I am wrong, but weren't you basing most of your decisions on the field joint on analysis of what was the maximum, what you believed to be the maximum possible erosion, and you had that analysis for the field joint and for the nozzle joint. When you saw the complete erosion of the primary
O-ring on the nozzle joint, that showed you that your analysis on the nozzle joint wasn't any good, I would think. That would indicate to you that your analysis on the field joint wasn't very good, either, or at least should be suspect.
MR. MULLOY: The conclusion, rightly or wrongly, for the cause of the secondary O-ring erosion on the nozzle joint, it was concluded from test data we had that 100 psi pressurization leak check, that the putty could mask a primary O-ring that was not sealing. The conclusion was-and that one was done at 100 psi. The conclusion was that in order to get that type of erosion that we saw on the primary O-ring, that that O-ring never sealed, and therefore the conclusion was that it never was capable of sealing.
The leak check on subsequent nozzles, all subsequent nozzles was run at 200 psi, which the test data indicated would always blow through the putty, and in always blowing through the putty we were guaranteed that we had a primary O-ring seal that was capable of sealing, and then we further did, and we already had that on the field joints at that time.
DR. RIDE: The 200 psi check on the field joints were started about a year earlier. Is that right?
MR. MULLOY: I don't recall specifically where. I don't recall the dates.
MR. WEAR: I believe it was-I think there was a separation of about three flows, I believe.
[1511] Mr. MULLOY: So what concerned us about the nozzle joint was that we had, as you have stated, Dr. Ride, we had to go back and then look at the analysis for a violated primary O-ring, so the rationale for the proceeding with flight, having looked at the occurrence on the nozzle joint, another analysis was run which included violation of the primary O-ring to determine what is the maximum erosion that could occur on the secondary O-ring.
That analysis matched very well with the observations that we had from the flight that that occurred on, and that was corroborated by test, and that was the rationale for proceeding with the next flight, and whether or not it was treated as launch constraint in the past, it was treated as an issue to be discussed in each one of the flight readiness reviews on the field joints as well as the nozzle joints.
DR. WALKER: Were you at all concerned about violations of the integrity of the putty, which was really the first line of defense in this joint? In fact, apparently you were deliberately trying to violate the putty by going to 200 pounds per square inch.
MR. MULLOY: Yes, sir. We were concerned with
violation of the putty, and as has been laid out during this investigation, there was a program under way to find something as an alternative to the putty. We were not deliberately trying to violate the putty. What we wanted to be sure was that we got a leak check on the primary O-ring, and the only way you could be sure you got a leak check on the primary O-ring was to put a pressure between the two rings that would, if the primary did leak, that it would blow through the putty.
DR. WALKER: Well, all right. Then if you found that-well, if you were going to violate the putty, there was no procedure you had which could determine that. You could violate the putty in your test, and yet you would never know that, and yet that was an integral part of your protection, and no actions were instituted to address that problem, as I understand it.
MR. MULLOY: Oh, yes, sir, that was the whole test program that was laid out, to find an alternative to the putty.
DR. KEEL: Mr. Mulloy, didn't it occur to you that if the putty could be masking a leak check, then the putty could sure be masking pressurization actuation of the primary O-ring, and hence your whole analysis about how that primary O-ring sealed and the time it
takes to seal could be just as suspect?
MR. MULLOY: No, sir, it did not. The pressurization is an entirely different direction.
DR. KEEL: Of course, the putty is still a barrier in both instances.
MR. MULLOY: Yes, sir. And the putty has a substantially different pressure carrying capability, depending upon which side you pressurize it from.
DR. KEEL: That was your assumption?
MR. MULLOY: That is a fact.
DR. KEEL: Do you still think the putty doesn't have anything to do with the pressurization of the primary O-ring?
MR. WEAR: You are speaking of today?
DR. KEEL: Yes.
MR, WEAR: Well, today there has been quite an extensive testing conducted.
DR. KEEL: What is your answer?
MR. WEAR: Well, I think that the testing that has been done has demonstrated that the time delay factor to the putty, as I understand, and I haven't been that close to the investigation per se
[1512] DR. KEEL: So regardless of which side you pressurize, it has an effect of delay.
MR. MULLOY: The putty is highly variable, and
we understand that its ability to sustain or not sustain pressure is unpredictable.
DR. WALKER: But you were not aware of that earlier. Is that correct?
MR. MULLOY: Yes, sir, we were aware of it, because sometimes we saw paths through the putty, evidence of paths through the putty. Sometimes we saw evidence of paths through the putty and O-ring erosion. Sometimes we saw no paths through the putty and no evidence of hot gas past the putty, and so we knew that it was performing in that variable manner.
DR. WALKER: The evidence of paths through the putty that you speak of, was that evidence directly looking at the putty after demating, or was that by implication because you saw O-ring erosion?
MR. MULLOY: Both. It was looking at the putty and at the joint, looking for evidence of soot between the putty and the primary O-ring, and the distribution of that soot, and the evidence of the path, hot gas path through the putty.
DR. WALKER: So you knew that in most cases pressurization was occurring by leaks through the putty?
MR. MULLOY: I don't believe it is in most cases. No, sir, because we had a very limited number of observations of the 170 some odd joints. There is a
very limited number of observations where you have pressurization by breakdown of the putty. The rest of them, there is no evidence of paths through the putty.
DR. WALKER: But there was a memo from the Titan program suggesting that pressurization, at least in the case of Titan, was occurring primarily through blow holes in the putty. Did you receive that memo?
MR. MULLOY: Yes, sir. You are speaking of, I think it was on your list for about a March, 1984, memo that I got from George Morefield. Yes. [Ref. 5/2-5]
DR. WALKER: And I think that was passed on to you from the chief engineer's office at headquarters. Is that correct?
MR. MULLOY: No, sir, that was written directly to me from Mr. Morefield at my request, observing this problem, asking CSD and Hercules to look at this problem and give me their observations about what they thought was happening and what could be done to rectify it, and so that is what we were working on.
DR. WALKER: So did you accept their view that pressurization in the shuttle was occurring primarily by blow holes, or did you think that the situation was different in your booster than the Titan?
MR. MULLOY: Yes, sir, I thought it was different based upon the evidence that we had that we
had a lot of joints that didn't have any blow holes in them, and evidence that-no evidence of any kind of a hot gas past the putty. And that is an established fact, I believe.
CHAIRMAN ROGERS: Assuming that you were advised as you were by Thiokol that they opposed the launch on the 27th because of weather, would you have reacted differently?
[1513] MR. MULLOY: No, sir. Frankly, I was not aware that this erroneous entry had entered in the PAS because it did not come up.
CHAIRMAN ROGERS: To you, what does a constraint mean, then?
MR. MULLOY: A launch constraint means that we have to address the observations, see if we have seen anything on the previous flight that changes our previous rationale, and address that at the Flight Readiness Review.
CHAIRMAN ROGERS: When you say "address it," I always get confused by the word. Do you mean think about it? Is that what you mean?
MR. MULLOY: No, sir. I mean present the data as to whether or not what we have seen in our most recent observation, which may not be the last flight, it may be the flight before that, is within our experience base and whether or not the previous analysis and tests that previously concluded that was an acceptable situation is still valid, based upon later observations.
CHAIRMAN ROGERS: Okay. So in this case, because you didn't know that the constraint had been closed due to an error, if the constraints were still on, if no error would have been made, you would still have reacted the same way?
MR. MULLOY: Reacted to what, sir?
CHAIRMAN ROGERS: Reacted as you did on the telecon.
MR. MULLOY: Yes, sir.
CHAIRMAN ROGERS: And so each time one of these flights took off that you knew there was a constraint on, you had to make a decision to waive it, what went through your mind?
MR. MULLOY: Okay. What went through my mind is, we looked at the most recent observation of recovered hardware. We compared what we were seeing to our previous successful experience with the joint, realizing that we were having some O-ring erosion, looking if we were seeing anything that changed the previous rationale. That is what led to opening the problem report, when we saw that we violated the primary O-ring. That was something that was different and therefore required additional analysis and test and until that was done.
CHAIRMAN ROGERS: But what did you do about
it, though? It seems to me in that case, when you say that you addressed it, no change was made in it. All you did on these waivers was to waive it. You just apparently-there is no indication.
What did you do? There is nothing in this chart that suggests that you corrected the joint.
Each time there was further experience, further erosion, and we don't see any examples of correction or effective action taken or anything of that kind.
MR. MULLOY: I guess I would have to take issue with that, sir. On page 2- -
CHAIRMAN ROGERS: That is why we want you here.
MR. MULLOY: On page 2 of 3 of document No. 2, if you look at-and I will just pick one and I think there's a rationale here. It says this problem is not considered a constraint to 510-F; a 100 psi leak check is performed which confirms seating of the secondary O-ring. Nozzle O-rings have been shown to survive erosion depths of 125 mils in hot subscale test. The math model predicts maximum erosion depth for the secondary O-ring to be 75 mils in the worst case condition. [Ref. 5/2-2]
The erosion on the secondary O-ring was 32
mils, well below the demonstrated sealing capability.
[1514] What was done was analysis and test to substantiate that rationale for accepting that condition for that flight. It is not just a matter of nothing was done.
DR. WALKER: You are relying on the secondary O-ring.
MR. MULLOY: No, sir. What we were saying was if we got a repeat, we didn't think we would get a repeat of the condition because we had gone to 200 psi and it was concluded that, we knew the putty could mask 100 psi leak check and we could have had a bad O-ring primary to begin with on that one.
We were sure on the 51-F flight because of the 200 psi leak check that we had a good primary O-ring was a substantial part of the rationale. The second then was if the primary O-ring was violated, the maximum erosion that could occur on the secondary O-ring was only 75 mils, which tests had shown could sustain 125 mils.
Now, rightly or wrongly, that was the rationale.
DR. WALKER: So you then were relying on the secondary O-ring in that case.
MR. MULLOY: We were relying on the
redundancy, yes. We showed that we had redundancy that, should the primary O-ring fail, the secondary would function.
DR. WALKER: Well, I guess we are going to address the case of the criticality later, and so I think I will save those questions.
MR. RUMMEL: Mr. Mulloy, I wonder how detailed your investigations were when you applied the experience of one flight to a subsequent flight.
For example, did you consider the differences in dimensional tolerances and dimensions of the various joints-out of roundness, they varied in diameter, they varied in numbers of ways and so when you evaluated a specific flight, did you look into where the previous flight or flights stood in this regard and then analyzed what would be applicable on a specific flight under consideration?
MR. MULLOY: Out of roundness? No, sir. We gave no specific consideration to a variation in gap that could occur due to the out-of-round condition.
What we did do was look at the dimensional tolerances for the specific flight, the tang dimension and the clevis dimension, to assure that we had the minimum O-ring squeeze that was specified and assure that that was within our experience base.
We did not go in and look at what would be the effect of out-of-roundness and possibly a higher squeeze on O-rings in a local area of the joint.
That is a revelation, I think, that has come from the investigation of the 51-L failure.
DR. FEYNMAN: Mr. Mulloy, when you use a math model, do you have any idea of how accurate it is?
MR. MULLOY: We did not just use the math model. What we did was build a math model that was correlated to test. There was a test fixture that was built to empirically determine the maximum erosion that could occur while filling the annulus between the putty and the primary O-ring and the annulus between the primary and the secondary O-ring.
Then Thiokol's, Dr. Salita's math model was shown to correlate very well with that, and I guess I can't put a percentage accuracy on that. But the fact that the math model correlated pretty well with the test results gave us some confidence in that and the fact that the test demonstrated that there was a significant margin that was tolerable in terms of the amount of erosion, given the dimensional tolerance.
[1515] DR. FEYNMAN: I think that the math model determined how the constants were determined and a line was put through the previous data on a somewhat similar
material. And the line that was put through deviated. It doesn't always give the same answer. You took an average rather than the maximum, so that there were factors of 2 above and factors of 2 below on the original data. If you would have known that, you could have appreciated that what this thing predicted could easily be a factor of 2 below the right answer, because in fact it didn't even fit with the data on which it was constructed.
You weren't aware of that?
MR. MULLOY: No, sir. I was not aware of that.
DR. KEEL: Well, Mr. Mulloy, what was your previous conservative estimate of maximum erosion on the primary O-ring for a nozzle joint? Do you remember that? You briefed it at several Flight Readiness Reviews.
(Pause.)
DR. KEEL: Going back to the record, it is 90/1000, based upon what you've characterized as a conservative estimate, by which you presumably meant worst case. Is that correct?
MR. MULLOY: That is correct.
DR. KEEL: And what erosion was there on 51-B that caused the launch constraints?
MR. MULLOY: On the primary O-ring erosion it was 171/1000.
DR. KEEL: So 171 compared to the previous worst case analysis prediction of 90.
MR. MULLOY: Yes, sir. And what was different-that again is why we were concerned about this and established the understanding of this as a launch constraint.
What we observed was we were seeing a different type of erosion on this primary O-ring. And that different type-what we had been seeing previously was
DR. KEEL: By different type, you mean worse erosion than predicted?
MR. MULLOY: No, sir. I mean a different type. The erosion that we had been seeing was due to the hot gas direct jet impingement on the surface of the primary ring as it seated.
What we saw here, it was evident that the primary ring never sealed at all, and we saw erosion all the way around that O-ring, and that is where the 171/1000 came from, and that was not in the model that predicted a maximum of 90/1000. The maximum of 90/1000 is the maximum erosion that can occur if the primary O-ring seals.
But in this case, the primary O-ring did not seal; therefore, you had another volume to fill, and the flow was longer and it was blow-by and you got more erosion.
DR. KEEL: But I think that the bottom line of all that is that perhaps your analysis didn't consider all the physical phenomena, including how that primary O-ring seals and how long it takes to seal, and if it doesn't seal, what erosion should take place then.
MR. MULLOY: It did consider how long it takes to seal and how long it takes to fill that volume. What it didn't consider was that the primary O-ring did not seal and you had hot gas impingement for the additional period of time that it takes to fill the volume during the primary and secondary. It did not consider that.
DR. KEEL: And as a consequence, it was off by a factor of almost 2.
MR. MULLOY: No, sir. It was correct for the mode that it was analyzed for. It was not correct for an O-ring that was not sealing. It is absolutely correct and has been demonstrated to be correct for a primary O-ring that seals.
[1516] DR. KEEL: It's hard, I think, certainly for me and I think the rest of the Commission, to understand how you can say it's absolutely correct. I don't know
of any models you've done since then or tests you've done since then that can correlate any more accurately than Dr. Feynman has indicated with your erosion models, an analytic empirical model. And it has to have uncertainty about it.
MR. MULLOY: Absolutely. That is a very unfortunate adjective. I withdraw that. Nothing is absolute.
DR. KEEL: I would have thought when you had this experience that you wouldn't have immediately gone back to your analytic methods as the basis of your confidence for lifting these launch constraints because certainly this case of erosion was certainly worse than what you would hypothesize from the previous worst case.
MR. MULLOY: Yes, sir. And that is why we ran additional tests and expanded the analytical model to account for this mode where the primary O-ring did not seal.
DR. KEEL: And you expanded the analytical model and what was the worst case erosion that that model then predicted?
MR. MULLOY: I believe I said 75 mils on the secondary O-ring if you had a primary O-ring that was violated and never sealed. And that is what the problem assessment system report shows on page 2 of 3.
DR. RIDE: Did you consider that acceptable?
MR. MULLOY: Yes.
DR. RIDE: 75 mil erosion of the secondary O-ring.
MR. MULLOY: Yes.
DR. COVERT: Mr. Mulloy, at the time this joint was conceived of, did you envision that the O-rings would be eroded to this extent?
MR. MULLOY: I'm sorry, Dr. Covert?
DR. COVERT: At the time that the joint was designed, was it designed with the intent in mind that the O-rings would be eroded to this extent?
MR. MULLOY: No.
DR. COVERT: So in some way, then, the acceptance of this erosion as a fact of life represented a departure from margins of safety that you originally had in mind at the time you were designing it?
MR. MULLOY: Yes, sir. It was treated as an anomaly.
CHAIRMAN ROGERS: Going back just for a moment to the criticality I statement, one of the things that has troubled me from the start and still troubles me is, if I understand English at all, this says that the leakage of the primary O-ring seal is classified as a single failure point due to the possibility of loss of the sealing at the secondary O-ring, because of joint
rotation after motor pressurization. [Ref. 5/2-6]
MR. MULLOY: Yes, sir.
CHAIRMAN ROGERS: That to me says that if the primary O-ring fails, then there is apt to be a loss of mission, vehicle and crew. That is what it says.
Then you have experiences of one kind or another over a period of time with a seal which certainly causes a lot of discussion and a lot of concern.
[1517] At one point in the same document it says, based on the amount of charring and erosion paths through the primary O-ring, and what is understood about the erosion phenomenon, it is believed that the primary O-ring SMR-61A never sealed or never seated.
Now, why at that point, wouldn't you all have said we were lucky not to have a loss of mission and crew at that point because criticality I says if that primary O-ring fails, we will lose everything? Why wasn't that a cause for concern on the part of the whole NASA organization?
MR. MULLOY: It was cause for concern, sir.
CHAIRMAN ROGERS: Who did you tell about this?
MR. MULLOY: Everyone, sir.
CHAIRMAN ROGERS: They all knew about it?
MR. MULLOY: Yes, sir.
CHAIRMAN ROGERS: And they all knew about it at the time of 51-L?
MR. MULLOY: Yes, sir. You will find in the Flight Readiness Review record that went all the way to the L minus one review.
CHAIRMAN ROGERS: That's why I'm saying we want to talk to you this morning. That suggests that was pretty well glossed over; that they didn't really realize it. But in any event
DR. SUTTER: Well, could I ask a quick question? I'm confused about this thing being signed off and that was a mistake, apparently. But is it still on the books as being signed off.? When was that mistake discovered?
MR. WEAR: It is in the books today, just like you see it here.
DR. SUTTER: So then a lot of
MR. WEAR: We haven't gone back and fixed the books.
DR. SUTTER: A lot of people must read these constraints, and a lot of people could read it as saying, hey, that's signed off-, don't worry about it any more. Who reads this? It's in the books.
MR. WEAR: Let me explain how this occurs. There is what is called a Problem Review Board Meeting.
that is held within each project. There's one for each of the projects at Marshall as a result of Bob Lindstrom's letter that was read earlier that set this out, where they go over these items.
The reason you see Jim Thomas there, he's the alternate Chairman of that Board for me. That is discussed at that time and that is where these are recognized as being within our data base or perhaps not within our data base. And it is covered in the Flight Readiness Review and that is the process.
DR. SUTTER: So at the Flight Readiness Review, the people read this and understand it.
MR. WEAR: He or I in doing this activity know it's in the Flight Readiness Review or it's not, as the case may be. Plus the other participants there know that. And therefore it is listed as a result of being presented and discussed in the Flight Readiness Review.
And it would be within our data base or not.
CHAIRMAN ROGERS: What do you mean by data base, because Thiokol keeps saying that experience was not within your data base on Flight 51-L, and you all say that it was within your data base.
Which is right?
MR. WEAR: The Thiokol that I addressed, sir, says it is within their data base.
DR. KEEL: I think the Chairman's point, though, is with respect to the night of the telecon, Thiokol was arguing just what you are arguing now with respect to erosion. They were arguing that we want to stay within our data base and we want to go with an O-ring temperature not any lower than 53 degrees, just like you're arguing now that it's okay to fly because we are within our data base, implying you wouldn't fly outside of your data base.
CHAIRMAN ROGERS: What do you mean by data base?
MR. WEAR: The data base to me, sir, is the previous test and flight experience that we have, as supported.
CHAIRMAN ROGERS: You've never had experience in this cold weather so when you're talking about your data base on Flight 51-L you didn't have any. It seems to me it is used as sort of a slick way of just getting over the problem, saying it's within our data base. But you hadn't had any experience of that kind before and so you can't say it was within your data base.
The engineers at Thiokol were saying it is not. We have never had that experience. We warn you: Don't do it. We don't know what's going to happen.
And we keep hearing from NASA, it was within
our data base.
MR. WEAR: Well, going to that particular evening, which I think what you are referring to is the January 27th evening, on that particular evening in my process, in my mind process, we had faced a cold launch the year before, in which we had had some erosion.
CHAIRMAN ROGERS: The worst experience you had.
MR. WEAR: Which they had addressed and they, the Thiokol that you're referring to, they had said that that condition, that cold condition on that particular occasion the year before had-I forget the exact words, but in effect had aggravated the situation, but that it was acceptable and would perform. That was the conclusion.
CHAIRMAN ROGERS: But you agree, though, it was not within your data base, don't you?
MR. WEAR: That experience of the prior year was within my data base; yes, sir.
CHAIRMAN ROGERS: Could I say on that one, though, you had your worst result on that one.
MR. WEAR: Yes.
CHAIRMAN ROGERS: So you can say well, we almost had an accident, but we didn't quite. Therefore, it is within our data base. And then you get to another day when it is colder and you still argue it is within
your data base.
Thiokol said it is not in your data base because you had never tried it in this cold weather.
Now, what I'm asking is how do you explain that controversy? How do you explain that conflict?
MR. WEAR: If I may continue, their engineering organization and my engineering organization had agreed the prior year that that experience was-could be acceptable on the next launch; that if the same condition occurred, that that was the findings that were made at that time, if that same condition existed it could be accepted.
And that is the same engineering organization that was talking on January 27th, so their conclusions and report to us on the previous year was that at those conditions and what we [1519] observed on that particular launch, that if that condition-meaning that type of-that entire condition, that weather condition, whatever-occurred again, that that was acceptable.
MR. HOTZ: But you didn't have those conditions.
CHAIRMAN ROGERS: Let me finish. That was within your data base, you argue. But what I'm saying, you've got a new condition now which was not within your data base.
Now, how do you keep relying on it was within your data base? The Thiokol people said it's not. The engineers said it's not. We've never tried it at this cold a temperature.
How can you keep saying that it was within your data base? It wasn't. It exceeded your data base. You never had that kind of experience before.
MR. WEAR: Well, on that particular evening I heard what the Thiokol engineering people stated. I also know that they then counciled and they came back and they stated that the conditions could be accepted. And so I have to conclude some engineering people must have changed their minds.
CHAIRMAN ROGERS: I'm not taking issue with that. I'm taking issue with the slickness of the words "within our data base," as if that excuses everything.
What I'm pointing out is I don't think you ever had a data base of this kind in Flight 51-L.
MR. WEAR: That's true.
CHAIRMAN ROGERS: Well, that's really all I was pointing out.
DR. FEYNMAN: Can I ask something, too? I would like to understand this idea of within the data base. For instance, 51-C had many seals on which there was no erosion at all. Which seals should we take?
Why is it-suppose we had an accident? You see, we have an accident on five of the six seals. And maybe it wasn't five, but anyway, most of them-there's no erosion, statistically.
Accidentally, it could happen that there was no erosion on any of the seals, only a little bit, and then the next flight a lot of erosion. In other words, I don't understand how the logic works, that because something just made it, that the next time it wouldn't be a bigger variation.
Could you explain to me, therefore, why when you have a successful flight which is successful in the sense that the entire flight takes place, but which is unsuccessful in the sense that you get effects that you didn't expect, that you consider that the next time it isn't going to be accidentally a little bigger?
MR. MULLOY: Yes, sir. That is what we addressed with trying to determine when we would see a 10/1000 erosion or 20/1000 erosion or a 30/1000 erosion. That is why the test setup was made, to determine how much erosion can you physically get.
And that is where the 90 mil calculation came from, because that erosion is limited by the amount of time that hot gas can impinge on the primary O-ring, on the assumption that the primary O-ring seals.
So the logic was that there was a large margin against what were actually observing versus what the theoretically possible erosion was, and then there was a large margin between the theoretically possible erosion and that which you could sustain as demonstrated by tests.
Now, that was the logic.
DR. FEYNMAN: How many tests were made that would show that you really could sustain 90 mils of erosion?
[1520] MR. MULLOY: I can't recall. There were several hot gas tests made, maybe ten, with 125/1000 erosion on the primary O-ring seal.
CHAIRMAN ROGERS: How did the results of those tests compare with the results of the tests that have been done lately?
MR. MULLOY: They are correlatable in terms of the erosion. There are no differences, I don't believe. I haven't seen all of the test data.
DR. FEYNMAN: This isn't quite right, sir. You said there were tests with 125 mils erosion.
MR. MULLOY: Yes, sir.
DR. FEYNMAN: That isn't quite the way I remember it. I think the tests were made by cutting the ring away.
MR. MULLOY: There were two tests run. One of them was a hot gas impingement that sustained up -to 125 and still sealed.
DR. FEYNMAN: How did you make the test with 125?
MR. MULLOY: By allowing the hot gas to impinge for a longer period of time and having it impinge longer. And then there were cold gas tests made where the O-ring was-simulated erosion was put in there, and the maximum demonstrated there I think was 95/1000 as I recall.
But that was the logic. And I guess anyone can question that logic.
DR. SUTTER: These were all the five-inch?
MR. MULLOY: Yes, sir; with a full-scale gland and O-ring, but a short length. Well, a 10-inch diameter, I guess.
MR. HOTZ: Mr. Mulloy, I would like to try to understand this in somewhat simpler terms than you people are used to using.
Is it correct to state that when you originally designed this joint and looked at it, that you did not anticipate erosion of any of the O-rings during flight?
MR. MULLOY: That is my understanding. I entered this program in November of 1982 and I wasn't there on the original design of the joint, but when I took over the program there was no O-ring erosion anticipated.
MR. HOTZ: So that when you did run into signs of O-ring erosion, this was a bad sign.
MR. MULLOY: Yes, sir.
MR. HOTZ: You didn't like it?
MR. MULLOY: No, sir.
MR. HOTZ: So then you decided to introduce a standard based on the measurement or the possibility of the limits of O-ring erosion. And as those limits, as the experience went up, your criteria for safe flight went up too.
In other words, when you experienced more than maximum anticipated O-ring erosion, you waived the flight and said well, it's possible to tolerate that. We still have a margin left.
MR. MULLOY: Are you speaking of the case where we did not have a primary seal?
MR. HOTZ: Yes.
MR. MULLOY: Yes, sir. That is correct. There was another piece of logic that went into that though; that that flight, the next flight, we were
positive we did have a good primary seal because we increased the leak check pressure to 200 psi.
[1521] MR. HOTZ: But getting back to that, you mentioned that the ability of the putty to sustain pressure was pretty unpredictable.
MR. MULLOY: Yes. sir.
MR. HOTZ: And wasn't that a cause for concern that you had an unpredictable element in your equation?
MR. MULLOY: Yes, sir. And what was done there was to look for some alternative to the putty, such as MBR rubber strips or carbon fiber or carbon wool or steel wool or something that would allow pressure to go directly to the primary seal.
MR. HOTZ: But to continue flying in the meantime?
MR. MULLOY: Yes, sir.
MR. HOTZ: Then you finally, you're talking about these margins of safety, and I wonder if you could express in either percentages or actual measurement terms-you have used the term «wide margin". I wonder if you could give us a quantitative measurement as to what you consider a wide margin?
MR. MULLOY: Yes, sir. Well, as I said we had demonstrated that we could stand 125 thousandths of erosion and still seat. The maximum erosion that we had seen in the case joint was on SDS-2, which was 53 thousandths, so that is a factor of two and a half.
MR. HOTZ: But it is still based on a very narrow physical measurement between 125 thousandths.
DR. KEEL: Could I clarify something here, Bob? I think, Larry, if you go back and look at your Flight Readiness Reviews, that you were relying on smaller margins than that.
You were arguing in the Flight Readiness Reviews where you briefed the problems of primary O-ring erosion that for the worst case for the field joint also that it would be 90 thousandths.
MR. MULLOY: That is correct.
DR. KEEL: At that point you were pointing out that's okay, because you can seal at 95, not at 125 but at 95. It wasn't until later on during the process that
you determined you could seal at 125.
MR. MULLOY: That is when we got the hot gas test data.
DR. KEEL: So that's a five percent margin, roughly, five and a half.
MR. MULLOY: On the 90 to 95 on a max predictable, yes.
MR. HOTZ: Just one more question, sir. With all of this experience base, wasn't there any time in this history of the flight that you or anybody else connected with the solid rocket booster said we are getting a lot of anomalies here. We are getting thin that are outside our original predictions, and shouldn't we take a look at it and stop flying until we've fixed it or have a better feel for what is actually happening in the joint?
MR. MULLOY: Only the first part of that, that we are seeing something here that needs to be corrected. We continually emphasized to the contractor that we need to put more emphasis on resolving this problem. We did not recommend that we stop flying for the logic that was presented in the Flight Readiness Reviews and which I reiterated here today.
MR. HOTZ: Thank you.
CHAIRMAN ROGERS: Was consideration
given, as far as you know, to stop flying and fix the joint? Was there any discussions about that? Maybe we should stop flying for a while?
MR. MULLOY: Not to me, no sir. I have become aware of that in the course of some of the testimony since the 51-L accident that there are some memos that are internal memos to that effect.
MR. SUTTER: I still am confused with the answers I got to the question about the statements as the problem was closed. The question that's going on right now is how deep was the concern over the joint.
The statement here that says this problem is closed is against the whole NASA philosophy of really documenting, controlling and having checks and balances. When I read this statement that it was a mistake but it's left on the book, that gives me the impression that NASA did not believe this was a very big problem. [Ref. 5/2-3]
DR. RIDE: How serious do you consider a launch constraint?
MR. SUTTER: Do you have any comments on that?
MR. MULLOY: The comment I have, it's very unfortunate that that was erroneously entered. I had no intention of closing that problem, because I considered this to be a very serious problem.
MR. SUTTER: But did you know this?
MR. MULLOY: No, sir.
VICE CHAIRMAN ARMSTRONG: Am I right that you didn't close it because you didn't want to change anything after the accident? Isn't that right?
MR. SUTTER: But when was this signed off? This was signed off on 12/18/85.
MR. WEAR: This entry was made by this PAS system organization based upon the entry that they got from the contractor where he recommended its closure and they did it.
MR. SUTTER: I'm really confused, because the contractor was the guy that said don't launch, yet he wrote you a piece of paper saying close this item out? If you're going to depend on the paperwork as to controlling the operation-and I've heard this now for three months, all of these documents that, my God, we really controlled and we followed this to the point where we can't get into trouble, yet here's a piece of paper that says there's no problem.
How the-nobody reads this, I guess.
MR. WEAR: No, I think it is read.
MR. SUTTER: Do we have a copy of the letter?
MR. WEAR: Yes, sir, it's in the book here.
DR. KEEL: Mr. Chairman, we're going to ask the contractors exactly that question.
CHAIRMAN ROGERS: But I think we have to follow up on Joe's point. You mean just because somebody wrote a letter, the whole thing was closed out and you didn't know? Nobody knew anything about it? After all of this history of it, one letter would close it out?
MR. MULLOY: That was a failure of the human being within the system.
CHAIRMAN ROGERS: It was a little more than that. It's a failure of the whole system if one letter and one human being can close out a constraint that has been concerning you for many years.
DR. KEEL: Can I ask for one clarification before you go away from this, Mr. Wear? You're listed here as the project representative under the approval line?
[1523] MR. WEAR: That's right.
DR. KEEL: Does that mean you did approve this, or you didn't approve it?
MR. WEAR: No, I did not approve this. It was not brought forward for us to consider.
DR. KEEL: So even though your name is on here as approval, you didn't approve it?
MR. WEAR: That's right. I think if you will notice, there is not an entry here that says so. On the others, I think you will notice over here that there is
an entry of initial, either JWT or LOW on the others as having lifted that for that particular flight. I think most of them them put an L after it or something that refers to it's lifted. In their nomenclature there's an L entered for that purpose.
DR. KEEL: So as far as you're concerned you were still operating as if this was-remained a launch constraint?
MR. MULLOY: Yes, sir.
MR. WEAR: Yes, sir.
DR. KEEL: But may I follow up? That is correct, but as a matter of practicality even though this was a launch constraint it was being waived for each launch?
MR. MULLOY: That is correct, on the basis of the presentation at the Flight Readiness Review.
DR. KEEL: So you in effect waived this for 51-L?
MR. MULLOY: That is correct.
DR. KEEL: But it doesn't show up on your summary?
MR. MULLOY: No because the man assumed when the closure came in from Thiokol that this was going to close the problem, and that requires project concurrence, Mr. Wear's concurrence, which Mr. Wear and
I had discussed it and that was not going to happen.
DR. KEEL: As far as you were concerned, though, you still considered it a constraint in spite of this document?
MR. MULLOY: Yes, sir.
DR. KEEL: So the night of the telecon, for example, when you were arguing about whether the primary O-ring would seal, you still considered the fact that there was a launch constraint on the primary O-ring for 51-L?
MR. MULLOY: Yes, sir.
DR. KEEL: Hence, in effect, the fact that in a strict sense there was no launch commit criteria with respect to temperature, there certainly was a launch constraint with respect to the primary O-ring sealine.
MR. MULLOY: Launch constraint relative to an understanding of our previous history to go into the next flight.
DR. KEEL: Well, it's primary O-ring, isn't that what the launch constraint is on?
MR. MULLOY: Yes, it is related to the primary O-ring on the field joint.
DR. KEEL: And the reason you could have so much erosion is because it didn't seal on 51-B?
MR. MULLOY: On the nozzle joint, that is correct.
DR. KEEL: That's right, and they were arguing on the field joint. They had concern about it sealing or having time to seal.
[1524] MR. MULLOY: Yes, that was the presentation. The concern for the time that it would take for the primary O-ring to seal.
DR. KEEL: So in that sense, all that argument was in the context of there being a launch constraint.
MR. MULLOY: I didn't think of it in those terms at that time.
MR. ACHESON: But I take it it would have made no difference whether it was formally closed out or not because, as I see the way the system had worked, a flag raised on 51-L-I would have thought the determination of the problem was contained by the fact that the leak check had been changed to 200 psi from 100, and you could go on as before with that change in the procedure.
Is that not the way it would have been resolved?
MR. MULLOY: Yes, it was contained by the fact that with the 200 psi leak check we were sure that we had a good primary O-ring.
MR. ACHESON: So it really didn't make any difference whether it was formally closed or not as to what would have happened.
MR. MULLOY: I would like to go back to the point Mr. Wear made, because I think it is important in addressing your concern, Mr. Sutter, and it's a very important point that Mr. Wear made, is that you do not see the accepted- closure on that last entry. That is an entry that the guy who makes entries into the PAS system made, and you do not see project signature concurring in that.
So in essence, it is still open. It is open until Larry Wear concurs that it's closed.
MR. SUTTER: Do the people who read these look for all the signatures?
MR. MULLOY: No, sir. I agree, it is unfortunate that that error was made by the gentleman who make the entries into the PAS system on the basis of a submittal from the contractor that this was no longer a problem of significance to carry in the problem assessment system.
MR. KEHRLI: Mr. Chairman, there are some documents, four documents, that we haven't looked at that address these specific questions that have been asked for the last three or four minutes that I would like to address the witness' attention to and the Commission's on the chronology of this closure.
The first one is the launch constraint
document number 4, which is a letter dated December 10, 1985 to Mr. Wear from Mr. McDonald of Thiokol, which references the particular problem that we've been discussing and suggests that or requests that the subject critical problem be closed. [Ref. 5/2-4]
Additionally, the next document that follows that page is the letter from Thiokol to Mr. Jack Fletcher of Rockwell, and it indicates that there is further information that's being provided with regard to the closure of critical problems. [Ref. 6/2-7]
Back on O-ring history document number 34, there is a letter from Mr. Wear to Morton Thiokol; subject, SRM Problem Review Board. The date of that letter is December 24, 1985.
It reads: "During a recent review of the SRM Problem Review Board open problem list, I found that we had 20 open problems, 11 opened during the past six months, 13 opened over six months; one three years old, two two years old, and one closed during the past six months. As you can see, our closure record is very poor. You are requested to initiate the required effort to assure more timely closures, and the MTI personnel shall directly coordinate with the S&E personnel the contents of the closure reports." The letter is signed by Mr. Wear. [Ref. 5/2-8]
The final document is attachment number 30 to
the O-ring history. This is, again, part of the Marshall package that was received when the launch constraint document that we've been talking about, the problem assessment documents were received. [Ref. 5/2-9]
MR. WEAR: What's the number, please?
MR. KEHRLI: Number 30 in the O-ring history. It gives a chronology. As you see, it notes on the entry of 51-B there is an asterisk next to the Marshall tracking number A09288, which indicates that "this problem contains secondary O-ring erosion of the nozzle joint and constrains launch anomalies."
Then down at the bottom, on STS-61-C, by the date, 1/12/86, there is an indication of erosion or an O-ring anomaly problem on the field and nozzle-to-case which was not reported and not given a problem tracking number in January of 1986.
CHAIRMAN ROGERS: What do you conclude by that?
MR. KEHRLI: Well, one, I am wondering if the closure of the item is the reason that the January 12, 1986 erosion or O-ring anomaly was not reported or given a tracking number.
MR. WEAR: I'm afraid I didn't follow your question there.
MR. KEHRLI: Well, on document number 30, the field and nozzle joint erosion problem, on STS-61-C on the
primary ring, unlike the other problems-except for STS-2-it has an indication that it was not reported.
Is that because the problem had been closed as a result of those previous letters we just referred to, the Thiokol request to close it and your letter indicating that you want open items closed?
MR. WEAR: I'm afraid I can't tell you why this one was not reported, looking at this right here, now.
As far as my letter is concerned, the thrust of my letter was we had problems. They are within this tracking system. It's more than the normal deviation, the deviation records. What the thrust of my letter is is that they are not making adequate progress to satisfy me in resolving problems. We were continuing to have the problem but we weren't making sufficient progress to suit me in resolving them, that they were hanging on for too long and we weren't closing them out.
MR. MULLOY: Let me take a try at answering this question as to why it was not reported. I see this was revised 3/19/86. This would be speculation, but I didn't get the information on the inspection of 61-C until after the Flight Readiness Review for 51-L. That came in-I believe we got those joints demated just two or three days before the L-1, and I reported this in the L-1 review.
Given that timing and given the events of 51-L, I just-and perhaps you could address this question to Thiokol, but I suspect they haven't written up this four mils of erosion that occurred on 61-C and put it into the system since the 51-L accident.
There is no reason why it wouldn't be reported because there was four mils of erosion.
DR. KEEL: Can I ask one followup question with respect to these memos and the documents that Randy has referred to here? Based on our staff interviews with Mr. Thomas from Marshall, he indicated that he had told Thiokol on the telecon to in fact close out all issues that were over six months old, and Mr. Russell at Thiokol confirmed that.
[1526] Was that your understanding, also, that engineering at Marshall was saying close out the problems over six months old?
MR. WEAR: I'm afraid I don't know the context of that discussion. He and I had had conversations.
DR. KEEL: Well, it's the same context here, I assume.
MR. WEAR: Well when you say close them out, I say I'm not sure what the context was of the statement he made. My intent-and I trust it is the same with Jim's because he and I usually communicate well-my
intent was let's get off our duff and work and reach solutions for these problems, not continue to drag them out for six months, five months, et cetera. That is the thrust of my direction to Thiokol.
DR. KEEL: Did ultimately Thiokol recommend, in fact, closing out by taking the joint, SRB joint problem off the problem report list?
MR. WEAR: That's the nature of McDonald's letter back here where he does recommend taking them off, and as I understand the thrust of the letter the way I read it now and the way I read it then was that we are discussing these problems in the Flight Readiness Review. Let's not also put them someplace else, so that in effect he has to report them two places.
As I read his letter, he's saying we are doing the tracking job in the Flight Readiness cycle. Let's take them out of this other tracking system. He's saying let's do it once. I think that's the thrust of his letter, the way I read it.
DR. KEEL: What was Marshall's response to that?
MR. WEAR: My response to that was no.
MR. MULLOY: Mine, too.
MR. WEAR: We will keep it in the system because it a formal check and balance. That is what its
intention was, to be sure if something didn't slip through a crack someplace. That's the way I've always looked at it, was to create a double check by the quality organization to see to it that something wasn't overlooked by the project. Perhaps the project might not even be aware that it could be overlooked, that it had to be faced.
DR. FEYNMAN: Is this answer of yours to McDonald in some document?
MR. WEAR: Pardon?
DR. FEYNMAN: How did you answer Mr. McDonald, in a document?
MR. WEAR: I never have.
DR. FEYNMAN: The answer you've just given us as to how you answered Mr. McDonald, in what document is that where you say you won't close it out?
MR. WEAR: I say I never have answered his memorandum; therefore, the system stands, the Marshall system stands.
DR. FEYNMAN: Except for the mistake.
MR. WEAR: Yes, sir. It makes mistakes, but it stands. Frankly, it wouldn't be within my power to accept his recommendation and take this out anyway, because the establishment of the system is well above me, and so it is not my prerogative.
DR. FEYNMAN: Nobody answered this letter of Mr. McDonald's?
MR. WEAR: No. 1526
[1527] DR. RIDE: How many of the problems in your tracking system carry launch constraints? How many launch constraints have you been waiving flight to flight?
MR. WEAR: I haven't researched it specifically, but I think you would find every one of them having to do with SRM or related to the joint putty, the joint and/or the putty type of affairs that we've discussed here. I don't recall anything else that would be there as a launch constraint.
DR. RIDE: Say on flight 61-C, how many launch constraints did you waive?
MR. WEAR: Offhand I can't tell you.
DR. RIDE: I mean, it must be, and I mean I hope the answer is one. If the answer is more than one, then there is more that we don't know.
MR. WEAR: Well, you've got the record here, and I would just stand on that. I mean, you've got the records from the past system.
CHAIRMAN ROGERS: Well, let's talk about it while you're here,
DR. RIDE: What we've got, I think, is the record of the launch constraints as a result of the
erosion problems on the joint.
MR. KEHRLI: That's correct.
DR. RIDE: I guess my question is are there other problems in the SRM or SRB system that also carry launch constraints that you have been waiving at your FRRs.
Maybe I'm wrong, but I consider a launch constraint to be a big deal that you have to address in a significant, substantial way before every launch, and I would think that if there is more than one you would know that there is more than one.
MR. WEAR: Well, you asked me how many, and that infers that I can pull off the top of my head exactly how many there are, and I can't right here, but we are required
CHAIRMAN ROGERS: Would you be willing to say it's very few?
MR. WEAR: Yes, sir. Absolutely.
CHAIRMAN ROGERS: Would you be willing to say they all relate to the SRB? All the waivers would relate to the solid rocket booster?
MR. WEAR: Do you mean that are going through the whole FRR process, is that what you're saying?
CHAIRMAN ROGERS: Yes.
MR. WEAR: No, because I'm only exposed to the
SRM, per se, so I couldn't answer that.
DR. RIDE: Was there more than this one?
MR. WEAR: In each Flight Readiness Review we addressed problems that occurred on a past flight and/or significant waivers or deviations that occur during the manufacturing process, the manufacturing acceptance process of the hardware. Those are categorized as being within previous data experience or not.
DR. RIDE: I'm trying to understand how you deal with the launch constraint. How important do you think a launch constraint is and how unusual it is in your system?
MR. WEAR: I think a launch constraint is a significant event in our system, and it is one that has to be addressed within the Flight Readiness cycle because I don't have the authority to not do that.
CHAIRMAN ROGERS: The question is how many have you waived?
MR. WEAR: Well, of course each of these joint and putty entries that you've seen, those are there.
[1528] CHAIRMAN ROGERS: Can you think of any others?
MR. WEAR: Yes, sir. I believe I've got one. I think if you check the record I believe it would be categorized as one. On a nozzle, where we had de-bond on a nozzle, I believe it is somewhere in the system and
we had to process it.
CHAIRMAN ROGERS: Is that the only other one you can think of?
MR. WEAR: Well, I know in the history of the program we have addressed several.
CHAIRMAN ROGERS: We're not talking about addressing. We're talking about waivers.
MR. WEAR: There has been a lot of discussion in the Flight Readiness Review process regarding the nozzle itself. I think some of you are aware that we struggled with a severe problem on the nozzle erosion.
CHAIRMAN ROGERS: I'm talking about how many have you waived?
MR. WEAR: Specifically I can't answer that.
CHAIRMAN ROGERS: So probably most of them were in connection with this joint.
MR. WEAR: I think the preponderance of them have to do with this joint and/or the nozzle-to-case joint, and I think it is the preponderance of everything that has been waived is there.
The reason I'm struggling here, I think if I went back and went through the whole record there would be some related to the nozzle because we did have severe nozzle erosion problems about two and a half years ago. We struggled with that problem for a while, and I suspect I would find some there.
VICE CHAIRMAN ARMSTRONG: Is it an easier question if you ask how many waivers were there on 51-L, or is that still a hard question?
MR. WEAR: I just don't recall it, and I don't recall that there were any on 51-L that we haven't discussed.
DR. FEYNMAN: In your letter of the 24th you said there were 20 open problems. These open problems are not all launch constraints?
MR. WEAR: That's right, they are not all launch constraints because the PAS system is concerned with problems other than Crit 1, Crit 1R problems. It is an attempt to keep the record clean for lesser problems as well.
DR. RIDE: That's exactly the point, because you've got the system that records open problems, and you have to have some way of distinguishing unimportant problems from important problems from very important problems, and it seems to me the one that says launch constraint next to it must be the very most important problem.
MR. WEAR: That's right, and that is why it has to be cleared by this PAS system before we can proceed.
DR. RIDE: What I'm trying to understand is
how many problems are in the launch constraint category in your system?
MR. WEAR: I can't give you a precise answer.
CHAIRMAN ROGERS: A followup question which really does surprise me is that I would think that if you have waivers and there weren't many, you would remember them all.
MR. WEAR: When you asked me how many, I would have to go back over history, and historically the only other problem of consequence that I recall we have wrestled with on the SRM has either been related to nozzle erosion and/or these joints.
[1529] DR. WALKER: Have you ever refused to waive a launch constraint because you thought the problem was so serious?
MR. WEAR: No.
DR. WALKER: What would happen if you didn't waive a launch constraint? Let's say you or Mr. Mulloy had looked at this O-ring situation and said, well, I don't think we can waive this because it is too serious. What would happen then?
MR. WEAR: Well, I think immediately there have been some times when we have looked at a problem and either I or someone else associated with my review have not been satisfied with the data that was presented.
On those specific occasions there was some additional work that was done, some additional data was provided, whatever, and then either I or that person was reconciled. But if I had said I refuse to accept this, I refuse to go forward, I refuse to accept this problem and fly with it, that would have gone to Larry and to Larry's board, and it would have had to have been reviewed with them.
DR. WALKER: Let's suppose it now goes to you, Larry, and you're not going to waive it.
MR. MULLOY: The effect of not waiving it the first time, it is presented as a launch delay, and we have had several of them. One of those was after STS-8 nozzle erosion. There were some people who felt that we could proceed with the flight of the next flight, which was STS-9, with the nozzles that we had on there. I did not accept that.
The consequence was we rolled STS-9 back and demated the aft segment and put in an aft segment with a nozzle that had, after understanding, spending some time to understand what the cause of that excess erosion was on STS-8, we changed out a segment.
So the effect of not waiving a constraint when it has first been presented has been launch delay. Up to this point, we have taken the time necessary to do
the analysis and tests to provide sufficient rationale to proceed with the next flight in the face of the open problem.
MR. HOTZ: Then you have refused to do it? On STS-8, you did it?
MR. MULLOY: On STS-9. On STS-9, after the problem was open on the nozzle, until we understood what the probable cause was, and as I say, the effect was a rollback from the pad and about a two-month delay in the launch of that vehicle.
DR. WALKER: So the nozzle was changed out?
MR. MULLOY: Yes, and that is what we would do in any case when there is insufficient rationale to proceed on the schedule that is proposed.
MR. HOTZ: Well, I am confused, Mr. Wear. You said you have never refused to waive a launch constraint, and I understand now that there was one instance when you did. Which is correct?
MR. MULLOY: I said both, sir?
MR. HOTZ: No, Mr. Wear said you never refused.
MR. WEAR: My point is, either the problem was resolved before it went forward, or else we wouldn't have launched, but it would be resolved before we would go forward. That is a case where we hit a problem, and I don't know exactly.
MR. HOTZ: But you did not waive it to go on with STS-9?
MR. MULLOY: As it was configured when it was on the pad. That is correct. And in fact I think that was a two-month launch delay, and that is what we would do in any instance when there has been insufficient time from the time that the problem was observed to do the test and analysis to understand whether you can proceed with the next launch, and as I say, the effect to date has been launch delays, but then ultimately that sufficient data is available to proceed, and then the waiver is before you can fly it. The waiver in the case for STS-9 was that the cause of the excessive erosion on STS-8 had been determined, and that condition did not exist on the nozzle in STS-9, and the rationale as to why the condition did not exist after we changed out the nozzle, and the material change.
CHAIRMAN ROGERS: Do you have ultimate responsibility for waiving the launch constraints?
MR. MULLOY: Yes, sir, I have ultimate responsibility for the launch readiness of the solid rocket boosters.
CHAIRMAN ROGERS: So there was a launch constraint, and you waived it.
MR. MULLOY: Yes, sir, all flights subsequent to.
CHAIRMAN ROGERS: All those you waived on your
own? Who did you consult with in connection with those waivers?
MR. MULLOY: Well, in terms of consulting on the waiver, the waiver is approved by my element managers, and the system that comes to me, the quality people who run the problem assessment system, when they come to my board, will say, you have an open launch constraint that has not been properly closed out. The rationale for closing out that launch constraint is presented by Mr. Wear to me in my flight readiness review, which is that rationale that exists throughout this document that says why it is okay, given this observation to proceed with the flight.
CHAIRMAN ROGERS: Well, I have trouble following this. Could you do it a little slower? There is a launch constraint put on by somebody, some decision.
MR. MULLOY: By me in this case.
CHAIRMAN ROGERS: By you. Okay. Now, who has the authority to waive it?
MR. MULLOY: I do.
CHAIRMAN ROGERS: Okay, you put it on and you take it off.
MR. MULLOY: Yes, sir.
CHAIRMAN ROGERS: Who do you consult with when
you take it off?
MR. MULLOY: Mr. Wear, in this case, who brings me the rationale as to why we can proceed for the next flight in light of the observation that we have seen.
CHAIRMAN ROGERS: So in all of these flights, Mr. Wear, you were involved in waiving it, too?
MR. WEAR: Sure.
CHAIRMAN ROGERS: Now, did you know anything about the telecon before Flight 51L? Were you involved in that?
MR. WEAR: The one on January 27th?
CHAIRMAN ROGERS: Yes.
MR. WEAR: I was there on January 27th. Yes, sir.
[1531] CHAIRMAN ROGERS: And did you point out to everybody there was a launch constraint, and therefore it was a little different than the usual launch? When the Thiokol people said we ought to, because of the weather, we ought to delay this, did that cause you any concern?
MR. WEAR: Did the existence of a launch constraint -
CHAIRMAN ROGERS: The combination of launch constraint and a warning by the manufacturer not to do
it, did that cause you any concern?
MR. WEAR: Let me make sure I answer that in the right context. The joint problem and its understanding had been a matter of concern for some time, and there was no one on that conversation that night to my knowledge-well, there was no one in that conversation that night that was not well aware of the problem and its seriousness.
CHAIRMAN ROGERS: Put aside the word "problem." Was everybody aware of the fact that there was a launch constraint?
MR. WEAR: There was a launch constraint to address the erosion. The problems on prior flights, yes.
CHAIRMAN ROGERS: Now, following along Dr. Ride's comment, we assume or at least I assume that that means there was something a little different or special, more serious, the fact that you had a launch constraint, this was a more serious problem than just an ordinary problem. She pointed out there must be gradations of problems, and this, when you have a launch constraint and it is in a Criticality 1 item, that is pretty serious.
Now, when the manufacturer then said we recommend don't launch to begin with did that cause you
any concern particularly in view of the fact that there was a launch constraint on it?
MR. WEAR: Yes. When the Thiokol engineering people expressed their concern, yes. That caused me some concern, yes.
CHAIRMAN ROGERS: And they suggested a slight delay until the weather was better.
MR. WEAR: That is what the Thiokol engineering people stated that night.
CHAIRMAN ROGERS: And you were willing to go ahead even in the face of that recommendation, even though the weather was not good, and even though you had no, what you have said, data, no data base that would say it was safe to do it under those conditions, you still were ready to go ahead and launch?
MR. WEAR: Yes. Now, let me explain one thing, though. In my dealings with Thiokol, I deal principally with Mr. Kilminster. There are other occasions, and beyond this one there are numerous occasions where the Thiokol engineering people and/or my engineering people have discussions and so forth. He occasionally has to deal with his people and understand the problem, and they give me recommendations. I mean, they are not always one for one. He is not just a mimic from them, and therefore I have to depend upon him to
present what Thiokol concludes.
DR. WALKER: But you also have to depend upon your own judgment, don't you?
MR. WEAR: Yes, I do.
DR. WALKER: It is not a situation where you are dealing with something you procured and you don't understand it and you have to accept it.
MR. WEAR: Ultimately you have to go with your own judgment.
[1532] DR. WALKER: So you concurred in that judgment?
MR. WEAR: Yes, sir.
DR. RIDE: You said that the thing that carries a launch constraint has to be presented at your FRR, and then you have the opportunity to waive it. You then take information from your FRR and make presentations to the Marshall center FRR and also to the Level 2 FRR, and I assume also to Level 1 FRR.
MR. MULLOY: Yes.
DR. RIDE: Did you ever mention in any of those cases that there was a launch constraint on the SRB nozzle joint that you had waived?
MR. MULLOY: No. What we did was, I am sure that we did address this recurring concern. We didn't address it in highlighted launch constraint, waive
launch constraint.
CHAIRMAN ROGERS: Did you ever say it in those words, though?
MR. MULLOY: No, sir.
CHAIRMAN ROGERS: If you read the documents, it seems to me everything was almost covered up, ever so slightly noted, and it seemed to be such a serious problem, and the papers reflect that a lot of you thought it was serious, and yet it doesn't seem that serious when you read the documentation. In other words, others that we have questioned said they didn't realize that it was serious, and they apparently didn't realize it was a launch constraint.
Well, anyway, that is not really in the form of a question. I don't have any others. Are there any other questions?
DR. KEEL: I just have one other, if I can, Mr. Mulloy, just so we might understand it. There are two bases on which you waive the constraint at least immediately after 51B. One was the analysis, which we have already questioned. The other was, as you have mentioned, going to higher leak check pressure at 200 psi. But you must have been aware at the same time that there were people at Marshall and your engineers in particular who were concerned about the effects of the
leak check pressure procedures on problems with the O-ring and on erosion.
Is that a correct statement? You were aware of that?
MR. MULLOY: No, sir, I was not then and I am not now. That 200 psi leak check was on the recommendation of our engineer.
DR. KEEL: Well, let me just read this memo then from Mr. Miller to Mr. Hardy and Mr. Coates that, Mr. Wear, you have got a copy of and, Mr. Mulloy, you have got a copy of. It is Tab 16. And it is "burned O-rings on STS-11." This was back in February, 1984, and the first sentence, "The recent experience of two burnt O-rings in STS-11 coupled with the missing putty finding at disassembly raises concerns with respect to STS-13." [Ref. 5/2-10]
If you drop down further in the paragraph the last sentence in that paragraph, "The O-ring leak check procedure and its potential effect on the ZCP," that is putty, "installation and possible displacement is also an urgent concern which requires an expedition of previously identified full-scale tests," so that to me, and clarify it if I am misinterpreting it, says we are concerned about the leak check procedure and the possibility of during a leak check displacing putty blow holes.
MR. MULLOY: Yes, sir, and filling in behind that their concern, of course, was that the 200 psi leak check is designed to ensure again that you do seat the primary O-ring to verify that it is capable of seating, although you are seating it in the wrong direction because you are pressurizing between the two, and what they are acknowledging is, what you are probably doing in the time that the primary O-ring is traversing, you are probably blowing by there and creating blow holes in the putty, but then following up on that the question was, do you recommend reducing the leak check pressure?
DR. KEEL: Can we just go step by step, though, because that is exactly what they are saying here, and in fact that was the advantage that you were mentioning previously of going to 200 psi.
MR. MULLOY: That is correct.
DR. KEEL: That the putty wouldn't mask, and a blow hole would be created, so they are saying we have got concerns about the leak check procedure creating blow holes which requires urgent concern, which requires expedition of previously identified full-scale tests, and what you are saying is, you are reassured by the fact that you were creating the blow holes, and hence not masking the leak.
MR. MULLOY: That is correct, and what I am
saying is, these engineers, and I presume you are going to have Mr. Miller here, will say that he would not recommend reducing the leak check pressure from 200 psi.
DR. KEEL: Well, we will ask him that question, but the point is that certainly they were concerned about the leak check procedures putting blow holes through.
MR. MULLOY: Yes, sir.
DR. KEEL: And that in fact was what reassured you from the standpoint of making sure the primary O-ring was seating, that you wouldn't mask-that the putty wouldn't mask, to assure it wouldn't mask you would go to a higher pressure, and there you were sure that you would get a blow hole through the putty, and hence not mask the leak check.
MR. MULLOY: That is correct, to assure that you have a primary leak check.
DR. KEEL: The penalty was this concern, to the degree that it is concern.
MR. MULLOY: That is correct.
DR. KEEL: They are also concerned about the cavity volume size, and obviously that is a critical quantity to predict erosion. You have to know that presumably with some accuracy to be able to predict
absolutely, as you said previously, on an analytic basis.
MR. MULLOY: I withdrew that absolute.
DR. KEEL: I'll withdraw that. The other thing we found out, Mr. Mulloy, and I am not sure you are aware of it, and I believe your Marshall quality assurance people weren't aware of it, is that after you went to the higher pressure, if you look back at problems with the joints, the nozzle joints, and the problem being blow by or erosion or heat effect, that for the eight flights where you had a 50 psi leak check you had one problem, for the eight flights where you had a 100 psi leak check you had five problems, for the eight flights where you had a 200 psi check you had seven problems.
MR. WEAR: Yes, sir.
[1534] DR. KEEL: That seems to correlate between the problems that these were-that Mr. Miller at least was raising with respect to the leak check procedures, doesn't it?
MR. MULLOY: No, sir. I don't think you can take that single variable and make any conclusions from it. You have to also look at the putty changes, and the changes in the layup of the putty, and probably changes in the roundness of the segments as we begin to reuse
them.
DR. KEEL: Is it enough to cause you concern?
MR. MULLOY: Oh, yes, yes, and that is what was being addressed.
MR. WEAR: The specific question you raise there about does this lead you to think, to have a second thought about some things you have done, yes, that was all addressed by our engineering people and by Thiokol's engineering people, about what is the correlation between leak check pressure, the change in putty manufacture, the changes we have made in putty layup patterns. Those were all questions that were raised, and you have got one item here addressing one particular aspect of it, but that question was raised, and attempts were made between these two engineering organizations to try to correlate and trace out what is the significant factor, and they were unable to do so.
DR. KEEL: I guess the reason for going through all of that is that you are using these two bases, the higher leak check pressure and your ability to analytically predict erosion, as the basis of waiving these launch constraints, and it is fairly clear if you go back through the history of all of this that they introduce problems, too, and the tradeoffs weren't quite that clear that that was absolutely reassuring that this
was a basis upon which you could waive these launch constraints.
DR. WALKER: Do you agree that the primary cause of the erosion is the blow holes in the putty?
MR. MULLOY: I believe it is. Yes.
DR. WALKER: And so your leak check procedure created blow holes in the putty.
MR. MULLOY: That is one cause of blow holes in the putty.
DR. WALKER: But, in other words, your leak check procedure could indeed cause what was your primary problem. Didn't that concern you?
MR. MULLOY: Yes, sir.
DR. COVERT: Mr. Mulloy, what happens if you have, make a leak check and you discover that the primary O-ring leaks? Do you now stop and destack it?
MR. MULLOY: Yes, sir.
DR. COVERT: And then put a new O-ring in?
MR. MULLOY: Yes, sir.
MR. WEAR: You clean it all up back to zero and reputty.
GENERAL KUTYNA: Larry, let me follow Art's question. Blow holes in the putty cause erosion. The Titan joint doesn't have a lot of putty. Does the Titan joint show erosion?
MR. MULLOY: Yes, it does, a higher rate than is happening on the SRM.
GENERAL KUTYNA: It is not necessarily blow holes that are causing erosion, but could I pursue just a quick line of questioning? This joint, is it the only one of this type in the industry?
MR. MULLOY: No.
GENERAL KUTYNA: Which one is the next closest to it?
[1535] MR. MULLOY: The Titan.
GENERAL KUTYNA: As a matter of fact, this was fairly highly derived from the Titan.
MR. MULLOY: Yes, that was the basis.
GENERAL KUTYNA: What are the primary areas where it is the same as the Titan?
MR. MULLOY: It is the same in that it is a tang and clevis pin joint that uses a bore seal O-ring sealing.
GENERAL KUTYNA: It doesn't have the putty?
MR. MULLOY: It has a small amount of putty, but not filling a large gap like we have on the SRM.
GENERAL KUTYNA: Now, isn't it usual in the industry and between agencies that if we have a problem we come running to each other and advise if we have similar pieces of equipment? When we had the inertial
upper stage we worked together very closely on that. When the PAMs failed, we were the program office on PAM and got together on that, and even when you had the burn-through on the nozzle on STS-8 you alerted us about that.
I had the program office for Space Shuttle for two years, 1982 to 1984, and of course was in charge of all Titans at that time, and I never once heard about problems with the O-rings. When I called the program office this morning on the Titans, they had received no prior indications that there was a large problem with the O-rings. Why would that be? Why wouldn't you have notified them?
MR. MULLOY: Well, the approach taken was, I did contact someone from CSD, and I think there is a memo in the stack of documents here that is in response to that back in March of 1984, and there were some interchanges at lower levels between engineers who know each other in the field, but nothing, no official letter that went to Space Division.
GENERAL KUTYNA: But Larry, here you are, and you have grounded this thing almost. You have a flight constraint that you can't fly unless you clear it, last year, 1985. I in headquarters was not aware of it. The program manager out at Space Division was not aware of
it. It just wasn't made that big a thing, and yet you have got a flight constraint. Why is that? MR. MULLOY: It would be because as far as I know I didn't contact you or the program manager directly to share that information or to request information on the Titan experience. There were at other levels some interchanges in that regard, and I think Larry Wear can talk to them.
MR. WEAR: Yes, there were. Specifically, Miller put his signature on this particular letter, has quite particular and specific contacts within the ETC echelon regarding the Titan. There was considerable exchange of data of what Titan has done and what Titan's experience is and why they think it is. You might want to ask him about that.
GENERAL KUTYNA: Who in the Air Force?
MR. WEAR: I know of no specific contacts with the Air Force per se, but we did hold a session at Thiokol. We spent a whole day with Joe Banna from Aerospace, and a group of six or seven people, which were principally from Aerospace and some other organizations.
GENERAL KUTYNA: When was this?
MR. WEAR: About a year and a half ago at Thiokol, where we discussed, we laid out for them, we spent a whole day laying out with them all of the problems we have had with our [1536] joint, the design details of it, the idiosyncrasies of it, and we asked them to exchange back to us comparable data from the Titan program, and their thoughts of what we were doing, and to be honest with you, I never received a response from Mr. Banna.
GENERAL KUTYNA: Now, during this time, a year and a half ago, was there not a competition for a follow-on launch vehicle to complement the Shuttle? Were you not aware of that?
MR. WEAR: There were some discussions, yes. There was an exercise for another
GENERAL KUTYNA: Larry, did Marshall put an entry in the competition?
MR. WEAR: Yes, there was a Marshall design submitted to that.
GENERAL KUTYNA: And is that sort of unusual because Marshall was competing against two industry guys?
MR. WEAR: That is not normal for Marshall's line of business.
GENERAL KUTYNA: And what was the construction
of that competitor? What was the makeup of that competing rocket?
MR. WEAR: It would have been United Technologies.
GENERAL KUTYNA: But what were the segments of that rocket made or.
MR. WEAR: They were derived from our SRM.
GENERAL KUTYNA: There were three solid rocket boosters?
MR. WEAR: That is right.
GENERAL KUTYNA: That is all.
MR. ACHESON: I have one question. I have never understood why it was that cold was not originally thought to be a variant which could be important or might conceivably become a launch constraint. Reading these documents, we see an explicit flag in the night readiness review presentation from Thiokol, February of 1985, on 51E pointing out that low temperature enhances the probability of erosion.
You have the presentation at headquarters in August of 1985 that made it plain that resilience was an important factor in the function of the O-ring. If you accept that concept, it is very hard for me to see why a lot of difference in temperature wouldn't be understood
to make substantial difference in the functioning of the O-ring, and when you come to the coldest day you have ever had for a launch by a margin of 20 degrees, why it was not thought important, and yet you have the description of the telecon in which reference was constantly made to the higher temperatures at which erosion had been observed, as if this somehow established that temperature was not a discriminator.
I have never understood that. You are dealing with an elastomeric material which, certainly, you don't have to be an engineer to know is affected by temperature, I have difficulty understanding, and perhaps you could explain, why that has been a missing factor in these equations, seemingly, all through the history of this project.
MR. MULLOY: Are you speaking that it seems to be missing in the Marshall documentation? Where concerns have been expressed in other areas, concerns are not expressed in the cold temperature areas?
MR. ACHESON: This is a concern Marshall apparently had for that problem.
MR. MULLOY: I can give you my thought process, and again, that can be questioned. Looking at the total context of Thiokol's conclusions on 51E, where it was concluded that cold temperature does affect or
enhance the possibility of erosion. If you continue, though, to read that total Thiokol presentation, the conclusion is that what they saw on 51C really fits within previous experience, and so it turns right around and kind of refutes that there is much different, even though the temperature was a little colder.
They further conclude that that type of erosion can be expected on subsequent flights no matter what the temperature is, I believe, if I am remembering what they said in that presentation to me.
DR. KEEL: Can I read you the conclusion, so we all know what it was? [Ref. 5/2-1]
MR. MULLOY: 51E.
DR. KEEL: It is the 51E flight readiness review, after 51C, of course, which was the previous coldest launch, and I won't read all of this, because it is an extensive review they gave to you.
MR. MULLOY: It is.
DR. KEEL: Based upon an urgent request from you to review all of the O-ring history that you sent out a few days after the 51C launch? [Ref. 5/2-12]
MR. MULLOY: All my messages are urgent, by the way.
DR. KEEL: I will go through your conclusion, and I recognize there is a lot more to the analysis and the scenarios in trying to explain what happens, but the conclusion is that STS-51C is consistent with the erosion data base. That means that the erosion there presumably in your vernacular hasn't exceeded the previous experience or is greater than worst case, but then in a subbullet they say low temperature and hence probability of blow-by. STS-51C experienced worst case temperature in Florida history, and so that is the conclusion they draw there.
CHAIRMAN ROGERS: Now, getting back to Mr. Acheson's question in light of that statement and others on weather-why wasn't more consideration given to the weather, particularly on Flight 51L, why wasn't-why didn't that become extremely serious in light of all the other things that the documents reflect?
MR. MULLOY: Yes, sir. I had started to explain my thought process in that, and I go back to the extensive look at that we did with 51C after experiencing the worst, the coldest temperature in Florida history, that the erosion was not outside of what we had experienced at 80 degrees, and that also the conclusion that that type of erosion which we all understood could be anticipated in the future because we
know we can get paths through the putty and hot gas impingement through those paths will cause O-ring erosion, and the conclusion that 51E could be flown under those circumstances.
The next thing in the August presentation, if you look at that total presentation, I think there is maybe one short reference again about the same thing that was covered in the 51E FRR, and then you get to the night of the discussion of January 27th, where the engineers were essentially citing the same data relative to the effect of resiliency, and that their concern was increased blow-by of the primary O-ring seal, which I took to be what engineers always do, realize what risk you are taking because you could have increased blow-by, thus higher erosion.
The ultimate conclusion of that was that the data did not specifically conclude that you would have higher erosion on the primary O-ring seal, and the second part of the conclusion was that during that pressurization sequence the joint was redundant. Now, that was the thought process which, as I say, has been and will continue to be questioned.
[1538] DR. KEEL: Can I just ask again, Larry, for a clarification? You talk about erosion, of course, but the footnote of the conclusion was a reference to
blow-by. This was the worst case of blow-by.
MR. MULLOY: That is correct.
DR. KEEL: It was the largest arc. There was black soot, black grease, and what it said was, it enhances the probability of blow-by.
MR. MULLOY: Yes, and it further states the condition is acceptable, I believe.
DR. KEEL: That is correct. You are exactly right.
MR. MULLOY: And that was my basis.
DR. KEEL: They do say that the condition was acceptable throughout all of the flight readiness reviews, that the risk was acceptable.
CHAIRMAN ROGERS: But they don't say that the worst condition would be acceptable. In other words, they don't say if the weather was even worse it would be accepted, and when the weather got worse they said, we recommend against launch, and so I have trouble with your logic.
MR. MULLOY: I understand, sir.
DR. KEEL: Could I ask just a couple more questions on this, Mr. Chairman, since we are on this subject?
What we have determined from our individual staff investigations and staff interviews, Mr. Mulloy, is
that apparently after this extensive briefing that Marshall gave to you-excuse me, that Morton Thiokol gave to you, and in fact at your request to a large extent, my understanding is that the center board-and I assume that's Dr. Lucas-wasn't given that briefing.
MR. MULLOY: No, they were given a summary briefing.
DR. KEEL: My understanding is, in fact, that the erosion problem wasn't mentioned.
MR. MULLOY: Yes, sir, on 51E it was mentioned At the Level-it was discussed at the Level 1, 2 FRR at the Marshall center board, the Shuttle projects office.
DR. KEEL: Can you tell us what was said at the center board?
MR. MULLOY: No, sir. It is the type of chart, I think, that Mr. Rogers was referring to, which is a summary chart.
DR. KEEL: Well, I know it was mentioned at Level 1. Maybe you could provide that, because we have been unable to get that center board briefing. We have Level 1, but our information was in fact from people there that it was not mentioned, so if you could clarify for the record, that is the purpose.
MR. KEHRLI: We do have the center board
briefing, and in that briefing I have not seen the chart. If you could provide it, that would be fine.
DR. KEEL: That is even more important. We have a briefing that doesn't show it, but then the next question, and then a final one on this is, you are right, at the Level 1 it was mentioned, but in fact you had this extensive briefing, which was made up of six or eight charts, I guess, from Morton Thiokol going into the whole history of ring erosion and mentioning that conclusion I just mentioned.
[1539] Then at the Level 1 briefing it was condensed down to one entry under problem summary. That basically said evidence of hot gas passed primary O-rings on two case joints, and then concerned mission safety resolution acceptable risk because of limited exposure and redundancy. Can you explain why?
MR. MULLOY: Does it reference another flight readiness review?
DR. KEEL: It references STS-41C, flight readiness review.
MR. MULLOY: That was a more extensive review then that documents that rationale again, which is the rationale for the limiting of the erosion, and the Thiokol conclusion, of course, that what they saw on 51-C was consistent with the previous experience, and it
could be anticipated on subsequent flights. So you would have to go to the 41G FRR.
DR. KEEL: I have got the -
MR. MULLOY: I'm sorry.
DR. KEEL: It is two charts that were mentioned there on 41C, and they didn't go through all of the O-ring history, of course, because that was several missions before that Thiokol had presented to you, so it made a reference to that, and presumably you didn't have the charts there, and secondly there were two charts, and they didn't reflect everything that had happened since 41C flight readiness review, and so I guess it is still difficult to understand from my viewpoint why a more extensive reference wasn't made to this problem.
MR. MULLOY: Well, I can give you-there is a reason. There is a very practical reason for that, and that is that the Level 1 FRRs are generally limited to four hours and not two days. So there is a screening and a summary of the information as it goes forward. You have the total FRR packages. You will see the FRR I take is this-as you will see the FRR that goes to the shuttle is this thick, and you will see the one that goes to the center is this thick, and you will see the one that goes to Level 2 and 1, a very abbreviated form
of that, and then what happens is, folks who take those briefings ask and assign actions usually say, I would like to hear more about that and have the total briefing, in which case we accommodate that outside the flight readiness review activity, but that is the practical reason.
DR. KEEL: I understand that when you condense from six to eight pages to one bullet it is a little difficult to understand, especially when you actually trigger the assessment based upon an urgent request message sent back to Mr. Wear, in fact, who then presumably relayed that to Morton Thiokol. The other aspect, I guess, that is puzzling is that you-even that one bullet, the resolution is based on one acceptable risk, but also redundancy.
Can you explain how redundancy is a factor for resolving this concern when Criticality 1 says it is not redundant?
CHAIRMAN ROGERS: Just to take that question a little more directly, doesn't that fly directly in the face of the Criticality 1?
MR. MULLOY: No, sir. I don't think so, and I know that we have had-you all have had difficulty understanding that.
CHAIRMAN ROGERS: I sure do.
MR. MULLOY: I don't know that we will have any more success with it, but it is again-I can see your interpretation of the words in the CIL. I would acknowledge that from-that those words, the wording could have been clearer. The intent was to show what is the physical phenomenon. The physical phenomenon is that after motor pressurization under "worst case [1540] conditions," the secondary O-ring may not be in a position to seal if called upon to do so by failure of the primary O-ring.
CHAIRMAN ROGERS: And the conclusion was, it was not redundant.
MR. MULLOY: After motor pressurization. That is the key. After motor pressurization. Those words are in there, and the sentence structure is probably very poor, but the fact is that on all flights that we have flown to date, with the exception of one case joint on STS-4, given the dimensional tolerances that exist, that with after motor pressurization, the secondary O-ring still has positive squeeze on it, and is indeed a redundant seal.
CHAIRMAN ROGERS: So you relied on redundancy then ever since?
MR. MULLOY: No, sir. We relied first on the test and analysis that said that the primary O-ring
erosion would not cause failure of the primary O-ring to seal. That is the first thing. The second thing then is, one has redundancy.
CHAIRMAN ROGERS: So you relied on redundancy since then?
MR. MULLOY: Yes, sir.
MR. KEHRLI: Mr. Chairman, there is one document that addresses that specific question that I would like the witnesses to examine, and that document is from Thiokol, from Brian Russell to J.W. Thomas. It is Attachment 32 to the O-ring history, and I am wondering if you would care to expand on your answer in light of that document, which specifically addresses the timing function of the secondary seal and also raises resiliency as an issue. [Ref. 5/2-13]
CHAIRMAN ROGERS: Try to keep your voice up. What is the question?
MR. KEHRLI: The question is, I am wondering if the witnesses would respond to the same question with regard to the timing function of the O-ring secondary seal in light of the letter from Brian Russell to Jim Thomas at Marshall.
CHAIRMAN ROGERS: Do you conclude that this letter is contrary to what Mr. Mulloy just said?
MR. KEHRLI: Well, that is what I am
wondering, whether it is or not, in light of Mr. Mulloy's interpretation.
MR. MULLOY: I have seen this memo. I first saw the front page. I didn't think I had. But if you would go to the second question, if the primary O-ring does not seal, will the secondary seal seat in sufficient time to prevent joint leakage? Answer, on the next page, MTI has no reason to suspect that the primary seal would ever fail after pressure equilibrium is reached, i.e., after the ignition transient. If the primary O-ring were to fail from zero to 170 milliseconds, there is a very high probability the secondary O-ring would hold pressure.
Since the case is not expanded appreciably at this point, if the primary seal were to fail from 170 to 330 milliseconds, the probability of secondary seal holding is reduced. From 330 to 600 milliseconds, the chance of the secondary seal holding is small. This is a direct result of the o-ring's slow response compared to the metal case segments as the joint rotates.
That is the same thing, I believe, as was in the August 19th briefing here at NASA headquarters, and so if your question is, does that change the conclusion relative to the secondary seal being able to hold
MR. KEHRLI: Let me be specific with the
question. The question is, and you have repeated it several times, and other witnesses have as well, that this redundancy is affected by the worst ca se tolerance. The secondary O-ring is still there so long as you don't have bad hardware.
MR. MULLOY: Yes.
MR. KEHRLI: There is no mention of that in the answer to this question.
MR. MULLOY: That is correct.
MR. KEHRLI: What is your understanding? Does this answer incorporate that worst case tolerance?
MR. MULLOY: Yes, I think so. I think it does. The worst case would be that you start out with minimum O-ring squeeze of 7.5 percent, and now what you are doing is reducing the squeeze, and at 170 to 330 milliseconds that squeeze is reduced just somewhat, and then the joint rotates to 32/1,000ths. If you started at 20/1,000ths you don't have any squeeze on it, and that minimum squeeze, though, would only occur in hardware that is outside of the base, as we have flown, is my understanding from Thiokol.
DR. WALKER: What do you understand by the term "high probability?"
MR. MULLOY: I think my understanding would be that the high probability is related to a case where
either the O-ring is not tracking the metal due to reduced resiliency or a case where you started with a minimum 20/1,000ths squeeze, and due to tolerances you have-well, you started with a minimum 20/1,000ths with tolerances. You rotate 32/1,000ths. You don't have any squeeze at all.
DR. WALKER: I don't think you understood my question. If, for example, I were to say to the President that I think the Shuttle has a high probability of working, do you think he should have the program continue in that state? I mean, is that a precise enough and sufficiently safe enough situation that he should continue to fly? What does high probability mean? Does that mean 75 percent, 80 percent, 82 percent?
MR. MULLOY: I don't know. I can't quantify that.
DR. WALKER: But surely you are basing your decision to proceed on this assertion that the secondary seal has high probability of working.
MR. MULLOY: Yes, sir, and the reason was that the secondary seal would be energized in the zero to 170 millisecond or 330 millisecond time frame.
DR. WALKER: Does that mean 90 percent of the time or 70 percent of the time?
MR. MULLOY: I don't know, sir.
DR. WALKER: But you are basing your decisions on that.
CHAIRMAN ROGERS: Dr. Walker's question is a very good one in that it said, if we report to the President that we think he should continue the program because it has a high probability of being safe, would that be satisfactory, or there is a high probability that it is not unsafe, and maybe that is not your question.
DR. WALKER: To me, a high probability is not good enough for the operations of something like this.
CHAIRMAN ROGERS: Particularly when you have a constraint on it, and you really haven't solved the problem, and you are still speculating on whether it is going to work under certain conditions or not.
[1542] Well, unless there are any other questions--
MR. RUMMEL: I have one, just to return for a moment, if I may, to an earlier point.
If I understood you correctly, you indicated that out of roundness was not considered when evaluating the squeeze or the relative fit of the segments.
MR. MULLOY: That is my understanding, sir.
MR. RUMMEL: But you, of course, understood and knew that out of roundness to varying degrees
existed from Day One due to shipping and so forth, as has been explained a number of times, that these items were out of round, and some required squeeze and some not.
MR. MULLOY: Yes, sir.
MR. RUMMEL: And I guess indeed that is one of the reasons for the destacking operating at the moment, is to see whether any out of roundness might have adversely affected the seals.
MR. MULLOY: Yes, sir.
MR. RUMMEL: My question is, knowing that, why wasn't it considered, why wasn't out of roundness measured and taken into account, or why wasn't it taken into account in your calculations when you were estimating, or your judgmental considerations with respect to the adequacy of the O-rings.
MR. MULLOY: It was an obvious oversight, sir.
MR. RUMMEL: It is a pretty obvious oversight, isn't it?
MR. MULLOY: Yes, sir, it is.
CHAIRMAN ROGERS: Okay, if there are no further questions, we will break for lunch.
(Whereupon, at 12:15 p.m., the hearing was recessed, to reconvene at 1:10 p.m. of the same day.)
[Please note that some of the titles to the references listed below
do not appear in the original text. Titles are included to identify
and clarify the linked references- Chris Gamble, html editor]
[1543] [Ref. 5/2-1 1 of 4] NASA (MSFC) Letter dated Sept. 15, 1980. Subject:
Assigning Launch Constraints on Open Problems Submitted to MSFC
PAS-.
[1544] [Ref. 5/2-1 2 of 4] NASA (MSFC) Letter dated Sept. 15, 1980. Subject: Assigning Launch Constraints on Open Problems Submitted to MSFC PAS- (continued). [Ref. 5/2-1 3 of 4] Letter dated Sept. 15, 1980. Subject: Assigning Launch Constraints on Open Problems Submitted to MSFC PAS- (continued).
[1545] [Ref. 5/2-1 4 of 4] Letter dated Sept. 15, 1980. Subject: Assigning Launch Constraints on Open Problems Submitted to MSFC PAS- (continued). [Ref. 5/2-2 1 of 3] MSFC Problem Assessment System: Problem Title: O-Ring Erosion in the Case to Nozzle Joint.
[1546] [Ref. 5/2-2 2 of 3] MSFC Problem Assessment System: Problem Title: O-Ring Erosion in the Case to Nozzle Joint.(continued) [Ref. 5/2-2 3 of 3] MSFC Problem Assessment System: Problem Title: O-Ring Erosion in the Case to Nozzle Joint. (continued).
[1547] [Ref. 5/2-3 1 of 6] MSFC Problem Assessment System: Problem Title: Segment Joint Primary O-Ring Charred. [Ref. 5/2-3 2 of 6] MSFC Problem Assessment System: Problem Title: Segment Joint Primary O-Ring Charred. (continued).
[1548] [Ref. 5/2-3 3 of 6] MSFC Problem Assessment System: Problem Title: Segment Joint Primary O-Ring Charred. (continued). [Ref. 5/2-3 4 of 6] MSFC Problem Assessment System: Problem Title: Segment Joint Primary O-Ring Charred. (continued).
[1549] [Ref. 5/2-3 5 of 6] MSFC Problem Assessment System: Problem Title: Segment Joint Primary O-Ring Charred. (continued). [Ref. 5/2-3 6 of 6] MSFC Problem Assessment System: Problem Title: Segment Joint Primary O-Ring Charred. (continued).
[1550] [Ref. 5/2-4] Morton Thiokol Letter to L.O. Wear Regarding Closure of Critical Problems Numbers.
[1551] [Ref. 5/2-5] United Technologies United Space Boosters Internal Correspondence. To Larry Mulloy From George Morefield. Subject: Zinc Chromate Putty in SRM Joints.[Second part appears on p.1553, C.Gamble, html editor].
[1552] [Ref. 5/2-6 1 of 2] SRB Critical Items List. [Ref. 5/2-6 2 of 2] SRB Critical Items List.
[1553] [Ref. 5/2-7 1 of 2] Morton Thiokol Letter to Jack Fletcher Regarding Closure of Critical Problems Numbers. [Ref. 5/2-7 2 of 2] United Technologies United Space Boosters Internal Correspondence. To Larry Mulloy From George Morefield. Subject: Zinc Chromate Putty in SRM Joints (continued). [First part appears on p1551, C.Gamble, html editor]
[1554] [Ref. 5/2-8] Letter from L.O. Wear to Joe Kilminster at Morton Thiokol Inc. Subject: SRM Problem Review Board.
[1555] [Ref. 5/2-9] SRM Seal Erosion Problems.
[1556] [Ref. 5/2-10 1 of 2] NASA Routing Slip To Mr. Hardy, Subject: Burned O-Rings on STS-11.
[1557] [Ref. 5/2-10 2 of 2] NASA Routing Slip To Mr. Hardy, Subject: Burned O-Rings on STS-11 (continued).
[1558] [Ref. 5/2-11 1 of 18] Morton Thiokol Inc. (MTI) STS-51E Solid Rocket Motor (SRM-16) Flight Readiness Review.
[1559] [Ref. 5/2-11 2 of 18] SRM-HPM Field Joint. [Ref. 5/2-11 3 of 18] STS-51C (SRM-15) Postflight Hardware Damage Assessment.
[1560] [Ref. 5/2-11 4 of 18] STS-51C (STS-20) (SRM-15) Performance Preliminary Postflight Hardware Damage Assessment. [Ref. 5/2-11 5 of 18] STS-51C (SRM-15) Postflight Hardware Damage Assessment.
[1561] [Ref. 5/2-11 6 of 18] STS-51C (STS-20) (SRM-15) Performance Preliminary Postflight Hardware Damage Assessment. [Ref. 5/2-11 7 of 18] O-Ring Erosion Patterns.
[1562] [Ref. 5/2-11 8 of 18] Analysis of Soot from STS-51C LH Forward Field Joint. [Ref. 5/2-11 9 of 18] History of SRM O-Ring Damage.
[1563] [Ref. 5/2-11 10 of 18] Comparison of STS-51C and STS-51E. [Ref. 5/2-11 11 of 18] O-Ring Erosion Scenario.
[1564] [Ref. 5/2-11 12 of 18] STS-51C (STS-20) (SRM-15) O-Ring Erosion Scenario. [Ref. 5/2-11 13 of 18] HPM Predicted Pressure During Ignition.
[1565] [Ref. 5/2-11 14 of 18] STS-51C (STS-20) (SRM-15) O-Ring Erosion Scenario (continued). [Ref. 5/2-11 15 of 18] Volumetric and Thermal Analysis and Test Results.
[1566] [Ref. 5/2-11 16 of 18] Why do most o-rings not erode and why don't we see soot behind primary o-rings more often? [Ref. 5/2-11 17 of 18] Rationale for Acceptance.
[1567] [Ref. 5/2-11 18 of 18] Flight Readiness Assessment for STS-51E. [Ref. 5/2-12] Message from Larry Mulloy. Subject: 51C O-Ring Erosion RE: 51E FRR.
[1568] [Ref. 5/2-13 1 of 2] Morton Thiokol Inc. (MTI) from Brian Russell, Manager, SRM Ignition System to James W. Thomas, MSFC.Subject: Actions Pertaining to SRM Field Joint Secondary Seal.
[1569] [Ref. 5/2-13 2 of
2] Morton Thiokol Inc. (MTI) from
Brian Russell, Manager, SRM Ignition System to James W. Thomas, MSFC
(continued).
CHAIRMAN ROGERS: Will the Commission come to order, please?
Gentlemen, good afternoon. I guess all of you have been sworn in, and those of you who have not been sworn in, maybe you had better be so.
(Witnesses sworn.)
CHAIRMAN ROGERS: The Commission is coming to the end of its tenure. In fact, we think this is probably the last hearing that we will have, and in the course of preparing our report, we have-our staff has looked at a lot of documents, and we wanted before we closed off the hearings to give you gentlemen a chance to make any comments you want to about the documents, and I guess some of the documents have already been shown to you, and so if that is all right we will just get started and ask questions.
Do you want to say anything?
MR. KEHRLI: I would just like to-there is a book that is in front of you, the red book, that some of you don't have, with additional copies of documents. What we will be referring to is Documents Number 2 and Number 3 in the very back of the book. They are tabbed according to number.
And what I would like to start off with are those two specific documents which are Marshall problem assessment system tracking reports, and one was started after STS-41B in 1984, and that is Tab Number 3, and Tab Number 2 was the problem report that was started after the 51B nozzle O-ring erosion problem, which was discovered at Thiokol on June 25th, 1985, and you will see from that particular problem that a launch constraint was applied as a result of the 51B O-ring nozzle erosion problem. [Ref. 5/2-14] [Ref. 5/2-15]
What I would like to ask you gentlemen is if you have ever seen those two Marshall tracking reports before, or are you familiar with them?
MR. RUSSELL: I am Brian Russell. I'm familiar with them. I can't recall if I have seen exactly these, although I would receive in the mail in my office a
monthly problem report which would be similar to that, and whether it was this one exactly I can't say without checking my notes back at the plant, but we would get these every month, and they would contain these problems as well as others that were being tracked.
MR. KEHRLI: Now, some of these entries, isn't it correct to say that the entries themselves, the information came from Thiokol that was eventually placed on these documents?
MR. RUSSELL: That is correct.
MR. KEHRLI: How would that work? Could you tell the Commission how that system worked, please?
MR. RUSSELL: We have our reliability engineering department, who is responsible to complete the monthly problem report, and in addition to that we have our monthly problem review board telephone conference with NASA and the contractors, of which we are apart, and the monthly problem review or the monthly problem report that reliability prepares they get the information from engineering or from the program office as necessary to complete their status of what has happened during that month, whether the problem originated that month or what has been done to close the problem out, and that is submitted every month, and I for one do review that before it is submitted to the
Marshall Space Flight Center, and so much of the information that I would read in these reports would be the same information that we had given in that monthly problem report or over the telephone on the teleconference.
CHAIRMAN ROGERS: Mr. Russell, when you say close the problem out, what do you mean by that? How do you close it out normally?
MR. RUSSELL: Normally, whether it takes engineering analysis or tests or some corrective action, a closeout to the problem would occur after an adequate corrective action had been taken to satisfy those on the problem review board that the problem had indeed been closed out. That is the way that that happens. For example, we had found a loose bolt on the recovery one time, and we had to take corrective action in our procedures and in the engineering to make sure that that wouldn't happen again, and then to verify that corrective action, and at that point that problem would be ready to be closed out. It generally involves a report or at least a mention by the review board stating what had been done to adequately close it out, and then it is agreed upon by the parties involved.
CHAIRMAN ROGERS: Without reference to a particular flight, if you had a problem with erosion and
blow-by, how would you expect it would be closed out? How would you normally close that out?
MR. RUSSELL: The normal closeout procedure would be after there had been changes to prevent the problem or to control the problem or to prevent it from recurring. That would be the normal system.
MR. KEHRLI: Now, as I mentioned earlier, one of these problems, Tab 2, the problem on the nozzle erosion, the O-ring erosion after 51B, did contain a launch constraint. Is that your understanding?
MR. RUSSELL: Well, my understanding was, until we adequately handled any of the O-ring erosion problems in the subsequent FRR, that I don't know if it would be so formal as a launch constraint, but I know that when we discovered any O-ring erosion or soot blow-by or whatever might have been beyond our experience base, there was always some analysis or some justification [1572] in the flight readiness review to continue to fly, and I guess I didn't realize that that one was any different than the others.
MR. KEHRLI: What do you understand a launch constraint to mean?
MR. RUSSELL: My understanding of a launch constraint is that the launch cannot proceed without adequately-without everyone's agreement that the
problem is under control.
CHAIRMAN ROGERS: Under control meaning what? You just said a moment ago that you would expect some corrective action to be taken.
MR. RUSSELL: That is correct, and in this particular case on this 51B nozzle O-ring erosion problem there had been some corrective action taken, and that was included in the presentation made as a special addendum to the next flight readiness review, and at the time we did agree to continue to launch, which apparently had lifted the launch constraint, would be my understanding.
DR. WALKER: What was the corrective action which was taken in that case?
MR. RUSSELL: One of the corrective actions was an increase in the leak check pressure because one of the ways we thought that could happen, our analysis said that the blow-by across the O-ring which added to the erosion of that particular O-ring, started right at ignition, indicating the O-ring didn't seal right from the very beginning.
And one of the ways that we thought that could have happened was that the putty could have masked our leak check, and we thought it was of utmost importance to have a verified primary O-ring and so we increased leak check pressure to 200 psi to make sure that we would blow through the putty, realizing that blow holes are not desirable either, but yet it is more important to know that you have a good O-ring and have some putty blow through than otherwise.
DR. COVERT: Mr. Russell, where did 200 psi come from? Is that twice 100?
MR. RUSSELL: No. We did some bench testing of some putty and some gaps that showed we tried at 50, 100 and 150 psi and since we have a ten-minute stabilization period, the question was can the putty hold these certain pressures for ten minutes? And in no case did 150 psi, in no case could the putty hold 150 psi for ten minutes. And we put a factor on top of that
of another 50 psi, and that was the basis for that.
DR. COVERT: Thank you.
DR. WALKER: The analysis that some of our staff has done suggests that after you increase the test pressure to 200 pounds, the incidence of blow-by and erosion actually increased.
MR. RUSSELL: We realized that.
DR. WALKER: Did you realize that only after the accident?
MR. McDONALD: Could I try to answer some of that because I was very involved in this, and in fact this is the first time I realized that was a launch constraint.
CHAIRMAN ROGERS: Could you speak a little louder, please?
MR. McDONALD: This is the first time I realized that this was flagged as a launch constraint and the others weren't, because the problem of the O-ring erosion was predominant all the way from STS-2. The uniqueness of the 16A nozzle which was 51 Baker, that was the first time we had violated a primary O-ring. And that erosion that caused that violation is different [1573] than we had seen previously and was different for a significant reason, in that it had blow-by at ignition which caused the erosion of the primary
O-ring.
It just wasn't jet impingement on the sealed O-ring which was what we had seen before. And because of that jet impingement on the unsealed O-ring, it made it unique. And we had to understand for the next flight, before we could launch it, as to what happened to that flight that we don't think will happen to the next one.
The point that Brian made was that in looking at the records for the assembly and leak check of that flight, it was the last one which we had one with 100 psi stabilization pressure by the leak check. That was the last one.
And we knew that we had to have blow-by at ignition to consume as much of the O-ring as we consumed on that particular flight. Therefore, we felt that we missed the leak check on that flight because the putty may well have masked it, or the fact that the groove in the nozzle is actually wider than it is in the field joint, and it moves farther, and therefore there is a higher probability that you may lose the leak check there because when you run the leak check it is on the wrong face.
It has got farther to travel, and either contamination or whatever, when you finally pressurize it to the other side, maybe it won't seal when it
pressurizes the right side where it may have sealed and passed the leak check.
So the conclusion was that may have been unique to that assembly and that all subsequent flights have had the 200 psi leak check to eliminate the potential for the putty masking it. And that, I think, was the basis for proceeding on with the next flight.
Now, I notice this is dated February 1986. I've seen a lot of these and I don't recall that that was a flight constraint earlier.
CHAIRMAN ROGERS: I think it was, wasn't it?
MR. KEHRLI: It's my understanding that that is a February 26th printout, but from those dates on the left-hand column there, the dates of the entry, you will see that the launch constraint was applied in July of 1985.
CHAIRMAN ROGERS: Let's ask some specific questions.
Did you all know that there was a launch constraint placed on the flights in July 1985?
MR. McDONALD: Brian is right as far as every erosion we had. We had to address it prior to the next flight, and I always felt that it could always become a launch constraint if it was not adequately understood and explained why this problem could not get worse on
the next flight.
CHAIRMAN ROGERS: But really my question is: Did you gentlemen realize that it was a launch constraint?
MR. RUSSELL: I would like to answer for myself I didn't realize that there was a formal launch constraint on this one, any different than some of the other erosion and blow-by that we had seen in the past.
MR. EBELING: I agree.
CHAIRMAN ROGERS: Isn't there a letter recommending that the launch constraint be taken off from you?
MR. KEHRLI: No. There's a letter recommending the problem be closed, the
[1574] CHAIRMAN ROGERS: What's the difference? [Ref. 5/2-16 through 19]
MR. KEHRLI: Well, let's ask about that letter. Mr. Russell, you wrote a letter, did you not, or a memorandum indicating that the problem should be closed.
Could you explain to the Commission what you meant by that?
MR. RUSSELL: Yes. In our December telephone call on the Problem Review Board and I can't remember the date it was around the 9th or so there was a request to close the problems out and particularly the ones that had been open for a long time, of which this
was one, and a long time meaning six months or more.
There was a request from the Director of Engineering, as I recall it, that we close these problems out.
CHAIRMAN ROGERS: What did you think that meant? You've got a lot of problems; close them out. I don't understand that language.
DR. WALKER: That was the Director of Engineering at Marshall?
MR. RUSSELL: Yes, at Marshall Space Flight Center. Now, he wasn't in that telephone call. My understanding is what they told us and my recollection was that Mr. Kingsbury would like to see these problems closed out.
Now, the normal method of closing them out is to implement the corrective action, verify the corrective action, and then the problem is closed, it comes off the board and is no longer under active review.
CHAIRMAN ROGERS: That's what you just said. Now, you get a request to close it out. What did you think then?
MR. RUSSELL: Well, it was expanded. As we talked in that telephone call, we were looking for ways to close out any and all of the problems and we talked
about each one separately.
DR. WALKER: Is there any other way besides fixing them?
MR. RUSSELL: Well, the way that we discussed is what I'm leading to; that we talked about the fact that this telephone call and monthly Problem Review Board as well as the Problem Report, really in our eyes and in those apparently of the people on the phone call at the Marshall Space Flight Center, was not adding any more visibility to the problem.
The fact was that we had a task force created with full-time people on it. We had constant reviews going on. We had an active program plan that we were working and were doing some test analyses and so forth to come to a solution of the problem.
And so we talked about that and pretty much jointly agreed that we would not be losing any visibility by closing out, and agreed that that would be rationale to close it.
CHAIRMAN ROGERS: I have trouble. Losing any visibility to close it. What does that mean?
MR. RUSSELL: What I mean to say is that from my understanding of why there would be a Problem Review Board is to make sure that things don't inadvertently become lost in the system or ignored for any reason;
that you have a problem and you adequately attack it, fix it, and implement it and verify it.
And that is my understanding of the purpose of such a board. On this problem, however, I personally felt that there was no chance of possibly ignoring the problem; that the visibility - [1575] what I mean by that is that management at NASA and management at Thiokol were adequately aware of the problem and also understood what was being done about it to fix it.
And that is what I mean by visibility.
CHAIRMAN ROGERS: What was being done to fix it?
MR. RUSSELL: Well, we had a task force created of full-time people at Thiokol, of which I was a member of that task team, and we had done some engineering tests. We were trying to develop concepts. We had developed some concepts to block the flow of hot gas against the O-ring to the point where the O-ring would no longer be damaged in a new configuration.
And we had run some cold gas tests and some hot gas motor firing tests and were working toward a solution of the problem and we had some meetings scheduled with the Marshall Space Flight Center. We had weekly telephone calls where we statused our progress and there was a team at Marshall also of engineering
people who were monitoring the things that we were doing to fix the problem with the goal of implementing a fix in our qualification motor No. 5, which was scheduled at that time in January, this time frame being about the December time frame of last year.
CHAIRMAN ROGERS: Can I interrupt? So you're trying to figure out how to fix it, right? And you're doing some things to try to help you figure out how to fix it.
Now, why at that point would you close it out?
MR. BOISJOLY: I think if I may
CHAIRMAN ROGERS: Just let me finish. Why would you write a letter saying let's close it out now?
MR. RUSSELL: Because I was asked to do it.
CHAIRMAN ROGERS: I see. Well, that explains it.
MR. RUMMEL: That explains it, but it really doesn't make any sense. On the one hand you close out items that you've been reviewing flight by flight, that have obviously critical implications, on the basis that after you close it out, you're going to continue to try to fix it.
So I think what you're really saying is, you're closing it out because you don't want to be
bothered. Somebody doesn't want to be bothered with flight-by-flight reviews, but you're going to continue to work on it after it's closed out.
Is that right?
MR. RUSSELL: I would like to respond to that. Now, this was not a flight-by-flight review. This was a month-by-month review, regardless of what was going on in the flight cycle.
MR. RUMMEL: Excuse me. But each flight up till then for some time had been reviewed. You reviewed prior experience and this had been on a list which really mandated that each flight be reviewed with respect to the O-ring problem. So I assume you did in fact review each flight.
MR. RUSSELL: Yes, but we were not requesting that it be taken off of that review. That was a separate set of reviews, and in fact it was one of the rationale items for removing it from the Problem Review Board. That was just one activity in the system that we had made a move to take it off of that review system, the Problem Review Board.
It had not been removed from the Flight Readiness Reviews.
DR. FEYNMAN: It says here that there was a formal reporting system used to evaluate the problem
resolution part of it, and that is why you wanted to close it out of the problem assessment system.
Maybe Mr. Boisjoly could explain that.
MR. BOISJOLY: Basically, closeout may be a poor term because nobody was going to close it out. Instead of tracking the same problem through two or three different channels, we were going to track the problem now in the highlight of the task team, plus the Flight Readiness Reviews, so there was no need to track it in several different locations. It was the same problem.
So we were tracking it as it came up in Flight Readiness Reviews and we were statusing and tracking it within the Seal Team. So it was a simple matter of bookkeeping to take it out of one area because we were already doing it in another area.
DR. SUTTER: Well, was this task force, did it have the priority and ability to really charge out and move on this problem? Was it receiving top management attention?
MR. BOISJOLY: As top as we could get; yes. We had statuses every week. And we were reporting the weekly activity.
DR. SUTTER: Well, there were memos written saying they couldn't get parts and you couldn't get
tests.
MR. BOISJOLY: That's right, but it was just to take it off of one area instead of tracking the same thing in two manners with two different groups of people, two sets of manpower, to put it and consolidate it,
DR. SUTTER: Well, taking it off of that list, maybe that would remove it as a pressure point in upper management to move out and make sure this task team was given the ability to do their job.
MR. BOISJOLY: No. If anything, the task team turned the gain up on it a little bit more, as indicated by my memos.
DR. SUTTER: It is just hard to know why, taking it off of any list, when it was one of the more critical items. Why not leave it there and have everybody get bugged by it?
CHAIRMAN ROGERS: Well, Mr. Russell said they did it because they asked him to. It's as simple as that, I guess.
MR. EBELING: It was being tracked in three separate locations, three written and teleconed descriptions every month or weekly in many cases, in the case of the O-ring Task Force as it was called, and it was a poor expenditure of people's time tracking it in
three different places simultaneously.
CHAIRMAN ROGERS: But I say, though, so that you understand, the confusion on the part of some of us is we have a constraint which was placed on July 1985 and Marshall and Mr. McDonald is listed as being one of those who knew about it.
And Mr. Mulloy says-and he testified this morning-he has the right to waive the constraint every time, so on all of these flights he waived it.
Now, during that time, as I understand it, you were doing some tests and so forth. And so each time nothing happened. I mean Mr. Russell said you were working on the problem, but as far as the joint is concerned, things continued on just the way they had been going and you had been having a lot of trouble.
[1577] This was a very troubled joint, obviously. And getting back to the group that is working on the problem, trying to fix it, I don't know, my word is repair it or fix it, do something about it, apparently that wasn't going very well either because there's a memo here dated October 1, 1985 on Tab 33 from Mr. Ebeling which starts out-in connection with this it says: "Help. The Seal Task Force is constantly being delayed by every possible means. People are quoting policy and systems and not working on them. Marshall is correct in
saying that we do not know how to run a development program." [Ref. 5/2-20]
And later on, after a lot of discussion about O-rings and putty he says: "The allegiance of the O-ring Investigation Task Force is very limited to a group of engineers numbering eight to ten. Our assembled people in Manufacturing and Quality have the desire, but are encumbered with other significant work. Others in Manufacturing, Quality, Procurement, who are not involved directly, but whose help we need, are generating plenty of resistance. We are creating more instructional paper than engineering data. We wish we could get action by verbal request, but such is not the case. This is a red flag."
Now, how does that happen? And then there are constraints on the launches. The constraints were waived and there seems to be total lack of coordination between Marshall and Thiokol.
MR. KILMINSTER: Mr. Chairman, if I could, I would like to respond to that.
In response to the concern that was expressed-and I had discussions with the team leader, the task force team leader, Mr. Don Kettner, and Mr. Russell and Mr. Ebeling. We held a meeting in my office and that was done in the October time period
where we called the people who were in a support role to the task team, as well as the task force members themselves.
In that discussion, some of the task force members were looking to circumvent some of our established systems. In some cases that was acceptable; in other cases it was not. For example, some of the work that they had recommended to be done was involved with full scale hardware, putting some of these joints together with various putty layup configuration; for instance, taking them apart and finding out what we could from that inspection process.
DR. SUTTER: Was that one of these things that was outside of the normal work, or was that accepted as a good idea or a bad idea?
MR. KILMINSTER: A good idea, but outside the normal work, if you will.
DR. SUTTER: Why not do it?
MR. KILMINSTER: Well, we were doing it. But the question was, can we circumvent the system, the paper system that requires, for instance, the handling constraints on those flight hardware items? And I said no, we can't do that. We have to maintain our handling system, for instance, so that we don't stand the possibility of injuring or damaging a piece of flight
hardware.
I asked at that time if adding some more people, for instance, a safety engineer-that was one of the things we discussed in there. The consensus was no, we really didn't need a safety engineer. We had the manufacturing engineer in attendance who was in support of that role, and I persuaded him that, typical of the way we normally worked, that he should be calling on the resources from his own organization, that is, in Manufacturing, in order to get this work done and get it done in a timely fashion.
[1578] And I also suggested that if they ran across a problem in doing that, they should bubble that up in their management chain to get help in getting the resources to get that done.
Now, after that session, it was my impression that there was improvement based on some of the concerns that had been expressed, and we did get quite a bit of work done.
For your evaluation, I would like to talk a little bit about the sequence of events for this task force.
CHAIRMAN ROGERS: Can I interrupt? Did you know at that time it was a launch constraint, a formal launch constraint?
MR. KILMINSTER: Not an overall launch constraint as such. Similar to the words that have been said before, each Flight Readiness Review had to address any anomalies or concerns that were identified at previous launches and in that sense, each of those anomalies or concerns were established in my mind as launch constraints unless they were properly reviewed and agreed upon by all parties.
CHAIRMAN ROGERS: You didn't know there was a difference between the launch constraint and just considering it an anomaly? You thought they were the same thing?
MR. KILMINSTER: No, sir. I did not think they were the same thing.
CHAIRMAN ROGERS: My question is: Did you know that this launch constraint was placed on the flights in July 1985?
MR. KILMINSTER: Until we resolved the O-ring problem on that nozzle joint, yes. We had to resolve that in a fashion for the subsequent flight before we would be okay to fly again.
CHAIRMAN ROGERS: So you did know there was a constraint on that??
MR. KILMINSTER: On a one flight per one flight basis; yes, sir.
CHAIRMAN ROGERS: What else would a constraint mean?
MR. KILMINSTER: Well, I get the feeling that there's a perception here that a launch constraint means all launches, whereas we were addressing each launch through the Flight Readiness Review process as we went.
CHAIRMAN ROGERS: No, I don't think-the testimony that we've had is that a launch constraint is put on because it is a very serious problem and the constraint means don't fly unless it's fixed or taken care of, but somebody has the authority to waive it for a particular flight.
And in this case, Mr. Mulloy was authorized to waive it, which he did, for a number of flights before 51-L. Just prior to 51-L, the papers showed, the launch constraint was closed out, which I guess means no longer existed. And that was done on January 23, 1986.
Now, did you know that sequence of events?
MR. KILMINSTER: Again, my understanding of closing out, as the term has been used here, was to close it out on the problem actions list, but not as an overall standard requirement.
We had to address these at subsequent Flight Readiness Reviews to insure that we were all satisfied with the proceeding to launch.
CHAIRMAN ROGERS: Did you understand the waiver process, that once a constraint was placed on this kind of a problem, that a flight could not occur unless there was a formal waiver?
MR. KILMINSTER: Not in the sense of a formal waiver, no, sir.
CHAIRMAN ROGERS: Did any of you? Didn't you get the documents saying that?
MR. McDONALD: I don't recall seeing any documents for a formal waiver.
[1579] MR. KEHRLI: Now, on the letter to Larry Wear, where Thiokol recommends to Mr. McDonald-where Thiokol recommends closing out the problem, that letter specifically references these AO numbers. If you look at Attachment No. 4 under launch constraint, the subject of the letter written December 10th, 1985, is closure.
CHAIRMAN ROGERS: Which one is this?
MR. KEHRLI: Number four of the launch constraint attachment, the letter from Mr. McDonald to Larry Wear of Marshall, and this is the request to close the problem. Under the subject, it specifically references the Marshall tracking numbers AO-7934, and those other numbers as well.
When you look at those numbers, those are the documents that you have in front of you, that report specifically lists the launch constraint on the nozzle joint and on the field joint number-the field joint problem, it says: "launch constraint, none".
So, isn't it accurate to say, based upon this letter, you must have seen those documents if you are referencing them, is that correct?
MR. RUSSELL: Answering for myself, yes, and I will have to also say that my understanding was that we were closing it off at the problem review board, and I did not know that there was a launch constraint that had to be waived every time, and I did not realize that this kind of an action would lead to removing any such constraint that I did not know about.
MR. McDONALD: I wrote the letter, and I would like everyone to read the letter to see what it says. It says, the subject critical problems are ongoing problems which will not be resolved for some
time. So, right away, I don't think that tells anybody that we are going to forget about this and take it off of anything.
It also says that, we request that subject critical problems be closed and removed from the next PRB agenda list, and the reason being that we spend more time each month going through reading all of that same thing that we have been reading every month for two years, because somebody colors a square in down at Marshall, please keep track of this each month.
We said, we've got a full-time task force working this problem. We have weekly meetings. We need more people solving the problem and less people keeping tracking and statusing it, and there's no sense us continuing doing that when we've got a very heavy activity doing that, and we got a letter a couple of weeks later from Larry Wear.
In fact, I was called before that letter, saying we were going to get it, that Marshall was very upset with us because we've got problems that have been on there for several years, and we haven't gotten them off of this list because it keeps getting thicker and thicker.
And, if you ever want to get something where nobody will read it, you get it so thick that they
finally pay no attention to it and that is exactly the thing the Problem Review Board was doing. It was getting so thick it had problems on there we knew we weren't going to solve for some time. And, they were the same problems we're having today.
And so, we said, okay, if you want to get them off the list, then just take them off and we will handle them through this other mechanism that we're addressing with everybody that really knows about this problem. And, I was unaware that a waiver was ever written for every flight after this problem.
[1580] CHAIRMAN ROGERS: So, your idea was really to cut down the paperwork?
MR. McDONALD: Yes. We're spending a lot of time in going through this matter where we had nothing to add, nor did anybody else.
DR. FEYNMAN: Mr. McDonald, this letter is a request. Did you ever get an answer that permitted you to do this?
MR. McDONALD: No, I didn't get any answer that permitted me to do that, nor did I know that anything was done about it. I just said, we've got a lot of things to do. This is one thing we don't need to do.
MR. ACHESON: Did you take the lack of answer
to mean that it had been closed out, that your proposal had been accepted?
MR. McDONALD: I don't know, because I believe we cancelled the subsequent Problem Review Board and I think the one after that or something, and so I don't know whatever happened after that letter was written. I wasn't involved in the next meeting.
MR. RUSSELL: I would like to add, we wouldn't have expected to get a letter back to this. It was a Marshall tracking system and, like Mr. McDonald said, we would have heard about the action in the next Problem Review Board meeting, which would have then been scheduled in early February, which was after the accident and did not take place.
MR. FEYNMAN: You mean, they come that far apart? This was December 1985.
MR. RUSSELL: But, we wrote the request-oh, they come every month.
MR. FEYNMAN: Well, then what happened?
MR. RUSSELL: Well, my recollection of being there is that
MR. FEYNMAN: Oh, okay. January 1986 is right after December 1985. I made a mistake. Excuse me.
MR. SUTTER: The task force was trying to improve the joint?
MR. RUSSELL: Yes.
MR. SUTTER: That in effect does not tie into what you do for the next launch? It seems to me that this conveys an attitude then that, okay, we've got troubles with this joint but it is okay to keep flying with it even though we want to fix it, that you're willing to keep flying with the joint even though you know you've got problems? And, I'm just wondering, what was the attitude.
There wasn't really undue concern about continuing flying.
MR. RUSSELL: The attitude was that it was an undesirable condition but still acceptable for flight, based upon the history that we had seen up until that point.
MR. SUTTER: The history of the erosion and the blow-by and the changing the leak checks and all of that did not really build into your people that should have been responsible, that this was a critical item?
MR. RUSSELL: I don't think that's true. I think we considered it to be a critical problem, which is the reason that we had the task force created to fix it. But, critical to stopping the program, it obviously wasn't.
MR. SUTTER: Well, you were going to run, say
another 20 or 30 flights without doing anything?
[1581] MR. BOISJOLY: No. Those who were intimately familiar with it-that is why we were writing these memos, to try to get a little bit of flame turned up underneath it, to be able to get something done, because we were concerned, we were extremely concerned.
I think that is what those memos reflect. They reflect frustration and concern both, to continue to fly in a condition that is marginal or a condition after the 16-nozzle erosion, if that ever occurred in a field joint, we would have a terrible problem on our hands and that is what we were trying to do.
We were attempting in that team to short-circuit company procedures. We were attempting to get things done that are not normally done, so that we could try things that we could tweak the system and get as many pieces of information in as short a period of time as possible, so that we could effect a change and get it implemented as quick as possible, and that was a big source of frustration because we weren't getting that support.
CHAIRMAN ROGERS: Mr. Kilminster, did you realize at the time you changed your vote on Launch 51-L, about this frustration that Mr. Boisjoly is speaking about?
MR. KILMINSTER: Yes, sir.
CHAIRMAN ROGERS: And you were not familiar with the fact that there was a constraint that had been placed on it in July?
MR. KILMINSTER: That would have required a waiver, no, sir.
CHAIRMAN ROGERS: Would that have made any difference to you, if you had known that?
MR. KILMINSTER: I don't believe so. We would have reported the information, the engineering evaluation that we had that evening, I think in the same fashion that we did.
CHAIRMAN ROGERS: What would have required you to vote differently, as far as the weather is concerned? If the temperature had been a lot colder, would you still have changed your vote?
MR. KILMINSTER: I've asked myself that same question. It is a difficult question to answer, of course. But I think that if I understood that the temperature was going to be in the low 20's, that I may indeed have changed my mind, but I have to say
CHAIRMAN ROGERS: You would have changed your mind twice, because you changed your mind once.
MR. KILMINSTER: Well, if we would have known in the first place
CHAIRMAN ROGERS: You wouldn't have changed
your mind, I guess since the answer?
MR. KILMINSTER: If we would have known in the first place it was in the low 20's.
CHAIRMAN ROGERS: But didn't the fact that you had these concerns, the papers show, you were terribly worried about it? And, it also shows you didn't really have any knowledge about what would happen if the weather became colder at that time, didn't you? At the time you made the decision, wasn't that a terribly difficult decision for you to make?
MR. KILMINSTER: Again, the information that we had available to us on the sub-scale work that had been done recently before that, the fact that there was no direct correlation with the full-scale data and the limiting analytical work that had been done, we used that as the basis-I used that as the basis for my recommendation.
CHAIRMAN ROGERS: Is it fair to say that when Mr. Boisjoly and others said that launching under those conditions on that date, January 28th, that you were exceeding your data base?
Would you agree with that?
[1582] MR. KILMINSTER: Specifically, for the SRM? That's true, yes, sir.
CHAIRMAN ROGERS: So, you were willing to
exceed your data base to get this launch going?
MR. KILMINSTER: Yes, sir, as we have been from the first launch on. Every launch, of course, extends the data base to a certain extent.
DR. WALKER: I thought the data base should be established by tests, and you shouldn't exceed the date base in your operations until you understand from tests or analysis or whatever that it is safe to proceed.
GENERAL KUTYNA: Bob Rummel, you're in the airlines. Do you exceed the data base every time you fly passengers?
MR. RUMMEL: No, sir.
DR. WALKER: What's the purpose of testing, then?
MR. KILMINSTER: Your first comment had to do with testing, and then you eventually said analysis.
DR. WALKER: Test or analysis.
MR. KILMINSTER: And we had indeed run analyses early in the program to evaluate the effects of a cold soaked motor at 40 degrees and a warm soaked motor at 90 degrees. That was analysis. The system had agreed that we would not test or attempt to cold soak motors and conduct those tests, so analytical qualification or certification, if you will, was acceptable.
DR. WALKER: So, you had some analytical data then, that extended your knowledge to 25 degrees or 20 or 30 degrees?
MR. KILMINSTER: No analytical data. We had some test data at 30 degrees. We also, in my mind, we have other solid rocket motors that utilize the same material in an O-ring at 30 degrees and had been qualified.
MR. SUTTER: But, wasn't your main thrust on the testing, to make sure that the engine had enough thrust and that most of the concerns over the temperature was the temperature of the propellant, to make sure the thrust was right? Is there much of a data base analyzing the joint at 28 degrees, 40 degrees or 50 degrees?
There is not a hell of a big data base.
MR. KILMINSTER: That is correct. There is not.
DR. WALKER: How could you proceed, then, if you didn't have either the analytical or experimental data base with the joint?
MR. EBELING: I have something I can add.
DR. WALKER: Well, let's just respond to the question.
MR. EBELING: We did that 40 degree test data.
DR. WALKER: But we're talking about 30 degrees, right. We're talking about even colder than that.
MR. EBELING: We had a statement of fact. We had 40-degree test data on the same elastomers.
DR. WALKER: Let's talk about 30 degrees. What did you have there?
MR. EBELING: We had on the 30-degree regime, we had information on one of the components called the safe and armed device, and that's the only thing I'm aware of.
DR. WALKER: So, you didn't have anything at 30 degrees?
[1583] MR. KILMINSTER: We also had information on the steel at 20 degrees from a fracture mechanics standpoint.
DR. WALKER: But you didn't have anything on the O-rings?
MR. KILMINSTER: On the seals, we relied upon the material capability, which was stated.
DR. WALKER: What material capability are you talking about?
MR. KILMINSTER: We're talking about the O-ring material capability that's identified in the mil spec.
DR. WALKER: The mil spec says specifically that the O-ring has to be qualified for each use, doesn't it?
MR. KILMINSTER: It does say that.
DR. WALKER: So, you didn't have anything?
MR. KILMINSTER: In that sense, that's correct.
DR. WALKER: So, you were not within either your analytical or your experimental data base in proceeding to launch?
MR. KILMINSTER: On this specific program, no. As I mentioned, we have other programs where it has been tested.
DR. WALKER: Do you think that your colleagues in NASA understood that?
MR. KILMINSTER: Yes.
DR. WALKER: Do you think they understood that too?
MR. KILMINSTER: I believe they did.
DR. WALKER: So, everybody knew that they were proceeding outside of your knowledge?
MR. KILMINSTER: Specific testing at 30 degrees?
DR. WALKER: Testing or analysis.
MR. KILMINSTER: I believe so.
DR. WALKER: Then, why did you do it?
MR. KILMINSTER: On the basis of the best
information we had available at the time, we did have this testing data, as we mentioned, at 30 degrees on a subscale that did function without any leakage indicated.
MR. SUTTER: Do you think that knowing what everybody knows now, how much credence should really be put into the subscale testing device that is fired with-that is ten inches in diameter and does not take into account all of the manufacturing tolerances, the usage effects, the fact that for instance even this metal yields a little bit at the loads imposed on it, even in the pressure tests, due to the firing?
Could you draw the conclusion today with what is known that the kinds of subscale testing that had been used to qualify these large solid rocket boosters is way inadequate to prove safety?
MR. KILMINSTER: Specifically in regards to this seal, I absolutely agree with that based upon what we've learned since the accident.
MR. SUTTER: I would like to put that in the record, because that is a finding that I'd like to arrive at. I think one of the things that was missed is, subscale testing is inadequate and I know it is expensive to run full-scale tests with a gadget 12 feet in diameter and 100 feet long, but to continue without doing it, no matter what it costs, I think is a mistake
and I know we're not drawing conclusions here but I feel so strong about that one, I would like to get it at least in the written record so it is addressed before we're done with this thing.
[1584] GENERAL KUTYNA: Joe, you're going to have it in the record twice, now that you've said it, and the second time by our independent tests. We've looked at the tests and essentially recommended the same thing.
CHAIRMAN ROGERS: Who in Thiokol is working on the new design?
MR. BOISJOLY: I am.
MR. RUSSELL: I am.
MR. EBELING: I am.
CHAIRMAN ROGERS: Do you all agree with the comment Mr. Sutter just made?
MR. BOISJOLY: Absolutely, that blow-by rig was a static rig and it was never intended to show sealability of a joint.
DR. WALKER: Is this the same thing?
MR. BOISJOLY: He's referring to the blow-by rig, the rig data that was used on the charts that night, was strictly a blow-by rig. It is not a dynamic rig. It never was intended to be a dynamic rig.
DR. WALKER: This was the 30 degree?
MR. BOISJOLY: Yes, it is, and it was never
intended to show sealability. It was to show blow-by.
DR. WALKER: Well, why was it raised, for the test, for this purpose? And I wasn't asking you that question specifically.
MR. BOISJOLY: It was put into the presentation that night because of a series of disjointed efforts under an umbrella by different people, trying to put together a presentation without a chance before presentation to go in and polish.
If I had my druthers, I would have taken that thing out of there after the fact because it adds absolutely nothing to the presentation that night.
DR. WALKER: That was our impression.
CHAIRMAN ROGERS: Mr. McDonald, are you working on the new design?
MR. McDONALD: Part-time, when I can. I have had some ideas that I've turned in on how to fix it.
CHAIRMAN ROGERS: Has your assignment changed since the accident?
MR. McDONALD: Yes.
CHAIRMAN ROGERS: What are you doing now?
MR. McDONALD: I used to be the Director of the SRM project. I have a title now called Director of Special Projects. The people that all worked for me work for somebody else. I am involved in reviewing some of
the failure data that is provided to me and coming up with some ideas on how to fix it, and define a test vehicle that will give us meaningful information which is a full-scale hot-fire test.
CHAIRMAN ROGERS: Were you given any reason for the change in assignment?
MR. McDONALD: Well, that is my second change in assignment since the hearing started. My first change after the 25th-well, the 14th of February testimony in Florida, I was pulled out of my position and given the assignment of scheduling.
CHAIRMAN ROGERS: Was any reason given?
MR. McDONALD: No reason other than, that is what I was going to do and my people again were put aside and assigned under somebody else. And I wasn't to be involved.
CHAIRMAN ROGERS: Who notified you to that effect?
[1585] MR. McDONALD: Mr. Cal Wiggins. He was general manager of the Space Division at that time.
CHAIRMAN ROGERS: Who?
MR. McDONALD: Mr. Cal Wiggins.
CHAIRMAN ROGERS: Were your people given a different assignment too?
MR. McDONALD: They were put under another
individual, in fact, one that used to work for me, and for a while I was under him too, during that time.
CHAIRMAN ROGERS: What was your second assignment?
MR. McDONALD: Well, when Mr. Dorsey came in and took over as the new general manager of the space division, and Mr. Wiggins was made his assistant, he gave me the assignment as Director of Special Projects reporting directly to him rather than to Mr. Wiggins.
CHAIRMAN ROGERS: What do you do in that capacity?
MR. McDONALD: In that capacity, he told me I can work on reviewing some of the information that has been provided on the failure analysis generated at Thiokol and some at Marshall, and to help Mr. Pelham in coming up with a test article for the recovery program, full-scale type test article for the seals, and to feel free to make what recommendations that I might want to relative to improvements in the program.
CHAIRMAN ROGERS: Mr. Kilminster, did you concur in the decision to change Mr. McDonald's assignment?
MR. KILMINSTER: My assignment was also changed, Mr. Chairman. I've been located at Marshall Space Flight Center, working on the investigation team.
That was done while I was down at Huntsville, working there.
CHAIRMAN ROGERS: Do you have any reason, Mr. McDonald, to think that you were given another assignment because of the testimony you gave before the Commission?
MR. McDONALD: Yes, I do. I feel that I was set aside so that I would not have contact with the people from NASA again because they felt that I either couldn't work with them or it would be a situation that wouldn't be good for either party, and so I was taken out of the failure analysis work that I was doing at Huntsville prior to that assignment.
CHAIRMAN ROGERS: So, you were in effect punished for being right?
MR. McDONALD: I feel I was.
CHAIRMAN ROGERS: Mr. Boisjoly, your assignment hasn't changed at all?
MR. BOISJOLY: In one respect yes, in one respect no. I have been designated as seal coordinator. I have been preparing a lot of information for input. But I too have been put on the sideline in that loop with relationship to the customer.
CHAIRMAN ROGERS: Do you feel that may be in retaliation for your testimony?
MR. BOISJOLY: I think that is a possibility, a distinct possibility.
CHAIRMAN ROGERS: How about any other engineers who testified?
MR. BOISJOLY: That I have personal knowledge?
CHAIRMAN ROGERS: Yes, any change in their assignment?
MR. BOISJOLY: No, not that I have personal knowledge of, no.
CHAIRMAN ROGERS: How about Arnie Thompson? Is he in the same assignment?
MR. BOISJOLY: Arnie is in the same assignment.
CHAIRMAN ROGERS: Does anybody from Thiokol want to comment on what Mr. McDonald or Mr. Boisjoly said?
MR. EBELING: I'm Bob Ebeling. Prior to this effort I was involved with the night of January 27th. Brian Russell worked for me at that time, and he basically does now. But the Seal Task Force, to give you an instance, was under my auspices as a manager at that time prior to the incident, and since then it has been-that Seal Task Force has been dissolved and has been reorganized under a different manager. My helper here, Brian Russell, now has been assigned to this new manager.
It is just one of the anomalies that has gone on since then, and I think it has got something to do with testifying-it isn't all bad, because there are several of us that already have got a lot of work, and there are some people that don't have as much work, therefore, because the program has been brought to its knees.
These are very qualified managers and qualified people, so therefore why not take advantage of them and redelegate some of this workload to these other
people? At first when it happened I was very upset, and I challenged the new general manager, Ed Dorsey, with this.
He explained to me that, hey, Bob, you've got more than you can handle anyway. This other fellow, he doesn't have nothing now that we've brought his program to its knees. Why not give this portion of it for him to manage? At the time it seemed logical.
MR. KILMINSTER: Mr. Chairman, I would also like to comment that since the new general manager has come on board, Mr. Dorsey, there has been a basic organizational concept change, structure change, in that we have engineering now who in the past had been in a support organization, supporting all the programs in the plant, including SRM.
Now we have identified specific individuals from that core organization, and they now report directly into the Space Shuttle SRM program. So in that concept or context, there has been some shifting of responsibility in the engineering staff.
CHAIRMAN ROGERS: Well, having said all of that, you know what I'm driving at. I mean, if it appears that you're punishing the two people or at least two of the people who are right about the decision and objected to the launch which ultimately resulted in
criticism of Thiokol and then they're demoted or they feel that they are being retaliated against, that is a very serious matter.
It would seem to me, just speaking for myself, they should be promoted, not demoted or pushed aside. Do you want to comment on that?
MR. KILMINSTER: There was certainly no demotion involved that I know of.
CHAIRMAN ROGERS: Well, you heard what Mr. McDonald said.
MR. McDONALD: I was not demoted. They just took my people away and gave me a more menial job as far as I was concerned.
CHAIRMAN ROGERS: All right, I'll withdraw the use of the word "demoted". It sounds as if you were demoted.
MR. McDONALD: I felt like it.
[1587] MR. KEHRLI: I have a question for Mr. Russell and Mr. Boisjoly on a document, tab 32 in the book, which is the August 9, 1985, letter to Mr. Jim Thomas at Marshall from you, Mr. Russell.
MR. RUSSELL: Yes.
MR. KEHRLI: Could you explain to the Commission what the purpose of that letter was and what the genesis of the letter was, what you were trying to
do there?
MR. RUSSELL: In the Problem Review Board telecon we had a specific or we received a specific question or two specific questions from Jim Thomas. He wanted a specific answer to those, and he asked us to submit that in writing.
So I talked with Mr. Boisjoly, who does not attend that Problem Review Board meeting, and I wanted his expertise in drafting the letter, which I got and submitted it, and it was in response to a direct request to send in a written answer.
I don't know what the motivation was at the head of that.
MR. KEHRLI: The letter refers to resiliency tests. What resiliency tests were those, sir?
MR. RUSSELL: Those were the ones that were performed in the spring of 1985 that were done with a fixed rate as opposed to a variable rate test. They were done at 100, 75 and 50 degrees to understand the O-ring's response in a nonpressurized condition, because there was some concern with the joint rotation; that even though the O-ring dimensionally should still have squeezed or maybe a timing function there or a time for that O-ring to recover where there is no squeeze.
That was the purpose of these tests, and that
was pretty much-well, it was driven by the inspection results that we saw on the 51-C flight and in January, and I guess that we inspected in late January or early February.
MR. BOISJOLY: If you look at the answer to the second question, you'll find that is the form of the chart that was used the night before the launch, and that was the chart. There was the chart that was prepared for the August 19th presentation that Al gave to headquarters.
MR. KEHRLI: In conversation with our staff investigators, Mr. Boisjoly, you indicated that this was a new theory or there is a difference between the old theory on the timing function and the new theory on the timing function, or that this letter somehow was a different way of looking at the timing function.
Could you explain that, please?
MR. BOISJOLY: Yes. What had happened is that we had made a presentation as a result of SRM-15, soot past the O-ring, which was the coldest flight at that time, the next Flight FRR we presented that information.
There was a curve in that presentation, that we broke it up into zones A, B and C. When I found out Al was going down to give the presentation at Washington
in August, I was trying to rework that information into a more concise form and to try to give probabilities to those zones. I'm the one that selected zero to 170 milliseconds as the basis of looking at that curve and said that at those pressure levels we would have a high probability of sealing.
Then I selected the 170 to 330 and from 330 to full ignition and assigned probabilities to those. That's when Al went down, and that was to point out the concern that if we were always [1588] in the initial phases we had a very good probability of success. If we moved into the next region, you lessened that probability. If you moved the event into a third region, you increased the probability towards a failure, and that was the whole impetus of the whole presentation.
DR. WALKER: What did you mean by high probability?
MR. BOISJOLY: Well, for instance, if you have a secondary seal lifted off the sealing surface in the later timing functions and you had crippled the primary O-ring, you would not have had the capability to pressure actuate the secondary O-ring when you take resiliency into consideration.
Prior to this effect we had always termed it in terms of geometry. We always talked of the geometry
of the gap opening in relationship to an O-ring that always had the capability of following in the seal surfaces and being pressure actuated.
But when you take that same group of data which was in this time frame and apply the resiliency data to it, as we knew it at that time, on a limited number of tests, that said that if the O-ring did not have the ability to follow the loading surfaces as they were pulling away as the gap was opening, the longer the period of time that you were in that regime the less chance you had to seal.
So I tried to define it in a three zone region, to say a high probability, a moderate probability, and a low probability.
DR. WALKER: I understand that. By high probability did you mean 95 percent or 75 percent?
MR. BOISJOLY: In the beginning on the basis of the limited test we had, I felt there was 100 percent if it happened within the first 150, 170 milliseconds because there would be an impingement problem and then the margin would truly be a margin of impingement and not blowby.
DR. WALKER: But you didn't use the word likely. You used high probability.
MR. BOISJOLY: Yes.
DR. WALKER: Which implies that there is some chance.
MR. BOISJOLY: Well, there always is.
CHAIRMAN ROGERS: Mr. McDonald, as I remember your testimony in public session, it was to the effect that you refused to sign the telefax or refused to go along, and normally when you were at Kennedy you were the one that would have signed. Is that correct?
MR. McDONALD: I said I hadn't had an experience and wasn't aware of one where anybody was ever asked to sign anything, but I felt that was my responsibility if it ever came up there because I am the senior official for Thiokol.
CHAIRMAN ROGERS: Mr. Kilminster said, as I recall, he normally would be the one to sign that and that seemed to kind of conflict with your testimony. Is that what you testified to, Mr. Kilminster?
MR. KILMINSTER: I think what I said, Mr. Chairman, was that that whole business was unusual in the sense that we were talking about something being signed the night before the launch. However under other circumstances the piece of paper that is signed to identify flight readiness, as far as Morton Thiokol is concerned, that is a document that I normally signed.
CHAIRMAN ROGERS: Even when Mr. McDonald is
there?
MR. KILMINSTER: Yes.
CHAIRMAN ROGERS: Is that correct, Mr. McDonald?
MR. McDONALD: That is true for every formal Flight Readiness that we have prior to the launch. The reason I was there and Joe wasn't there is we alternate doing that. In case something comes up after the last formal review, and the last formal review is the L-1 Review, it is signed by anybody and if anything comes up after that then that's why I'm there and the other contractors have people like me there who can resolve those issues and are responsible for going ahead with the launch.
As I said in my testimony, Jesse Moore makes an oral poll subsequent to that L-1 Review because in many cases there are items brought up at the L-1 that are finally resolved, and then makes a poll to ask each of the contractors if they are ready to fly because now everything is supposed to be in.
I'm the one that has to answer yes, Thiokol is ready to fly. If something happens after that time, it is my impression the reason I'm there is that I'm the guy that's going to have to get up and say yes, we are ready to fly.
I have never been in a condition where we ever had to sign anything, and I don't think there was ever any condition where we had to sign anything after L-1. I expected, because of this telecon, that I would have to do that because I felt that I had the same responsibility at the launch site as he had when he was there or I shouldn't even take the time in going.
CHAIRMAN ROGERS: You told them that you would not sign?
MR. McDONALD: I told Mr. Mulloy I wouldn't sign and that it would have to come from the plant. Now, he didn't ask me to. I just told him I wouldn't do that because I felt that I was going to have to do that. That was my responsibility. That's why I was there.
CHAIRMAN ROGERS: Did you know that, Mr. Kilminster?
MR. KILMINSTER: No, I did not.
CHAIRMAN ROGERS: You didn't know that he had said he wouldn't sign?
MR. KILMINSTER: No, sir.
CHAIRMAN ROGERS: Why did you think that they required your signature in this case? You said you had never done it before.
MR. KILMINSTER: Because we had not had a
later than L-1 identified problem.
CHAIRMAN ROGERS: Why would it have to be in writing?.
MR. KILMINSTER: I guess the only thing I can say is I was not surprised when Mr. Mulloy asked for that, but again the whole thing was unusual.
CHAIRMAN ROGERS: I'm interested in why you weren't surprised if you had never done it and you had a man there that normally would orally approve this. Why weren't you surprised when they asked you to sign it?
MR. KILMINSTER: I guess I can't comment other than what I've said.
CHAIRMAN ROGERS: Didn't it occur to you that they might be wanting to put the monkey on your back in writing?
MR. KILMINSTER: Well, the monkey is always on my back under normal circumstances.
CHAIRMAN ROGERS: Not normally. It would be on the man who said it.
[1590] MR. KILMINSTER: Well, again, I go back to the formal paper that is signed for the flight readiness review sequence, and it normally is my name that is on there unless I am gone at the time that signature was to be made, in which case Mr. McDonald or someone else signs it.
CHAIRMAN ROGERS: In any case, the fact that they never asked you to do anything like that before and that Mr. McDonald was there when normally he would be authorized to say okay, that didn't strike you as particularly alarming at all?
MR. KILMINSTER: Well, the rationale came from the plant. The information that was presented came from the plant, and that is where I was, is at the plant.
MR. HOTZ: I wonder if we could poll the Thiokol delegation as to whether they were or were not aware that a launch constraint was being waived on a formal waiver for each of the flights after 51-B.
MR. EBELING: I was not aware.
MR. KILMINSTER: I was not aware.
MR. RUSSELL: I was not aware.
MR. McDONALD: I was not aware.
MR. BOISJOLY: I was not aware.
MR. HOTZ: Thank you.
DR. WALKER: Let me just follow up on that if I might. Although you weren't aware that there was a formal waiving of the launch constraints, you were aware that anomalies from previous flights had to be explained satisfactorily before the next flight could occur.
As I understood Mr. Mulloy this morning, that was based on the analysis of the amount of damage that
was acceptable and the amount of erosion that was acceptable versus the amount of erosion which had been seen.
There were some models used to predict the amount of erosion which was to be expected. As Dr. Feynman has pointed out, these models really were not precise models. There is a considerable amount of variation, because all of the parameters were beyond your ability to measure or know, so on what basis did you feel the erosion could be predicted sufficiently well to allow the flight to be continued? What models, what data, what analyses?
MR. McDONALD: I would like to try to respond to that. Dr. Salita developed those models at my request, I guess, starting in the spring of 1985 and published two different parts of that.
At that time, prior to the problem we had on 51-Baker, 16-A we had not seen any erosion other than direct impingement on the seal, and that is what his model was based on.
When we recovered that hardware and saw that we had violated the primary seal, we immediately were very concerned because his model, being a jet impingement model, is a function of time, and it has to have a forcing function which is the delta pressure between
the chamber and the cavity.
That's a function of ignition time, and so it does have a limit within the model that you can't get any further erosion because there is no other time in the pressure trace, the pressure trace of the motor, that you get that kind of a condition.
Based upon that, we thought we had a limiting model. However, there are a lot of unknowns as far as the fidelity of that model is concerned, but his predictions, models in five-inch [1591] tests that we ran and fairly good engineering terms to define how accurate it was and within about a 12 percent spread, he predicts all that data very accurately.
Now, the amount of erosion that we had, we then went and tested to see where we in the real world lose the O-ring relative to what we think would be the maximum with our model, and we did that two ways.
The first way we took an O-ring and actually sliced pieces out of it just like we figured it would be eroded away until we could no longer hold it as a seal. In doing that we found out that we could lose nearly two-thirds of that O-ring before we lost seal. That was cold, a cold gas test.
GENERAL KUTYNA: Was that a dynamic test?
MR. McDONALD: No, that was a static test.
GENERAL KUTYNA: No rotation of the joints?
MR. McDONALD: Right, no rotation of the joints.
GENERAL KUTYNA: Did you feel that was valid in that case?
MR. McDONALD: We didn't feel it was totally valid, and so we wanted to run some hot scale tests because you eroded differently as well during hot scale, and so we then took some hot scale tests of our motors and focused the jets so there were more and more higher heat transfer rates to keep eating away the O-ring until it failed.
We found that in the evolution of that test series that we could erode up to 125 thousandths, and we had a lot of tests there and below that we never saw a seal ever fail. We went up to 150 thousandths of the material removed, and it still sealed.
However, we had two tests, one at 145 and one at 160, that did fail. Now, both of those failed the primary seal and eroded something like less than 10 thousandths off of the secondary seal.
Now, these joints did not rotate but they did show the sensitivity and the capability of the O-ring to jet impingement erosion. If you took our worst measured erosion on the O-ring relative to what it took
to really fail it, it was nearly a factor of three to one.
Recognizing the fidelity of the math model is not real good, we did not feel it was that bad at three to one, and as long as we could retain the secondary seal during a good portion of the erosion time period, we felt good. Our concern was if we ever got it past the time that we could have a good secondary seal. That was our concern.
The meeting that occurred on August 19 came about as a result of this problem with the nozzle eroding through, and that is what drove that meeting. Headquarters wanted to hear about that. We lost the primary seal and eroded some secondary.
We all sat down together and got together with the engineering people and put together that presentation and collectively said, you know, we ought to address the whole seal issue, not just that failure, because we all felt that if that ever happened in the field joint we were in bad trouble because the nozzle has a much better secondary seal than the field joint does.
We decided we would put it together as a total pressure seal presentation and highlight the field joint even though there was a nozzle joint that caused the problem that
apparently drove the restriction to launch I was unaware that we retained.
MR. SUTTER: At this meeting on August 19th at headquarters, that was called because of Thiokol's concern that the joint was really in trouble?
[1592] MR. McDONALD: No, it was called-we had had another meeting scheduled at Washington headquarters at that time that had a problem with the mixer fire earlier in the year, and there was a review of that.
I believe Mike Weeks either called Joe or I or one of us and said well, you're here, you ought to come and address a couple of other issues that have happened recently that we are very interested in.
One of them is we had broken the structural test article on the filament wound case I believe in July down at Marshall, and they wanted to hear about that.
The other one was they were made aware that we had violated the primary seal in the nozzle and wanted to hear about that and what our rationale was to continue.
So then I called Mr. Mulloy and told him that we had been requested for this presentation, and he said we're going to have to review that with them before we can go to Washington, which is the normal sequence of things, and so we were prepared to have to go down to
review it with the Marshall people.
We finally held that on a telecon and faxed down all the charts that we planned to take to Washington and reviewed it with all the Marshall people before we went to Washington. He came up and met us here in Washington.
MR. SUTTER: What was the tone of the meeting in Washington? That it was a serious problem or just business as usual? Apparently it wasn't the primary subject on the agenda.
MR. McDONALD: It was not the primary subject on the agenda.
MR. SUTTER: There was another meeting in Washington, too, wasn't there, on this problem? Weren't there two meetings there in the fall of 1985?
MR. McDONALD: Well, I'm not aware of the other meeting.
MR. SUTTER: Well, what I'm curious about was NASA headquarters fully informed about or what was their attitude based upon what they heard from your or anybody else as to how serious this joint problem was?
MR. McDONALD: Well, I think they felt it was serious. They felt that we did a good job of focusing the history of that problem; where we were at, what we understood about it, what we felt we needed to do to fix
it, because we not only addressed the history, I think, and we focused it quite well.
MR. SUTTER: Would it have put them in the frame of mind that when another launch came up they would ask how is this joint for this flight? Is it okay? Or would they be that concerned?
MR. McDONALD: I don't think so, unless the previous flight showed an erosion problem or something different than they had heard. They, I think, felt that we needed to establish this task force in support of that which we had had in place about that time to put forth a focused effort on trying to resolve the problem because they felt it was the same problem. It didn't matter whether it was in the nozzle or the field joint.
MR. SUTTER: Then apparently your presentation to them is a lot less or a lot more cooled down than some of the internal paperwork as to the severity of this problem.
MR. McDONALD: Well, I'm frankly surprised that this piece of paper here generates as much interest as that presentation, because that gets lost in the stack of those that's probably that thick (indicating).
CHAIRMAN ROGERS: Well, I'm not sure when you say more interest, we're just trying to complete the record. I don't think you should judge our relative
interest. I mean, we're pretty interested in all aspects of this, and we wanted to find out more about this piece of paper because we had never been alerted to it.
No one told us before. We asked NASA and everyone else to be sure to let us know everything there was that might relate to this, and we discovered this ourselves. So we're just asking about it.
I'm going to first let Gene go ahead, and then I have a question.
DR. COVERT: I had several questions on the sequence. I guess, Mr. Kilminster, I will ask you first.
Were you involved in the original design of the SRM and the joints in any way?
MR. KILMINSTER: Yes, sir.
DR. COVERT: Was it at that time the goal of the design to protect the integrity of the O-rings against thermal distress of any kind?
MR. KILMINSTER: Yes, sir, that's always been the goal.
DR. COVERT: In view of that, do you feel that as the problem has progressed and you now live with a situation that you hadn't anticipated at all, does this give you a feeling of discomfort that this is not the way O-rings ought to be treated?
MR. KILMINSTER: Yes. I believe that especially since this recent testing has been done that we have learned quite a bit from the subscale testing that has been done of higher fidelity than we were doing before, and I think we have learned a lot more about that joint. I think we have learned a lot more about the effect of squeeze on the O-ring. And there has been a move, if you will, from the time that this joint was initially designed, to getting rid of the putty, if you
will, getting rid of the thermal barrier, the protection.
I believe that perhaps in some motors and in some circumstances, maybe even in the field joints of our motor, that might be appropriate, where you can convince yourselves you're not going to have circumferential flow or a replenishment of the heat, if you will. And that is something we will have to take a very quick look at and deep look at.
DR. COVERT: Have you ever been associated with the use of O-rings in a rocket before where there was continued and repeated thermal distress to the O-rings?
MR. KILMINSTER: Well, usually, Dr. Covert, you don't get the opportunity to look at fired hardware to the extent that we have here.
DR. COVERT: How about Mr. Boisjoly, were you involved in the original design of this joint?
MR. BOISJOLY: No, I was not. I came to work at Thiokol in 1980.
DR. COVERT: Mr. Ebeling?
MR. EBELING: Yes, I was.
DR. COVERT: Does your feeling about the use of O-rings coincide with Mr. Kilminster's?
MR. EBELING: No, it doesn't.
DR. COVERT: Do you want to tell me the difference?
MR. EBELING: Well, I am a hydraulics engineer by profession, and O-rings and seals in hydraulics are very sacred, but for the most part, a hydraulics or pneumatics engineer controls the structure, the structural design, the structural deformation to make sure that this neat little [1594] part that is so critical is given everything it needs to operate. In solid rocket motors I have been there now pushing 25 years. They had a different attitude on O-rings when I came there, and it is not just Thiokol, it is universal.
DR. COVERT: By universal, you mean the solid rocket industry?
MR. EBELING: The entire solid rocket industry. It gets around from one, the competitors' information eventually gets to me by one track or another, and mine to theirs, but my experience on O-rings was and is to this date an O-ring is not a mechanism and never should be a mechanism that sees the heat of the magnitude of our motors, and I think before I do retire, I'm going to make sure that we continue to fly with round seals which I am against round seals anyway. I think seals with memories, not pressure-activated, but energized through mechanical means, and in all cases,
keep the heat of our rocket motors away from those seals. Whatever it is, you do not need chamber pressure to energize a seal.
DR. COVERT: In this regard, then, did you have an increasing concern as you saw the tendency first to accept thermal distress and then to say, well, we can model this reasonably and we can accept a little bit of erosion, and then etc., etc.? Did this cause you a feeling of-if not distress, then betrayal in terms of your feeling about O-rings?
MR. EBELING: I'm sure sorry you asked that question.
DR. COVERT: I'm sorry I had to.
MR. EBELING: To answer your question, yes. In fact, I have been an advocate, I used to sit in on the O-ring task force and was involved in the seals since Brian Russell worked directly for me, and I had a certain allegiance to this type of thing anyway, that I felt that we shouldn't ship any more rocket motors until we get it fixed.
DR. COVERT: Did you voice this?
MR. EBELING: Unfortunately, not to the right people.
DR. COVERT: How do you feel about inspectability of this kind of a sealing device?
MR. EBELING: Like the ones that-I think the pressure testing that we do, the leak pressure testing is the only practical method that we can do for this type of a seal. There is a possibility you might invent an N-ray system to rotate the motor and make sure everything is all right by N-ray, because N-ray is opposite of X-ray. It washes out metal, and elastomers and plastics show up real keen.
DR. COVERT: N-ray is neutrons?
MR. EBELING: Neutrons. Such a system might be devised, but it would be very expensive, but the pressure testing and the inspection of the joint the way we are doing it is probably the only practical way of doing it.
MR. SUTTER: Do you think that pressure testing it 28 days before the rollout and the thing sitting out there with the temperature working on it
MR. EBELING: No problem. You're from Boeing. I am sure your people would say the same thing, no problem.
MR. SUTTER: Well, maybe some people from Boeing would say it. I still have reservations. When you take into account
MR. EBELING: Then quit flying. You had better quit flying.
MR. SUTTER: Wait a minute, now. We've got four systems flying everything in our bucket, and a single failure doesn't do anything except give you a yellow light. There's a difference.
MR. EBELING: It makes you tense though.
CHAIRMAN ROGERS: You said you didn't voice your views to the right people.
Did you voice your views to anybody?
MR. RUSSELL: I heard them.
MR. BOISJOLY: I heard them.
CHAIRMAN ROGERS: Did you put it in writing at all, any of your views?
MR. EBELING: No, not exactly, not in the words I just told you.
MR. BOISJOLY: I think I can express the fact that he must have expressed it at least two or three times a week during the times of November and December in our seal meetings.
CHAIRMAN ROGERS: I want to make a comment to Mr. Kilminster, I guess, but to the company as a whole. I am very upset about the testimony Mr. McDonald gave. It's a very serious matter. In this kind of an accident where people come before a Commission and tell the truth and then they are treated as he believes he has been treated, which obviously is in some way punishment or
retaliation for his testimony, it is extremely serious, and the whole idea of the program is to have an openness and to have an honest exchange of views.
And in this case, Mr. McDonald and Mr. Boisjoly and others, Mr. Thompson and others, were right. If their warnings had been heeded that day and the Right had been delayed, there's no telling what would have happened. We might never have had the accident. And to have something happen to him that seems to be in the nature of punishment is shocking, and I just hope that you convey that to management.
I don't know how the others feel, but that is how I feel. I would think you would want him in all of your discussions, and Mr. Boisjoly and he shouldn't be treated that way. He should be treated the other way, that he was right and you were wrong, and others who changed their decisions were wrong, and they were right, and to have something that seems to me to be in the nature of punishment is very, very distressing, and I just wanted you to know that.
There may be other questions.
GENERAL KUTYNA: I had one question. The briefing on the 19th in Washington, Mike Weeks was there.
Who were the other folks who were there, do
you recall?
MR. McDONALD: I had a list at one time.
GENERAL KUTYNA: Could you send me that for the record?
MR. McDONALD: Yes.
GENERAL KUTYNA: Secondly, there has been some question that people understood that there was a temperature problem. I remember your conclusions chart, your file chart, and the very first bullet of that chart had the word "resiliency" in it.
Do you feel when you talked about resiliency at that meeting people got the connection between resiliency and temperature, that resiliency was a function of temperature, or was that lost?
[1596] MR. McDONALD: It may have gotten lost because we hadn't run a very long range of temperatures when we got that data.
GENERAL KUTYNA: So it is possible that people at headquarters from that briefing did not understand temperature was a concern?
MR. McDONALD: I guess it is possible they could have.
GENERAL KUTYNA: Is it probable?
MR. McDONALD: I don't know if it is probable, because we put it as the first bullet of why
we thought that was our highest concern, and if that hadn't have happened, we wouldn't have had that concern.
DR. KEEL: Could I follow up on that, just one quick question?
In fact, in your 51-E flight readiness review after 51-C, which, Mr. Boisjoly, you have testified was the worst case of blow-by, and in fact, you drew a correlation to temperature in that worst case, the conclusion of that that was briefed to Marshall at the project level, as we discussed this morning in testimony, in fact says that explicitly. It says STS-51C, consistent with erosion base, and then it says sub-bullet under that conclusion, low temperature enhanced probability of blow-by. STS 51-C experienced worst case temperature change in Florida history.
Now, that is an unequivocal reference to temperature and the fact that it makes things worse.
Can you explain, and perhaps you, Mr. Kilminster, since you had testified again today that you sign off on these flight readiness reviews, how it was that that information never got up to headquarters, or did it get to headquarters?
MR. KILMINSTER: I believe that that information was a direct quote from a flight readiness
review statement that we made for acceptability to fly. From that point on, that's up to Marshall Space Flight Center, what they take forward.
DR. KEEL: Did you sit in on the Center board briefings of that flight readiness review?
MR. KILMINSTER: Either Al or myself, and I don't recall specifically on that one.
DR. KEEL: Al, did you sit in?
MR. McDONALD: Joe sat in as a member of the Center board. I stayed there because I gave the briefing you just read from.
DR. KEEL: So you stayed at the Center board briefing? So you were there at the Center board briefing?
MR. McDONALD: Yes.
DR. KEEL: Who briefed that?
MR. McDONALD: Mr. Mulloy.
DR. KEEL: And Mr. Boisjoly, were you there?
MR. BOISJOLY: Yes, I was.
DR. KEEL: Now, was it any of your recollections, can you recall what was briefed, and was this temperature concern or even the extensive brief you have given now on O-ring erosion problems, was that briefed at the center board?
MR. McDONALD: No, it wasn't. It was kind of
a one-liner that we had some erosion, but it was within our experience base or something to that effect.
[1597] CHAIRMAN ROGERS: I want to ask one more question.
When you said before that you answered my question that there was reference to how your customer would feel about you, what was the basis for that?
MR. McDONALD: Well, prior to the testimony I gave in Florida, as I indicated, I was spending full time at Huntsville, and subsequent to that time I was not allowed to go back to Huntsville. I wanted, too, to review some of the
CHAIRMAN ROGERS: Was any reason given?
MR. McDONALD: The only reason given was they didn't think it would be in the best interests of either party that I do that.
CHAIRMAN ROGERS: Who told you that?
MR. McDONALD: I talked with Mr. Dorsey for a while on that, and he felt that I would be better off with working with problems in the plant.
CHAIRMAN ROGERS: Did he indicate anyone from NASA had said that to him?
MR. McDONALD: No, he did not. I do know that even after the recovery team started and I had submitted some ideas on how to fix the problem, before
we went to our first formal meeting at Marshall, that one of the fixes that we proposed was one that I had developed. I wasn't asked to even go and support that.
CHAIRMAN ROGERS: Any reason given?
MR. McDONALD: No. In fact, I didn't even get a copy of the presentation. I had to go borrow somebody else's, and I also did not get copies of some of the material that was being generated in failure analysis, and I had found out in the hallway that that had been done, and so I went and found some of the data. But I got the distinct impression that it was not being sent to me on purpose because some people knew that I did not agree with all of the conclusions that were being drawn on some of the data.
MR. HOTZ: Mr. McDonald, did subsequent to the accident, in your testimony before this Commission, did you receive any personal comment from people at NASA about your testimony?
MR. McDONALD: I think, I'm not sure of the exact meeting, was when part of the Commission came out, I believe it was Mr. Sutter's team, reviewed the development, qualification, certification of the SRMs. It was a two-day meeting, and the second day I was there, Mr. Mulloy came into my office and slammed the door, and as far as I was concerned, was very
intimidating to me. He was obviously very disturbed and wanted to know what my motivation was-and I won't use his exact words-for doing what I was doing, and I asked him what's his problem? Do you mean what my testimony was? And he said no. As I understand it, you're giving information to the Commission without going through your own management, without going through NASA, and what's your motivation for doing that?
And I told him to calm down, that I didn't think I had to get a note from my mother or anyone to give anybody information, and I felt it was appropriate to give them information. And I asked him what specifically, and it was the information that I had generated on the fourth of February while I was at Huntsville when I got frustrated with regard to the failure analysis on what I thought would be some real, potential causes of the failure, that I had given that to Dr. Covert when he was there, I think before my testimony, at the end of February, and had not gone through I guess the proper channels or something in doing that, and for some reason he was very upset about that and was very intimidating. But I ignored him.
On the other hand, he said that, you know, he never was against me, and he had a high regard for my capability, and he wanted to let me know that. But I
could see no reason for him doing what he did.
MR. HOTZ: Did you get the feeling that there might be some feeling on the part of the Huntsville people that they wanted to control this flow of information to the Commission?
MR. McDONALD: I got the feeling that that was happening from things that I was reading, the data that I was looking at, and the conclusions that were drawn, and how they were drawn, instead of focusing it, to me was making it fuzzy.
GENERAL KUTYNA: Mr. Kilminster, you are then
CHAIRMAN ROGERS: Don, before we leave that, Mr. Mulloy, do you have any comment to make on Mr. McDonald's statement?
MR. MULLOY: Yes, sir. If you would like for me to give you my version of that discussion.
CHAIRMAN ROGERS: Why don't you?
MR. MULLOY: Yes, sir, I would be glad to.
As often happens, I guess, when we have a meeting, we don't all remember it being the same. It must have been a different meeting. I came to Mr. McDonald's door and asked him if he had a moment, and he said he did, and I then closed the door. I didn't realize I slammed it. I was not upset. I think I
started by shaking Al's hand and said, before I say anything, I want you to know that I don't have any personal feelings one way or the other about what has occurred subsequent to the launch. However, I have a curiosity, and the curiosity I have is why you have taken the approach that you have in circumventing your own management and the customer in voicing concerns about the launch of 51-L that were never voiced to his management or to me. His response to that was that he was very upset about the way the investigation was going at Marshall when he was down there, but when he got back to HOSC, after the incident, when he came back to Huntsville, he found NASA organizing a great number of teams and laying out a broad spectrum of areas to look at, which included the external tank and the SSME and the SRB and forming teams to look at in a broad way across the total system to try and determine what caused the accident.
Al I think mentioned to me that he had obtained the information about the temperature readings on the right hand SRB from the IR gun at 9 degrees, and he was trying to introduce that into the team to which he was assigned, which was Mr. Swinghammer's SRM team, and he was very frustrated, that the team seemed to be more interested in getting a structure to look across
the total spectrum at the possible failure causes as opposed to picking up the obvious thing, which was the low temperature on the right hand booster.
And that is the way I recall the discussion.
GENERAL KUTYNA: May I ask who directed you to look across the whole spectrum at the accident structure? Where did you get that guidance?
MR. MULLOY: I did not do that. I was not participating.
GENERAL KUTYNA: Well, let me tell you where I got-where the guidance came. Our panel gave you the guidance to look across the whole spectrum of the accident structure and look at every facet of it rather than home in on something that you might think was the conclusion.
MR. MULLOY: I think that was the proper approach.
[1599] CHAIRMAN ROGERS: But the discussion that Mr. McDonald referred to was initiated by you?
MR. MULLOY: Yes, sir, it was.
CHAIRMAN ROGERS: And what was the purpose of it again? And don't go through the whole thing, but what was the purpose? Why did you do it?
MR. MULLOY: I have known Al for some time and had worked with him for some time, and why I did it was
based upon his testimony that he had objected to the launch and continued to protest the launch after the discussion on the night of the 27th when that was not a fact.
CHAIRMAN ROGERS: In other words, you were challenging his veracity at this point?
MR. MULLOY: I wondered why he said that when he did not pass that on to his management, nor did he pass it on to me, nor did at any time during the launch process when he was on the console during that morning did he make any comments or continue to object to the launch. As a matter of fact, as he testified, he left his console and left the loop during the launch process.
CHAIRMAN ROGERS: So you had no particular motive to start this discussion other than to satisfy your own curiosity?
MR. MULLOY: Yes, sir. As to why he felt that there was a concern now that he did not consider it worthy of passing on to his management or to the customer.
DR. KEEL: May I ask one question, Mr. Chairman? We did give directions with respect to a broad investigation, but we never gave a direction with respect to anyone who had information that they thought should be passed to the Commission, that they could pass it directly to the Commission. There was never any direction that that should go through management, or through NASA, or through anyone.
DR. WALKER: I had a question for Mr. Kilminster. In the original Thiokol proposal, the seal was different from the present seal. It was a face seal and a joint seal.
MR. KILMINSTER: That is correct.
DR. WALKER: I wonder if you could or if you are aware of the evolution of that design and why the change was made to two bore seals, and what the rationale for that change was.
MR. KILMINSTER: As I recall, Dr. Walker,
there were probably three things that entered into that decision. Number One was that if we maintained that configuration, as you will recall, at the end of the clevis leg, where there would be an O-ring groove, that would become a very delicate item to have on the end of the segment, and subject to easy damage. That was one consideration.
DR. WALKER: Do you mean the thickness of the walls was insufficient, you felt?
MR. KILMINSTER: No, not the thickness of the walls, the O-ring groove. There was a notch, as I recall, in that clevis where the O-ring sat and then fitted against a face that was on the tang side. That was one side.
DR. WALKER: I haven't understood the objection yet. Are you saying that the walls of the O-ring groove were too thin, or what was the problem there?
MR. KILMINSTER: Instead of having a total thickness of the clevis end, you would have a groove there where the side walls of the groove, if you will, are thin and therefore subject to damage.
[1600] MR. EBELING: We set the cases periodically on-end, and said if you would have done that you would have ruined that $100,000 plus component because you would have deformed that fragile edge, and the distance on
that O-ring groove is what he is replying to.
DR. WALKER: But of course a face seal is a much more reliable seal. In fact, it is one that satisfies your criteria, as a matter of fact.
MR. KILMINSTER: Let me address the other two items. The analysis showed that we were going to get elongation and therefore unseating of that face seal on the order of 20 to 30/ 1,000ths during pressurization. That is in the wrong direction. It wouldn't maintain the seal, the squeeze, if you will, during pressurization. It would tend to move away.
And then a third consideration, and I believe those are the three main ones, was that at that time we were trying to extend the total length of those segments so that we would not have to have a weld in those segments, in that high strength D6 material, and we were exceeding what the fabricator had previously done. To put that other notch on there, to form the face for the O-ring to match against would have required more beef in the billet at that point, and there was some question whether he would be able to give us the length of segment that we desired, so those were the three main things, I believe.
DR. WALKER: Were you concerned in making that change that you were going away from the standard way in
which O-rings were to be used? You were going away from a seal which was a positive seal, which was at least in principle independent dynamics? And I am not sure that I agree with your second point, but you were going toward a configuration which was not recommended by the seal industry?
MR. KILMINSTER: To the contrary, going the way that we did was in keeping with the existing Titan configuration with the bore seal, with all of the history that they had behind that.
DR. WALKER: I am not talking about the solid rocket motors. I am talking about the seal industry. I am talking about recommendations from Parker and other people who had developed O-rings.
MR. KILMINSTER: Subsequently we did have discussions with Parker, and initially they had the same comments that you just mentioned. Subsequently, however, they came back to us and said, with the data base that we had at that time, with the work that had been done, they could understand our utilizing the seal in that configuration.
DR. WALKER: Are those comments documented?
MR. KILMINSTER: Yes, sir.
DR. WALKER: Could you see that we get copies of those?
MR. KILMINSTER: Yes, sir.
DR. FEYNMAN: Mr. McDonald, Marshall has been making some tests, and so have you been making some tests of the seal and so on in various kinds of jigs and small-scale and that sort of stuff. Do you have something to tell us in which you presumably disagree in some way with the conclusions or the attitudes that we have been getting, because we got most of our information directly from them and not directly from you, and I wanted to correct that, if possible, if you had something to suggest about the way we interpret the results, and have you seen the kind of results that they have had?
[1601] MR. McDONALD: Yes, I have seen, I believe, most of them. I don't know if I have seen them all. I don't know what their final recommendations-I guess my comments would be based on the last presentation I am aware of that was made on the 10th of April. I think it was made by Marshall to the task force, and the conclusions from that presentation were that if you look at the conclusions, they don't even mention temperature in there at all, but even where they do mention temperature as having an effect, that it by itself couldn't explain the problem. It had to be in conjunction with other things.
DR. FEYNMAN: Do you mean like the seal fitting into the groove?
MR. McDONALD: That is correct, and it is obviously, I think, biased towards potential assembly problems, either through the assembly itself or contamination of things that one can't specifically prove other than that there is no indication in any of the records or any of our prior history that this was outside of that, and if you look at their own chart on the dynamic O-ring test, you can go across that chart and see at 25 degrees where they never had a single success. It was 100 percent failure in both the primary and secondary O-rings.
If you go across the same chart and look at 55 degrees and up, there was not a single failure. There was 100 percent successes, and it was like 17 tests out of 17 at the higher temperatures and ten out of ten at the lower temperatures.
GENERAL KUTYNA: But, Al, if I can interrupt, the chart was presented as a compilation of your data and their data, and what is the ordinate on that chart? It's temperature, right?
MR. McDONALD: Yes.
GENERAL KUTYNA: And the bottom line of that chart is, boy, when it gets cold, you start failing. Is
that not true?
MR. McDONALD: Well, that is absolutely true.
GENERAL KUTYNA: Then how can you say they have not considered temperature?
MR. McDONALD: Well, they considered it, but looking at that chart
GENERAL KUTYNA: Where were the reds on that chart?
MR. McDONALD: That is exactly the point I am making. And how you can look at that chart and then not conclude-I mean, you are making my point.
GENERAL KUTYNA: I think we are saying the same thing.
MR. McDONALD: I look at that chart and I don't know how anybody can not conclude that that wasn't the major driver, if not the whole thing.
VICE CHAIRMAN ARMSTRONG: Well, Mr. McDonald, it is this Commission that is going to make the conclusions.
CHAIRMAN ROGERS: That is right. I wonder, the discussion that you had with Mr. Mulloy the one you were talking about a moment ago, when was that?
MR. McDONALD: I can't recall the exact date. It was when Mr. Sutter's team was out to our plant.
MR. DUPREE: March the 17th timeframe.
CHAIRMAN ROGERS: At that time Mr. Mulloy was advised he was not to have anything to do with the investigation, and not to take part in the investigation. And you testified that he advised you about how you should convey information to the Commission?
[1602] MR. McDONALD: Well, he asked me why I was doing it, what I was doing, and why I was taking the liberty of giving information directly to the Commission without going through my own management and through NASA first. That is exactly what he asked me. And it wasn't in a nice voice.
CHAIRMAN ROGERS: Do you feel now that you have had the opportunity to present as much information that you want to the Commission? We want to be sure that anybody that has any comments or information, that they have direct access to us. Do you feel that you have had?
GENERAL KUTYNA: Mr. Chairman, let me interject.
CHAIRMAN ROGERS: Well, let him answer first.
MR. EBELING: Does that apply to all of us?
CHAIRMAN ROGERS: To Mr. McDonald.
MR. McDONALD: Well, in violation of Mr.
Mulloy's concern again I gave some information directly to the Commission without going through the proper channels. I sent a copy to Dr. Keel and also to General Kutyna on a memo that I drafted to my boss about a week ago, and I informed Mr. Garrison last night about that on my conclusions from the failure analysis from the data that I have had access to, recognizing that I haven't seen everything as to what I concluded I think happened and what may have happened, and what seems very conclusively has happened, and what I feel is necessary to make sure that we understand those things that are well substantiated and those that may be speculative.
CHAIRMAN ROGERS: Well, will you feel free to give the Commission any information you want to give us, and if you consider what Mr. Mulloy told you as a direction, forget it, because we are running this investigation, and you have access to us any time you want, and that applies to all of you.
GENERAL KUTYNA: Just for the record, sir, I learned of this possibility that some things had not gotten through early last week, and Mr. McDonald and I have had a conversation where I wanted to assure that his concerns, both his and Mr. Boisjoly's and whoever else-is that not true?
MR. McDONALD: It is true.
MR. RUMMEL: Mr. Chairman, just in the interest of completing the record and accuracy, Mr. Mulloy did sit in on some of our meetings up at Thiokol and did participate in the discussion quite freely. I thought personally that it was generally more helpful than not. Whether he should have or not, I don't think we chose to impose any restrictions.
CHAIRMAN ROGERS: Well, to make it clear, and I hope it is clear, and I think it has been, the Commission's decision to disqualify those who might have been involved in the decision making to launch 51-L was that they should not be responsible for matters involving the investigation. That was not to preclude them from giving information or preclude us from asking for information from them, but any instructions from Mr. Mulloy about how the investigation was going to be conducted or how information was going to be conveyed was directly opposite what we told them. They all knew that, And I was very surprised to hear Mr. McDonald say that.
Now, Mr. Mulloy, I guess, says he didn't say that.
MR. MULLOY: Mr. Chairman, may I make a clarifying comment?
CHAIRMAN ROGERS: Sure.
MR. MULLOY: From my vantage point, what I was talking about was not any information related to the investigation. The discussion I was having as to why Mr. McDonald didn't express any of the concerns that he expressed initially in the closed door hearing on a Friday at KSC in the initial discussion where he stated that he had continued to protest against the launch after the decision was made, and my question was why, if he had those concerns, did he not express those concerns to his management on the night of the 27th or on the morning of the 28th?
Now, what I heard him say was, he interpreted something different than that.
CHAIRMAN ROGERS: It wasn't an interpretation. He said you questioned him about why he was giving information directly to the Commission and not going through NASA and his own management. Did you say that?
MR. MULLOY: Yes, sir. What I said was related to the events of the 27th. My question was, if he had all those concerns, why he did not relay those concerns to his management on the night of the 27th and up to launch time on the 28th.
CHAIRMAN ROGERS: So there was a misunderstanding. You were talking about why he didn't
do something on the 27th. And he construed that to mean something else.
MR. MULLOY: And waiting until the Commission hearings on that Friday to express those concerns when they had not been expressed to his management in time to do anything about the launch of 51-L.
MR. ACHESON: I don't want to stop this episode, but I do have two questions for Mr. Kilminster.
CHAIRMAN ROGERS: Go ahead.
MR. ACHESON: One goes to the design of the joint. A lot of the material we have received, one reads that the designers, presumably both the corporate designers and the NASA supervisors, believed that the joint was designed to compress and seal in the gas tight under combustion pressure. And it turned out very quickly in the joint history that it did the opposite. It opened up.
Now, to a layman it is kind of hard to understand why you wouldn't at that point say this thing works the opposite of what we thought, so there is a lot about it we don't understand, so let's go back to the drawing board and design a new joint that works the way we want it to. If you design an aircraft and you discover the windows are almost but not quite pressure tight, you don't tell the passengers to wear oxygen over
20,000 feet. You send it back to the manufacturer and you say, make it right.
I just don't understand why the program then developed to go into a lot of little fixes to see if you could compensate for a fundamental error, and maybe you can explain that to us.
MR. KILMINSTER: Again, I think we have to refer back to the experience base that we used to design that configuration in the joint, and that was the Titan joint. The Titan joint has the same gap tolerances, the same diametral tolerances that we have in our joint. It has the same bore seal. It has the same material in that seal, the O-ring seal. However, I believe their O-ring is slightly smaller than ours in cross-sectional diameter, .275 versus .280.
When we first found out, and I think it was due to our hydrotesting, hydroburst test program, and eventually the structural test, Article 1 testing we discussed before, when we found out there that there was not a closing of the gap on pressurization, there was an opening of the [1604] gap, then we started to look, and I understand at that time we found out that the Titan gap also opened, and the number that I recall is 28/1,000ths, where ours is in the 40/1,000ths, 42/ 1,000ths or thereabouts.
So, again, the basis that that system had been functioning, functioned acceptably with basically the same material, the same O-ring, the same configuration, same diametral tolerances, we thought it was reasonable to proceed with that basic design.
DR. WALKER: You are telling me this accident was the fault of the Air Force and the Titan program?
MR. KILMINSTER: No, sir, I am just answering the question.
DR. WALKER: I just do not understand that remark. You have got a system which you designed, and it behaves exactly the opposite, as Mr. Acheson has said, and it is clearly a serious problem. I don't understand why you are pointing at somebody else who might have the same problem. What does that have to do with your problem?
MR. KILMINSTER: They did not have a problem that was known at that time. They had a long history of success, and it was functioning just fine.
MR. SUTTER: How many successes did they have when they only shoot it into the air once and they don't pick it up and refurbish it? It seems to me that is a hell of a skimpy data base.
MR. KILMINSTER: Well, the analytical work that supported that continued to show, and we have
recently been surprised about some information that you are aware of about diametral growth, but the information, the analytical information continued to show.
MR. SUTTER: Well, don't you know about the information down at Marshall? You inferred that I know something you don't know about diametral growth. That data was presented down at Marshall when we were down there. From the test firings and from the actual firings you get a deformation that stays built in. You are aware of that now, aren't you?
MR. KILMINSTER: That is just what I was speaking to. That is recent information.
MR. SUTTER: Well, that gets back to the comment that I made before. I think part of this accident was caused by not a sufficient data base to design solid rocket booster joints, even today, and before anything flies there ought to be five or ten of these full-scale things shot off, and after you get about ten of them that work right, then you can say you should fly again.
I will go back to the remark-I asked a question of your people in one of the open sessions and I didn't get a satisfactory answer, and it has been bugging me ever since.
My question was, looking back on things, do you feel that that testing that had been done prior-well, with all of these problems going on, was that testing adequate to really give you a sufficient knowledge to fly? And the answer I got is, there is not enough money in the Treasury of the United States to do the kind of testing I want.
I really sort of took issue with that comment, and I would like to just comment back to you now. I am going to see that before anything flies again, there is going to be a lot of full-scale testing or something suitable to back it up, not these little dinky ten-inch tests where a whole bunch of scale effects are missing.
And, you know, I don't see why we all don't say we didn't know what this joint was doing. It worked backwards to the design. There was a mistake made due to lack of knowledge, and let's [1605] get the knowledge and get on with it. There is a lot of pussyfooting going on about what we did or didn't know.
My view of all of the testing that is done, you don't know whether it is temperature, you don't know whether it's tolerances, you don't know whether it's the screwing ups when you have to take
these things and put hydraulic jacks on them and squeeze them into position. I still question the fact that something sitting out there for 28 days, and the temperatures going up and down, and it was sealed once-whether that seal is still holding. The knowledge base, the data base that is referred to on joints and solid rocket boosters 12 feet in diameter is about 100th of what you need to say the article is safe, and somebody ought to recognize that before they say they are going to fly this thing again.
CHAIRMAN ROGERS: Fine. Unless we have a lot of urgent questions-did you want to go ahead?
DR. WALKER: I just had one more. Is the Titan a manned system?
MR. KILMINSTER: No, sir.
DR. WALKER: Do you think that there ought to be a higher standard applied to systems where there are people present? I think that can be answered yes or no.
MR. KILMINSTER: The answer is yes. Obviously, I think I have to ask the question, though, not knowing the Titan safety factors, whether they were established for a manned system. I believe early on there was some talk about using it.
GENERAL KUTYNA: Whether these-have that same factor or not I don't know. However, you know,
when I have a $600 million payload on board, whether I have got a man on board or not, I put a fairly high safety factor in there.
CHAIRMAN ROGERS: Well, thank you very much, gentlemen. I think that will be all.
(Pause.)
CHAIRMAN ROGERS: Let's come back to order.
Gentlemen, the Commission is in the process of concluding its hearings, and in that connection we are reviewing documents particularly as they relate to the joint and the O-ring history. I guess you probably-the gentlemen have been given the documents.
So we just want to refer to them and ask you a few questions about those documents to be sure you have a chance to respond and make any comments you want about them.
(Witnesses sworn.)
[Please note that some of the titles to the references listed below
do not appear in the original text. Titles are included to identify
and clarify the linked references- Chris Gamble, html editor]
[1606] [Ref. 5/2-14 1 of
3] MSFC Problem Assessment System:
Problem Title: O-Ring Erosion in the Case to Nozzle Joint.
[Ref. 5/2-14 2 of
3] MSFC Problem Assessment System:
Problem Title: O-Ring Erosion in the Case to Nozzle Joint.
[1607] [Ref. 5/2-14 3 of 3] MSFC Problem Assessment System: Problem Title: O-Ring Erosion in the Case to Nozzle Joint. [Ref. 5/2-15 1 of 6] MSFC Problem Assessment System: Problem Title: Segment Joint Primary O-Ring Charred.
[1608] [Ref. 5/2-15 2 of 6] MSFC Problem Assessment System: Problem Title: Segment Joint Primary O-Ring Charred. [Ref. 5/2-15 3 of 6] MSFC Problem Assessment System: Problem Title: Segment Joint Primary O-Ring Charred.
[1609] [Ref. 5/2-15 4 of 6] MSFC Problem Assessment System: Problem Title: Segment Joint Primary O-Ring Charred. [Ref. 5/2-15 5 of 6] MSFC Problem Assessment System: Problem Title: Segment Joint Primary O-Ring Charred.
[1610] [Ref. 5/2-15 6 of 6] MSFC Problem Assessment System: Problem Title: Segment Joint Primary O-Ring Charred.
[1611] [Ref. 5/2-16 1 of 2] Morton Thiokol Inc. Interoffice Memo From Manager, SRM Special Projects to A. J. McDonald. Subject: Closure of the SRM O-Ring Erosion Critical Problems.
[1612] [Ref. 5/2-16 2 of 2] Morton Thiokol Inc. Letter From Manager, SRM Special Projects to A. J. McDonald. Subject: Closure of the SRM O-Ring Erosion Critical Problems (continued).
[1613] [Ref. 5/2-17] Morton Thiokol Letter to L.O. Wear Regarding Closure of Critical Problems Numbers.
[1614] [Ref. 5/2-18] Morton Thiokol Inc. Interoffice Memo From Manager, SRM Special Projects to D.E Thompson. Subject: Rationale for Closure of the O-Ring Erosion Problem.
[1615] [Ref. 5/2-19 1 of 2] Morton Thiokol Inc. Letter From Brian Russell, Manager, Special Projects to L. O. Wear. Subject: Request for change order on SRM seal anomalies effort.
[1616] [Ref. 5/2-19 2 of 2] Morton Thiokol Inc. Letter From Brian Russell, Manager, Special Projects to L. O. Wear. Subject: Request for change order on SRM seal anomalies effort (continued).
[1617] [Ref. 5/2-20 1 of 2] Morton Thiokol Inc. Interoffice Memo From Manager, SRM Ignition System, Final Assembly, Special Projects and Ground Test to A.J. McDonald. Subject: Weekly Activity Report, 1 October 1985.
[1618] [Ref. 5/2-20 2 of 2] Morton Thiokol Inc. Interoffice Memo From Manager, SRM Ignition System, Final Assembly, Special Projects and Ground Test to A.J. McDonald. Subject: Weekly Activity Report, 1 October 1985 (continued).