Ch7-2

Computers in Spaceflight: The NASA Experience

- Chapter Seven -
- The Evolution of Automated Launch Processing -

Launch processing in the Saturn era

[210] A Saturn V rocket with an Apollo spacecraft on top presented a magnificent sight, which engineers nonetheless viewed with a mixture of prideful awe and dread. No earthly booster since the Skylab launch has been as large or as powerful. Shuttles being mated in the Vehicle Assembly Building originally designed for the Saturn appeared as dwarfs in houses made for giants. It looked as though there was nearly enough room to stack them two high. The dread came from the fear of failure among the thousands of components, many capable of bringing disaster and killing a crew in flight. Early in the Saturn program automation began to be introduced in the testing of the gargantuan rockets. Marshall Space Flight Center acquired computers for Saturn vehicle checkout. Marshall also had responsibility for the Launch Vehicle Digital Computer (LVDC) housed in the Instrument Unit that [211] was the last stage below the Apollo spacecraft on both Saturn IB and Saturn V configurations. NASA's Launch Operations Center, later renamed Kennedy Space Center, acquired computers for telemetry data reduction and display and began work on the checkout systems used for the Apollo spacecraft, a project later transferred to the Manned Spacecraft Center in Houston. Each of these computer-controlled systems contributed to the concepts and development of the Shuttle Launch Processing System, now wholly based at the Kennedy Space Center.


Checkout of the Saturn Vehicle


Marshall Space Flight Center in Huntsville had primary responsibility for the design, manufacture, and flight preparation of the Saturn vehicles. In 1951, when Marshall was still the headquarters of the Army Ballistic Missile Agency, Kurt H. Debus formed a launch team that commuted to the Air Force's Eastern Test Range in Cape Canaveral, Florida. Within a short period of time, the frequency of launches made it necessary to establish a permanent group at the Cape, called the Missile Firing Laboratory. When Marshall was established on July 1, 1960, the Laboratory was renamed the Launch Operations Directorate. By 1962, the activities at Cape Canaveral grew to the level that the Launch Operations Center was formed separately from Marshall and given status equal to other NASA centers. However, its charter stated that the centers responsible for a particular vehicle or spacecraft had to perform its checkout and test, so during the Apollo era Marshall prepared Saturns and the Manned Spacecraft Center worked on the Apollos. Personnel at the Launch Operations Center performed facilities management and provided telemetry data reduction.

Computers were used both on-board the Saturn vehicles and in preparing them for flight.^* Ten Saturn I vehicles were launched [212] between 1961 and 1965. Each was unmanned, the series being used primarily to demonstrate that clustered-engine first stages and high-energy upper stages were feasible. The first five launches did not use a computer for guidance. Each was a suborbital mission utilizing a German-made mechanical time-tilt device for control². On the fifth flight, an ASC-15 computer, built by IBM originally for the Air Force's Titan, flew as a passenger and handled telemetry transmissions³. It guided the last five missions, several into earth orbit. When Saturn evolved into the IB and V series, an Instrument Unit containing the LVDC was mounted atop the S-IVB stage on each vehicle. Termed the "integrating element" of Saturn, IBM was not only responsible for its computer but for its construction⁴. Besides the computer, the Instrument Unit contained the Launch Vehicle Data Adapter as an I/O front end, analog control circuits and an ST-124 guidance platform. On lunar missions the LVDC guided the spacecraft until the S-IVB stage separation after the lunar trajectory insertion.

IBM's LVDC was architecturally quite similar to the Gemini guidance computer⁵. It used nearly the same instruction set, 26-bit data words and 13-bit instructions. One difference was that the memory had two-syllable locations instead of Gemini's three. Construction of the LVDC, however, was radically different. For reliability reasons, triple modular redundant (TMR) circuits were adopted. Even though the component count went up just 3.5 times, the reliability increased 35 times⁶! Three logic channels, each with seven functional modules, required 395 voters⁷. Packaging the computer used techniques developed under the Advanced Saturn Technology Program commissioned by Marshall and executed by IBM⁸. First of the "flat pack" integrated circuit series, IBM applied this silicon semiconductor technology in its System 360 commercial machines⁹.

Use of a computer in the launch vehicle led directly to using ones for checkout. Marshall bought an RCA 110 to communicate with the IBM ASC-15 used in the Saturn I. Later, RCA upgraded its machine by enlarging the memory to 32K 24-bit words of core and an additional 32K on an associated magnetic drum. When the Saturn IBs began to be launched, discrete circuits for interfaces with the rest of the launch vehicle were added¹⁰ .Renamed RCA 110As, these computers continued to be augmented to handle more communications circuits, so that by the time Saturn Vs appeared, the computers could maintain the status of each of 1,512 signal lines¹¹. At first the 110s simply handled communications and switching. Activating test procedures and conducting tests were still done manually. But in 1962, IBM suggested that Chrysler convert the 110s they used for stage checkout of the Saturn I to do the tests automatically¹². Even though the advantages of automating procedures seem obvious, chief among them the fact that all are done exactly alike, it was difficult to get people responsible for checkout to convert from doing things.....

[213]

FIGURE 7-2. IBM engineers work inside the Saturn launch vehicle Instrument Unit. (IBM photo)

.....manually, a theme repeated in other parts of the Apollo program¹³. At that time, computers were seldom used for on-line work, and most engineers were still unfamiliar with them and wary of any more innovations in an already innovative program. However, Chrysler converted some factory tests to automatic, using a special language, "HYLA," to define them. Additionally, several Packard Bell computers connected to a common memory automatically checked out parts of the Saturn I. Use of a common memory as a computer interconnection device reappeared in several later systems and is critical to the success of the Shuttle's Launch Processing System. Engineers wrote the language "SOL," or Systems-Oriented Language, for the Packard Bell [214] machines. By late 1962, the Saturn V stage contractors accepted the concept of automatic checkout and settled on a common system, the Control Data Corporation CDC-924A computer, as the factory test machine, with 110As assigned to the S-I stage and for the assembled vehicle at the launch site¹⁴.

By this time, it was clear to Ludie Richard, a NASA engineer, and his team at Marshall that preparing a language to help test engineers write automated procedures was the key to continued acceptance of the principle. A custom-designed programming language would leave control over the definition of the tests in the hands of the engineers, avoiding communication problems that might arise with computer programmers inexperienced in checkout techniques¹⁵. IBM eventually wrote routines for the RCA computers in assembly language, but the majority of the automated tests were ATOLL (Acceptance, Test, or Launch Language) programs stored on tape. Richard acquired the over two dozen RCA 110As that were eventually used. His deputy, Charles Swearingen, was put in charge of managing the flight computer, ground computer, and checkout software¹⁶. James Lewis and Joseph Medlock were instrumental in developing the checkout systems and defining ATOLL¹⁷. IBM wrote both the flight programs and the Saturn Operating System that ran on the RCA computers and executed ATOLL procedures.

By mid-1963 the final configuration of the Saturn checkout computers was set by Richard's group. At Launch Complex 34, the Saturn IB launch site, one master RCA 110A was in the blockhouse and a slave underground at the pad. For Saturn Vs at Complex 39, one RCA 110A was located in each of the four firing rooms in the Saturn Launch Control Center, which was attached to the Vehicle Assembly Building in which the Saturns would be stacked. Each of four mobile launchers also contained a computer. In addition to the 110As, the firing rooms also had a DDP-224 minicomputer as a display driver for the CRTs showing output data to the engineers, as well as a controller for slides and other visuals. Computers in the mobile launchers could be used for checkout in the Assembly Building as well as at the pads, a foreshadowing of the later Launch Processing System. Due to reliability problems with the 110As, the launcher computers used a dual memory configuration. Checkout programs filled just half the memory, so the other half acted as a duplicate for redundancy, the same principle as applied to the LVDC memory.

Part of the credit for the perfect success record of the Saturn vehicles (all Saturn I, IB and V boosters flew without a failure) must be due to the effectiveness of the checkout procedures. Without automatic testing the confidence in the rockets could not have been attained, since they were too complex for effective manual procedures. In addition to checkout methods specific to the launch vehicle, the launch directors in the firing rooms had access to automated test data....

[215]
FIGURE 7-3. A block diagram of the automated preflight checkout hardware for the Saturn launch vehicle. (From IBM, SLCC Programming System)

....from the spacecraft preflight test equipment developed by both the Launch Operations Center and Manned Spacecraft Center.


Development of Apollo's Acceptance Checkout Equipment

From the first Apollo earth orbital flights through the lunar missions, Skylab, and the Apollo-Soyuz Test Project, ground testing and countdown support of the spacecraft and its associated systems were the responsibility of the ACE, or Acceptance Checkout Equipment^** .

[216]
FIGURE 7-4. A block diagram of the Saturn Operational Display System. (from IBM, SLCC Programming System)

[217]
FIGURE 7-5. Typical firing room layout during the Saturn era. Nearly 250 engineers would crowd this area during a countdown. (From IBM, SLCC Programming System)

[218] ACE stations were located in the Apollo Operations and Checkout Building in the Industrial Area of the Kennedy Space Center, at Launch Complexes 34 and 37, at the Johnson Space Flight Center, and at North American Aviation and Grumman Corporation assembly plants. Two of the North American stations were modified from use in assembling the Shuttle in Palmdale, California¹⁸.

ACE resulted when a spacecraft checkout engineer figured that there had to be an alternative to manual methods. Thomas Walton transferred from a job in the computer room at Langley Space Flight Center to the Cape Canaveral Launch Operations Center in early 1961. Assigned to the checkout of Mercury spacecraft, by September he had enough of manual testing and applied his background in computers to devising another way of doing things¹⁹. Walton convinced his boss, Harold G. Johnson, to let him build a digital ground station for telemetry from the capsule. Using the Mercury missions as prototypes, he proved that digital equipment could display the engineering data previously shown on dials and strip charts. His success led to a search for a computer system to handle the data in real time. With NASA's Gary J. Woods, he traveled to several companies in search of a machine. Walton did not believe that a computer like an IBM mainframe of that era could do the job. Since they were designed for large-scale batch processing, the difficulties of adapting such a computer to the real-time world of telemetry displays and automated checkout would be too great. Instead, he and Woods looked for simpler minicomputers such as the Digital Equipment Corporation PDP-1 and the Control Data Corporation CDC-168²⁰. Walton convinced the Gemini Project Office to buy a pair of CDC-168s to be used for checking out their spacecraft. Meanwhile, plans continued to create a system dedicated to the Apollo.

Marshall and Launch Operations personnel met in 1963 to determine whether the checkout equipment for both the Saturn vehicle and the Apollo spacecraft could be combined. Richard's and Walton's teams decided to continue separate paths²¹. The results were the Saturn checkout system and the first ACE unit using General Electric discrete equipment and CDC-168 computers going on line in late 1964.

Although the first ACE stations were under construction, a small political battle was raging over who would have ownership of the program. Joseph Shea, then at NASA Headquarters in Washington, wanted to control it from the Apollo Project Office there. Transferred to Houston in 1963 to take over management of Apollo, he moved the ACE development group of between 50 and 60 persons there instead²². This act reinforced the feelings that the Kennedy Space Center was to be strictly an operations center, staying clear of research and development activities.


[219]
Figure 7-6. An ACE station with twin Control Data computers. (NASA photo 107-KSC-67C-919)


Each ACE station used two digital computers with a common memory. One was the Digital Command Computer, which processed commands from the control or firing rooms to the spacecraft and was interconnected with the ground support equipment. A second machine was the Data Processing Computer, which drove the displays and controlled peripherals. Memory could be directly accessed by discrete circuits in the ground station, so data for both computers could be placed there²³. ACE stations could function in manual command mode, semiautomatic, or fully automatic with manual override. Stations in the Kennedy Industrial Area serviced the spacecraft both before and after mating with the Saturn V boosters. When Launch Complex 39 went into use, checkout wires carrying digital formatted data ran over 15 kilometers from the ACE stations to and from the pad and firing rooms in the Saturn Launch Control Center using a video wideband transmission system²⁴. Of course ACE had to cooperate with the RCA 110As at various points, so interfaces between the different computers consisted of dedicated I/O registers, sense lines, and priority interrupts²⁵. ACE also had to talk to the Apollo guidance computers in the command module (CM) and the lunar excursion module (LEM). On the average, the CM computer operated for 50 hours in support of the countdown. A CRT display controlled by ACE duplicated the data shown on the display and keyboard while the Apollo computer was in operation.²⁶

[220] Walton judged that the development of ACE did a lot to stimulate the technology of on-line processing. Certainly it helped create techniques of interconnecting multiple different computer systems. Also, this was one of the first times that data transfers in the megabit range were accomplished over distance.


Digital Displays of Telemetry

Telemetry transmissions from the vehicle are one important source of data for rocket engineers. In the early days of rocket flight research, the causes of failures often could only be guessed. When the larger size of later rockets made it possible to carry radios for sending back data, sensors were added to supply engineering data from critical components throughout the flight of the vehicle. If a failure occurred, it was often possible to determine which components contributed most to it by examining the reams of data sent back and originally recorded in analog form on the ground. However, Tom Walton's pioneering digital ground station for Mercury displayed the data in processed digital form. In 1962, the Atlas-Centaur project automated postflight telemetry data reduction²⁷. By the mid-1960s, digital telemetry displays were standard at Kennedy Space Center, provided by a pair of mainframe computers in the Central Instrumentation Facility.

Kennedy acquired two General Electric 635 computers for telemetry monitoring and batch processing of institutional programs. GE 635s were 36-bit processors capable of double-precision arithmetic²⁸. Programmers prepared separate code for each of the Delta, Atlas-Centaur, and Saturn flight vehicles. Delta and Atlas launch pads, as well as Complexes 34, 37, and 39, could be switch connected to the computers at any one time. Forty different displays were possible and could be transmitted to the appropriate blockhouse or Launch Control Center firing room²⁹. NASA's Bruce Miller was in charge of systems programming for the GE computers, with Bradley Hughes as the chief scientific programmer.

These computers had the longest operational life of any installed at the Kennedy Space Center. GE delivered the first machine in late 1965. A second came on line in early 1966. Until May of 1983-18 years later!- one was still in use driving blockhouse displays for Delta and Atlas-Centaur. GE had long been out of the computer business by then, having sold its digital computer division to Honeywell in the early 1970s. Kennedy retained a permanent systems programmer from GE (who later moved to Honeywell) to keep the operating systems going and used a retired blockhouse 635 from Wallops Island as a source of parts³⁰. From the beginning the computers had a dual operating system. Batch institutional jobs could be run at the same [221] instant a real-time telemetry program was running, except when a Saturn was being supported, as its program was so big it pushed out the batch programs. When Kennedy Space Center officials searched for a second machine for the Central Instrumentation Facility, they considered other vendors. IBM's branch manager in Cape Canaveral, W. O. Robeson, sent a letter suggesting a System 360/50 as an administrative computer, pointing out that evidence from prior telemetry computers indicated that they rarely failed³¹. The dual operating systems could then be abandoned. However, Kennedy bought the second 635 to provide a redundant backup anyway, accepting the loss of batch processing during Saturn operations.

Telemetry data reduction computers thus provided yet another source of information to the launch directors in the Apollo/Saturn era. Still, some engineers were convinced that the computer data were never accurate, just as their colleagues in the checkout world had to be dragged into automation³². Regardless, telemetry displays became an integral part of the technology of launch processing.


Impacts of the Apollo/Saturn Era on the Shuttle Launch Processing System

Developing the major computer components of the launch processing system for Apollo/Saturn provided software contractors such as IBM and the Kennedy Space Center staff valuable experiences later transferred to the Shuttle Launch Processing System and onboard software for the Shuttle program. Additionally, some techniques known in theory, but never properly applied, found justification during the Apollo/Saturn programs. The areas of impact included the modularization of software, lessons learned by IBM as a key future contractor, and Walton's continued influence on ground computer concepts.

Software written for the LVDC and the GE 635 computers started as single monolithic programs and evolved to modularized programs at just about the same time. Flight software for the ASC-15 computer used on the Saturn I vehicles was necessarily monolithic because it had to be sequentially executed and strictly timed³³. Any changes impacted on the execution time, and therefore had to be carefully integrated. The computer could not handle waiting for an interrupt to instigate an action. Actions had to be initiated by the program relative to its starting time. When the ASC-15 gave way to the LVDC, a more powerful and flexible machine, programmers continued in the monolithic mode. Finally, IBM staff realized that by preparing the software in essentially free-standing chunks, the impact of changes would be limited to the modules and not spread side effects throughout the software. This discovery came late in the Saturn program but [222] early enough to affect the development of the Skylab on-board software. Also, IBM separated the modules into groups consisting of the control subsystem and applications subsystem, which is a prototype of the Shuttle on-board software organization³⁴. IBM helped transfer this concept to the Shuttle by moving people such as Kyle Rone and Lynn Killingbeck from working on the Saturn computer directly to the Houston office to support the Shuttle software development. NASA also independently moved toward modularization when, in 1973, it broke down the programs used on the GE 635s to support telemetry data reduction. Before then, it took an average of 3 months to implement a simple change in the monolithic version of the program, because of the massive debugging necessary to eliminate side effects³⁵. Thus, modularization came to be expected by NASA as part of software design. If modularization was not used on the Shuttle on-board software, preparing new flight loads would have been impossible within the projected time between flights of an individual orbiter.

Besides modularization, Apollo/Saturn significantly influenced IBM's later work on the Shuttle's on-board software, especially the company's design of the system used for Shuttle launch processing. IBM summarized its conclusions in a document released in late 1972, just at the time both Shuttle ground and on-board software contracts were being let³⁶. The study recommended that the vehicle's flight software be capable of reloading all programs on board³⁷. This was implemented on the Shuttle, as the mass memory units (MMUs) contain all preflight and flight software for the primary avionics computers, the display computers, and the engine control computers. Ground software recommendations required that all checkout functions use a higher order language and that checkout be conducted using one computer system³⁸.

During Saturn, both ATOLL and machine language programs controlled preflight tests, the machine language routines absorbing a considerable amount of development and maintenance time. This lesson helped spur the creation of an improved checkout language, GOAL. In regard to consolidating all functions in one computer, IBM thought that the difficulties of integrating the two RCA computers, the DDP-224 display computer, and the telemetry reduction computers were excessive. By taking that position, IBM found itself squared off against the distributed concepts envisioned by Tom Walton and his team for the Shuttle system. Walton refused to move to Houston when Shea transferred the ACE team. By staying at Kennedy, he was able to influence the structure of the Shuttle Launch Processing System and help make the Center fully responsible for all checkout and launch operations for the entire vehicle, a significant change from the Apollo/Saturn program.

^* For a complete description of the evolution of the Saturn and its components, see Roger Bilstein, Stages to Saturn: A Technological History of the Apollo/Saturn Launch Vehicles, NASA SP-4206, 1980. Chapter 8 centers on the use of computers in checkout and the development of the Instrument Unit and its flight computer. Chapter 16 of Charles D. Benson and William B. Faherty, Moonport: A History of Apollo Launch Facilities and Operations, NASA SP-4204, 1978, describes the development of automated launch operations. Due to these prior treatments, my account will concentrate on briefly summarizing the use of the computers to provide the necessary introduction to the section on the Launch Processing System, rather than retelling the whole story. Some new evidence is presented where applicable, but the reader is urged to consult both previous works.

^** The acronym ACE evolved from PACE, or Preflight Acceptance Checkout Equipment, which appears in some of the literature. It was discovered that the name conflicted with a commercial product, so the "Preflight" was dropped. Prior to PACE, there was a short period when the equipment was known as SPACE, but apparently not officially.