Ch9-3

Computers in Spaceflight: The NASA Experience

- Chapter Nine -
- Making New Reality: Computers in Simulations and Image Processing -

Image processing



[287] Image processing is one area in which NASA, primarily through work done at JPL, clearly leads the field. Ironically, even though the production of high-quality images from space probes and Landsat earth orbiters has great scientific and public relations value, the concept of digital image processing was not incorporated in the original planning of a number of early missions. Instead, it had to gain acceptance as a "tack-on" to the Ranger and Surveyor programs⁵¹. Robert Nathan led the development of digital image processing in its early stages, and with the technical help of other JPL scientists, won for it a featured place on the planetary missions of the late 1960s and beyond. Of the early resistance, he later said that he "had to prove to [project management] each time what they needed" to get the most out of the first American pictures coming from space.

[288] Nathan came to the California Institute of Technology as a graduate student in 1952. He earned a Ph.D. in crystallography in 1955 and soon found himself running CalTech's fledgling computer center, where he received a good grounding in the potential of digital computers. In 1959, he went to JPL to help develop imaging equipment to map the moon. When he saw the Russian pictures of the far side of the moon, he thought he could do better and began developing digital techniques for image enhancement using a small NCR 102D computer. Nathan reasoned that analog equipment, such as television cameras, could only be controlled by hardware changes, just like an analog computer can only have its internal program changed by rewiring or switching components. However, digital processing allows changes to be made with software, allowing a wider variety of enhancements⁵².

Before an image can be processed, it must be put into digital form. Frederick Billingsley and Roger Brandt of JPL devised a Video Film Converter (VFC) that could transform analog video signals, such as those sent back by Ranger spacecraft, into digital data. While they supervised the construction of the device, John Morecroft of JPL used the NCR computer to begin programming processing algorithms. These events took place in 1963, and by the next year Howard Frieden had programmed the Laboratory's institutional IBM 7094 computer to process Ranger data. Success with Ranger images led the Surveyor project to use Nathan's techniques, as well as Mariner Mars 1964. By the Mariner Mars 1969 missions, the concept of digital image processing was fully accepted.

Why is image processing needed? Due to the resolution and design of the video cameras used to make the images, they must be processed in order to return the most information possible. The surface of Mars is such a low-contrast object that without enhancements, features would be lost in the wash of monocolor⁵³. Also, because the human eye cannot adjust to differences in illumination across a field of view, illumination must be normalized⁵⁴. The cameras operate by taking an instantaneous view of the scene; the values of the light impressed on the vidicon tube are then made into digital data. Since images are taken one after the other, very close together in time, residual images from prior "snapshots" affect the current view⁵⁵. These residual images must be removed, a technique that took several missions to perfect. Finally, noise from transmitting a signal over planetary distances must be accounted for.

To see how such processing is done, the real-time display system used for the Mariner Mars 1971 orbital mapping mission provides a useful example. A UNIVAC MTC 1230 computer extracted 9-bit pixel data from the telemetry stream. A pixel is a single picture element, or dot. The spacecraft had a camera capable of recording frames of 700 lines by 832 pixels, or 580,000 individual dots. Such large [289] numbers of pixels were only practical as interplanetary communication advanced. Mariner Mars 1964's 200 by 200 pixel imaging equipment transmitted at the rate of 8 1/3rd bits per second. Thus, it took nearly one entire shift at a Deep Space Network station to record a single frame. At that data rate it would take over 1 week for a Mariner Mars 1971 frame! But by 1971, the data rate increased to 16,200 bits per second, giving a complete picture in 5 minutes and 40 seconds. Even these rates increased by over seven times in the next few years.




Figure 9-7A. Image processing's decade of progress: Mariner Mars 1964 returns the first closeups of Mars.

[290]

Figure 9-7B. As the planetwide dust storm clears, Mariner Mars 1971 scans Nix Olympica in January, 1972.
[291]

Figure 9-7C. Details from the Valles Marineris canyon taken by the Viking Orbiter in 1976. (JPL photos P-7875A; P-13074; P-17872)



[292] Several techniques could be applied to the data by the computer. Contrast stretching helped increase the contrast of the single color Martian surface. Original values of the pixels ranged from 0 (black) to 255 (white). The computer truncated these to 6 bits, which yielded 64 levels. Since humans can only discern about 25 levels of grayness, this was more than enough. By increasing the brighter grays toward the white end of the scale and decreasing the darker grays toward the black end, the contrast was increased⁵⁶. Illumination could also be normalized using the computers. A "high pass filter" corrected the value of the pixels by averaging the immediately surrounding 125 pixels and then subtracting the running average from the value of the pixel⁵⁷. Another process compensated for geometric distortion. Simply because of the way the cameras were made, there was distortion in the image frames. Reference points marked on the image served to help distortion elimination algorithms properly square off the image. These techniques were also applicable to developing mosaic maps by taking images shot at oblique angles and flattening them out in any one of several projections⁵⁸. Noise elimination could be done by assuming that any pixel exceeding a difference of 32 levels of brightness from its neighbors was a spike and then changing the value of the spike to the average of its two immediate neighbors. From 20 to 10,000 spikes could be found on a single raw image, so without removal the image would be noticeably damaged⁵⁹.

Aside from the near-real-time imaging provided by the UNIVAC and other computers on later missions, long-term processing with a number of techniques is done in the Image Processing Laboratory at JPL. First established in 1965 with a new IBM 360/44 computer that lasted 10 years, the Processing Lab pioneered new imaging techniques and developed support software to implement them. Central to the success of image processing was the Video Information Communication and Retrieval language, or VICAR. Written in 1966 after a design by Stan Bressler and Howard Frieden, VICAR enabled users to define a pipeline of processes without having to use cumbersome job control language. For instance, VICAR could define an image file to be processed and then specify the type of processing to be performed on it in a sequential manner. Output from the stretching program could thus be directed to the input to the geometric transformation program. The existence of this language significantly increased the value of the imaging⁶⁰.

By 1975, when a 360/65 replaced the older computer, the Image Lab did roughly half of its work on planetary imaging and half on earth resources work using Landsat images⁶¹. Also, by that time numerous spinoffs from the program began to turn up in other fields, chief among them astronomy and medicine. Astronomers now use digital techniques to enhance their photographs of celestial objects in the same way spacecraft images are processed. Nathan left the....

[293]

Figure 9-8. Increasing contrast enhances a Mars image. (JPL 511-4353)

[294] ....planetary imaging to his colleagues in 1968, when he turned his attention to a series of grants from the National Institutes of Health to study applications of digital image processing to microscopy and medical diagnosis. Robert Selzer of JPL had applied the techniques to x-ray enhancement. For Nathan, with a background in x-ray crystallography, this was a natural step. Unfortunately, by 1973 the government canceled all fundamental research grants in the field and Nathan found himself without support and nearly without a JPL position⁶².

Nathan managed to hold on for a few more years at JPL on other projects until, in the late 1970s, he thought of a way to increase the speed of the then computer-time-hungry image-processing programs. With Mariner Mars 1971 it became possible to send images faster than they could be processed. Since then, the ratio between transmission time and processing time has gone way up in favor of transmit time. In general, it does not really matter, since instant images are not now a requirement, but for users of image processing other than planetary scientists, additional speed is attractive. Also, as the number of images has skyrocketed from Mariner Mars 1964's 22 to literally tens of thousands in the Voyager and Galileo projects, time to process the images is of interest even to the most patient. The problem is that as the number of pixels has increased, the number of individual computations also increases. A 1,000 by 1,000 pixel image weighted 35 by 35 times requires 1.225 billion multiplications⁶³! If these are done in sequence, the amount of processing time would be formidable.

To solve this problem, Nathan suggested putting 35 sets of 35 multipliers in parallel on very large-scale integration (VLSI) chips. By doing that, the amount of calculations is reduced by 1,225 to 1. Recently, he has begun design of a set of VLSI chips that will speed up the geometry or reprojection operations⁶⁴. Basically, the weighting algorithm is encapsulated in a single chip as a unit of hardware, rather than as software. Logic in hardware executes faster than logic in software because all 1,225 multipliers are operating simultaneously in parallel rather than one at a time serially as in a central processor. Nathan's chips have been plugged into Digital Equipment Corporation VAX 11/780 computers. When the computer is executing an image-processing program and reaches the point where it wants to do the algorithm on the chip, the computer "calls" the chip just as though it were calling a software subroutine.


[295]

Figure 9-9A. Mosaics combine detailed images into detailed maps: a Martian desert...


[296]

Figure 9-9B. ...Volcanic Io...


[297]
: Figure 9-9C. ...Heavily cratered Callisto. (JPL photos 211-4704; P-21278; P-21746)



Nathan sees his invention not only as the solution to a problem in image processing but also as the beginning of a new future in computing. Using this technique, special-purpose computers with a lot of logic embodied in hardware could easily outstrip the existing systems in speed and accuracy. In some ways, it would be like electronic analog computers, but better in that the rearrangement of components would be simpler.

It is fitting to end on this note, as Nathan's application of computers to fulfill a need in space exploration mirrors the entire story of NASA's use of computers. He approached his tasks in the late 1950s and early 1960s as a pragmatist. He had some computing background, as well as grounding in other fields, so he could see the possibilities of applications. He used equipment usually behind the state of the art but got beyond the state-of-the-art results with it. And, finally, he repays computing by finding one way to improve it on the path to solving yet another problem. Nathan himself said that "NASA is not to be given credit for initiating advances in image-processing technology, but NASA has supported the grass roots initiatives." In general, that is true. NASA never asked for anything that could not be done with the current technology. But in response, the computer industry sometimes [298] pushed itself just a little in a number of areas. Just a little better software development practices made on-board software safe, just a little better networking made the Launch Processing System more efficient, just a little better operating system made mission control easier, and just a little better chip makes image processing faster. NASA did not push the state of the art, but nudged it enough times to make a difference.