Apollo 11 Had a Hidden Hero: Software

Neil Armstrong hovered a few miles above the surface of the moon on July 20, 1969, searching for a safe place to make history. Only minutes of fuel remained to land the first men on another world. A power meter failed in Apollo 11’s cramped lunar lander. Communications faded in and out. Then, warnings began flashing: Program alarm. Program alarm. Five times the onboard computer signaled an emergency like none Armstrong and crewmate Buzz Aldrin had practiced. In that moment, the lives of two astronauts, the efforts of more than 300,000 technicians, the labor of eight years at a cost of $25 billion, and the pride of a nation depended on a few lines of pioneering computer code. An 18-inch-thick printout in Don Eyles’s loft shows some of the computer code that controlled the Apollo lunar module’s descent to the moon. Photo: Tony Luong for The Wall Street Journal Humans had never risked so much on zeros and ones. Yet they decided to trust the machine and the binary two-digit code, and Armstrong and Mr. Aldrin reaped the glory as the first people to walk on the moon. “The software saved the mission,” says Fred Martin, 85, who managed much of the Apollo software development. From the vantage of 50 years, we view the leap to another world as a singular triumph of humankind. By almost any standard, though, it is a victory for the machine, too, marking the most important 15 minutes in the history of computing. “They’d put the computer at the center of this hugely ambitious project,” says David C. Brock, director of the Computer History Museum’s software history center. “It was a real test of that technology and everyone’s beliefs and aspirations for it.” Apollo 11’s lunar module, Eagle, photographed from the command module after separating to begin its computer-guided descent to the lunar surface. Photo: NASA The Apollo guidance computer—the first digital general-purpose, multitasking, interactive portable computer—laid the foundations of much of the digital world we know today, from the fly-by-wire cockpits of commercial jetliners to the multitasking smartphones we carry in our pockets. In the airless void above the moon, wafer-thin silicon and the code that powered it came of age. Letting ‘the kids’ thrive Don Eyles unfurled a half-century-old, 18-inch-thick computer printout on a table in his loft on the Boston waterfront earlier this year. The 75-year-old traced the maze of terse commands with his finger, proud of how little memory the interlocking tasks and routines required to land on the moon. It was poetry in the enigmatic commands of machine language. “They had to gamble that the kids would rise to the occasion,” says Mr. Eyles. “We were brought into a sort of loose managerial situation and allowed to thrive.” Don Eyles, in his Boston loft, was just out of college when he went to work for MIT’s Instrumentation Lab and helped write the code that guided men to the moon. Photo: Tony Luong for The Wall Street Journal In the summer of 1966, he was a 23-year-old math major with a taste for opera and fast cars, looking for work. Fresh out of college, he had many things about life yet to learn. Computer code was one. So was space travel. He applied to the Instrumentation Laboratory at the Massachusetts Institute of Technology, whose job it was to guide the Apollo astronauts to the moon and back. It took more than big rockets to put humans on the moon, they told him. It took code. And, to his surprise, they hired him. Dan Lickly, shown circa the 1960s, says there was an art to finding people who could turn engineering equations into code for a journey to another world.. Photo: Draper The I-Lab, as everyone called the place, was housed in a former underwear factory overlooking the Charles River, now long since demolished. The Apollo engineers and programmers labored at scuffed metal desks in cubicles with code scribbled on the chalkboard, slide rules on the table, cigarette butts on the linoleum floor. Fanfold computer printouts were stacked up to 6 feet high, like termite mounds. The lab had pioneered inertial guidance systems for the nuclear-warhead-tipped missiles of the Cold War, such as the submarine-launched Polaris intercontinental ballistic missiles. Funded by the U.S. Air Force, it also developed a plan in the late 1950s to fly a computerized probe to Mars and back. MIT received the first major Apollo contract, the only one awarded to a university, and the only one given without competitive bidding. In an era when a computer used fragile tubes, ran on punch cards and filled an entire room, the I-Lab engineers had invented a briefcase-size digital brain packed with cutting-edge integrated circuits and memory so robust it could withstand a lightning bolt—a direct ancestor of almost all computers today. Ralph Ragan (left) and Eldon Hall, who helped lead the MIT lab’s Apollo work, checking components of the guidance computer in front of a mock-up of Apollo controls. Photo: Draper Unlike other machines of its era, it could juggle many tasks at once and make choices of which to prioritize as events unfolded. Apollo missions carried two of these computers, one aboard the command module and one in the lunar lander, running almost identical software. Only the lunar lander, though, required the extra code to set down safely on the moon. There was an art to finding people like Mr. Eyles who could turn engineering equations into code for a journey to another world, says Dan Lickly, 86, who oversaw the computer software development. “You can’t get a degree in how to fly to the moon,” says Dana Densmore, who joined the lab in 1965 and became a control supervisor for the lunar-lander software. “You had to get people who know how to think, who are creative and alert. It was all invented on the spot.” Saydean Zeldin, a physicist who had worked in GE’s missile division, answered a help-wanted ad from MIT’s lab and soon was managing a dozen guidance engineers. Photo: Dave Cole/The Wall Street Journal Physicist Saydean Zeldin, now 79, had worked on warhead re-entry for General Electric Co. ’s Missile and Space Division. After the birth of her third child in 1966, she answered a help-wanted ad placed by MIT’s moon project. “I was a little scared that I could do this job,” she recalls telling the program manager who interviewed her. She soon managed a dozen guidance engineers on matters involving many Apollo missions. “I was there at work early in the morning, as soon as the babysitter arrived. I used to come home late in the afternoon. Once they had their cake and milk and had their stories read, I hopped back in the car and went back to the lab,” she says. Staffing more than quadrupled from 130 people in 1966, when Mr. Eyles was hired, to 600 or so programmers by the time of Apollo 11. Whimsy in the code The power of code was intoxicating. Spaceflight engineers discovered they could use it to perform tasks that otherwise required rods, cables and actuators. Code was cheaper, more adaptable and, most important, weightless. “People started using the software to solve all their problems,” says James Kernan, 84, who oversaw assembly of the lunar-module software for Apollo 9. If the computer could plot a course, why not have it steer it, too? Desperate to lose weight, NASA in 1964 decided to eliminate electromechanical flight controls meant to operate the rocket engines and thrusters. An early version of the Apollo guidance computer, known as AGC-3, before the MIT lab condensed it into a usable model. Photo: Draper It increased the demand on the small computer, designed to fly to the moon and back using no more memory than the text in an average email, by up to a third. They redesigned the computer, doubled its memory—and still the software overflowed. “NASA was really in a bind,” says Don Fraser, 78, who worked on the digital autopilot. In a bitter series of so-called Black Friday sessions, NASA managers ordered the software squeezed to fit, dumping less essential tasks and battling over every change, the I-Lab engineers recall. The Apollo computer eventually required about 145,000 lines of code in all, compared with about 62 million lines of code required today to operate Facebook and more than two billion lines of code for Google. Among the many routines for the lunar lander, Mr. Eyles co-wrote a master ignition sequence that, with a series of related lunar-landing routines, calculated and controlled the descent to the moon. Mr. Eyles had inserted explanatory comments so that he and his colleagues could remember what the string of commands was intended to accomplish. There, whimsy crept in. He labeled the ignition sequence “BURN_BABY_BURN.” An Apollo guidance computer from an early lunar module. Photo: Jesse Rieser for The Wall Street Journal When the computer wanted to reposition the landing-radar antenna, the code noted: “ASTRONAUT PLEASE CRANK THE SILLY THING AROUND.” It then performed a calculation to determine if the astronaut had moved it correctly. “SEE IF HE’S LYING,” the code noted. When the antenna was aimed and the landing could proceed, the code said: “OFF TO SEE THE WIZARD.” “I did not imagine that anyone in the future might be looking at our code for historical purposes,” Mr. Eyles says. Whimsy disappeared when the dozens of MIT engineers heard Neil Armstrong report a computer problem as Apollo 11’s lander began its powered descent to the moon. “Program alarm,” the commander radioed Mission Control in Houston, a touch of urgency in his usually laconic voice. “It’s a 1202. What is it?” Shock and alarm Margaret Hamilton’s blood pressure soared when she heard the astronauts call out a 1202. She knew it all too well. She had programmed the alarm codes. “They were the never-supposed-to-happen alarms,” she recalls. “I was in a state of shock. How could this be happening now just before the landing? I’m thinking: Oh my God, this is not real.” Ms. Hamilton, now 82, and her colleagues crowded around the squawk box and telemetry relays that connected the laboratory’s Switching, Conference and Monitoring Arrangement room to flight controllers in Houston and the astronauts aboard the lunar lander nearly 250,000 miles away. Margaret Hamilton knew what the 1202 alarm was: She had programmed the alarm codes. Photo: Tony Luong for The Wall Street Journal The alarms, she knew, signaled that the computer was overloaded and, as designed, was dumping unimportant tasks in order to keep flying safely. It could handle interruptions and power outages, literally able to be turned off and back on without interrupting the landing or any other vital maneuver. About a year before the lunar landing, NASA had asked the already busy I-Lab engineers to build into the software this so-called restart protection. As the astronauts neared the surface of the moon, something kept overtaxing the computer. No one in the room knew what it was, how severe it might become, or how to fix it before the computer exceeded its margin of recovery. “People’s throats fell into their stomachs,” says Dr. Martin, who was a senior manager of the mission development group. In thousands of test runs no one had seen these alarms. “I remember looking at Margaret and we are all absolutely certain there would be an abort.” When she joined MIT’s Apollo effort in the early 1960s, Ms. Hamilton was the rarest of new recruits—an experienced programmer. She had an instinctive distaste for programming errors. A mathematician, she learned to program by running calculations for an MIT meteorologist named Edward Lorenz, who discovered a new field of study called chaos theory. She programmed his $46,000 LGP-30 computer by key-punching binary code composed of zeros and ones into a paper tape. Impatient with the time it took to recompose and resubmit a faulty program made this way, she learned to correct her errors by poking new holes in the tape with a pencil to create binary ones or by taping over existing holes to turn a binary one into a zero. Margaret Hamilton’s MIT Instrumentation Lab employee card. Photo: Draper Needing a raise, she next wrote code for the largest computer ever made, a 250-ton, four-story military behemoth that served as the brains of the $27 billion national air defense system. In the race to the moon, her first assignment was to write the code for an abort routine. She soon took responsibility for the spacecraft’s system software. By Apollo 11, Dr. Lickly had put her in charge of all the capsule and lunar-lander software. When it came to hiring women for engineering or management, NASA “had a few women, and they kept them hidden,” says Ms. Densmore. “At the lab it was very different,” and there were opportunities for women. As the moonshot approached, Ms. Hamilton explored worst-case scenarios. “I kept doing what-ifs? What if there is an error? What if there is an emergency in real time with real astronauts and there is no way to tell the astronauts they are in trouble?” she says. “I figured out a way to let them know in the software that there is an emergency.” Her 1202 program alarm was one of 29 that could be triggered during landing. As it flashed on the lunar lander’s digital computer display, Buzz Aldrin and Neil Armstrong knew only that they were in trouble. “Give us a reading on the 1202 program alarm,” Armstrong radioed with an edge in his voice. The men were 30,000 feet above the moon and descending. Twenty-seven seconds ticked off the mission clock with no answer from their home planet. ‘Go’ and ‘Go’ again In that moment, only one person on Earth—Flight Director Gene Kranz at NASA’s Mission Control in Houston—had the authority to stop the lunar landing. He strained to listen. Communications with the lander had been unusually distorted and static-filled. Whispering in his ears also were a dozen voices from six communications loops and the air-to-ground communications channels, Mr. Kranz, 86, recalls. “We’re ‘go’ on that alarm?” Mr. Kranz said, asking his team if he could let the landing proceed. After his team wrongly called for an abort in a practice run, flight director Gene Kranz pushed them to learn the computer alarms. It paid off during Apollo 11’s descent. Photo: Michael Wyke/Associated Press In their final practice run before the mission, Mr. Kranz and his team had been tested on these alarms and failed badly. Caught off-guard, they mistakenly ordered the astronauts to abort the landing. That dress rehearsal taught his team that these alarms meant the computer could still be relied upon. With the astronauts waiting, NASA engineer Jack Garman, 25, checked his notes from MIT and passed the word to Steve Bales, 26, who staffed the guidance console for the lunar descent. He gave Mr. Kranz the word. “Go,” Mr. Kranz ordered. Again, the alarm. Again, “Go.” Five times in all, the computer reset itself and continued flying. Mr. Eyles’s code performed flawlessly. Armstrong jockeyed for a safe spot to land. “We were just 17 seconds from calling an abort,” as the fuel supply approached empty, recalls Apollo astronaut Charlie Duke, who as the capsule communicator was the only one at Mission Control allowed to talk directly to the crew in flight. The astronauts brought their lander, which they had dubbed Eagle, to rest on the Sea of Tranquility. “Then Neil said very calmly after all that tension—it was amazing the calmness in his voice—‘Houston, Tranquility Base here. The Eagle has landed,’ ” Mr. Duke says. It would be the next day before anyone figured out what had set off the program alarms. A mismatched power supply to a radar unit had erroneously triggered a near constant series of calculations that overloaded the computer. The restart protection added to the software a year earlier had rescued the mission, preventing the astronauts from unnecessarily aborting it before they landed. Engineers (from left) Norman Sears, Margaret Hamilton, Phyllis Rye and Ain Laats, during Apollo 8. On the blackboard is a program for command-module navigation. Photo: Draper At MIT, the euphoria of the moon landing in July gave way to protests in November as students and police clashed outside the I-Lab over its military-weapons work. By 1973, MIT had severed ties to the laboratory, by then renamed the Charles Stark Draper Laboratory, and moved all the university’s classified military research off-campus. Around the same time, NASA canceled its last three Apollo missions, ending human exploration of space beyond low Earth orbit for the indefinite future. The computers were mostly scrapped. NASA never published its Apollo code. It can be found in museums in Cambridge and Washington, and has been posted online by computer enthusiasts. Today, though, its legacy is in just about every pocket, driveway, home and office. Its descendants helped to remake how the world learns, works, plays, communicates, spends and socializes. And thorny questions of control and responsibility it raised in 1969 resonate in an era of dawning artificial intelligence. The code itself perhaps best sums up the wonders it hastened. As billows of lunar dust settled around the lander 50 years ago, its onboard computer ticked through the instructions in its P68 lunar landing confirmation routine. Embedded in the final lines, where no outsider was ever likely to see it, the software said: “ASTRONAUT: NOW LOOK WHERE YOU ENDED UP.” Mr. Hotz is a science writer for The Wall Street Journal in New York. He can be reached at lee.hotz@wsj.com. Copyright ©2019 Dow Jones & Company, Inc. All Rights Reserved. 87990cbe856818d5eddac44c7b1cdeb8