Forget Crazy Earth,
Look to the Future:
Voyager and Fortran 5

2024

The Voyager spacecraft, Voyager 1 and Voyager 2, were not programmed in Fortran 5. Or in any Fortran for that matter. This widely accepted misbelief can be traced back to a widely misunderstood statement in a single magazine article.

1.  Introduction

The Voyager program took advantage of a rare planetary alignment to send two spacecraft on a tour of the solar system's gas giants: Jupiter, Saturn, Uranus, and Neptune. In addition to the cameras and scientific instruments, each Voyager has 3 computers (plus their backups, for a total of 6):

• Computer Command Subsystem (CCS) - is the basically the master controller of the spacecraft. As such, it "provides sequencing and control functions. The CCS contains fixed routines such as command decoding and fault detection and corrective routines, antenna pointing information, and spacecraft sequencing information." (Voyager: The Spacecraft, JPL) The CCS CPU is the same as used on the Mars Viking Orbiters.

• Attitude and Articulation Control Subsystem (AACS) - "controls spacecraft orientation, maintains the pointing of the high gain antenna towards Earth, controls attitude maneuvers, and positions the scan platform." The cameras and a couple of other instruments are mounted on the scan platform, hence adjusting their orientation must be done in conjunction with the spacecraft's orientation. (Voyager: The Spacecraft, JPL) The AACS computer is not the at-the-time experimental HYPACE computer. This turned out to be an unexpected rabbit hole, so see "AACS, HYPACE, and HYBIC" below for more details.

• Flight Data Subsystem (FDS) - "controls the science instruments and formats all science and engineering data for telemetering to Earth." (pre-launch Voyager Press Kit, August 4, 1977, p. 22, PDF) The FDS used a custom-designed CPU.

And, expanding our vocabulary:

• Flight Computers - are the computers onboard a spacecraft, like the three above.

• Flight Software - is the collection of programs and routines that run on the flight computers. Like any computer program, the flight software is composed of low-level CPU instructions such as "Load CPU register A with the value at memory address 1234". Flight software engineers write and maintain these programs.

• Spacecraft Commands - are binary commands that perform operations on the spacecraft. Uplinked to the spacecraft by mission control, they are not CPU instructions. In a spacecraft without computers, the commands are fed to a hardware decoder that effects the desired operations. In a spacecraft with computers, the software interprets the commands and runs the sometimes complex code needed to perform the operations. (Think of the Unix bash shell, where the "flight software" is the bash executable and the "spacecraft commands" are the command-line commands: cd, ls, etc.)

• Command Sequences - are sequences of spacecraft commands uplinked to the spacecraft as a singular bundle. A given sequence may be executed immediately, executed at a scheduled time, or executed only when the spacecraft's hardware or software detects some condition. Sequence engineers design and develop command sequences. (A simplified definition of sequences; see Sequences and Simulators below for more details.)

In the telemetry data radioed back to Earth:

• Engineering Data - has to do with the health and operation of the spacecraft; e.g., temperatures, power levels, spacecraft attitude, and readouts of flight computer memory for debug purposes!

• Science Data - is that returned from the scientific instruments. The prime example is the image data from the cameras.

"This artist's concept shows plasma flows around NASA's Voyager 1 spacecraft as it approaches interstellar space."
(Source: "Voyager 1 Enters Final Region of Our Solar System Before Interstellar Space", Mike Killian, AmericaSpace, 2013.)

A brief Voyager timeline (the fly-by dates are actually years of closest encounters):

    1977 - Voyager 2 and then Voyager 1 launched.
    1979 - Both Voyagers fly by Jupiter.
    1980 - Voyager 1 flies by Saturn.
    1981 - Voyager 2 flies by Saturn.
    1986 - Voyager 2 flies by Uranus.
    1989 - Voyager 2 flies by Neptune.
    2004 - Voyager 1 crosses the termination shock.
    2007 - Voyager 2 crosses the termination shock.
    2012 - Voyager 1 enters interstellar space.
    2018 - Voyager 2 enters interstellar space.

After the 1980 encounter with Saturn, Voyager 1 angled north from the ecliptic (planetary) plane at about 35°. After the 1989 encounter with Neptune, Voyager 2 angled south of the ecliptic plane at about -48°. (Voyager Fact Sheet, JPL)

Entering interstellar space means the Voyagers have emerged from the heliosphere, the region in which solar winds predominate. The spacecraft have not yet escaped the Solar System, as the Sun's gravity still holds sway. That will take another 30,000 years until the spacecraft cross the outer edge of the hypothetical Oort Cloud!

For a real treat, see the real-time Voyager Mission Status! I noticed that Voyager 1's distance from Earth was rapidly decreasing (!) while Voyager 2's was slowly increasing. At the same time, both spacecraft's distances from the Sun were increasing at a constant rate as expected, both at about 10 miles every second. I eventually figured this out. (Completely missing, like an idiot, the "Distance from Earth" link that pops up an explanation!) Earth is traveling around the Sun at nearly twice the speed of the Voyager spacecraft. For half of its orbit, Earth will be moving towards, for example, Voyager 1 and is thus "catching up" to the spacecraft. (That's a rough, simplified explanation. Given the Earth's and Voyager 1's full 3-D velocities, you must separate out the components along the changing Earth-Voyager 1 axis. When the Earth's speed along that axis exceeds Voyager 1's, the distance will decrease. So the distance will decrease on only a portion of that half-year half-orbit.)

And the DSN Now page shows the real-time status of the Deep Space Network antennas and which spacecraft they are talking to, VGR1 and VGR2 being the Voyager spacecraft.

"Voyager Project Manager Suzy Dodd shows off her spacecraft". (Source: Bill_D@Flickr)

2.  How It Started

Wired

In 2013, Wired published an article based on an interview with Suzanne Dodd, Voyager project manager at the time and, currently, also Director of the Jet Propulsion's Interplanetary Network Directorate. In the article, the author, Adam Mann, wrote (his words, not a direct quote from Dodd):

The spacecrafts' original control and analysis software was written in Fortran 5 (later ported to Fortran 77). Some of the software is still in Fortran, though other pieces have now been ported to the somewhat more modern C.

—Adam Mann, "Interstellar 8-Track: How Voyager's Vintage Tech Keeps Running", Wired, September 2013.

My impression upon first reading this was that the ground-system mission control and science data analysis software was written in Fortran and parts were later ported to Fortran 77 and C. (I now think Dodd was referring to mission control-related analysis instead of science data analysis.) I'd never heard of "Fortran 5", but the rest was unremarkable and tracked with my own experience. In the early 1980s, I worked on the image processing side of the LANDSAT-4/5 ground system, programming in Fortran 77 on VAX/VMS minicomputers. The mission-control developers on the other half of our cubicle farm at NASA's Goddard Spaceflight Center programmed in Fortran on DEC-2060 mainframe computers. And, in the late 1980s, I was part of a small team developing a generic, UNIX workstation-based ground system, TPOCC, for Goddard — programmed in C, of course!

So, aside from the "Fortran 5" oddity, Mann's article was spot-on and provided an excellent update for me on the Voyager project. Others, however, inferred more from the quote and, shortened to one sentence with the "control and analysis" phrase dropped, a misleading form of the quote spread in the popular press and in numerous hardware and software forums. The inference or "meme", easily identified by the "Fortran 5", is that the Voyagers' onboard computers were programmed in Fortran, etc., etc., as shown here:

The initial software program was Fortran 5, then they were reprogrammed during flight to Fortran 77, and finally C.

—South Australian Doctor Who Fan Club Inc. (SFSA), "Voyagers disco party!", The Wall of Lies, No. 169, Nov-Dec 2017, p. 2. (4-page newsletter, PDF)

(I'm not dumping on the fan club. For my example, the juxtaposition of Doctor Who and disco practically screamed, "Choose me! Choose me!")

Popular Mechanics

I originally dismissed the following 2015 article in Popular Mechanics as just another knock-off of the Wired article. However, I was wrong and, upon rereading it more closely, I have to say it is a superb article. Based on the author's own phone interview with Suzanne Dodd, the article is about the Voyager project seeking a new software developer to replace the soon-to-retire Larry Zottarelli, "the last original Voyager [software] engineer". From other information I gleaned from the web, it appears he was one of the original flight software engineers, joining the project pre-launch in the early-to-mid 1970s. Don't judge the article by this isolated quote (again, the author's words, not Dodd's):

Know Cobol? Can you breeze through Fortran? Remember your Algol? Those fancy new languages from the late 1950s? Then you might be the person for the job.

—John Wenz, "Why NASA Needs a Programmer Fluent in 60-Year-Old Languages", Popular Mechanics, October 29, 2015.

Between Wired and Popular Mechanics, I'm starting to wish Suzanne Dodd had been a bit clearer in these interviews about what the programming languages were used for. On the other hand, in her defense, Dodd has spent decades having to and trying to convey scientific and technical information in a meaningful way to the public and non-tech-oriented journalists (in general; I don't mean Wenz).

In this particular case, I expect that knowledge of Fortran is needed for some of the support software, but not for actually programming the onboard computers. (Keep in mind that Fortran is not hieroglyphs! Moderately experienced programmers should be able to pick it up fairly easily.)

Aside: Algol? Really?! I have long taken some small measure of pride in the fact that I am one of the few people in the U.S. who have actually programmed in Algol! When I first caught the computer bug in 1977, I read one of Daniel McCracken's Fortran books and his Algol book ... before I even took my first computer class and got access to a computer. In two classes where the other students used Fortran, I used Algol because I wanted to try it out — I liked it. This was the Norwegian University (NU) Algol 60 compiler on a Univac 1100-something mainframe.

Unrelated aside: In the Popular Mechanics article, Wenz contrasts the Voyagers' programmability with that of other spacecraft with fixed-hardware operation sequencers, such as ISEE-3. The generic, UNIX workstation-based, misson control system I worked on at Goddard, TPOCC (mentioned above), was used to replace ISEE-3's Xerox Sigma-based (if I remember correctly) control system in the early 1990s! Launched in 1978, ISEE-3 was repurposed in 1982 as the International Cometary Explorer (ICE) and underwent a complex set of maneuvers leading to a rendevouz with Comet Giacobini-Zinner in 1985. (See this cool graphic of the maneuvers.) (The mission control system was called the ICE/IMP control center, so it was controlling two spacecraft, the second probably being the Interplanetary Monitoring Platform, IMP-8.)

3.  What is Fortran 5?

A distinct detail in every replication of this meme is "Fortran 5". I haven't used Fortran since the 1980s, but the unknown-to-me Fortran 5 was an immediate red flag. In all the online discussions of this subject that I've seen, no one questions "Fortran 5". Even in a Fortran Discourse thread where the users are very knowledgeable about Fortran. It can't just be me ... please!

On Wikipedia's Fortran page, Fortran 5 is called an obsolete variant (i.e., non-standard version) of Fortran:

Fortran 5 was marketed by Data General Corporation from the early 1970s to the early 1980s, for the Nova, Eclipse, and MV line of computers. It had an optimizing compiler that was quite good for minicomputers of its time. The language most closely resembles FORTRAN 66.

Data General's Nova 840 minicomputer was introduced in 1973; a contemporary DG brochure, "840: The Loaded Nova", touts the Fortran 5 compiler in its feature list:

Fortran 5, an extremely thorough, multipass compiler that produces globally optimized code that's nearly as efficient as assembly language code

And Data General advertised Fortran 5 with "It's a Real Pig": "Pigs are the Smartest Animals in the Barnyard"! The Fortran 5 compiler was slow because it was taking the time to produce smaller and faster executables. (Flickr image)

Data General's 1978 Fortran 5 Reference Manual (PDF) lists 1972 as the first copyright date, incidentally the year the Voyager project officially began.

"PC or MAC? UNIVAC.", The Piranha Club by Bud Grace.
(Source: James McDonald Stewart's Response to Fankuchen Award 2001)

4.  Fortran V?

I've seen no mention of Data General computers in connection with the Voyager project in my research, so the use of Fortran 5 on the project seems highly unlikely.

My guess is that, when speaking to Wired's Adam Mann, Suzanne Dodd said "Fortran Five", meaning "Fortran Roman Numeral Five", or Fortran V. This is consistent with Fortran II being known as "Fortran Two" and Fortran IV as "Fortran Four". (Fortran III, in case you're interested, was an unreleased, internal IBM compiler dating to 1958.)

Fortran V, like Fortran 5, is one of Wikipedia's non-standard, obsolete variants of Fortran:

[In addition to Control Data Corporation,] Univac also offered a compiler for the 1100 series known as FORTRAN V. A spinoff of Univac Fortran V was Athena FORTRAN.

Now things fall into place. Remember that Suzanne Dodd spoke of "control and analysis" software. She joined the Voyager team in 1984 as a sequence engineer in the lead-up to Voyager 2's January 1986 fly-by of Uranus. These engineers designed/developed sequences of spacecraft commands for upload to the Voyagers ... work done on one or more Univac 1100-series mainframe computers. From an interview:

DODD: When I first started, I started on a— I don't even remember the name of it, but I do recall it had an eight-inch floppy drive. That was our command medium, an 8-inch floppy drive. Not a memory stick, not even a CD. When we did these designs and plots that showed—there was a program that you could design an observation, like you want to make a mosaic over Uranus, and you could lay out the observation, and then it would tell you what the commands you need to do it are. That was done on a UNIVAC computer, so kind of more of a mainframe refrigerator-size computer.

David Zierler, "Suzy Dodd (BS '84), Engineer and Deep Space Pioneer"

Caltech Heritage Project, June 9, 2023.

I found more specific information about JPL's mission control computing facilities in a pre-launch press kit (Voyager Press Kit, August 4, 1977, pp. 106-107, PDF):

• Mission Control and Computing Facility (MCCF) - three IBM 360-75 mainframes used for the day-to-day operations of the spacecraft, including command uploads and tracking.

• General Purpose Computing Facility (GPCF) - three Univac 1108 mainframes used for "navigation and mission sequence systems ... [and] prediction programs and detailed spacecraft engineering performance analysis". These are the systems that Suzanne Dodd would have used. (Assuming JPL hadn't upgraded the 1108s to newer models in the intervening 7 years.)

• Mission Test and Computing Facility (MTCF) - "three strings of UNIVAC and Modcomp computers to receive, record, process and display" the engineering and science data downloaded from the spacecraft. (The Univac computers here were minicomputers such as the UNIVAC 1230. The Modcomp minicomputers included the MODCOMP II and MODCOMP IV; for some reason, Tomayko [Chapter 8, Section 3, p. 265] calls them the ModComp 2 and 4, respectively, though everyone else on the web uses the Roman numerals!)

Sun Kang Matsumoto is a CCS flight software engineer who joined the Voyager program in 1985, also, like Dodd, in the ramp-up for the Uranus fly-by. In a 2016 paper about the Voyager Interstellar Mission (VIM), she gives some more details about the evolution of Dodd's "control and analysis" software from Univac to Unix (and thus from Fortran to C, presumably):

During the prime mission and early VIM, Voyager had been using the JPL-developed software programs called SEQTRAN (to generate sequences) and COMSIM (to simulate sequences and CCS FSW changes). They ran on now-antiquated UNIVAC mainframe computers. Shortly after the start of VIM [in 1990], these programs were converted over to more modern UNIX-based SEQTRAN and High Speed Simulator (HSSIM). They were rewritten to maintain the same functionality of the old SEQTRAN and COMSIM, and tailored for VIM. Rewriting and testing required significant effort from the developers and the project personnel; however, the end result is much improved speed and efficiency.

Sun Kang Matsumoto, "Voyager Interstellar Mission: Challenges of Flying a Very Old Spacecraft on a Very Long Mission" (PDF)

p. 4, 2016 SpaceOps Conference, Daejeon, Korea.

SEQTRAN and COMSIM/HSSSIM are programs Dodd would have used as a sequence engineer. As a connoisseur of good program names, I can't fail to mention Matsumoto's mention of two new programs: VAMPIRE (Voyager Alarm Monitor Processor Including Remote Examination) and MARVEL (Monitor/Analyzer of Real-time Voyager Engineering Link)! Matsumoto's paper is well-worth reading not only for the technical information, but because she provides a valuable portrayal of how the Voyager team has managed to overcome, by hook and by software crook, the very real problems of aging hardware at a great distance. And have done so despite steep technological and budgetary challenges.

I think (or hope) it is clear by now that, in the 2013 Wired article, Suzanne Dodd was referring to the ground system software and not the software running on the flight computers in space.

Okay, Univac and Fortran. But what about Fortran V? I could only find one instance of "Fortran V" being explicitly mentioned in connection with a Voyager mission operations program. (I do not include academic papers whose authors used Fortran V programs to analyze the Voyagers' science data at institutions outside of JPL; e.g., universities.)

The scarcity of "Fortran V" and "Voyager" on the web is not surprising. Most programming languages maintain backwards compatibility between versions, so, for example, a Fortran IV program usually can be recompiled with a Fortran V compiler without error and with identical functionality. Consequently, programmers do not in general specify versions when speaking about languages. A C programmer doesn't say, "I wrote this program in C89 and later ported it to C99." Instead, it's just an unadorned "C program". (This is not a slight on the Dodd-Mann remark.)

Also, Univac's Fortran V compiler was available as early as 1966, so it probably would have been the default Fortran compiler on Univac mainframes throughout the 1970s. Many Fortran users may therefore have been unaware that the specific version they were using was Fortran V. (The documentation that I could find online showed that a generic @FOR invoked the Fortran V compiler and @FTN invoked the ASCII Fortran compiler discussed in the second aside below.)

Aside: I anticipate eventually having to eat my pontificating words on programming language versions! In the meantime ... I couldn't find an online copy of Univac's Fortran V manual, but the 1970 UNIVAC 1108 System Description (Section 10, p. 7) had this to say: "FORTRAN V, being an outgrowth of the earlier FORTRAN languages (in particular, UNIVAC 1107 FORTRAN IV and IBM FORTRAN IV as announced in IBM form C-28-6274-1), accepts these languages as compatible although the reverse is not necessarily true."

Aside: The transition to Fortran 77 on a Univac mainframe would have been non-trivial. Univac's Fortran 77 compiler, ASCII FORTRAN, largely accepted Fortran V source code with a few, documented, mostly Univac-specific exceptions. The big problem, I think, would have been character sets. Univac's 36-bit mainframes used 6-bit FIELDATA characters stored 6 per 36-bit word. The ASCII FORTRAN compiler used the incompatible 7-bit ASCII character set. (On the Univac 1108 computer I worked on c. 1980, ASCII characters were stored as 9-bit quantities, 4 per word.) Vanilla character strings in the source code would just be converted to ASCII without complaint by the compiler, but programs that depended on specific characteristics of FIELDATA characters would require changes. That includes programs that must read or write FIELDATA-compatible tapes. (I have no experience with ASCII FORTRAN, so these are just my thoughts from perusing the ASCII FORTRAN Reference Manual; Appendix A addresses the differences between FORTRAN V and ASCII FORTRAN.)

Unrelated aside: Circa 1980, I overheard a graduate student remark that the shell pipe, |, was Unix's gift to mankind and @ADD was Univac's gift to mankind. He was right!

The Voyager ground-system software I found that was explicitly stated to have been written in Fortran V is the Orbit Determinaton Program (ODP), which was also used on other missions. The mathematics in the following 1983 paper is way beyond me, but it establishes the ODP's host system:

To give an idea of the computational burden that is involved, consider a typical radiometric SRIF/SRIS solution with 67 state variables (Table I). This model contains only 4 process noise states (line-of-sight acceleration and 3 camera pointing errors); there are 3500 data points and 132 time propagation steps. The problem run on a UNIVAC 1110, in double precision, used 275 CPU s[econds] for filtering; smoothed solutions and covariance computation used 265 CPU s[econds]. The entire run scenario including trajectory variational equation integration, observable partials generation, solution mapping, and generation of smoothed residuals used 4320 CPU s[econds].

...

We note in closing this factorization algorithm discussion that the SRIF and U-D algorithms that were used in this application have been refined and generalized, and are commercially available in the form of portable Fortran subroutines.

—James K. Campbell, Stephen P. Synnott, and Gerald J. Bierman; "Voyager Orbit Determination at Jupiter" (PDF), IEEE Transactions on Automatic Control, Vol. AC-28, No. 3, March 1983, pp. 259-261.

For those not mathematically inclined, the paper is still worth skimming just for the enumeration of some of the esoteric details of the Voyager spacecraft that they had to account for in the calculations. We learn what language was used in a 2008 presentation by Lincoln J. Wood of JPL:

The mainframe computers used include the IBM 7090, IBM 7094, UNIVAC 1108, and UNIVAC 1100/81. During the 1980s the ODP was transported to minicomputers, with the software maintained in both mainframe and minicomputer operating environments to fulfill the desires of various flight projects. The ODP consisted of 300,000 lines of code as of 1979, with FORTRAN V being the primary language.

—Lincoln J. Wood, The Evolution of Deep Space Navigation: 1962-1989" (PDF), p. 6, 31^st Annual AAS Guidance and Control Conference, 2008, Breckenridge, Colorado.

Shortly after writing the above, I came across this entry in NASA's long-gone, COSMIC software catalog:

Calculating Trajectories and Orbits
Two programs calculate the motions of spacecraft and landers.

The Double Precision Trajectory Analysis Program, DPTRAJ, and the Orbit Determination Program, ODP, have been developed and improved over the years to provide the NASA Jet Propulsion Laboratory with a highly reliable and accurate navigation capability for their deep space missions like the Voyager. DPTRAJ and ODP are each collections of programs that work together to provide the desired computational results. DPTRAJ, ODP, and their supporting utility programs are capable of handling the massive amounts of data and performing the various numerical calculations required for solving the navigation problems associated with planetary fly-by and lander missions. They were used extensively in support of NASA's Voyager project.

...

DPTRAJ-ODP is available in two machine versions. The UNIVAC version (NPO-15586) is written in FORTRAN V, SFTRAN, and ASSEMBLER. (A processor is supplied for SFTRAN, a structured FORTRAN.) DPTRAJ and ODP have been implemented on a UNIVAC 1100-series computer. The VAX/VMS version (NPO-17201) is written in FORTRAN V, SFTRAN, PL/1 and ASSEMBLER. It was developed to run on all models of DEC VAX computers under VMS and has a central-memory requirement of 3.4 Mb. The UNIVAC version was last updated in 1980. The VAX/VMS version was developed in 1987.

—NASA Tech Briefs, September 1989, p. 33.

A whopping 3.4 MiB of memory required for a lousy 300,000-line program? Software hogs like that were the reason why we couldn't have nice things back then ...

Adam's Voyager Spacecraft for Kerbal Space Program.

5.  So What Language was Used?

Let's ask an actual Voyager flight software engineer what language was and is used to program the onboard computers. Perhaps the Voyager Fault Protection and CCS Flight Software Systems Engineer who's worked on Voyager since 1985:

Both the AACS and FDS use assembly language. The CCS uses assembly language and Voyager-unique pseudo code (interpreter). As a result, it is difficult to attract younger programmers to join the project.

Sun Kang Matsumoto, "Voyager Interstellar Mission: Challenges of Flying a Very Old Spacecraft on a Very Long Mission" (PDF)

p. 6, 2016 SpaceOps Conference, Daejeon, Korea.

In the acknowledgements at the end of the paper, Matsumoto thanks fellow flight software engineer Larry Zottarelli, now retired, and Suzanne Dodd, among others, for reviewing the paper. I assume Zottarelli, Dodd, or one of the others would have spoken up if they took issue with the above paragraph.

That would be two or three assembly languages: one for the CCS CPU, possibly a variant for AACS's modified CCS CPU, and definitely another for the completely different FDS CPU.

The phrase "pseudo code (interpreter)" makes me think of an embedded scripting language, but Matsumoto is probably referring to executing uplinked spacecraft commands and sequences. The CCS also must process internally generated power codes received from the AACS computer:

A "power code" is a 6 bit message sent to the CCS computer, which may be only informational or may cause a power command to switch power to an AACS component. Such power switching commands are usually the means by which redundant elements are exchanged. These power codes are an important part of the fault protection logic, allowing the CCS computer to issue commands in response to a fault condition. These commands may be a simple power command (A gyro on) or a command sequence which will turn the spacecraft in a pattern designed to re-oriented the spacecraft towards the sun from an entirely random attitude. Some serious faults result in an OMEN power code, which causes CCS to save the next three power codes (normally lost) for later analysis.

Assessment of Autonomous Options for the DSCS III Satellite System

Prepared for the U.S. Air Force by JPL personnel (Donna L. S. Pivirotto and Michael Marcucci?), "Volume III: Options for Increasing the Autonomy of the DSCS III Satellite" (PDF), p. 179, Aug. 6, 1981.

In her paper, Matsumoto discusses a problem with the "power decoder relay matrix":

The power decoder relay matrix problem that first manifested in 1998 makes commanding of the spacecraft extremely difficult. Basically, the faulty decoder may cause an issuance of extraneous power commands in addition to the intended command.

Sun Kang Matsumoto, "Voyager Interstellar Mission: Challenges of Flying a Very Old Spacecraft on a Very Long Mission" (PDF)

p. 8, 2016 SpaceOps Conference, Daejeon, Korea.

I wonder if this is the interface used to convey the power codes from the AACS to the CCS. Later, she describes a patch for this error made to AACSIN, the CCS routine that she notes receives and processes power codes, but the specific description of the patch again uses the term, "power commands".

6.  Why Assembly?

Since so many commenters in discussion forums had no problem with "Fortran 5", it might be valuable to examine some possible reasons why the flight software was written in assembly language.

The CPUs

There is not much information available specifically about the Voyager CCS computer, borrowed from the Viking Orbiter. Fortunately, Tomayko does provide details about the Viking CCS:

In general, the design of the processor was exceedingly simple, yet fairly powerful, as indicated by the use of direct addressing, a minimal set of registers, and a reasonably rich set of 64 instructions. The key is that the design placed relatively light demands on spacecraft resources while replacing both the programmable sequencer and the command decoder used in the Mariners. The fact that the processor was later adopted by the Voyager project as its Command Computer and modified for use as the attitude control computer is not only a statement of JPL's frugality but also a testament to the versatility of the design. (p. 157)

...

[On the Viking Orbiter, the] Command Computer's central processor contained the registers, data path control and instruction interpreter. The machine was serial in operation, thus reducing complexity, weight, and power requirements. It had 18-bit words and used the least significant 6 bits for operation codes and the most significant 12 for addresses, as numbered from right to left. This permitted 64 instructions and 4K of direct addressing, both of which were fully utilized. Data were stored in signed two's complement form, yielding an integer range from -131,072 to +131,071. Average instruction cycle time came to 88 microseconds. Thirteen registers were in the Command Computer, mostly obvious types such as an 18-bit accumulator, 12-bit program counter, 12-bit link register that pointed to the next address to be read, and a 4-bit condition code register that stored the overflow, minus, odd parity, and nonzero flags. (Box 5-3, p. 159)

James E. Tomayko, Computers in Spaceflight: The NASA Experience, 1988

Chapter Five, Section 6.

Note the 12-bit addresses, giving an addressing range of 0..4095. And note, in the following, that the Viking CCS flight software engineers masked most interrupts in routines, thus simplifying code (which reduced memory usage) and increasing performance:

Viking Orbiter software had to be written in an assembler, which fortunately had relocatable addresses, simplifying the maintenance task. The 64 instructions were mostly common to other computers, but there was no multiply or divide. There were two sets of loads, stores, increments, and subroutine calls: one used during independent operation and one aimed at dual operation, so that the two memories could be kept equivalent. Even though many interrupts were available, most routines as coded had all but the internal error and counting interrupts disabled. Many routines were free to run out without being interrupted, in contrast to the highly interrupted Apollo and shuttle software. Programmers avoided the memory and processing time overhead required to preserve the current accumulator and register contents during an interrupt.

James E. Tomayko, Computers in Spaceflight: The NASA Experience, 1988

Chapter Five, Section 6, Box 5-4, p. 162.

The Voyager AACS computer uses a modified CCS CPU (e.g., with the added index register), so most of the CCS details above apply as well to the AACS. (In Box 6-1, p. 179, Tomayko uses "HYPACE" to refer to the AACS despite pointing out two paragraphs earlier that the term is a misnomer.)

The FDS CPU was developed especially to meet the high-rate data handling requirements of the Voyager spacecraft. Designed by Jack L. Wooddell, the CPU, in the form of a breadboard prototype, evolved in a collaboration between Wooddell and flight software engineer Richard J. Rice. For some reason, I picture Wooddell seated in front of a really large breadboard with a soldering iron in one hand and a wirewrap tool in the other. Rice is standing beside him with an open laptop in the crook of his left arm, his right hand is typing on the keyboard, and the changing flight software is downloaded via wi-fi to the breadboard computer for testing. Wrong era, I realize, and Rice may well have been left-handed!

Voyager's data computer is different from most small general-purpose computers in several ways. Its special registers are kept in memory, permitting a large number (128) of them. Wooddell also wrote more powerful shift and rotate instructions because of data-handling requirements. Despite its I/O rate, the arithmetic rate is quite slow, mostly due to byte-serial operation. This meant 4-bit bytes are operated on in sequence. Since the word size of the machine is 16 bits, it takes six cycles to do an add, including housekeeping cycles. If all the arithmetic, logic and shifting were not done in the general registers, the machine would have been even slower. Reflecting its role, in addition to the usual ADD, SUB, AND, OR, and XOR instructions found on most computers, the data computer has many incrementing, decrementing, and branching instructions among the 36 defined for the flight version of the machine.

Overall, the Flight Data System requires 14 watts of power and weighs 16.3 kilograms. Its computer needs just one third of a watt and 10 volts, less than the power required for a temperature sensor! At first the estimated throughput required was 20,000 16-bit words per second. By flight time, the instruction execution rate was 80,000 per second, with data rates of 115,000 bits per second, much higher than previous Flight Data Systems.

James E. Tomayko, Computers in Spaceflight: The NASA Experience, 1988

Chapter Six, Section 2, Box 6-2, p. 184.

Note that Tomayko uses the term byte-serial for non-8-bit "bytes", in this case, when describing how the FDS CPU's arithmetic logic unit processes 16-bit operands 4 bits at a time. The last two sentences about throughput, while not wrong, seem to ask the reader to compare apples and oranges.

The breadboard prototype began with the same 4K-by-18 plated wire memory as the CCS computer. I don't know if this means the CPU was originally designed as a 18-bit processor, but, in any case, it ended up as a 16-bit processor with 8K-by-16 CMOS memory. From Tomayko's description on p. 183 of Chapter 6, Section 2, my impression is that the CPU used the same 12-bit addresses as the CCS, but two, independent-of-each-other address lines were added that effectively provided a 13-bit (8K) address range for instruction fetches and a 13-bit (8K) address range for data accesses. I don't know if these extra address bits were incorporated into the CPU's instruction set or if they had to be explicitly set/reset in a separate operation by the programmer.

If my understanding of Tomayko's description on p. 183 is correct, an FDS CPU can run code anywhere in its lower and upper 4K words of RAM and, likewise, can read and write data anywhere in those lower and upper 4K of RAM. (Accessing a sequence of code or data that crosses the 4K boundary may or may not be seamless depending on the computer architecture.) About a month after I wrote the previous paragraph, a commenter in an online discussion (probably about the Voyager team's workaround for the November 2023, Voyager 1 FDS memory problem) said that code is stored read-only in the lower 4K of RAM and the data is stored in the upper 4K of RAM. I thought either the commenter had a different interpretation of p. 183 or that JPL simply followed a convention of maintaining a strict separation of code and data. (Tomayko doesn't mention read-only.) However, rereading Tomayko another month later revealed an interesting twist I had overlooked.

Tomayko has three relevant descriptions of FDS memory in Chapter Six, Section 2:

... JPL's project staff believed that equipment would last longer if unpowered. Although both CCSs are always powered, rarely are both Flight Data Systems running, and both attitude control computers are never turned on at the same time. Full bit-for-bit redundancy is not maintained in the dual memories. For example, "expended" algorithms, such as the deployment sequence executed shortly after separation from the booster, need not be maintained. Both memories are accessed by the single active processor in each system. The Flight Data System keeps a copy of its instructions in both memories, but intermediate data and variables can be stored in either memory. (p. 174)

... Along with the new chips, the memory changed with an expansion to 8K. Two "external" address bits were added to flag whether the top or bottom half of the memory is being accessed. One bit is used to select the memory half used for data access; the other, for the half used for instruction access. (p. 183)

... Also, since the machine can directly address the lower 4K of memory, programs were to be kept there, with the upper portion for transient data. Later the flight configuration of the computer evolved to one processor accessing both memories. Therefore, a copy of the programs is kept in the lower portion of each memory, but both upper portions are usable by the single processor as a scratch pad. If dual mode is required, the memories are separated. (p. 185)

James E. Tomayko, Computers in Spaceflight: The NASA Experience, 1988

Chapter Six, Section 2.

According to the descriptions on p. 174 and p. 185, an FDS CPU runs code from its own lower 4K of RAM and can read and write data both in its own upper 4K of RAM and in the other CPU's upper 4K of RAM. So 4K of code and 8K of data. Note that this is not an increase in physical memory — each CPU still only has 8K of RAM and the first CPU is just borrowing 4K from the second CPU. If Tomayko's description on p. 183 of the two additional address bits is accurate and these bits only select the lower and upper banks of a CPU's own 8K of RAM, it is not clear how an FDS CPU addresses the upper 4K of the other CPU's RAM.

Or, more generally, how any of the spacecraft's CPUs — CCS, AACS, or FDS — access their alternate CPU's RAM, as the description on p. 174 seems to suggest they are all capable of doing. On a related note, how are software (and associated data) updates received from the ground routed to and stored in their target CPU's memory? For example, if the active CCS computer is the one that receives and interprets an upload as a software update for FDS CPU 2, how does the CCS computer directly or indirectly store the code in FDS CPU 2's RAM?

Dual mode, p. 185, is when both FDS computers are powered up and running independently of each other. Tomayko states that the two CPUs' memories are separate in this mode of operation. That may have been JPL's nominal intention, but the practical needs of the mission would supersede this convention when necessary. During Voyager 2's flybys of Uranus and Neptune, the secondary FDS CPU compressed the image data and I think it would have been most efficient for the primary CPU to directly access the compressed data in the other CPU's RAM for packaging and downlink to the ground. (Tomayko's report was published in 1988 in between the 1986 Uranus and 1989 Neptune encounters and he may have written the Voyager section much earlier.)

Don't be dismayed by the CCS and AACS CPUs' lack of multiply and divide instructions and the FDS CPU's slow arithmetic operations. Spacecraft computers don't crunch numbers. (Out of necessity, in my ground-software experience up to the mid-2000s. More modern flight computers with more powerful processors and abundant RAM may have changed this dynamic.) Sensors and detectors return unit-less raw counts. If these readings are needed in real-time for autonomous spacecraft operations, it is faster and more memory-efficient for command sequences or flight software to work with these counts directly. Otherwise, the ground system can apply its sizeable CPU, RAM, and disk resources to perform what are called engineering unit conversions; e.g., the conversion of a hypothetical temperature sensor's raw 0-255 count to degrees Celsius. The FDS computer is focused on assembling and formatting science and engineering data for transmission back to Earth, so fast bit and byte manipulation is more important than fast arithmetic operations.

Memory Constraints

Wise words:

The Voyager flight software design was very heavily impacted by the limited memory space. Extensive effort and ingenuity was required to perform the necessary functions in the available space. The flight software was written in assembly code.

Assessment of Autonomous Options for the DSCS III Satellite System

Prepared for the U.S. Air Force by JPL personnel (Donna L. S. Pivirotto and Michael Marcucci?), "Volume III: Options for Increasing the Autonomy of the DSCS III Satellite" (PDF), p. 179, Aug. 6, 1981.

Words that annoy me:

The computers aboard the Voyager probes each have 69.63 kilobytes of memory, total.

—Adam Mann, "Interstellar 8-Track: How Voyager's Vintage Tech Keeps Running", Wired, September 2013.

An otherwise good article, but ... First, I'm old-school, so the use of "kilobytes" in the modern 1,000 sense annoys me. Second, the pointless (and truncated) precision of 69.63 is annoying. Third, the sentence is wrong because of the confusing wording. The computers don't each have that much memory. The sum total of memory for all 3 computers and their 3 backups is 69,632 bytes:

     CCS - 4,096 18-bit words =  73,728 bits =  9,216 bytes
    AACS - 4,096 18-bit words =  73,728 bits =  9,216 bytes
     FDS - 8,192 16-bit words = 131,072 bits = 16,384 bytes
                                               ------
                                               34,816 bytes
                                              x     2 for the backup computers
                                               ------
                   Total for all 6 computers:  69,632 bytes

When examining the memory limitations from a programming standpoint, expressing the memory size in bytes can be misleading. The CCS computer, for example, has 9,216 bytes of RAM. Readers may mistakenly infer from that figure that there are 9,216 addressable code/data units when, in fact, there are only 4,096 addressable code/data units, a significantly lower number that more accurately reflects the limits in which the programmers had to work.

For example, if you want a variable that counts from 1 to 10, you can't use a memory-efficient single byte and must instead use 2¼ bytes (18 bits). Yes, you can pack multiple data items into an 18-bit word and I'm sure the Voyager assembly language programmers did just this, but doing so incurs added memory/performance costs for packing and unpacking the individual data items. And, yes, clever coding can reduce those costs and or even avoid the need for packing/unpacking altogether. Which makes a strong case for using assembly language rather than a higher-level language like Fortran 5!

A given computer's memory holds the flight software code (CPU instructions), the software's dynamic variables, and uplinked data (tables, command sequences, and such). Regarding the AACS, Dr. Tomayko wrote (emphasis added by me):

The programmers must have done an outstanding job, considering the slow processor and limited memory. At launch, only two words of free space remained in the 4K of plated wire. Tight memory is now a problem because the scan platform actuators on Voyager 2 are nearly worn out, and software has to compensate for this during Uranus and Neptune encounter periods.

James E. Tomayko, Computers in Spaceflight: The NASA Experience, 1988

Chapter Six, Section 2, p. 178.

(Interestingly, Tomayko's 1988 report was published after the Uranus encounter and before the Neptune encounter! He was writing not a history of the distant past, but an account of a still very much active and prominent project.)

A big chunk of Voyager 2's CCS memory is consumed by the Backup Memory Load (BML), identified by name in Matsumoto's paper and described by Tomayko as follows:

As pioneered on Mariner X, a disaster backup sequence was stored in the Voyager 2 CCS memory for the Uranus encounter, and later for the Neptune encounter. Required because of the loss of redundancy after the primary radio receiver developed an internal short, the backup sequence will execute minimum experiment sequences and transmit data to earth; it occupies 20% of the 4K memory.

James E. Tomayko, Computers in Spaceflight: The NASA Experience, 1988

Chapter Six, Section 2, p. 176.

In other words, if Voyager 2 loses the ability to receive commands from the ground, the BML ensures that, on its own, the spacecraft will continue to perform science operations and to downlink science and engineering data.

Fault protection is very important in a remote unmanned spacecraft and is, incidentally, one of Sun Kang Matsumoto's areas of expertise. In the 1981 Assessment of Autonomous Options for the DSCS III Satellite System, Volume III, we learn that 8 CCS fault routines take up 26.5% of the CCS memory (Table B-3, p. 176) and 9-plus AACS fault routines take up 19.4% of the AACS memory (Table B-4, p. 180). This suggests that nearly half of Voyager 2's CCS memory was devoted to the fault routines (26.5%) and the BML (20%), leaving only a little more than half of the memory, about 2,200 words, for other CCS functions such as command sequence storage and processing:

  Fault S/W     BML        Everything Else!     

(That's a graph, not a diagram of the memory layout.) This seems hard to believe and the sizes could well have changed up or down over the decades since this 1981 study and Tomayko's 1988 report.

A couple of weeks after I wrote the above, I came across a 1987 paper by the Voyager Flight Operations Manager, published between the Uranus and Neptune encounters. This paper seems to indicate that, on Voyager 2, there was even less memory available for sequence storage and processing, about 1,250 words:

Out of the total CCS memory, the fault recovery routines, other fixed routines, and flight reserve leave only about 2,500 words available for sequencing of preplanned activities. These activities involve spacecraft maneuvers, scan platform movements, data mode changes, data recording or playback, scientific observations, etc.

... skip one paragraph ...

Since the failure of one of the Voyager 2 command receivers in 1978, that spacecraft has had a back-up sequence stored in its CCS that occupies approximately one-half of the available sequencing space. In the event of a failure of the remaining receiver, this back-up sequence will execute until late in 1989, allowing for limited mission return including data-taking during the Neptune encounter.

Terrence P. Adamski, "Command and Control of the Voyager Spacecraft" (abstract)

p. 3, American Institute of Aeronautics and Astronautics (AIAA), 25th AIAA Aerospace Sciences Meeting, March 24-26, 1987, Reno, Nevada.

(And a couple of more weeks later, I found the exact number: 1,239 words per CCS computer! See Morris's 1986 paper, p. 173.)

The "fixed routines" would probably have included the core software such as the executive discussed below, interrupt handlers, communications with the AACS and FDS computers, ground communications, etc. At the beginning of the Voyager Interstellar Mission in 1990, a BML was installed on Voyager 1 too (see Matsumoto's paper, p. 3). It and probably an updated Voyager 2 BML would have been tailored to the post-planetary-encounter, interstellar mission. (I suspect that Voyager 2's original BML, intended for the Uranus and Neptune encounters, could have been reduced in scope and size afterwards.)

Performance Constraints

I couldn't find much detailed information about the actual flight software. Both Tomayko and the Autonomous study describe the cyclic executive used on the AACS computer to schedule routines (functions) in regularly-spaced time slots:

A flow chart of the AACS flight software is shown in Figure 4-14 in the main body of this report. Normal program execution occurs in three different rate groups having periods of 10 ms, 60 ms, and 240 ms. The fourth rate group shown (20 ms) was used only for the Propulsion Module operation. Functions requiring high rates such as thruster activation and scan platform stepper motor operations are performed by the 10 ms logic. The bulk of the attitude control functions, such as attitude sensor 'reads' and control law algorithms, are accomplished by the 60 ms logic. The 240 ms logic performs a variety of tasks that do not require the higher execution rates, such as decoding CCS commands from the input buffer, fault monitor and correction, and "power code" processing.

Assessment of Autonomous Options for the DSCS III Satellite System

Prepared for the U.S. Air Force by JPL personnel (Donna L. S. Pivirotto and Michael Marcucci?), "Volume III: Options for Increasing the Autonomy of the DSCS III Satellite" (PDF), p. 179, Aug. 6, 1981.

In a more visual form, here are the 10-ms time slots and the 60- and 240-ms intervals:

     .-- 10 ms
     v
    |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
    |   <- 60 ms ->   |                 |                 |                 |
    |                              <- 240 ms ->                             |

Difficult-to-update, hand-drawn flowcharts are a thankfully retired relic of the 1970s software industry and I provide here a pseudocode version of the scheduling algorithm on the AACS. It is not apparent from the Autonomous study's flowchart, but I do make an assumption that the 60-ms routines, for example, are not all run in a single 10-ms time slot every 60 ms. Instead, the per-slot load is lessened by distributing them over the six 10-ms time slots in the interval. So, given 60-ms routines ABC and DEF, then ABC might be run in the 10-ms slots 2, 8, 14, 20, 26, ... and DEF might be run in slots 5, 11, 17, 23, 29, etc.

    Offset60 = 0
    Offset240 = 0
    EVERY 10 ms DO {
        Run all of the 10-ms routines.
        Run any 60-ms routines scheduled for time slot Offset60.
        Increment Offset60.  If 6 then reset Offset60 to 0.
        Run any 240-ms routines scheduled for time slot Offset240.
        Increment Offset240.  If 24 then reset Offset240 to 0.
    }

Within a time slot, all the scheduled routines must be sure to finish before the start of the next time slot. The programmers must quantify beforehand the worst-case execution times for each of the routines and ensure that the sum does not exceed 10 ms. The programmers probably are counting instruction cycles for the assembly statements in each routine. (I use the present tense because this would have been done for the routines prior to the 1977 launch and still needs to be done for routines updated/uploaded nearly 50 years later.) If the time exceeds 10 ms and offloading some 60-ms and 240-ms routines to other time slots is not a possibility, the programmers must speed up the routines. This is most easily done working directly with the assembly language. It would be awkward to do this in a higher-level language because the quanitification still has to be done at the assembly language level.

Both Tomayko and Autonomous also mention the FDS computer's P periods, with Tomayko providing a little more detail. With 2.5-ms time slots, the execution constraints were even more stringent than those on the AACS. There is no indication of whether or not the FDS had multiple levels of intervals like the AACS. And apparently the CCS software was also built on a cyclic executive?

Like the command computer, the data computer has a simple executive. Time is divided into twenty-four 2.5-millisecond intervals, called "P periods." Each 24 P periods represent one imaging system scan line. Eight hundred of those lines is a frame. At the beginning of each P period, the software automatically returns to memory location 0000, where it executes a routine that determines what functions to perform during that P period. Care is taken that the software completes all pending processes in the 2.5-millisecond period, a job made easier by the standardization of execution times once the direct memory access cycle was added.

James E. Tomayko, Computers in Spaceflight: The NASA Experience, 1988

Chapter Six, Section 2, Box 6-3, p. 185.

Unrelated to performance concerns, the use of an executive proves the truth in Richard Rice's wry quip:

Rice characterized the unique nature of the data computer software this way: "We didn't worry about top-down or structured; we just defined functions."

James E. Tomayko, Computers in Spaceflight: The NASA Experience, 1988

Chapter Six, Section 2, pp. 183-184.

The executive provided the top-level structure, so the programmers really were just writing functions to be called when their time slot came up!

Simulators & Other Random Thoughts

I'm sure you'd just love the reputation of being the guy who lost 2 deep space probes that had gone the furthest of any manmade object and had been doing just fine for decades. Through your bright idea for how Things Could Be Done Better.

—Mike Pellatt, comment on The Register forum, October 31, 2015, for "Think Fortran, assembly language programming is boring and useless? Tell that to the NASA Voyager team" (based on John Wenz's Popular Mechanics article).

In the comments on an Ars Technica article, Humanity's Most Distant Space Probe Jeopardized by Computer Glitch" (February 6, 2024), there was an interesting subthread about the lack of simulators for the flight computers. (Actually just for the AACS and FDS computers? Matsumoto mentioned the COMSIM/HSSIM simulator for command "sequences and CCS FSW changes".) I wanted to address some of the issues raised in this section, but then I decided to get into some of the more general issues; hence the "& Other Random Thoughts" in the title.

In every online discussion of anything touching on the Voyager computers, there are always a number of questions such as "Why didn't they do this?" and "Why don't they do that?" The answer to the first question is more than likely, "Well, you had to have been there." The second question is ably answered by Mike Pellatt above.

Shoestring Space Exploration

The Voyager spacecraft had more than fulfilled their nominal missions after Voyager 2 flew by Neptune, so they've been living on borrowed time ever since. However, NASA had long been keeping a close eye on budget concerns after being raked over the coals by Congress in 1974 for Viking cost overruns. (See Viking project hearings in the bibliography.)

Suzanne Dodd spoke of a bathtub effect in personnel levels between the Saturn and Uranus flybys:

DODD: Now, Voyager went through Jupiter and Saturn, and then it had a bathtub, because it had five years to get out to Uranus. So a lot of people left. Because Jupiter and Saturn were like 18— maybe two years apart, 18 months to two years apart in encounters. Then there was this lull, so they had to cut— a lot of people left, and/or they just needed to trim staffing. A bathtub in staffing, because there was such a long duration.

David Zierler, "Suzy Dodd (BS '84), Engineer and Deep Space Pioneer"

Caltech Heritage Project, June 9, 2023.

(McLaughlin and Wolff also refer to the bathtub effect and describe how JPL tried to reduce the effect between Uranus and Neptune by spreading the expected work out more evenly in the interval.)

The "bathtub" after Neptune would be endless and the Voyager team was very small in the succeeding decades. For those of us interested in the software side of things, keep in mind that programming would only have been one of the areas of engineering expertise needed on the team. And the 10-12 full-time employee equivalency means that some, if not many, of the team were and are working only part-time on Voyager. As knowledgeable people retired, the remaining members by necessity assumed multiple roles.

The Voyager program was almost cut off completely in the 2000s:

NASA officials said the possibility of cutting Voyager and several other long-running missions in the Earth-Sun Exploration Division arose in February, when the Bush administration proposed slashing the division's 2006 budget by nearly one-third — from $75 million to $53 million.

—Guy Gugliotta, "Historic Voyager Mission May Lose Its Funding", The Washington Post, April 4, 2005.

(Voyager's share of the $75 million was $4.2 million.) Thankfully, this didn't happen, but program leaders like Dodd (who returned to Voyager in 2010) have probably had to fight for funding every year.

Between the lack of personnel, the part-time nature of the work, and the sword-of-Damocles-like budget cut-off hanging over the project, the time and money resources available for writing new simulators, rewriting the flight software in C++ or Python, etc. were scarce. And both would run up against an insurmountable obstacle: "If it ain't broke, don't fix it!" We've got a very tenuous connection with two spacecraft, one 15 billion miles away and the other 12 billion, and you want to do what?! Paging Mike Pellatt ...

• • •

Finally, a few more words about spacecraft operations. The constraints on flight operations range from the mundane to out-of-this-world. Examples of the mundane are staffing levels. Adamski (p. 4) notes that during the long, hopefully uneventful, cruise intervals between planets, operations were limited to daytime hours since the control center was not manned around the clock; this was a contraint taken into consideration in the design of a sequence.

Skipping in-this-world concerns such as earthquakes (see Kohlhase, p. 120) and going straight to out-of-this-world constraints, everyone seems to know that the Voyager spacecraft are very far away and that it takes about 40 hours to send a command and receive a response back. (This is called the Round Trip Light Time, or RTLT.) However, I sometimes wonder if people also understand that the Voyager team is not in constant contact with the two spacecraft. You can't just make a change to the flight software, recompile it, and uplink it to the spacecraft.

I use 40 hours as a nice round number for RTLT. More specific values in 2024 are about 45 hours for Voyager 1 and about 38 hours for Voyager 2. If you need even more accuracy, the real-time Voyager Mission Status page has the hours, minutes, and seconds of the one-way light times. As I noted in the "Introduction", the distance of each spacecraft goes up and down depending on whether the Earth in its orbit is falling behind or catching up with the spacecraft. As the distance changes, so does the transmission time.

Communications with the Voyager spacecraft are handled by NASA's Deep Space Network (DSN), which consists of three tracking stations spaced about 120° of longitude apart in Goldstone (California), Madrid (Spain), and Canberra (Australia). Each station has a 70-meter dish antenna, multiple 34-meter antennas, and others. According to Matsumoto's predictions (p. 6), by now (2024), a 70-m antenna is required to send commands to either Voyager spacecraft and a 70-m antenna or two 34-m antennas are required to receive engineering and science data.

Aside: A 70-meter dish antenna is 230 feet in diameter, ¾ of the length of an American football field, as you can see in this fanciful picture of the Goldstone 70-m antenna placed in Pasadena's Rose Bowl stadium! (Pasadena is also home to JPL. Source: The Voyager Neptune Travel Guide, p. 28.)

The Voyagers are not the only spacecraft using the DSN, so the Voyager team must request and compete for transmitting and receiving time on the DSN. (At DSN Now, you can see in real-time which antennas are talking to which spacecraft.) When scheduling time, the team must take into account the 40-hour round trip light time. Depending on a variety of factors, it is possible that they might have to settle for time slots more than 40 hours apart, in which case the team must ensure they can still verify the results of an operation. Matsumoto gives an example:

Command Loss (CMDLOS) timer, the timer that triggers CMDLOS FPA entry when it reaches to the pre-set value, is set for six weeks. It is still desirable to send the command to reset the timer every week but it is a best effort approach based on the DSN resources, usually without downlink coverage for command receipt verification. (The receipt of a command is verified indirectly by the CCS hourly status data when the telemetry is available.)

Sun Kang Matsumoto, "Voyager Interstellar Mission: Challenges of Flying a Very Old Spacecraft on a Very Long Mission" (PDF)

p. 5, 2016 SpaceOps Conference, Daejeon, Korea.

After passing Saturn, Voyager 1 angled north of the ecliptic (planetary) plane as it headed out into space. After passing Neptune, Voyager 2 angled south. Consequently, communications with Voyager 2 are only possible via the Canberra DSN station. This became a problem in 2020 when Canberra's 70-m dish was taken offline for 11 months for upgrades. Voyager 2 continued transmitting data to the smaller dish antennas, but commands could not be sent to the spacecraft. Contact was successfully reestablished in early 2021 when the 70-m dish came back online. (The prior preparation and the actual downtime are featured in the 2022 film, It's Quieter in the Twilight, about the Voyager team.)

The DSN stations send commands to the Voyagers at a radio frequence of about 2.1 gigahertz. Because of the Earth's rotation (as well as the revolution of the Earth around the Sun and other factors), a dish antenna as a whole moves around in space. As a result, the relative motions of the antenna and the spacecraft (which is also moving) cause a continuously varying Doppler shift in the radio signal's frequency as seen by the spacecraft. (Kobele, p. 13, says that the Doppler shift can be up to 3 kilohertz during a DSN pass.) The Voyager's receivers were designed with this in mind and use a phase-lock loop (PLL) circuit to detect, lock onto, and maintain the lock on a varying frequency signal within a 100-kilohertz range.

In 1978, less than a year after launch, ground control forgot to reset Voyager 2's Command Loss timer and the aforementioned CMDLOS function was tested for real when the timer expired. The CCS computer thought the primary receiver was no longer working, so it switched to the backup receiver. Unfortunately, the backup receiver's PLL circuit had failed partially prior to this switchover and the receiver could not lock onto the command signal. After 12 hours, the CMDLOS function switched back to the primary receiver and contact was reestablished. "Phew!", as Wordle likes to say. A half-hour later, the primary receiver failed completely. Back again, a week later, to the backup receiver that couldn't receive. (The CMDLOS timeout value was 1 week for the primary receiver and 12 hours for the backup receiver.)

(Note that human error triggered these two problems, it didn't cause them. The partial failure of the backup receiver's PLL circuit had already happened and the complete failure of the primary receiver was just waiting to happen. The project was actually fortunate that this occurred early on and not later, for example, in the middle of flying by Jupiter.)

I don't know how they did it, but somehow the engineers figured out that the backup's PLL circuit was limited to a single, nominal frequency with a tolerance of only 100 Hz — that's 100 cycles-per-second in a 2.1 billion cycles-per-second radio frequency. Wow! Since Voyager 2's receiver could no longer compensate for the Doppler shift (up to 3 kHz during a pass), the DSN transmitter had to compensate by sending a varying frequency signal that canceled out the pre-calculated, expected Doppler shifts, thus delivering a constant-frequency signal to the spacecraft.

Note: The 100 kHz and 100 Hz figures are from Ludwig and Taylor, p. 39-40; other sources have slightly different values.

Would that things were so simple! The frequency at which the backup receiver is listening for commands is called the best-lock frequency (BLF). The BLF is extremely sensitive to the temperature of the receiver and small changes in temperature cause shifts in BLF exceeding the 100-Hz range. Many activities on the spacecraft cause these temperature changes. To cope with this added difficulty, the ground system must (1) limit uplinks to no-activity periods when the receiver's temperature is stable and (2) regularly recalibrate the BLF. The former is accomplished by following a period of activity with an uplink-free, 24- to 72-hour command moratorium that gives the temperature time to stabilize. The latter is done with a scatter-shot approach, sending something (probably a test command of some kind) at multiple frequencies and seeing which one actually gets through. (McLaughlin and Wolff, p. 5, describe using the scatter-shot approach to uplink emergency or time-sensitive commands during active periods: send the same command at multiple frequencies in hopes that one will get through. This technique may have been used in planetary fly-bys to upload onboard sequence updates.)

Amazingly, all this was made to work and Voyager 2 successfully flew by Jupiter, Saturn, Uranus, and Neptune and returned valuable images and science data under these operational constraints. And continues to do so. (The 2020 MAG-roll-gone-awry in It's Quieter in the Twilight was probably followed by a command moratorium.)

Addendum: Various sources attribute the crippling of the backup receiver to the failure of a tracking-loop capacitor in the PLL. Kobele, 1989, p. 14, was more specific: "a resistive short in the tracking loop capacitor due to particle migration through the dielectric material". This necessitated a periodic maintenance operation to prevent complete failure: "It has been demonstrated that particles which have not yet completed their migration can be destroyed when the capacitor charge is maximized by periodically pulling the uplink signal to the extremes of the loop bandwidth." (I have not seen this mentioned elsewhere.)

Sequences and Simulators

Regarding simulators, you'll find comments such as, "Give me the CPU details and I'll whip up a simulator over the weekend." These comments and the earlier questions are not bad and I admit to often having similar thoughts myself. However, as software developers, we of all people should have — but don't have — an instinctual reaction that nothing is ever as simple as it first seems.

In this case, simulating the CPU may be the least complicated part of a flight computer simulator, itself part of a larger system. It turns out that, much to my amazement, the engineers back in the 1970s — without JavaScript and Rust, without gigabytes of RAM, and without 55-MB web pages — somehow managed to write sophisticated and complex programs!

The following discussion is largely based on the papers by McEvoy (1975) and Adamski (1987) about the Viking and Voyager simulators, respectively. The descriptions here are my understanding of the systems, which may likely be imperfect. The intended takeaway is that the systems were complex and I think that will be made clear in spite of any errors. (Linick and Weld's 1992 paper examines the streamlining of the prior sequencing process, as described in Adamski, for the Voyager Interstellar Mission, an effort that successfully met two goals, among others, of moving off the Univac mainframes and operating with greatly reduced personnel in the long term. The paper provides a somewhat managerial evaluation of the changes and doesn't provide the new hardware and software details I would have liked.)

Note something that I completely missed in my initial readings. The Viking Orbiter had only one computer (and its backup), the CCS. Its Attitude Control Subsystem (ACS) and its Flight Data Subsystem (FDS) did not have their own computers. The Viking FDS did have a 1K-by-8-bit plated-wire memory:

The FDS memories were required to perform a number of essential tasks and were additionally used to accomplish a variety of FDS housekeeping functions, including reading engineering identifiers for the flexible formats; buffering MAWDS, VIS, and IRTMS (A/PW) science data; storing and updating VO time; storing PN sequences for the VIS and science formats; executing memory-alteration commands from CCS; performing most of the science format multiplexing; and counting TV pictures taken and controlling related FDS logic.

—Neil A. Holmberg, Robert P. Faust, and H. Milton Holt, Viking '75 Spacecraft Design and Test Summary, Volume II: Orbiter Design (NASA Reference Publication 1027), p. 61, November 1980.

Because of signal transmission delays, the Viking and Voyager spacecraft were designed to function autonomously, a form of operation that depends heavily on stored command sequences as opposed to single, separate, "real-time" commands. Consequently, simulation is especially geared towards the testing of command sequences.

In the introduction, I defined a command sequence as a sequence of commands, but they are much more than that. A sequence is composed of blocks, each of which consists of:

• Events - are actions (?) that may or may not be associated with spacecraft commands. In McEvoy's sample Viking Orbiter block (Figure 9, p. 414), events with CCS spacecraft commands are actions targeted to the external subsystems such as the ACS and FDS. In one entry, the CCS instructs the ACS: "Inertial mode; stop roll turn; stop yaw turn; negative turn direction". Events without CCS commands are actions taken internally by the CCS, as in this CCS event: "Disable the ACS accelerometer calibration within the CCS".

• Data - any data required by the block; e.g., tables.

• Flight software - is the non-core software needed to support the commands/events in the block.

In developing a sequence, engineers such as Suzanne Dodd could draw on previously defined blocks in the block dictionary for subtasks, much like programmers use functions from libraries. Blocks used in such a manner are customized via block options and pull in their own required flight software. A Viking block's flight software was stored as OSTRAN macros (McEvoy, p. 415); OSTRAN was the Viking counterpart to Voyager's SEQTRAN. The output of OSTRAN is assembly source code that is passed on to an assembler and linker, ultimately resulting in the memory load (containing sequence, data, and flight software) that will be uplinked to the spacecraft. It appears that the flight software resident in memory consists of (i) the core software that is always present (executive, sequence processing, fault protection, etc.) and (ii) only the non-core software needed to support the currently loaded sequences.

Throughout the development and simulation of sequences, there are automated and human checks for physical and operational constraint violations. When a violation is detected, engineers determine how serious the violation is and, if necessary, go back to the drawing board to remedy the problem. The engineers also need to verify that a sequence actually does what it is supposed to do; e.g., take a picture of Neptune. Consequently, the design-develop-test cycle for sequences is an iterative process.

A block diagram for the Voyager simulator (COMSIM) would look similar to that of the Viking Orbiter simulator (OCOMSM) below. (I think — I haven't been able to find detailed information about Voyager's simulator. Incidentally, the Viking Lander simulator was called LCOMSM.) Note that the nested CCS and FDS simulators are subsystem simulators. I'm puzzled by the absence of an ACS simulator. The Viking Data Storage Subsystem, DSS, handled the Digital Tape Recorder (DTR), a function taken over by Voyager's FDS.

The simulator outputs on the right side of the diagram are very important and provide:

• Visibility into the internal sequence processing to the sequence and flight software engineers.

• A step-by-step timeline for the flight controllers to follow (after the fact because of the time delay!) as the sequences are actually executed on the spacecraft.

The step-by-step timeline is called the Sequence of Events (SOE) on Voyager:

In addition to the blocks of load commands, the sequence generation process provides other products that are used in the command and control of the spacecraft. A Sequence of Events (SOE) is produced that is a printed, time-ordered listing of the activities contained in the CCS sequence. Performance predictions are generated for power usage, scan platform pointing, and other spacecraft functions. Additional uplink commands may also be generated along with the main sequence. These commands are held for subsequent transmission at pre-identified points in the sequence in order to perform discrete functions or enable checkpointed events.

Terrence P. Adamski, "Command and Control of the Voyager Spacecraft" (abstract)

p. 5, American Institute of Aeronautics and Astronautics (AIAA), 25th AIAA Aerospace Sciences Meeting, March 24-26, 1987, Reno, Nevada.

A spacecraft is more than just its flight software and simulators must also simulate the hardware. McEvoy suggested APL for both describing and simulating hardware, although he didn't come right out and say they used APL. I wasn't aware of APL's role in the hardware realm, but it's a real thing. (I do question the widely asserted notion that APL was originally a hardware description language. That doesn't seem to agree with Kenneth Iverson's contemporary writings and later recountings of APL's history, but it is a Fortran 5-like rabbit hole that I'll leave to someone else to explore!)

One difficulty in modeling digital hardware is the different languages used in the hardware and software disciplines. Hardware description languages that describe hardware at the logic-gate bit-time register-transfer level provide a commonly understandable detailed definition that can bridge the gap. The APL language itself provides an excellent basis for describing digital hardware. Power-on-reset states and hardware idiosyncracies, that could affect a sequence, must be considered in the modeling stage. Any of these that could affect operation of the hardware, software, timing, or other subsystems should be modeled.

Maurice B. McEvoy, "Viking Orbiter Uplink Command Generation and Validation via Simulation" (PDF)

p. 417, The Institute for Operations Research and the Management Sciences (INFORMS) Winter Simulation Conference 1975 Conference Proceedings.

Voyager's CCS subsystem simulator would also have looked similar to Viking's:

Each of the CCS, AACS, and FDS subsystem simulators would have had to simulate:

• the two CPUs,
• the internal elements of the subsystem (e.g., memory banks),
• interfaces with elements external to the subsystem (including interfaces with the other computer subsystems),
• cross-strapping (switching) between primary/redundant elements both internal and external,
• and the timing of everything.

The two CCS CPUs could also work independently of each other and not in a strict primary/backup relationship. Despite my listing of the simulation "requirements" above, there is a lot I don't know about the simulators:

• Were CPUs simulated in software or were actual hardware spares used?

• For CPUs simulated in software, was the simulation at the instruction-set level or was it a simulation of the CPUs' actual electronic circuits (e.g., using APL)?

• And similar questions about interfaces with the various non-CPU elements: Was an interface simulated at a logical level or as an electronic circuit? Or was a real interface with a hardware spare used?

• What elements could be cross-strapped? And was cross-strapping simulated?

In other words, my ideas about the simulators are largely guesswork. McEvoy does provide a few more details about the CCS simulator, although some of my questions are still unanswered:

The Computer Command Subsystem (CCS) simulator (Fig. 12) accurately models the Block redundant CCS processors, output units, and memories at the instruction and register transfer level. The accuracy with which software programs (in the memories) are simulated is dependent primarily on the accuracy with which the bit-time clocks can be calibrated during flight and timing accuracies of the 32 priority interrupts and 24 level indicators. Output simulation consisting of discrete commands (relay closures), coded commands, and telemetry data can be routed through either or both output units. A hardware and software self-test must be satisfied before access can be obtained to an output unit. (p. 418)

...

The large amount of time used to simulate uplink command decoding is due primarily to the CCS hardware and software self-test activity, which in turn is affected by the FDS engineering telemetry rate and format. (p. 419)

Maurice B. McEvoy, "Viking Orbiter Uplink Command Generation and Validation via Simulation" (PDF)

The Institute for Operations Research and the Management Sciences (INFORMS) Winter Simulation Conference 1975 Conference Proceedings.

(I don't understand the purpose of the "hardware and software self-test" in the last sentence of the first paragraph. Was this done on the actual spacecraft? The later sentence seems to indicate "Yes". I can understand self-tests performed repeatedly in the background or prior to a set of operations, but I am perhaps overestimating the frequency of the self-tests.)

The validation end of the software development process was handled by the Capability Demonstration Laboratory (CDL). Completed after the initial software was produced, it was a collection of either breadboard or flight surplus computer and science hardware, and its interfaces interconnected in the same way as those on the actual spacecraft. Its function is identical to that of the Shuttle Avionics Integration Laboratory (SAIL), in which both software and hardware changes could be tested to see if they functioned successfully.

James E. Tomayko, Computers in Spaceflight: The NASA Experience, 1988

Chapter Six, Section 2, p. 175.

And the CDL lapsed into disrepair:

In addition, the [Voyager flight team] also has to work around a lack of a hardware test bed, the limited memory of on-board computers, and antiquated programming languages.

The Capability Demonstration Lab (CDL), the testbed used during the prime mission, could not be maintained and had to be abandoned at the start of VIM. The failures of the testbed were too often, even in pre-VIM, due to aging hardware and disappearing repair expertise. The project had to move to a new location in early VIM and the CDL did not survive the move. There are no simulators for the AACS or FDS and only the CCS has a simulator, i.e., HSSIM. As a result, any FSW changes other than something very simple have to be done in the CCS.

Sun Kang Matsumoto, "Voyager Interstellar Mission: Challenges of Flying a Very Old Spacecraft on a Very Long Mission" (PDF)

p. 6, 2016 SpaceOps Conference, Daejeon, Korea.

The last sentence above suggests to me that a non-trivial change to the AACS or FDS flight software is considered risky and that, if possible, it is better to leave the existing AACS/FDS software as-is and have the CCS take over responsibility for the changed functionality. That's my impression at least; given the lack of rigorous simulation capabilities, I think it's a sensible decision.

Oddly, searching on Google for JPL's "Capability Demonstration Lab", mentioned by both Tomayko and Matsumoto, only brings up a couple of references to Matsumoto's paper; on Bing, you get a single result ... for JPL's Cyber Defense Laboratory! By a strange coincidence, a couple of hours after reading the previous sentence, I was reading McLauglin's & Wolff's paper about the preparations for Voyager 2's Uranus fly-by and I came across the most detailed description of the CDL yet. Which is not saying much, but the passage below does give a good picture of a flight software engineer's life!

The physical actions and movements in the Digital Tape Recorder (DTR) produce "equal and opposite reactions" in the spacecraft and thus affect the spacecraft's attitude. After the Saturn encounter, a routine, DSSCAN, was added to simultaneously fire attitude control thrusters when performing high-rate tape operations, thereby greatly reducing the time it takes for the spacecraft attitude's rate-of-change to slow enough to begin the next science operation. (A savings of over 3 minutes in the case of the end-of-track turnaround, valuable time during the all too brief hours of closest approach.) That's my understanding; it's not clear which computer DSSCAN was added to, AACS or CCS.

Note how Voyager 1, its mission already completed, was used as a test bed for the Voyager 2 FSW modifications. Note also that the AACS and CCS programs were not reassembled, but instead patched, a practice that had been ongoing since launch. What a nightmare the onboard software must have become! I don't know how long the practice continued, but it makes the VIM team's reluctance to modify the AACS software, per Matsumoto, even more understandable.

The DTR is mounted on the spacecraft such that its angular momentum is introduced into the yaw and pitch axes of the spacecraft with almost none going into the roll axis. DSSCAN was first programmed to introduce cancelling momentum in the yaw axis only. The modification to the AACS and CCS software took place in an environment of a scarcity of available memory so that, from a programming point of view, it had to be carefully fit in. The "patch" was carefully tested in the Voyager Capability Demonstration Laboratory (CDL) before loading onboard Voyager 1. (The AACS and CCS programs were modified without being reassembled as is the case with all AACS and CCS changes since launch.) The CDL is a digital/analog simulation of many of the spacecraft capabilities. Modifications or tests of any degree of complexity are done first, whenever possible, on Voyager 1 before implementation on Voyager 2, a reflection of the fact that Voyager 2 still has two planetary encounters scheduled while Voyager 1 has none.

Once onboard the spacecraft, DSSCAN was tested during periods of DTR operation to determine empirically how many thruster pulses should be used to counteract a DTR event. For example, tests on Voyager 1 in March of 1984 showed that five thruster pulses were the optimum number. These pulses each have an electronic-activation length of 10 ms, and a single pulse will change the drift rate about the spacecraft yaw axis by approximately 11 µr/s. See Fig. 12.

Now that the effect of DSSCAN on yaw-axis momentum introduced by DTR events is known, the subroutine is being modified to incorporate momentum (near) cancellation in the pitch axis. Although the modification is, in principle, straightforward in view of the yaw axis experience, in practice this is not the case. The relative complexity arises from the tight programming that must be done because of the shortage of AACS and CCS words and existing programs that have already been heavily modified; with resultant complexity of code. Add to this the potentially serious consequences that could arise from any mistake in modifying the control law of the spacecraft, and one appreciates the fact that even seemingly simple changes can require extensive time for testing and experienced personnel for safe implementation. (pp. 11-12)

...

During implementation of the sequence, certain critical or complex subsequences may be checked for validity by running them through a sophisticated analog/digital simulation of extensive portions of the spacecraft: the Voyager Capability Demonstration Laboratory (CDL). The CDL contains, for example, duplicate versions of CCS, AACS, FDS, scan platform actuators, and the DTR. (p. 17)

—W.I. McLaughlin and D.M. Wolff, "Voyager Flight Engineering: Preparing for Uranus" (abstract), pp. 11-12, American Institute of Aeronautics and Astronautics (AIAA), 23rd Aerospace Sciences Meeting, January 1985, Reno, NV.

Aside: I'm not sure how to square the "patching-only" of the CCS code (or if it even needs to be squared) with McEvoy's description of assembly source code being pulled from the sequencing block dictionary, code that would be assembled and placed in a sequence's uplink memory load.

The High Speed Simulator (HSSIM) is a flexible spacecraft simulator developed at JPL in the 1990s and written in C++. Spacecraft components are represented by C++ objects and a component/object may have one or more interfaces. Interfaces are strongly typed and connections between components can only be made over compatible interfaces. For example, you can connect a CPU's data bus to a memory bank's data bus, but not to the memory bank's address bus. The link, at the C++ level, between two interfaces is called a splice. Each component has its own Tcl interpreter that allows scriptable behavior in addition to other functions:

C++ was chosen as the implementation language for simulation components, because of its run time efficiency and its support for object-oriented programming. A processor component, for example, may emulate a hardware instruction set and the code which implements this must be sufficiently fast to meet the performance objectives of the simulator.

Most components contain a[n] embedded interpreter which recognizes the language Tcl. Tcl is a freely available embeddable language and concrete interpreter implementation developed at the University of California, Berkeley. Tcl was augmented to make it multi-thread ("MT") safe and to add additional features, in particular, a C++ class wrapper and a fast "remote procedure call" that can execute commands remotely on named interpreters (many of the components within HSS contain named interpreters, which are used to manipulate the component) and return the result of the execution to the local interpreter. The HSS Tcl subsystem is used for many things, including model state examination and alteration, establishment of model interconnections, and model state monitoring and display (using Tk, a toolkit based on Tcl which can quickly create Graphical User Interfaces). As a general rule, Tcl is used unless other performance or robustness concerns dictate the use of splices.

—K. Patel, W. Reinholtz, and W. Robison, "High Speed Simulator — A Simulator for All Seasons" (PDF), p. 751, SpaceOps 1996, Proceedings of the Fourth International Symposium, Munich, Germany.

HSSIM supports multiple implementations of a given spacecraft component/object, with the choice of which implementation to use being made at the time of a simulator run. Depending on their need, an engineer can choose between, for example, a slower, bit-level (logic-gate) simulation of a component or a faster, functional-level simulation. HSSIM was successfully used in various roles for the Jupiter orbiter, Galileo, beginning some time after the spacecraft's high-gain antenna failed to fully deploy in 1991. According to the paper, versions for the Saturn orbiter, Cassini (to be launched in 1997), and Voyager were to be delivered "later this year" (1996?).

As a software developer, I admire and respect the design of HSSIM. And my knowledge of the capabilities and implementation of the earlier COMSIM and CDL is almost entirely inferred from McEvoy's description of the Viking simulator; i.e., I could be way off-base. Keeping this in mind, I suspect that the HSSIM delivered to and used on Voyager is a bare-bones CCS simulator. Building a software-based simulator for a specific spacecraft is a labor-intensive task, with or without HSSIM. You need people knowledgeable about the spacecraft and buy-in from a management willing to invest in the effort. The nearly 20-year-old Voyager spacecraft had already fulfilled their primary missions and the program lacked some key requirements for building a simulator:

• Many people with intimate knowledge of the spacecraft hardware and flight software had moved on or retired.
• JPL as an organization was probably losing interest in the program as newer — much newer — programs vied for attention.
• Machine-readable documentation (e.g., hardware specifications) was probably non-existent.
• HSSIM developers themselves were probably none too eager to work on technology from the ancient past!

Knowing C++ and Tcl would have been helpful. The HSSIM paper did mention automatic code generation using an expert system and, under "Future Direction", proposed other means of automatically building parts of the simulator. Those would lessen the need for C++/Tcl software expertise, but there's still the laborious task of assembling and manually entering all the information needed as inputs to the expert system and to the other proposed mechanisms.

So, no AACS or FDS simulators. Only a bare-bones CCS simulator. (Again, that is my impression from my readings and I hope I am wrong.)

AACS, HYPACE, and HYBIC

NOTE that the AACS computer is not the at-the-time experimental HYPACE (Hybrid Programmable Attitude Control Electronics) computer. For budgetary reasons, Voyager instead used a CCS computer running at a higher clock rate and with an added index register found useful on HYPACE. Dr. Tomayko wrote:

Documentation for the system still refers to the attitude control computer as HYPACE, even though its heart was the [CCS] command computer.

James E. Tomayko, Computers in Spaceflight: The NASA Experience, 1988

Chapter Six, Section 2, p. 178.

(General Electric made both the CCS and HYPACE computers.) With respect to HYPACE, Dr. Tomayko wrote:

Index registering meant that the same block of code could be used for all three axes, reducing memory requirements.

James E. Tomayko, Computers in Spaceflight: The NASA Experience, 1988

Chapter Six, Section 2, p. 177.

This sounds like the typical use of an index register: add the value in the index register to addresses when accessing memory. By changing the value in the index register, a program can switch between each axis's set of variables prior to calling a common subroutine.

I suspected a simple way of implementing an index register so as to avoid changing the CPU's instruction set would have been to place a memory-addressable chip or electronic circuit between the CPU and memory to hold the index value. The following description of the AACS computer found elsewhere gives a name to a circuit that may or may not be such an implementation:

The Attitude and Articulation Control Subsystem used an augmented version of the CCS computer that inserted a unit (the Hybrid Buffer Interface Circuit (HYBIC)) between the CPU and RAM, which intercepted instructions to add indexed addressing capability (at the expense of other instructions), and accelerated instructions that used idle cycles.

—John Culver, "The CPUs of Spacecraft Computers in Space", The CPU Shack.

Unlike other (incorrect) descriptions, Culver's doesn't conflate the AACS with HYPACE, so I think he knows what he's talking about. His description does seem to indicate that changes were made to the CCS CPU and its instruction set for the AACS. Strangely, Tomayko's report doesn't mention HYBIC at all.

In her 2016 paper, Matsumoto describes a problem in a decidedly more complex HYBIC:

The Hybrid Buffer Interface Circuits (HYBIC) on V1 was switched to the backup unit in 2002 due to a failing component inside the HYBIC Analog to Digital (A/D) converter. The star tracker on the original HYBIC was performing well at the time of the switch but each HYBIC has a unit (A/D converter, sun sensor, and star tracker) dedicated to the HYBIC. The star tracker on the current HYBIC is degrading rather rapidly and closely monitored; it is quite likely that a HYBIC switch back to the original is needed before the end of the mission

Sun Kang Matsumoto, "Voyager Interstellar Mission: Challenges of Flying a Very Old Spacecraft on a Very Long Mission" (PDF)

p. 3, 2016 SpaceOps Conference, Daejeon, Korea.

Okay, HYBIC is a lot more than just an index register or registers. (See the picture of the HYBIC Logic Subassembly at the Smithsonian.) A 2006 JPL presentation about the Voyager 2 HYBIC problems included a functional block diagram of the AACS (click image to enlarge):

The diagram doesn't show an interface between the HYBIC and its associated memory bank. However, a 1981 U.S. Air Force study, Assessment of Autonomous Options for the DSCS III Satellite System, has a similar diagram (Figure B-3, p. 178) that adds a 6th arrow between the HYBIC and RAM. And, interestingly, has a dashed box labeled "HYPACE" around the upper portion of the diagram!

The role of HYBIC in the AACS is up in the air for me. Online information about HYBIC is extremely scarce and what little I could find is basically a rehash in one form or another of the information above.

Miscellaneous Information

Ben-Hur Rides Again!

Eat your heart out, Lew Wallace, author of Ben-Hur! I don't recall ever seeing the whole movie, but I did read the novel. My memory is that the chariot scene was even more exciting in the book than in the movie, but neither hold a candle to Voyager 2's launch troubles!

Based on Tomayko's telling (Box 6-1, p. 179-180, in Computers in Spaceflight), my understanding is that the Voyager 2 spacecraft had separated from the launch vehicle and was in the process of extending the booms while at the same time trying to correct its whopperjawed attitude. Things began to go wrong, but I'll let Rasmussen and Litty tell the story:

The first launch provided more excitement with little delay. An unbelievable combination of events shortly after injection initiated a chain reaction that ultimately triggered nearly every single fault routine in the AACS repertoire. Briefly, the sequence happended as follows.

The spark to the fuse was a partially latched boom, resulting in high tip-off rates at propulsion module separation. Meanwhile, reconfiguration activities and a precautionary restoration of isolation value states had temporarily disabled attitude control. While thrusters were waiting to fire, the tip-off effects were accumulating, so that by the time the isovalve power requests reached the top of the queue, a large angular offset existed. By coincidence, thruster control was re-enabled just before sequence activation of thruster fault detection. This quickly responded to the large error by again disabling thruster firing in preparation for a switch to backup thrusters. More isovalve activity ensued causing further delays, three more repetitions of the error reponse, and finally a self-inflicted heartbeat failure, all within sixteen seconds of separation!

This last action was pivotal in resolving the situation. The new system, with no memory of past difficulties, including the large error, quickly brought the rates under control. Finding the sun nowhere in sight, this system then proceeded automatically to search for and acquire celestial references, achieving success less than three hours after launch. Ground operations spent the next several days recovering.

—R.D. Rasmussen and E.C. Litty, "A Voyager Attitude Control Perspective on Fault Tolerant Systems" (abstract), p. 246, American Institute of Aeronautics and Astronautics (AIAA) Guidance and Control Conference, August 19-21, 1981, Albuquerque, New Mexico.

That last exquisite sentence is priceless!

Suzanne Dodd: Early Voyager Years

A couple of additional paragraphs from the Caltech Heritage Project article:

DODD: It was quite a bit different. But, you come in, and it was a fairly routine thing. You would be working on a sequence, and it would take six weeks to develop that sequence, and then there was different steps, and different software programs, and different reviews. That process then repeats itself every six weeks. From a work standpoint, you're doing a fair amount of the same thing, although you sometimes had special events. You sometimes worked with a particular science team on a special calibration they wanted to do. What I do remember is the Voyager team, the sequence area had a lot of pranksters in it. They would like fill a person's drawer with water and put fish in there, and so when they came in the next day, the water would splash and they'd have these goldfish swimming around in their desk. [laughs] It's that kind of a team camaraderie. Most people would eat lunch together, too. You just really were cohesive as a team.

ZIERLER: What areas did you have real responsibility, even in the early years?

DODD: I hired in when four other people hired in. Now, Voyager went through Jupiter and Saturn, and then it had a bathtub, because it had five years to get out to Uranus. So a lot of people left. Because Jupiter and Saturn were like 18— maybe two years apart, 18 months to two years apart in encounters. Then there was this lull, so they had to cut— a lot of people left, and/or they just needed to trim staffing. A bathtub in staffing, because there was such a long duration. So, I was part of the crew that came back in, new group of crew that came back in during Uranus. There were four of us that hired in all at the same time, to do basically the same job, which was sequencing. It was basically a very junior job on Voyager, sequencing engineer, but a super exciting mission. Really gets you going for liking this type of work. Then, when it came to Neptune, I got to be the lead sequence engineer. So, of the four of us that were doing it, I got the closest approach sequence.

David Zierler, "Suzy Dodd (BS '84), Engineer and Deep Space Pioneer"

Caltech Heritage Project, June 9, 2023.

Larry Zottarelli

In an article about astrophysicists debating whether or not Voyager 1 actually had entered interstellar space, the following encapsulation of a CNN report on Larry Zottarelli appeared:

[Determining if Voyager 1 has really entered interstellar space] will also be just a little bit harder than it was last week, because the last original member of the Voyager team has retired. Larry Zottarelli, aged 80, left NASA's employ this week after 55 years on the job. Zottarelli helped to develop Voyager's on-board computers and has worked on the mission since 1975. CNN reports that he was sent on his way with a handshake from actress Nichelle Nicholls, Star Trek's Lt. Uhura. NASA is reportedly seeking a replacement fluent in FORTRAN, Algol and assembly language for the Voyagers' 250 KHz General Electric 18-bit TTL CPUs, complete with single register accumulator and bit-serial access to 4096-word plated-wire RAM.

—Simon Sharwood, "Has Voyager 1 escaped the Sun yet? Yes, but also no, say boffins", The Register, October 30, 2015.

Boffin is British slang for a scientist or engineer! And the original CNN report has pictures, especially of Zottarelli with Nichelle Nichols AKA Lt. Uhura! (The Register gets it wrong, however. The picture with Nichols is from the Voyager 30th anniversary celebration in 2007, not Zottarelli's retirement.)

The following is from an (unpublished?) article about the Voyager team:

The quiet voice, that's Larry Zottarelli, the programmer and "computer whisperer", understanding his machines like no one better does.

—Joachim J. Kehr, Editor SpaceOps News of the Journal of Space Operations & Communicator; April 2016, "Voyager — The Right Staff" (PDF), p. 4.

I wasn't fortunate enough, but I think I would have liked working with him! (The Kehr article includes an interview with Suzanne Dodd.)

I don't quote it here, but this widely cited, 2017 article in the New York Times leads with a picture of and story about Larry Zottarelli: "The Loyal Engineers Steering NASA's Voyager Probes Across the Universe", by Kim Tingley, New York Times, Aug. 3, 2017.

Real Programmers Don't Use Pascal

Some of the most awesome Real Programmers work at the Jet Propulsion Laboratory in California. Many of them know the entire operating system of the Pioneer and Voyager spacecraft by heart. With a combination of large ground-based FORTRAN programs and small spacecraft-based assembly language programs, they can do incredible feats of navigation and improvisation, such as hitting 10-kilometer wide windows at Saturn after six years in space, and repairing or bypassing damaged sensor platforms, radios, and batteries. Allegedly, one Real Programmer managed to tuck a pattern-matching program into a few hundred bytes of unused memory in a Voyager spacecraft that searched for, located, and photographed a new moon of Jupiter.

—Ed Post, "Real Programmers Don't Use Pascal", Datamation, July 1983, "READERS' FORUM", pp. 263-265. (HTML version; the full magazine issue is available as a large PDF.)

Clash of Viking Mainframes

This section is not related to Voyager. While searching for information about the Voyager flight computers, I landed on Herb Johnson's web page, "COSMAC 1802: history of microprocessors in space", and his accompanying notes about the Viking flight computers. The notes rely mostly on Tomayko's report and a Martin Marietta Corporation report, Viking Software Data. (The links to the report in Johnson's notes don't work anymore.)

Preceding the Voyager program by several years, the Viking mission to Mars began in 1968. Two identical pairs of spacecraft were launched in 1975 and arrived at Mars in 1976. Each pair consisted of (i) an Orbiter which went into orbit around Mars and (ii) a Lander which landed on Mars.

JPL was responsible for the Orbiter. Martin Marietta designed and developed the Viking Lander and its flight software. They also developed the ground-based, mission operations (mission control) software for the Lander. Afterwards, Martin Marietta wrote an evaluation of the software development process for the U.S. Air Force, the document used by Herb Johnson above.

Viking Software Data turned out to be very interesting reading. After the introductory material, the document consists of individual evaluations of different techniques used in the development and testing process. Each evaluation is structured as follows: name, summary, application considerations, recommendation, history, description, qualitative results, and quantitative impact.

I was motivated to write "Clash of Viking Mainframes" upon being taken aback by the description of the second technique, "DIFFERENT DEVELOPMENT/INTEGRATION SITES". Martin Marietta developed and tested the mission operations software on Control Data Corporation (CDC) mainframes at their Denver, Colorado facility (denoted MMC) and then converted the source code to be compiled and tested again on the target Univac and IBM mainframes at JPL's Pasadena, California Viking mission control center! This is somewhat akin to developing a native Windows application under Linux because you only had a Linux PC; though possible, it would be unthinkable today. However, as you'll see, there were some very good reasons for this approach — some known beforehand and some uncovered during the process — and I can't fault Martin Marietta or JPL for taking this approach.

NASA management feared JPL overextending itself, so it assigned the Orbiter to JPL and the Lander to Martin Marietta. However, the Lander integration and test and the actual operations would still take place at JPL. So:

Martin-Marietta Corporation, the Viking Lander contractors, had to do some dangerously unique software development when NASA decided to move control of the Lander from Langley to JPL. Since Orbiter software development and giving support to other missions tied up JPL's computers, Martin took the chance of developing the Lander software in a "minimal higher order language," specifically a hopefully transportable subset of FORTRAN. Martin's solution reflected its recent migration to IBM 370 series and Control Data 6500 series computers at its Denver plant. These were technologically more advanced than the JPL computers and could not be trusted to produce directly transportable software. The idea worked, but Martin admitted that the requirement for delivering mission support software 10 months before the flight provided strong motivation.

James E. Tomayko, Computers in Spaceflight: The NASA Experience, 1988

Chapter Eight, Section 3 (Unmanned mission control computers), p. 267.

And getting into the "DIFFERENT DEVELOPMENT/INTEGRATION SITES":

APPLICATION CONSIDERATIONS: The forecasts for the loads on the computer sets at JPL indicated that the MMC portions of the Viking operational software system could not be developed on them. The MMC facility contained CDC 6400, CDC 6500 and IBM 360/75 computer sets, whereas the JPL facility contained UNIVAC 1108 and IBM 360/75 computer sets. The operating systems of the IBM computers at the two facilities were different. No equipment similar to the JPL UNIVAC 1108 system was available in the Denver area. Pathfinder studies indicated software could be developed on non-target computers without creating any serious conversion problems. (p. 30)

HISTORY: One of the problems which faced the management of the Viking mission operations lander software development was that the development computers differed from the operational computers. The Martin Marietta facility consisted of CDC 6400 and CDC 6500 and IBM 360/75 whereas the JPL computer facility consisted of IBM 360/75s and UNIVAC 1108s. Thus the software conversion task became a management concern very early in the software development activities. (p. 31)

Martin Marietta Corporation, Viking Software Data: Final Technical Report (15MB PDF)

May 1977, RADC-TR-77-168.

Keep in mind that lacking the target Univac mainframe could not be remedied by running down to the local computer store, picking a computer off the rack, and then shoving it in a spare closet with an ethernet cord back at the office. A mainframe computer and its peripherals were very expensive and you had to commit to sizeable initial and long-term investments in real estate (new or existing housing for the installation), utilities (power, etc.), supplies (drums, disks, tapes, punch cards, etc., etc.), and especially all the support personnel needed to manage and run the whole thing.

I should note that my only experience with mainframes was with a Univac 1100-series mainframe in the University of Maryland's Computer Science department in the late 1970s. (They had two 1100s, but I only ever used one of them.) Supporting hundreds if not thousands of students, faculty, and staff meant that the department's mainframe installation fit my perhaps overbroad characterization above, but I realize that smaller groups or companies could have much leaner installations.

The CDC 6500 had two 6400 CPUs; the separate 6400 computer was apparently not used for Viking development. I don't know why the report says here that Denver had an IBM 360/75 when elsewhere it speaks of a 370, specifically a 370/155 on p. 260. The computers from the three companies had widely differing CPU architectures:

Pathfinder studies showed the feasibility of developing the software on the CDC mainframes and converting it for the IBM and Univac mainframes, as well as providing guidance and recommendations on practices and techniques. I think two particular practices contributed to the ultimate success of the development process: (i) determining a minimal subset of Fortran that, if adhered to, would significantly reduce the level of manual effort in the conversion process and (ii) educating the programmers in the whys of the minimal subset, in the 3 different architectures, and in the importance of writing portable code.

The conversion process was successful, but not without a number of serious problems. The minimal subset of Fortran couldn't encompass some of the different but needed operating-system-specific capabilities. Assembly language code was required in some cases for performance reasons or to stay within memory limitations. IBM assembly code could be assembled and tested to some extent on MMC's IBM 370, but the Univac assembly code could only be developed and tested on JPL's 1108. Fortran programs tested and changed at JPL had to have the changes back-ported to the CDC source code, an annoying and error-prone task.

I was very surprised at how difficult the IBM computers made everything, and not just in the conversion process. JPL had an unhappy history with IBM 360s according to Tomayko (Chapter 8, Section 3, p. 265). In late 1969 and early 1970, JPL received two hand-me-down 360s from two other NASA centers; JPL later purchased a third, brand-new 360. The one previously used on the Apollo program had a bespoke, real-time operating system from IBM. At the time of Mariner 9's launch in May 1971, the ground software failed every 5 hours. By the time Mariner 9 reached Mars in November, the failure rate had thankfully dropped to only every 20 hours! (I am reminded of a colleague in the 1980s who went to work at an IBM facility in Maryland for a low-earth-orbiting satellite. Per his telling, the ground system would usually crash once during a 20-minute satellite pass, but the system rebooted fast enough to keep the pass from being a total waste. Again, this was in the 1980s and I don't remember if the system used an IBM or non-IBM minicomputer.)

Some mistakes were self-inflicted, although I would not count among those good-faith decisions that just didn't turn out as planned. All the logic and reason in the world can't foresee, for example, a faulty IBM Fortran compiler on JPL's 360. However, Martin Marietta should have foreseen the major hit the Viking effort would take from switching the CDC 6500's operating systems from CDC's unofficial MACE OS to its official SCOPE OS in the middle of development. Viking would not have been the only project supported by the CDC mainframe and perhaps MMC calculated that the benefits to other projects which needed SCOPE would outweigh the cost to Viking and other ongoing projects.

Despite MMC not having a Univac 1108, the development, conversion, and testing of the 1108 software was much smoother than for the IBM software. Being able to compile and test the Univac programs at JPL anytime day or night was probably a significant factor in this.

QUALITATIVE RESULTS: ... This occurred in the IBM 360 conversion effort. Due to the operational procedures at JPL, in order to receive the necessary turnaround time to do program conversion and testing, blocks of computer time had to be scheduled. These blocks of time were usually 4 to 6 hours in duration during the week starting at 9:00 PM to 4:00 AM, and up to 48 hours duration on the weekends. The pressures of trying to make as many runs as possible and meet delivery schedules forced many extra errors into the software. Instead of doing a detailed analysis of the code and the dump of the program the programmer or engineer would shot-gun many runs to try to fix errors. During longer block-time stretches many engineers and programmers would work until they introduced new errors due to physical and mental exhaustion. This altering of normal work habits did not occur on the 1108 conversion effort. The turnaround was excellent and the machine was available for use during normal working hours. (p. 34)

... The JPL version of OS was a real-time system with standard OS features but was a different release than the MMC's IBM OS. This caused problems in the conversion effort. In one instance the difference in FORTRAN compilers between MMC IBM and JPL IBM caused schedule slippages due to errors in the target compiler that were not discovered until the conversion process began. This problem happened early in the conversion process and two steps were taken to help solve the problems: 1) the release of the compiler JPL had was installed at MMC for use by the software still in development and 2) the release of the MMC compiler was installed at JPL for use when needed. (pp. 34-35)

... The operating system [on MMC's CDC computers] was changed from MACE to SCOPE during the period in which the software was being developed. This change caused many schedule impacts in the Viking software development. Much time was spent by the development programmers changing their job control cards, their file naming conventions, their file structures, and in learning how the system worked. This coupled with extra down time and running of two operating systems caused much confusion. (p. 35)

QUANTITATIVE IMPACT: ... The conversion process represented approximately five percent of the development effort for 1108 programs, and approximately ten percent of the development effort for 360 programs. (p. 36)

Martin Marietta Corporation, Viking Software Data: Final Technical Report (15MB PDF)

May 1977, RADC-TR-77-168.

I haven't read Martin Marietta's report from cover to cover, but I have read substantial amounts in the operations software and flight software sections. The prose is not dry and Martin Marietta was honest about what techniques and practices worked and which didn't and why. Again, another sort-of-Voyager-related-but-not-really document you can dive into on any page and enjoy.

I was unable to find an online copy of 1973's "Characteristics of FORTRAN" (download page for brief-description PDF) by W.R. Garner of Martin Marietta, which detailed the differences between Fortran on the IBM 360, Univac 1108, and CDC 6000s. Nor was I able to find a copy of Martin Marietta's Viking Flight Operations Programmer Guide (mentioned in the Viking Software Data report), which described the minimal subset of Fortran to be used at MMC.

However, I did came across another, slightly later instance of Martin Marietta converting a Fortran program developed on the CDC 6500 to run on a Univac 1108. In this case, the program implemented a contamination model for the ESA Spacelab flown on multiple Space Shuttle missions. The target 1108 was located at NASA's Marshall Space Flight Center in Huntsville, AL. This 1976 requirements study detailed the differences between FORTRAN IV on the CDC 6500 and FORTRAN V on the Univac 1108:

This study was performed in order to determine the modifications necessary to convert the Martin Marietta Aerospace (MMA) Spacelab Contamination Computer Model, written for a CDC 6500 computer, for use on a Marshall Space Flight Center (MSFC) computer ...

Major differences between the CDC 6500 and UNIVAC 1108 are memory capacity, word length, and various specific functional capabilities. The CDC 6500 uses Fortran IV whereas, the UNIVAC 1108 uses Fortran V control language. None of the differences are such as to create any great difficulties in program conversion and a good programmer, familiar with the contamination program, should be able to process the necessary changes with minimum difficulty, knowing the differences.

—L.E. Bareiss, V.W. Hooper, E.B. Ress, and D.A. Strange, "Payload/Orbiter Contamination Control Requirement Study - Computer Interface Study", Final Report, p. ii, September 30, 1976.

Bibliography

Essential Reading

Command Sequences & Flight Software

Other Sources

Multimedia!

I believe Aaron Cummings's YouTube video showed up in search results when I was looking up information about the Voyager computers. I learned about the two movies (and lots of other information) from the Ars Technica community.