On January 9, Canada’s CHIME (the Canadian Hydrogen Intensity Mapping Experiment) radio telescope team described the detection of a repeating, milliseconds-long FRB (fast radio burst), only the second to ever be detected. The announcement appeared in two letters published in the same January 9 issue of Nature magazine and a third paper presented that same day at the 233rd American Astronomical Society meeting in Seattle. The FRB repeated six times, and all six FRBs came from the same location in the sky, about 1.5 billion light years distant, based on the observed signal dispersion.

More than 60 FRBs have been observed to date by radio telescopes around the world. However, only one repeating burst had been observed prior to the CHIME discovery. The Arecibo radio telescope in Puerto Rico made that one previous repeating FRB observation in 2015. CHIME detected the repeating FRB along with 12 other FRBs over three weeks during the summer of 2018, which was prior to CHIME’s full commissioning. At that time, the telescope’s processing capabilities were running at only a fraction of full capability. All of its servers had yet to be installed.

Dude, It’s a Half-Pipe Observatory

CHIME looks like a series of parabolic, side-to-side snowboarding half-pipes made out of chicken wire – er – galvanized steel mesh. (See Figure 1.) The wire-mesh half pipes, which measure 100m by 20m, are antenna reflectors that feed signals to the telescope’s 1024 antennas. CHIME’s developers chose wire mesh as a reflecting material to compromise between RF reflectivity and snow-shedding capability. (It’s in Canada, remember?)

Figure 1. The CHIME Radio Telescope resides in the heart of Canada’s wine country on the south end of Okanagan Lake in British Columbia. (Image credit: CHIME)

A walkway suspended above each mesh reflector and located precisely at each parabolic half-pipe’s focus line holds a linear series of polarized feed antennas and LNAs (analog, low-noise amplifiers). Each antenna has two polarized RF outputs, and each RF output is amplified by an LNA. The amplified signals travel over coaxial cables to a large, shielded RF enclosure (actually, it’s a repurposed, weatherproof shipping container) for filtering, further amplification, and signal processing. The CHIME telescope has 1024 dual-polarization antennas (256 antennas per half-pipe) for a total of 2048 antenna feeds.

Needless to say, there’s a lot of signal processing involved with correlating signals from thousands of individual antenna feeds. CHIME processes the signals received across the massive radio-antenna array using a technique called interferometry. Most interferometry telescopes reconstruct images across a small region of the sky by mechanically pointing several individual dish antennas in an array. However, CHIME has no moving parts. It reconstructs an image by digitally processing the information its 2048 antenna feeds have captured as the Earth sweeps the sky over a 24-hour period.

FPGAs, GPUs, and CPUs, Oh My!

And what’s used to process, sort, and correlate the signals from all those antennas? FPGAs, GPUs, and CPUs of course.

The CHIME telescope wasn’t originally designed to look for FRBs. It was designed to aid the search for dark energy. According to NASA: “In the early 1990s, one thing was fairly certain about the expansion of the universe. It might have enough energy density to stop its expansion and recollapse, it might have so little energy density that it would never stop expanding, but gravity was certain to slow the expansion as time went on.” Granted, the slowing had not been observed, but, theoretically, the universe had to slow. The universe is full of matter, and the attractive force of gravity pulls all matter together. Then came 1998 and the Hubble Space Telescope (HST) observations of very distant supernovae that showed that, a long time ago, the universe was actually expanding more slowly than it is today. So the expansion of the universe has not been slowing due to gravity, as everyone thought; it has been accelerating. No one expected this, no one knew how to explain it. But something was causing it.

“Eventually theorists came up with three sorts of explanations. Maybe it was a result of a long-discarded version of Einstein’s theory of gravity, one that contained what was called a ‘cosmological constant.’ Maybe there was some strange kind of energy-fluid that filled space. Maybe there is something wrong with Einstein’s theory of gravity and a new theory could include some kind of field that creates this cosmic acceleration. Theorists still don’t know what the correct explanation is, but they have given the solution a name. It is called dark energy.

“More is unknown than is known. We know how much dark energy there is because we know how it affects the universe’s expansion. Other than that, it is a complete mystery. But it is an important mystery. It turns out that roughly 68% of the universe is dark energy.”

In addition to scientists not knowing what dark energy is, there’s currently no known way to detect dark energy directly. Any evidence must be collected indirectly. Observing and measuring periodic fluctuations in the distribution of baryonic matter, specifically neutral hydrogen, across hundreds of millions of light years, is thought to be one indirect method for studying dark energy. These fluctuations, called baryonic acoustic oscillations (BAO), serve as a “standard ruler” for cosmological distance. That’s why CHIME’s chicken-wire metal-mesh half-pipes ended up being built in the heart of Canada’s wine country on the south end of Okanagan Lake in British Columbia.

Teflon Antennas

Each of CHIME’s dual-polarization antennas is made from a cloverleaf-shaped PCB. Tuned baluns combine differential signals from two adjacent cloverleaf petals to form one single-ended output. These are all four-leaf clovers, so there are two RF outputs per cloverleaf. Conventional PCB materials like FR4 are unacceptably lossy for astronomical instrumentation, so CHIME’s designers used Teflon-based PCB material for both the balun stem and support base.

The RF antenna signals are routed to co-located LNAs (low-noise amplifiers), and each LNA drives 60m of coaxial cable that leads to a central shielded enclosure made from a weatherproof shipping container, which holds the rest of the telescope’s analog electronics. Within the enclosure, the analog antenna signals are first bandpass-filtered and then amplified by three more analog gain stages.

Another 2m of coaxial cable delivers the filtered signals to ADC daughtercards plugged into digital back-end processing cards known as ICE motherboards, where the bandpass-filtered analog channels are digitized by individual 8-bit, 1.25Gsample/second ADCs operating at 800 Msamples/sec. The CHIME telescope’s input data rate is staggering:

2,048 channels x 8 bits/channel x 800 Msamples/sec = 13.11Tbps

The 2048 filtered, amplified, and digitized antenna signals feed an FPGA-based “F-engine,” which converts each microsecond of recorded, time-based data into 1024 frequency channels spanning the 400MHz to 800MHz range. The FPGAs, which are Xilinx Kintex-7 devices, channelize the input data from the ADCs into frequency bands using a custom polyphase filter bank followed by FFTs. Over the 400-to-800MHz frequency band of interest, CHIME’s spectral frequency resolution was originally 390kHz per channel.

The F-Engine organizes the spectral data by frequency bin and ships these frequency bins to a GPU-based X-engine for spatial correlation. The X-Engine consists of 256 processing nodes, each receiving 25.6 Gbps of frequency-channelized data over four 10GbE ports. An Intel Xeon E5-2620v3 CPU manages the data transfer using Intel’s DPDK4 (Data Plane Development Kit) to achieve the required throughput. Each X-engine processing node also incorporates 64Gbytes of DDR4 RAM, representing a 20-second data buffer for the incoming spectral data. CHIME’s designers plan to upgrade the X-Engine’s SDRAM to at least 96Gbytes to expand the buffer’s capacity to 31 seconds.

For CHIME’s original BAO cosmology mission, each X-engine processing node crunches four 390kHz frequency channels using two dual-chip AMD FirePro S9300x2 Server GPUs. Each GPU has a memory bandwidth of 1Tbyte/sec. Each GPU in the X-engine operates independently and performs all spatial and in-band processing for one 390kHz frequency channel. The processed data, including the resampled stream for the FRB search, are exported to backend science engines over a pair of GbE links. Figure 2 shows the CHIME telescope’s processing architecture.

Figure 2. CHIME’s FPGA-based F-engines use FFTs to channelize the frequencies from the telescope’s antenna data. The GPU-based X-engine correlates the F-engine data. The X-engine output feeds several search engines including the FRB engine. (Image source: CHIME)

The CHIME design team selected GPUs rather than FPGAs for the correlation and averaging tasks because they’re easier to program, while admitting that FPGAs used for the same tasks would draw far less power. The entire CHIME telescope consumes 250 kilowatts, so power consumption is no small consideration.

Although not originally designed to search for FRBs, the output of CHIME’s X-engine could be adapted for such phenomena, so appropriate code was developed for the X-engine, and a new FRB search engine was developed. This FRB search engine continuously scans 1024 separate regions or “beams” in the sky looking for evidence of an FRB. CHIME samples each beam over nearly 16,400 frequency bands every millisecond.

The X-engine synthesizes these 1024 search beams by forming 256 beams along the telescope’s North-South direction using an FFT. The X-engine forms four beams in the East-West direction via exact phasing. Together, these X-engine computations divide the observed section of sky into a 2D array of 1024 discrete, closely spaced, static search beams.

From BAO to FRB

The original CHIME design, intended to delve into the details of BAOs, lacked the spectral resolution needed by the additional FRB search mission. Consequently, the X-engine’s correlation duties were extended to improve spectral resolution by exploiting the GPUs’ programmability. The upgraded X-engine gathers 128 successive, 2.56-microsecond voltage samples and applies an FFT to produce a spectrum. It then computes the magnitude of this spectrum, squares it, downsamples the newly generated spectrum by a factor of 8, averages three successive downsampled transforms, and sums the two orthogonal polarizations to produce a data stream with a cadence of 0.983 milliseconds and a spectral resolution of 24.4 kHz per band.

The X-engine’s output is shipped via a high-speed network to a specifically designed FRB search engine (see Figure 2), which is housed in its own 40-foot shipping container beneath the CHIME telescope. The FRB search engine consists of 132 compute nodes. Each of the 132 nodes contains two 10-core Intel Xeon E5-2630 v4 CPUs and 128Gbytes of RAM, for a total of 2640 Xeon CPU cores and 32Tbytes of RAM.

The FRB search engine devotes 128 of its compute nodes to RFI rejection, “dedispersion” processing, sifting, and grouping to find candidate FRBs, which are then passed to a second processing stage built with the remaining nodes. This second stage combs the information from all 1024 beams to determine if a burst has been detected. If so, this second stage computes the burst’s location, distance, and other characteristics.

Once the FRB search engine identifies an FRB event, it sends out an automatic alert. Within seconds, the CHIME team and the wider astrophysical community are alerted to the event, permitting a worldwide, rapid response by the astrophysics community. For example, it might be possible to correlate an FRB with visual phenomena, with X-ray or IR activity, or with a gravity wave using the LIGO and Virgo gravitational-wave observatories.

FRBs are a New, New Thing

Worldwide, only a few dozen FRBs have been observed since the first one was identified in 2001 from archival data obtained by the Parkes radio dish in Australia. Current best estimates suggest that FRBs may arrive here on Earth roughly a thousand times per day over the entire sky. Although not in the original design, the CHIME telescope’s large collecting area, wide bandwidth, and enormous field-of-view make it a superb FRB detector, which is probably why it started to detect FRBs even before it was fully operational. The only concern when the decision was made to upgrade CHIME’s capabilities for FRB detection was the radio telescope’s 400-800MHz bandwidth. Earlier FRBs detected by other radio telescopes had not been within this frequency range. CHIME’s FRB detections in 2018 demonstrated that this fear was unfounded.

Astrophysicists do not currently understand the sources of FRBs and, especially, repeating FRBs but it’s clear they require enormous amounts of energy to generate. They could be natural. They could be ET (or several ETs) phoning home or swiping right on some intergalactic Tinder app using a not-so-local area network (INSLAN). This new tool in the astrophysical toolbox, namely the CHIME radio telescope, may shed more light on this subject.