Scientists since Albert Einstein have believed that when major galactic events like supernova explosions or black hole mergers occur in the universe, they leave a trace. That trace, it is believed, takes the form of gravitational waves, ripples in the curvature of space-time that propagate as a wave, travelling outward from the source.

For over a decade, scientists and engineers have engaged in one of the most ambitious research efforts ever undertaken: to design, build and operate the Laser Interferometer Gravitational-wave Observatory (LIGO) to identify signs of gravitational waves.

From 2002 to 2010, hundreds of scientists worked together to bring the experiment to life. It required two gravitational wave observatories, 1,865 miles apart, working in unison: the LIGO Livingston Observatory in Livingston, Louisiana, and the LIGO Hanford Observatory, located near Richland, Washington. LIGO was developed and is managed by Caltech and MIT, and is funded by the National Science Foundation (NSF).

“Gravitational waves offer an unprecedented opportunity to see the universe from a new perspective, providing access to astrophysical insights that are available in no other way,” said Stuart Anderson, a research manager for LIGO based at Caltech.

To date, no waves have been detected. Yet most astronomers believe they’re still out there, waiting to be discovered if only we use the right tools. For that reason, the NSF, along with the UK Science and Technology Facilities Council, the German Max Planck Society, and the Australian Research Council, have supported the creation of a follow-up to LIGO called Advanced LIGO (ALIGO), which replaces the original detectors with ones more than 10 times more sensitive. ALIGO is expected to go online in 2015.

“Advanced LIGO’s goal is to pioneer a new field of gravitational-wave astronomy, and its planned sensitivity will allow this very significant step beyond first detections,” Anderson said. “The massive, relativistic constituents, causing violent perturbations of space-time itself, will be revealed through this unique probe.”

In addition to all of the astronomical considerations that must be attended to in a project like this — the placement of the observatories, the sensitivity of the detectors, the modeling of the expected signals — Advanced LIGO is also a “Big Data” problem.

The observatories take in huge volumes of data that then must be analyzed to determine their meaning. In the case of a positive identification of a gravitational wave candidate, this process must happen quickly, so the location of the waves can be identified and used for further study.

Among the many goals of the project was developing an offline system for binary neutron star and black hole detection, Anderson said. “We wanted to dig deeply into signals, since we have a good idea of what those signals are.”

Since gravitational waves are still in their theoretical infancy, scientists require additional information to learn about their existence. Combining observations of electromagnetic radiation, neutrinos or gravitational waves — a process known as multi-messenger astronomy — provides complementary information, but it makes a messy data problem even messier.

When they were planning Advanced LIGO, the researchers realized they needed more computing and data processing power than they had used previously. But how much more did they need? And where would it come from?

Discussing the matter with program officers at the NSF, they decided to see if the Extreme Science and Engineering Discovery Environment (XSEDE) could help. XSEDE, another NSF-supported project, oversees the nation’s network of supercomputers and digital resources. These resources allow scientists to solve otherwise impossible computing problems and pursue really big science.

Perhaps it could do the same for Advanced LIGO.

Scientists from Caltech, Syracuse University and Abilene Christian University worked with computing experts at the Texas Advanced Computing Center (TACC) to develop and test new methods for transmitting and analyzing data with an eye towards Advanced LIGO’s future operations.

To achieve their goal, the team overlaid LIGO’s computing environment on top of the Stampede supercomputer at TACC, one of the most powerful in the world. They then used a tool called Condor to break their large computing problems into smaller parts that could be distributed and solved on individual nodes on the supercomputer. (A node consists of a number of processors connected together and to the larger system.) Miron Livney’s High Throughput Condor group at the University of Wisconsin-Madison provided critical support (including software updates) to allow this computational challenge to work, and the Advanced LIGO team was given priority access to the supercomputing queue, so their jobs could run efficiently with few interruptions.

At first, they used a data analysis code from the initial LIGO for the experiment, since it was easy to validate. However over time, they began using the new Advanced LIGO version of the analysis code. Through detailed benchmarking, they were able to verify the reliability of the process and provided the team with a preliminary sense of how much computing and data-crunching capability they needed to transmit, store and analyze data from the observatories for use by the astronomy community.

“The XSEDE and TACC teams where extremely helpful in collaborating on a successful port of a large scale LIGO data analysis pipeline to the Stampede system — all the way from network transfers of instrument data, through large scale number crunching, to generating and presenting web accessible scientific results,” Anderson said.

“The solution we implemented allowed individual LIGO data analysts to use their existing computer identities and software tools to quickly and efficiently leverage the Stampede co
mputing resources. I am very excited by the possibility of using XSEDE resources to maximize the scientific reach of the Advanced LIGO experiment.”

Not only were the researchers able to prove they could compute on a big national supercomputer, in the process they ended up making the Advanced LIGO software four times more efficient, ultimately saving money and time.

According to Peter Couvares, a computer scientist from Syracuse University who worked on the project, the experience showed LIGO researchers that large high-performance computing, or HPC, systems like Stampede can work as well as the high-throughput computing, or HTC, systems they were familiar with, which use a number of clusters distributed across LIGO Scientific Collaboration institutions.

“Given that our foremost computing challenge has always been to enable scientists to automatically and reliably execute large complex workflows consisting of millions of jobs and big data, NSF-supported tools like Condor and Pegasus have been invaluable,” Couvares said. “We’re increasingly realizing the value of HPC expertise at places like TACC to optimize our codes to make efficient use of shared XSEDE resources like Stampede.”

Borne out in tests run over the last few months, the results proved that using the nation’s supercomputers for gravitational wave analyzes is feasible. In fact, in the end, the team was able to run more than 10,000 analysis jobs simultaneously on Stampede.

They presented their results at the HTCondor Week 2014 workshop in April.

“TACC staff not only got the LIGO team up and running on Stampede, they helped make the LIGO code substantially more efficient,” said TACC Executive Director Dan Stanzione. “Over the course of a year of computing with advanced LIGO, that could equate to millions of dollars in savings, or a huge increase in the amount of processing we can do on the same budget. This shows the value of software engineering being interjected into the process.”

Contact:
Faith Singer-Villalobos
TACC Communications Manager
The University of Texas at Austin
+1 512.232.5771, cell: +1 512.663.7237
faith@tacc.utexas.edu