Current Campus Champions

Champion Leadership Team

Terminology at XSEDE

2020

2019

2018

2017

2016

2015

2014

XSEDE-allocated Stampede2 supercomputer simulates star seeding, heating effects of primordial black holes 

By Jorge Salazar, Texas Advanced Computing Center 

Supercomputer simulations have probed primordial black holes and their effects on the formation of the first stars in the universe. Black holes can help stars form by seeding structures to form around them through their immense gravity. They also hinder star formation by heating the gas that falls into them. XSEDE-allocated Stampede2 simulations show these effects basically cancel each other out. Shown here is an artist's concept that illustrates a hierarchical scheme for merging black holes. Credit: LIGO/Caltech/MIT/R. Hurt (IPAC).

Just milliseconds after the universe's Big Bang, chaos reigned. Atomic nuclei fused and broke apart in hot, frenzied motion. Incredibly strong pressure waves built up and squeezed matter so tightly together that black holes formed, which astrophysicists call primordial black holes. 

And that's a minus for star formation, as gas needs to cool down to be able to condense to high enough density that a nuclear reaction is triggered, setting the star ablaze. 

Matter fields at the moment of cloud collapse (i.e. onset of star formation) as projected distributions of dark matter (top) and gas (bottom) in four simulations targeted at the same region but with different abundances of primordial black holes, measured by the parameter f_PBH. Primordial black holes are plotted with black dots and the circles show the size of the structure that hosts the collapsing cloud. The data slice has a physical extent of 2000 light years and a thickness of 1000 light years. The age of the universe at the moment of collapse first decreases with f_PBH for f_PBH<0.001 when the "seeding" effect dominates. Then it increases from f_PBH=0.001 to f_PBH=0.01 and above as the "heating" effect becomes more important. Credit: Liu et al.

Volker Bromm, Department of Astronomy, UT Austin (left); Boyuan Liu, University of Cambridge (right).

"A key effect of primordial black holes is that they are seeds of structures," Liu said. His team built the model that implemented this process, as well as incorporating the heating from primordial black holes. 

TACC's Stampede2 supercomputer.

The simulations use data from the universe's initial conditions to high precision based on observations of the cosmic microwave background. Simulation boxes are then set up that follow the cosmic evolution timestep by timestep.  

Images derived from crystal structures give neural network running on XSEDE-allocated system clues needed to predict ability to create a given crystal in the real world

By Ken Chiacchia, Pittsburgh Supercomputing Center

To create new electronic and other tools, materials scientists need new types of crystals with specific electrical and physical properties. But while they have been able to predict whether a given new material has the properties needed, they've been limited in their ability to predict whether it's possible to create it in the real world. A team led from the University of Illinois Chicago has taken a completely new artificial intelligence (AI) tack, coding crystal structures as abstract 3D images that powerful neural network AI programs, honed for image recognition, can "understand" in a way far beyond human experts' abilities. Their AI, run on the XSEDE-allocated Bridges-2 at the Pittsburgh Supercomputing Center (PSC), showed high accuracy in predicting synthesizability in a group of test materials.

Overall framework of the synthesizability-likelihood-prediction AI. a) The researchers obtained hypothetical crystal structures never synthesized using CSPD algorithms alongside those that are synthesized or naturally formed from the Crystallographic Open Database (COD). b) They converted the crystal structures (top) and properties data into digitized, abstract 3D images (bottom). c) The AI analyzed the 3D images without human supervision, first learning and then successfully predicting crystal synthesizability. From Davariashtiyani, A., Kadkhodaie, Z. & Kadkhodaei, S. Predicting synthesizability of crystalline materials via deep learning. Commun Mater 2, 115 (2021), reproduced under Creative Commons.

Why It's Important

Figuring out how materials act the way they do – and predicting how new materials will act – may not exactly be glamorous. But materials science lies at the center of the miraculous developments of the Information Age. It also provides us with new structural materials, whether they're making a military plane harder to see on radar, giving a high-stress component of a reactor greater strength or allowing an electrical circuit to operate with essentially zero resistance.

Predicting what properties new materials, particularly crystals, will have is really important for developing tomorrow's smartphones, energy storage, aircraft and many other useful devices. But such predictions, even with advanced computing, have been limited. Even with the best supercomputers, predicting the properties of a crystal from first principles is impossible. Instead, scientists had to take theoretical shortcuts, which posed their own problems. Scientists could often predict how changing a given atom in a crystal would change its properties. But that limited them to small changes in already-known materials. Other predictions, using a type of AI called machine learning, could predict when a given new crystal had the right properties and whether it was stable enough to exist. But often creating that crystal in the real world was virtually impossible.

"The reason we want to know if we can or cannot make a certain crystal form of a material is the properties of materials highly depend on their structure. So if you target a certain property – let's say very high electric conductivity or superconductivity or a very hard material … [you] should be able to synthesize a certain crystal form [with that property]. It should be thermodynamically stable, but that's not enough. It should also be accessible through [today's] processing techniques." – Sara Kadkhodaei, University of Illinois Chicago

That's why Sara Kadkhodaei of the University of Illinois Chicago assigned her very first graduate student, Ali Davariashtiyani, to use a completely new approach to machine learning on the problem. Her hope was that Davariashtiyani could create a computing solution that drew on the strengths of AI to predict required properties and the ability to be created in the lab. To do this, they turned to the XSEDE-allocated Bridges-2 advanced research computer at PSC.

How XSEDE Helped

As a new junior faculty member, Kadkhodaei was not exactly rich in physical research resources. But she did have access to a kind of dream team. She herself was a materials scientist. Davariashtiyani had studied computer science as a master's-degree candidate. And Kadkhodaei's sister, Zahra Kadkhodaie of New York University (NYU), specialized in computational neuroscience – particularly in neural network modeling. A kind of machine-learning AI, neural networks have made huge leaps in image processing and recognition by copying theories of how nerve cells in the brain communicate and process information.

The teammates wondered whether they could reduce a crystal's chemical structure, properties and above all ability to be synthesized to a kind of abstract, 3D image that encoded all that information. No human could ever create or understand such an image directly, without computers translating it. But once such an image existed, a neural-network AI might be able to "understand" it, much as neural networks can now successfully pick out images of cats from the Internet.

"I just want to emphasize how critical XSEDE resources are to my research … I can't really do my [work]  if I didn't have access to XSEDE … it is really a critical tool for me and my students to just do our research." – Sara Kadkhodaei, University of Illinois Chicago

The only problem was that their AI would be data- and computation-hungry, particularly in the graphics processing units (GPUs) that have powered the AI revolution since 2012. That's where Bridges-2, which was designed to excel at tasks combining "Big Data" and AI, came in. By writing a relatively simple proposal for a startup allocation on Bridges-2, Davariashtiyani was able to get going, eventually generating enough results to write a full research allocation that helped give him the computing time he needed to complete the work.

As an added advantage, he was able to leverage XSEDE's educational offerings to learn how to use the machine. He used both online tutorials at xsede.org and virtual online classes to get up and running.

Davariashtiyani started with a known group of crystals whose properties were labeled as synthesizable, "training" the AI to recognize which were and weren't buildable in the real world. Then he tested the AI on another known group of crystals without labeling their properties, to see how accurately it predicted synthesizability. The results were encouraging. Without human intervention, the AI was able to predict whether or not those crystals could be synthesized in the lab. His AI had "Area Under the ROC Curve" values – AUC, a measure of how much a prediction improves upon random guessing – above 0.9, not far from the 1.0 perfect score, with accuracies (not too many false positives or missed "good" crystals) also above 90%. The scientists reported their results in the journal Communications Materials in November 2021.

Today, Kadkhodaei has access to more computing resources. Her team is using an expanded supercomputer toolbox, including Stampede2 at the Texas Advanced Computing Center  –  another resource available through the XSEDE network of NSF supercomputing centers  –  for the next phase of their work. They're also continuing to use Bridges-2. The work will include expanding into predictions of how pressure can influence the synthesizability of crystals, as well as developing AI models that can discover crystalline materials with extreme hardness.

This research is based upon work supported by the National Science Foundation (NSF) under Award Numbers DMR-1954621 and DMR-2119308. The work also used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548562. Specifically, it used the Bridges system, which was supported by NSF award number NSF 14-45606 and Bridges-2, award number is: NSF 19-28147, at the Pittsburgh Supercomputing Center (PSC). It also used  resources at the Electronic Visualization Laboratory (EVL) at UIC available through the NSF Award CNS-1828265.

At a Glance

• To create new electronic and other tools, materials scientists need new types of crystals with specific electrical and physical properties. 

• While they have been able to predict whether a given new material has the properties needed, they've been limited in their ability to predict whether it's possible to create it in the real world.

• Scientists used a completely new artificial intelligence (AI) tack on XSEDE-allocated systems to code crystal structures as abstract 3D images.

• Their AI showed high accuracy in predicting the synthesizability in a group of test materials.

Domain Champions

New Findings Illustrate Possible Rationale for Some High-Elevation Diseases in Humans

By Henry Lemersal, SDSC REHS Program and Kimberly Mann Bruch, SDSC Communications

Photo by Alexas Fotos

UC Berkeley scientists Michael Nachman and Elizabeth Beckman have studied environmental adaptation of house mice for many years. Their latest project, which utilized XSEDE allocations on San Diego Supercomputer Center (SDSC) resources, looked at how house mice colonize and adapt to high elevations in Ecuador and Bolivia. They discovered that several hypoxia-associated genes were different in the mice at the higher elevations and that some of these genes exhibited a threshold effect – a large shift in how frequently certain forms occurred at only the highest elevations. Their findings were recently published in Genetics.

"This study is a continuation of earlier work on the genetic basis of environmental adaptation in house mice, and in a paper that we published last year we identified genes underlying adaptation to cold weather," said Nachman, a professor of integrative biology at UC Berkeley and director of the campus's Museum of Vertebrate Zoology. "In our earlier study we found that there was some predictability to evolution, both at the organismal level and at the genetic level.  However, in the present study, we found mostly distinct evolutionary responses to high elevation in mice from Ecuador and mice from Bolivia, showing that adaptation to high elevations can occur through changes in many different genes and pathways."

For their latest study, titled The Genomic Basis of High-Elevation Adaptation in Wild House Mice (Mus musculus domesticus) from South America, the researchers used XSEDE allocations on the Comet supercomputer at SDSC. Specifically, they utilized the processing software FastP to clean raw sequence reads and contrasted them with the house mice reference genome (GRCm38) using BWA (v 0.7.13), a tool for genetic similarity comparison. 

Why It's Important

"We were able to complete our large-scale bioinformatic analyses in a timely and robust way thanks to our XSEDE allocation and SDSC resources." – Elizabeth Beckman, postdoctoral researcher at UC Berkeley

The team's research has plenty of prospective applications for humans. An example is how the study could provide more clarity in the functionality of human traits.

"Humans have a lot of complex traits," noted Beckman, a postdoctoral researcher at Berkeley. "Every time we dive into the details of complex traits, including in a model system like house mice, it helps us understand more about how complex traits work, and it offers clues about the details of complex traits in humans."

She further explained that human disease associated with elevation increases around 2500 meters and occurrence increases the higher you go. Based on these similar patterns in humans and mice, a similar process may be occuring to both species as they get near 3000 meters. Studying this pattern in house mice could help the researchers better understand human diseases associated with high elevation.

How XSEDE Helped

The scientists used XSEDE allocations on Comet to index collected data in SAMtools and then used the Genome Analysis Toolkit (v 4.0.11.0) to successfully clean up and work with the aggregated information. 

"We were able to complete our large-scale bioinformatic analyses in a timely and robust way thanks to our XSEDE allocation and SDSC resources," Beckman said. "Now, we are using Expanse to continue our study of environmental adaptation across North and South America in house mice, as well as high-elevation adaptation in South American finches."

This research was supported by an NIH grant to Nachman (R01 GM127468). Computations on SDSC resources were allocated by XSEDE (TG-MCB130109).

At A Glance:

• UC Berkeley researchers used XSEDE allocations to examine at how house mice colonize and adapt to high elevations in Ecuador and Bolivia.
• They used a package called FastP to clean raw sequence reads and then aligned them to the M. musculus reference genome (GRCm38) with BWA (v 0.7.13) using the BWA-MEM algorithm. 
• The study was recently published in Genetics.

Student Champions

Project leader attributes XSEDE success, in part, to community building

By Dina Meek, National Center for Supercomputing Applications

Towns speaks at the recent PEARC22 conference in Boston.

As the final keynote speaker at this year's ACM Practice & Experience in Advanced Research Computing (PEARC) conference, July 10-14 in Boston, John Towns, XSEDE leader since 2011, spoke on his experience guiding the National Science Foundation project with a particular focus on how community building was critical to the project's success. Introduced by XSEDE's Campus Engagement Co-manager, Dana Brunson, his address "XSEDE and Beyond or How did we get here and where are we going (as a community)?" took the audience not only through the highlights of the project but also the insights he's gained along the way.

Starting on the national cyberinfrastructure work as the TeraGrid Forum chair, Towns was named principal investigator and project director for XSEDE in 2011. He recalled deep divides at the start of the project due to what he characterized as a difficult, disruptive awards process. "As a virtual organization of people who often compete for solicitations, they needed to understand how to compartmentalize their activities and align with the larger goals of the project," he said. From his perspective, it took XSEDE a bit more than three years to hit its stride. By then, it was time to analyze lessons learned as XSEDE2, which ran from 2016 to 2022, was being planned and then ramped up. 

The community did, indeed, benefit from coming together and today researchers using XSEDE-allocated resources number around 11,500 and domains have grown in diversity to include archaeology, finance, genomics and machine learning. The diversity of the partner institutions in XSEDE, Towns said, brought strength to the partnerships and also leveraged various strengths of different institutions – including smaller institutions. Trust, he said, was the most important factor contributing to XSEDE's success.

"What's most important to success is less about proximity to machines and more about proximity to humans." – John Towns, PI/Project Director, XSEDE 

As the 11-year project winds down, and the community looks ahead to Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS), NSF's follow-on project, Towns continues to extoll the virtues of building relationships and learning from each other. He said the community needs greater coherence and to celebrate its success the way other professions do to the convey the importance of what they do. To expand the community beyond those funded by NSF grants. To better prepare technical people for management roles and to create career paths beyond the technical. Ultimately, he said, "what's most important to success is less about proximity to machines and more about proximity to humans."

Read HPCWire's full account here.

 

At A Glance

• In the final keynote address at PEARC22, XSEDE PI Towns outlined XSEDE successes
• Towns contributes XSEDE successes to relationship-building
• PI/Project Director considers how community can improve going forward

XSEDE-allocated Stampede2 supercomputer helps find new properties of high-entropy alloys

Jorge Salazar, Texas Advanced Computing Center

Discovery of new high-entropy alloys. Shown is a data-driven workflow to map the elastic properties of the high-entropy alloy space. Credit: Chen et al.

When is something more than just the sum of its parts? Alloys show such synergy. Steel, for instance, revolutionized industry by taking iron, adding a little carbon and making an alloy much stronger than either of its components. 

Supercomputer simulations are helping scientists discover new types of alloys, called high-entropy alloys. Researchers have used the Stampede2 supercomputer of the Texas Advanced Computing Center (TACC) allocated by the Extreme Science and Engineering Discovery Environment (XSEDE). 

Why It's Important

Their research was published in April 2022 in Npj Computational Materials. The approach could be applied to finding new materials for batteries, catalysts and more without the need for expensive metals such as platinum or cobalt.

"High-entropy alloys represent a totally different design concept. In this case we try to mix multiple principal elements together," said study senior author Wei Chen, associate professor of material science and engineering at the Illinois Institute of Technology.

The term "high entropy" in a nutshell refers to the decrease in energy gained from random mixing of multiple elements at similar atomic fractions, which can stabilize new and novel materials resulting from the ‘cocktail.'

For the study, Chen and colleagues surveyed a large space of 14 elements and the combinations that yielded high-entropy alloys. They performed high-throughput quantum mechanical calculations, which found the alloy's stability and elastic properties, the ability to regain their size and shape from stress, of more than 7,000 high-entropy alloys.

Graph representations of association rules between elements and elastic properties of high entropy alloys. Results for (a) bulk modulus, (b) Young's modulus, (c) shear modulus, (d) Pugh's ratio, (e) Poisson' ratio and (f) Zener ratio respectively. Node colors and sizes represent different elements and fractions. Credit: Chen et al.

"This is to our knowledge the largest database of the elastic properties of high-entropy alloys," Chen added.

They then took this large dataset and applied a Deep Sets architecture, which is an advanced deep learning architecture that generates predictive models for the properties of new high-entropy alloys.

"We developed a new machine-learning model and predicted the properties for more than 370,000 high-entropy alloy compositions," Chen said.

The last part of their study utilized what's called association rule mining, a rule-based machine-learning method used to discover new and interesting relationships between variables, in this case how individual or combinations of elements will affect the properties of high-entropy alloys. 

"We derived some design rules for high-entropy alloy development. And we proposed several compositions that experimentalists can try to synthesize and make," Chen added.

How XSEDE Helped

High-entropy alloys are a new frontier for materials scientists. As such, there are very few experimental results. This lack of data has thus limited scientists' capacity to design new ones.

"That's why we perform the high-throughput calculations, in order to survey a very large number of high-entropy alloy spaces and understand their stability and elastic properties," Chen said. 

He referred to more than 160,000 first-principle calculations in this latest work.

"The sheer number of calculations are basically not possible to perform on individual computer clusters or personal computers," Chen said. "That's why we need access to high-performance computing facilities, like those at TACC allocated by XSEDE."

Wei Chen, Associate Professor of Material Science and Engineering, Illinois Institute of Technology.

Chen was awarded time on the Stampede2 supercomputer at TACC through XSEDE, a virtual collaboration funded by the National Science Foundation (NSF) that facilitates free, customized access to advanced digital resources, consulting, training and mentorship.

Unfortunately, the EMTO-CPA code Chen used for the quantum mechanical density function theory calculations did not lend itself well to the parallel nature of high-performance computing, which typically takes large calculations and divides them into smaller ones that run simultaneously.

"Stampede2 and TACC through XSEDE provided us a very useful code called Launcher, which helped us pack individual small jobs into one or two large jobs, so that we can take full advantage of Stampede2's high performance computing nodes," Chen said.

The Launcher script developed at TACC allowed Chen to pack about 60 small jobs into one and then run them simultaneously on a high-performance node. That increased their computational efficiency and speed.

"Obviously this is a unique use application for supercomputers, but it's also quite common for many material modeling problems," Chen said.

"Hopefully more researchers will utilize computational tools to help them narrow down the materials that they want to synthesize." – Wei Chen, Illinois Institute of Technology

For this work, Chen and colleagues applied a computer network architecture called Deep Sets to model properties of high-entropy alloys.

The Deep Sets architecture can use the elemental properties of individual high-entropy alloys and build predictive models to predict the properties of a new alloy system.

The Stampede2 supercomputer at the Texas Advanced Computing Center is an allocated resource of the Extreme Science and Engineering Discovery Environment funded by the National Science Foundation.

"Because this framework is so efficient, most of the training was done on our student's personal computer," Chen said. "But we did use TACC Stampede2 to make predictions using the model."

Chen gave the example of the widely studied Cantor alloy – a roughly equal mixture of iron, manganese, cobalt, chromium and nickel. What's interesting about it is that it resists being brittle at very low temperatures.

One reason for this is what Chen called the ‘cocktail effect,' which produces surprising behaviors compared to the constituent elements when they're mixed together at roughly equal fractions as a high-entropy alloy.

The other reason is that when multiple elements are mixed, an almost unlimited design space is opened for finding new compositional structures even a completely new material for applications that weren'tt possible before.

"Hopefully more researchers will utilize computational tools to help them narrow down the materials that they want to synthesize, Chen said. "High-entropy alloys can be made from easily sourced elements and, hopefully, we can replace the precious metals or elements such as platinum or cobalt that have supply chain issues. These are actually strategic and sustainable materials for the future."

The study, "Composition design of high-entropy alloys with deep sets learning," was published April 2022 in Npj Computational Materials. The study authors are Jie Zhang, George Kim and Wei Chen of the Illinois Institute of Technology; Chen Cai and Yusu Wang of the University of California San Diego.

This research was funded by the National Science Foundation OAC-1940114, OAC-2039794 and DMR-1945380. XSEDE ACI-1548562. NERSC Contract No. DE-AC02-05CH11231.

At A Glance

• Supercomputer simulations are helping scientists discover new high-entropy alloys.

• XSEDE allocations on TACC's Stampede2 supercomputer supported density function theory calculations for largest database yet of high-entropy alloy properties.

• Deep Sets architecture generated predictive models on Stampde2 for the properties of new high-entropy alloys.

• Study of high-entropy alloys represents an effort of materials scientists to develop new materials for a more sustainable future.