Preaching the HPC Gospel

XSEDE's Campus Champions Provide Vital Link between Researchers, Supercomputing Resources

To get the help you need, sometimes you have to break something first.

Dirk Colbry admits, a bit sheepishly, that he made his debut in Michigan State University's high performance computing (HPC) center the hard way. As a graduate student investigating 3D face recognition for possible security applications, he'd been stringing together every computer he could get his hands on to get results. But the computational burden was overwhelming his Rube Goldberg cluster.

"The preliminary algorithm I developed took about ten seconds" to analyze a single face, he says. "We thought that would be acceptable, say, in an airport security screening. But, I needed a test that could compare 5,000 scans with each other."

Not being willing to wait years for about 25 million comparisons (5,000 times 5,000)—let alone for his PhD—he took a suggestion from a colleague that he try the university's cluster.

 "I promptly submitted the biggest job they had ever seen, and shut the thing down," Colbry says. "Andy Keen, the guy who developed the cluster slapped my hand and said, ‘Don't do that again.' But he also said, ‘I want you to keep testing the system.'"

When he got his degree, Colbry didn't immediately go into HPC as a profession. But the seeds had been planted.

Colbry's experience isn't that unusual among HPC researchers and engineers, according to Kay Hunt of Purdue University, project coordinator for XSEDE's Campus Champions Program. A tip from a colleague, an experienced guiding hand when a project hits a wall, even simple encouragement are often the key ingredients in turning a struggling researcher into an enthusiastic consumer of HPC resources. XSEDE initiated the Campus Champions Program in 2008, she explains, to institutionalize some of the informal networking, consulting and mentoring that individual faculty had been contributing to the HPC brew.

"We identify a local person at a campus or institution who's going to do three things as a campus champion," Hunt says. "First, letting their colleagues on campus know that XSEDE exists, that it offers resources for them at no cost and how to get access to them; second, serving as the point person to help them get started; and third, giving feedback to the XSEDE staff on what could be improved, what documentation needs to be created or updated and what processes can be streamlined or made more efficient and make the whole XSEDE experience better."

Since the initiation of the program, it has expanded to include student champions, who help campus champions reach students on campus; regional champions, who target a multi-campus audience; and domain champions, who concentrate on spreading the XSEDE gospel to members of a specific scientific discipline. But the campus champions remain the anchor of the program, preaching the Word, making connections, referring to training courses and materials and sometimes just holding hands to help newcomers master the incredibly productive technologies represented by HPC.

The use of religious metaphor isn't a conceit; you run into it frequently when you talk with XSEDE's campus champions.

Dana Brunson, a campus champion at Oklahoma State University, in fact calls herself "the cyberevangelist for the campus." Her job at OK State focuses on user support at the university's HPC center. Hers was a quick conversion, she explains.

"I started my position in the Fall of 2007 and, very quickly, Jeff Pummill at the University of Arkansas introduced me to XSEDE," she says. At her first HPC conference—SC08—Pummill, himself a campus champion, introduced her to Kay Hunt, who immediately recruited her to the program.

"The program helps me offer local people any kind of resource they need," she says. XSEDE gives her the ability to connect people with a nationwide array of supercomputers, networking resources, archival systems and other specialized machines that no single university could ever match. "I need never say, ‘Oh, I'm sorry; you can't do that research because we don't have those resources.'"

Pummill's foray into large-scale scientific computing began in 2005, when the University of Arkansas acquired its first Top500 cluster.

"They quickly realized that a part-time graduate student was not sufficient to maintain and implement a system of that size," he jokes. "As a result, I spent the first few years as both systems administrator and user support."

Pummill was an early entry to the campus champions program, and was both elated and sometimes astounded at the program's growth.

"In the early days, you could fit everybody in a small room," he says. As the program has grown to more than 200 campus champions, he says, the informal nature of those original meetings had to evolve—though the passion in the group remains.

"You find that people are siloed," Pummill says, "developing moderately good solutions on their campus. It's good enough to get them by, but highly inefficient as everyone is duplicating the effort."

A key advantage to the campus champions' extolling XSEDE's virtues is that it breaks down those silos by encouraging networking and communication between members.

"This creates more elegant solutions via the champions' network of expertise," he says. "In turn, online media maintain this information for future reference and dissemination by others. It forestalls a lot of wheel-reinvention."

Dirk Colbry couldn't agree more.

He'd kept a hand in HPC through his research, but only considered making it his career in 2009, when his wife got a position back at Michigan State. The very HPC center he'd crashed as a graduate student was looking for a consultant to help researchers access the center; it proved a perfect fit. When Michigan State campus champion Brock Palen interviewed Kay Hunt for his podcast, Colbry's ears perked up: He saw another way he could help his researchers, and so he joined up.

"One of the best ways to enumerate the value of the program is to watch our email list," Colbry says. "If you put in a ticket for a complex HPC issue to any good help desk you'll get a response that says, ‘We're working on it,' and after a couple of days of back and forth you'll get the answer you're looking for. I'll put the same question on the campus champions list and I'll get seven or eight answers within five minutes. There's no place else to find that type of information so easily."