NSF Cyberinfrastruct ure User Survey 2005 Results

Introduction

To ensure the effective support of its broad user community, SDSC and NCSA have conducted annual user surveys for many years. In August and September 2005, the annual survey was conducted for the first time as a joint activity of the Cyberinfrastructure Partnership (SDSC and NCSA) and the TeraGrid. This year's survey not only showed the continued success of the NSF-supported cyberinfrastructure environment, but also demonstrated the user community's general satisfaction with the resources, software and services from all NSF cyberinfrastructure sites.

The 2005 NSF Cyberinfrastructure User Survey was prepared and administered by David Hart of SDSC and Dave McWilliams of NCSA, with help from Nancy Wilkins-Diehr (SDSC), John Towns (NCSA), Sergiu Sanielevici (PSC), Charlie Catlett (Argonne National Lab) and many staff members within the CIP and TeraGrid. The survey was open for four weeks during August and September 2005. The 39 questions cover the areas of computational environment, grid environment and services, programming environment, software and tools, data and I/O, visualization, allocations, consulting and help desk support, training and documentation, and some optional demographic information.

We advertised the survey by sending e-mail messages to active users at NCSA, SDSC, PSC and TeraGrid. The potential respondent pool included approximately 2,600 e-mail addresses from SDSC, 1,900 from TeraGrid, 2,000 from PSC, and 2,200 from NCSA; there is some overlap among these lists. A total of 412 complete and partial responses were recorded. Based on a high estimate of 7,000 unique addresses in the respondent pool, we achieved a response rate of better than 5%. CIP and TeraGrid sites will be able to make decisions about resources, services, and software based on this survey.

A detailed report with complete questions as well as the survey data is below. The various components of the survey are reviewed and the responses summarized. Specific questions from the survey are shown with a gray background. Quantitative responses are presented as tables. For questions that permitted lengthy text answers, the individual responses are shown in bulleted lists and are provided without editing other than correcting obvious spelling and grammar errors.

	Contact Information
	NSF division or directorate
	Scientific Successes
Computational Environment	NSF-supported systems: satisfaction level
	Mass storage systems: satisfaction level
	NSF-supported sites: satisfaction level
	Suggestions for improving the NSF cyberinfrastructure resource environment
Grid Environment and Services	Grid tools which are used
	Experience or plan to use grid tools and services
	Web-based portal: features desired
	Interest in pre-production and trial grid services
	Suggestions for improving grid environment
Programming Environment, Software and Tools	Nature of parallelism in your code
	How the code is parallelized.
	Mathematical software/libraries used.
	Parallel/performance tools used
	Third-Party scientific software used
	Suggestions for improving the software offerings.
Data and I/O	Data software, libraries or services used
	Interest in persistent online storage area
	Ideal access method for accessing persistent storage area
	Use or planned use of public data collections
	Suggestions for improving data storage and I/O environments
Visualization	NSF visualization systems: satisfaction level
	Scientific visualization applications used
	Suggestions for improving NSF visualization software offerings
Allocations	Allocations process: satisfaction level
Allocations	Suggestion for improving allocations process
Consulting and Help Desk Support	Phone consulting services: satisfaction level
	Email consulting services: satisfaction level
	Preferred method to contact help desk/consultants
	Suggestions for improving direct user support
Training and Documentation	Training activities in which you participated
	Documentation: satisfaction level
	Suggestions for improving training and documentation
Demographic Details	Your institution affiliation
	Your position
	Gender
	Race

1. Your contact information (optional, but required for iPod drawing):

Remarks/Plans: Almost all respondents included their contact information. CIP and TeraGrid sites will use this information to follow up with users.

2. Please select the NSF division or directorate (or corresponding area of science) that most closely represents your research:

Remarks/Plans: Nearly 40% of the respondents were in the top three areas: Chemistry, Molecular and Cellular Biology and Physics. (The table is sorted by number of responses in descending order.)

	Number of Responses	Response Ratio
Chemistry	57	17%
Molecular and Cellular Biology	37	11%
Physics	33	11%
Astronomical Sciences	20	10%
Other	18	6%
Computer and Network Systems	18	5%
Materials Research	17	5%
Atmospheric Sciences	17	5%
Chemical and Transport Systems	12	5%
Civil and Mechanical Systems	9	4%
Earth Sciences	7	3%
Bioengineering and Environmental Systems	6	2%
Computer and Communications Fundamentals	6	2%
Ocean Sciences	5	2%
Mathematical Sciences	3	1%
Information and Intelligent Systems	3	1%
Social, Behavioral and Economic Sciences	1	0%
Education and Human Resources	1	0%
Electrical and Communications Systems	1	0%
Environmental Biology	1	0%
Polar Sciences	0	0%
TOTAL	335

3. Please describe any scientific successes from your computational research as enabled by NSF cyberinfrastructure resources. Please provide URLs for further details, if available.

Remarks/Plans: Nearly 100 URLs are listed below. See the Survey Summary for a list of representative successes.

Publication in Biophysical Journal: In press
http://www3.interscience.wiley.com/cgi-bin/abstract/109745869/ABSTRACT Submission of "How Can a ?-sheet Peptide be Both a Potent Antimicrobial and Harmfully Toxic? Molecular Dynamics Simulations http://www.ks.uiuc.edu/Research/hemolysin/
of Protegrin-1 in Micelles. " to Journal of Peptide Science. Preparation of third paper " Introduction of leucine into ?-hairpin peptide causes antimicrobial activity: insights into the mechanism of action from molecular dynamics simulations."
Inverse modeling of Asian black carbon (published) and sensitivity analysis of ozone non-attainment in US (forthcoming).
TeraShake Visualization - Shown at Supercomputing conf and TV news. http://visservices.npaci.edu/projects/scec/terashake/ Puente Hills Visualizations shown on TV news and over the web http://visservices.npaci.edu/projects/scec/puente_hills
Using the TeraGrid and SRB systems at the San Diego Supercomputer Center (SDSC) and the TeraGrid system at National Center for Supercomputing Applications (NCSA), it has been a great success for my research results on the search for high energy neutrinos by analyzing data of the AMANDA experiment. More details about my research can be found at: http://www.ps.uci.edu/~silvestri
Able to elucidate the structure of hydrated, cationized amino acids in the gas-phase. Supercomputer usage aided in the publication of 6 scientific papers.
It enables our entire research program. http://maat.med.cornell.edu
The TeraGrid has allowed JHU Center for Imaging Science (CIS) to explore the Large Deformation Diffeomorphic Metric Mapping (LDDMM) algorithm at a very large scale. We are still analyzing our results and plan to duplicate our results with a larger population. Check: http://www.teragrid.org/news/news05/birn.html
I am just beginning to use the NSF cyberinfrastructure resources and so I have not reached a point that I could say I achieve some scientific successes
My advisor, Romeel Dave, and I have been able to run cosmological simulations in order to study the evolution of star formation and the heavy element enrichment of galaxies and the intergalactic medium.
I study solar magnetism through the use of large-scale simulations of the solar convection zone. These simulations require the latest generation of massively parallel supercomputers. The DataStar machine at San Diego Supercomputer Center and the Lemieux machine at Pittsburgh Supercomputing Center are vital resources in our simulation efforts.
Simulations of solar magneto-convection on granular and supergranular scales have helped clarify the nature of faculae, the excitation of oscillations, the validation of local helioseismic inversion techniques, the driving of convection, the emergence of magnetic fields in the quiet Sun. Results are available at http://www.pa.msu.edu/~steinr/talks http://www.pa.msu.edu/~steinr/papers
Recently, we have developed a new code form magnetohydrodynamics in full general relativity. See http://xxx.lanl.gov/abs/astro-ph/0503420 and http://xxx.lanl.gov/abs/astro-ph/0503421
I have two manuscripts currently in preparation based on work my students performed at the NCSA. This work has also allowed me to secure an ACS-PRF grant.
Development of C5 Landscape Database (not yet released. Version 2.0 is supported by SDSC academic associates program.). http://www.c5corp.com/research/demtool/index.shtml
A lengthy computation resolved an open question about the largest possible single deletion correction code on 10 bits. See http://www.research.att.com/~njas/doc/graphics.html and look under 1dc.1024.
We have managed to perform a large number of high dynamical range simulations of the formation of the first generation of stars in the universe in both a standard cosmology and using warm dark matter particles. In addition, we have followed the evolution of the HII regions (ionized gas) from these stars, and also their supernovae. This work is groundbreaking and will be published in upcoming issues of The Astrophysical Journal.
I have used the DataStar and BlueGene systems to run a seismological wave propagation code for investigation of earthquake sources and earth structure.
Enabled the development of neural network based material models incorporated within finite element analysis
Quantified terrestrial atmospheric, biological and other effects due to Galactic Gamma-Ray Bursts: http://www.nasa.gov/vision/universe/starsgalaxies/gammaray_extinction.html http://www.nature.com/news/2005/050531/full/050531-1.html http://www.agu.org/pubs/crossref/2005/2005GL023073.shtml http://arxiv.org/abs/astro-ph/0505472 http://www.journals.uchicago.edu/ApJ/journal/issues/ApJL/v622n2/19056/brief/19056.abstract.html?erFrom=-4419225545088408487Guest
highest resolution ever computed for solar turbulent convection in spherical shells allowing to explain solar differential rotation and meridional circulation
Class project was successfully completed
http://www.physics.ucdavis.edu/Text/Fong.html
Implemented and tested a very efficient diskless checkpointing scheme for scientific applications running on large parallel systems.
Currently working on support tools (Projections - http://charm.cs.uiuc.edu/research/parallel_perf/ ) that aid the scaling performance of NAMD (http://www.ks.uiuc.edu/Research/namd/ ) on large numbers of machines.
We have developed SYNSEIS application with integrating TeraGrid for the scientific simulation. https://portal.geongrid.org:8443/gridsphere/gridsphere
As a new NCSA user, I have yet to complete any scientific successes. However, the NCSA systems are being used to develop distributed data mining systems, which hope to be published about by the end of the year.
Paper making use of SDSC computing facilities for cosmological N-body simulations using adaptive mesh refinement published in Monthly Notices of the RAS, May 2005. http://jilawww.colorado.edu/~rimes/research
We have solved a time-dependent geophysical fluid instability problem by computing a complete set of time-dependent normal modes to a time-periodic flow. The modal structures have direct physical interpretations which illuminate the physics of time-dependent instabilities. These results are described in the manuscript "Normal-mode analysis of a baroclinic wave-mean oscillation" under review with the Journal of the Atmospheric Sciences. Work is ongoing to relate these results to the geophysical forecasting problem.
Engineering productivity for simulation of fluid dynamics in heavy duty diesel engines, including cooling internal combustion & emissions.
http://eol.sdsc.edu/ analysis of SNPs in human genome
HPSS is a fundamental part of the operation of the UCSD network telescope: http://www.caida.org/data/passive/network_telescope.xml
Publications in preparation.
Developed data used in the generation of Equations of State for high pressure and temperature geomaterials.
We have utilized supercomputing resources in uncovering mechanisms for various organic chemical reactions and in understanding various unusual molecular structures. For further details: http://www.chem.ucdavis.edu/groups/tantillo/djt.html
Most of my computations are being performed using the NSF cyberinfrastructure. We have used Molecular Dynamics calculations to understand protein-DNA interactions and obtained very interesting results which will be published soon. I also have used these facilities to understand various structural and energetic properties of DNA and RNA, the results of which have been published/submitted for publication.
We can simulate turbulent mixing at the highest Reynolds number up to date which helps resolve long standing problems in turbulence research. Also we can follow Lagrangian particles in a turbulent flow at high Reynolds number.
Our research deals with the multiscale modeling of biological nanocomposites to quantitatively understand the specific features and interactions responsible for mechanical properties. The idea to develop tools for simulations based materials design of biomimetic nanocomposites. Our group discovered the presence of interlocks in biological nanocomposite nacre (the inner layer of sea shells). The computational resources at NCSA were used to prove that these interlocks are the key to high toughness and strength in nacre. You can find more details at the following web site: http://www.ndsu.nodak.edu/instruct/dkatti/research/nacre/index_4.html
Nine eukaryote proteomes have been aligned to nine new organism genomes, the Daphnia pulex genome and 8 Drosophila species, with help from TeraGrid and Generic Model Organism Database projects. The eukaryote proteomes with over 200,000 protein sequences are drawn from organism genome databases, Ensembl and NCBI. Alignment is done using a Grid-aware version NCBI tBLASTn and run on the TeraGrid. This study is part of an assessment of the TeraGRid as a shared computational resource for genome database projects under the GMOD umbrella. URLs: http://wfleabase.org/maps/dpgenomesample/ , http://insects.eugenes.org/species/
We found (to our surprise) that linear alkynes do not have the same stabilization due to conjugation that alkenes have. This has stirred up some activity (and some controversy) in the computational study of hydrocarbon bond interactions, bond energies, and in the general nature of hydrocarbon bonds. -- A classic example of a new surprise in a field in which "everything is already known".
We have completed a number of successful research projects. Some of these are published (see for example recent years of publications on the Wuebbles website under http://www.atmos.uiuc.edu). This research relates to effects of human activities on stratospheric ozone, future responses of ozone to international policy, effects of climate change on air quality in the U.S., potential impacts of climate change on the Midwest and more recently on the N.E. United States.
I'm just getting started, but at the end of the day, I'm hoping to have a well-estimated model of retirement behavior in a developing country as well as some pension reform simulations!
We investigate the generation and evolution of the intermediate scale size irregularities in high-latitude plasma patches, which are of interest to the High Latitude Plasma Structure (HLPS) group of the NSF-sponsored CEDAR (Coupling, Energetics, and Dynamics of the Atmospheric Regions) initiative. We have convincingly demonstrated the nature of the mesoscale structuring of high latitude plasma patches by a combination of instabilities starting with the classical gradient drift instability (GDI) and secondary Kelvin-Helmhotz instability followed by a tertiary generation of a self-consistent shear flow. The high-resolution simulations obtained with our parallel MPI 3D code have a dynamic range that allows us to make detailed comparisons with the observations. The subsequent statistical comparison with the overall as well as spectral characteristics of the simulated density and electric field fluctuations and its agreement with the observations are documented through our publications [Gondarenko and Guzdar, 1999, 2001, 2003a, 2004a, 2004b; Gondarenko et al., 2003b]. Gondarenko, N. A., and P. N. Guzdar (2004a), Density and electric field fluctuations associated with the gradient drift instability in the high-latitude ionosphere, Geophys. Res. Lett., 31, L11802, doi:10.1029/2004GL019703. Gondarenko, N. A., and P. N. Guzdar (2004b), Plasma patch structuring by the nonlinear evolution of the gradient drift instability in the high-latitude ionosphere, J. Geophys. Res., 109, A09301, doi:10.1029 /2004JA010504. Gondarenko, N. A., and P. N. Guzdar (2003a), Structure of Turbulent Irregularities in High-Latitude Plasma Patches-3D Nonlinear Simulations, AGU Monograph on Storm-Substorm Relationship, 142, 205. Gondarenko, N. A., P. N. Guzdar, J. J. Sojka, M. David (2003b), Structuring of high latitude plasma patches with variable drive, Geophys. Res. Lett., 30, 4. Gondarenko, N. A., and P. N. Guzdar (2001), Three-dimensional structuring characteristics of high latitude plasma patches, J. Geophys. Res., 106, 24,611. Gondarenko, N. A., and P. N. Guzdar (1999), Gradient drift instability in high latitude plasma patches: ion inertial effects, Geophys. Res. Lett., 36, 3,345.
Dr Pedersen's group both develops computational chemistry methods and applies them to large blood coagulation protein systems. PSC resources have been instrumental both in developing high scaling molecular dynamics applications and using them to model protein systems.
Analysis of x-ray structures of myosin and actin.
Still running model simulations
We have been able to extend our results for 2048^3 grid points and thus obtain better understanding of Reynolds number effects for turbulence.
http://www.journals.uchicago.edu/ApJ/journal/issues/ApJL/v623n2/19316/19316.html http://arxiv.org/abs/astro-ph/0501175, http://adsabs.harvard.edu/cgi-bin/nph-bib_query?bibcode=2004AAS...20514707B&db_key=AST&data_type=HTML&format=&high=429f64035301404
Almost all the molecular simulations we run are conducted on NCSA's computation clusters. Our website is http://www.ks.uiuc.edu
Study of complex fluids with unprecedented size asymmetry between the components. See http://www.ncsa.uiuc.edu/News/datalink/0503/pecm.html
Development of protein loop structure algorithm; Continuum solvent models and long-time molecular dynamics simulations Calculation of electrostatic properties in proteins
Successfully use the TeraGrid to simulated large scale plasma physics.
Several peer reviewed publications. Development of a new molecular dynamics simulations algorithm for open systems
http://www.griphyn.org, http://www.ivdgl.org, http://www.opensciencegrid.org
high memory nodes at NCSA allowed for 21 million atom electronic structure simulation, the largest ever
So far a couple of papers submitted to different journals
Hurricane dynamics study, storm-scale data assimilation. http://redrock.ncsa.uiuc.edu/CMG/index.html
Scattering and propagation properties of electrically large objects have been analyzed using fast integral equation methods in time and frequency domain. Range of new functionalities of the methods have been introduced. Results published in IEEE Transactions on Antennas and Propagation and reported at IEEE APS Symposiums.
Experimental results on fault tolerance techniques (http://iss.cs.cornell.edu)
http://www.cfm.brown.edu/crunch/projects.html
The unfolding of complex for protein
Since my research involved huge computational work, it would not be able to be done without the parallel computers from TeraGrid.
Saving time and money using computational fluid dynamics to determine the best geometry of melt blowing dies.
The elucidation of design principles for constructing an oscillating gene network. The development of an equation-free probabilistic steady state approximation, which speeds up the simulation of dynamically stiff stochastic chemical kinetic systems without losing accuracy. Development of 'Hy3S: Hybrid stochastic simulation for supercomputers', an open sourced software package that uses hybrid stochastic simulation methods to compute the stochastic dynamics of arbitrary chemical/biochemical reaction networks. It is embarrassingly parallelized with MPI and works well with the TeraGrid resources. It also has a bunch of other features that are useful for studying/designing natural/synthetic biological systems. URL: http://hysss.sourceforge.net
enable us to do break through calculations for example, we are able to calculation positron- electron annihilation above the positronium formation threshold in low energy positron collision with hydrogen atom for the first time!
New explanation for the most intense energization of ions close to the Sun via shock waves.
Work on first principles molecular dynamics of enzymatic reactions, design and characterization of heparin-binding foldamers and non-hemolytic antimicrobial polymers was supported by NSF cyberinfrastructure resources. http://www.cmm.upenn.edu/~ivanov/research.htm
See "MILC in the News" at: http://physics.indiana.edu/~sg/milc.html
http://www.ks.uiuc.edu/
development of 3D radiation MHD codes application to variety of astrophysical systems Basically, everything listed at http://www.astro.princeton.edu/~jstone
Our lab regularly publishes on large scale Phylogenetics made possible by using MrBayes 3.0 and 3.1 software on the NCSA clusters (especially Copper and Tungsten).
Ab-initio calculations of large supercells, which are time consuming and require large amounts of virtual memory: Influence of grain boundaries on the magnetic coupling in Co doped TiO2. Co doped TiO2 is one of the so-called diluted magnetic semiconductors. However, the origin of magnetism in this system is still not fully clear. Sure is, that the microstructure (e.g. in form of grain boundaries) does have a crucial influence.
The cyberinfrastructure resources have allow our research team to compute the thermal conductivity of bulk and thin films of silicon. Typically this computation can take several days or weeks in a single processor computer.
Analysis of pile to pile interaction using 3D nonlinear soil models
Major simulations of a magnitude 7.7 earthquake on the southern San Andreas Fault, leading to new insights about seismic wave amplifications in the LA Basin area. http://sceclib.sdsc.edu/TeraShakeDev
I am an Associate Professor and have been an NCSA user for nearly 10 years now! All our successes, including many publications and a major NSF grant, are dependent on NCSA: http://rutchem.rutgers.edu/faculty/lee.html
Most accurate simulations of galaxy formation and evolution to date.
submitted one paper, writing a second and have a third planned
Big shared memory computers provided by NCSA enables us to run our AMR simulation.
Studying several reaction pathways in free-radical oxidation chemistry. Good quantum chemical results for large systems.
too numerous to list here
Computed (and published) the electric polarizability of some hadrons (with Walter Wilcox).
We perform computations that elucidate how pulsed optical plane waves and beams propagate through complex materials (specifically, sculptured thin films).
http://www.astro.virginia.edu/VITA/jetmovie.html
We have been able to compute large scale earthquake simulations which could occur in the greater Los Angeles area. http://epicenter.usc.edu/cmeportal/CyberShake.html
excellent comparison with experimental data for shock wave lithotripsy
recently accepted publication in Biophysical Journal, "Model-Driven Designs of an Oscillating Gene Network," by Lisa M. Tuttle, Howard Salis, Jonathan Tomshine, Yiannis N. Kaznessis (not yet published)
mechanism and pathways of O2 diffusion/transport in hydrogenase (http://www.ks.uiuc.edu/Research/hydrogenase); mechanism of Cl permeation in ClC chloride channels
We have recently completed a speed-up study of our multiphase eruption code with a development allocation. The code scaled very well, and some of the results will be included in a publication that is currently in preparation. In addition, a proposal was submitted for a more substantial allocation with our current approach. We are interested in understanding how different particles become concentrated during explosive volcanic eruptions in order to better access potential hazards and to interpret deposits of previous eruptions.
The Grist project is helping to build the "Hyperatlas", a grid-based astronomical image mosaicking system (see http://grist.caltech.edu).
Design and development of an algorithm for protein-structure prediction and ab initio protein folding
Not yet. Actually I have been trying to find a specific protein modeling software but I could not found until now. I keep searching.
The TeraGrid will help us solve challenge problems in materials processing, especially in solidification, medical imaging, especially in image reconstruction, and nano carbon tube structured materials properties, especially in bridging domain method which couples molecular domain to finite element domain.
Major findings about the coupling of rotation, turbulent convection and magnetisms within the solar convection zone, and in the operation of the solar magnetic dynamo.
None just yet. I'm studying how collaboratories develop and use CI.
Four published papers based on quantum chemical calculations performed with NSF computational resources: http://www.macalester.edu/~kuwata/text/pubs%20aug%2005.htm
Numerical prediction of storm-scale weather (http://caps.ou.edu) Creation of new radar technologies for observing the lower levels of the atmosphere (http://casa.umass.edu) Creation of service-based environments for studying the atmosphere in a dynamically adaptive manner (http://lead.ou.edu)
Successful works on high redshift galaxy formation models using large-scale cosmological hydrodynamic simulations. E.g., http://arxiv.org/abs/astro-ph/0502001 http://arxiv.org/abs/astro-ph/0406032 http://arxiv.org/abs/astro-ph/0503631 http://arxiv.org/abs/astro-ph/0407143 http://arxiv.org/abs/astro-ph/0312651
CAPS participated in the SPC (Storm Prediction Center, http://www.spc.noaa.gov) Spring Experiment where 2km weather forecasts were made and evaluated as new technology by the SPC. The forecasts were made by CAPS personal.
My NSF-funded research will be published in the October Issue of "Biophysical Journal."
several papers published in J. Phys. Chem. and Synthetic Metals
Continued access to computational resources enables me and my students to stay current in advanced technologies. My research which deals with deterministic cost performance measurements relies on dedicated resources since the matrix computations and the theory I use require it. When I have access to advanced machines and all current scientific libraries I am able to prove conjectures on performance. I and my colleagues have numerous publications in conferences and journals thanks to NCSA.
We have a paper submitted on diffusion-limited nucleation and growth phenomena, and have developed methods for the stochastic reconstruction of materials using gas adsorption and/or structure factor data (also submitted.)
I used the TeraGrid cluster of CPUs and storage to conduct wave-propagation simulations for the CyberShake project for the Southern California Earthquake Center (SCEC). The purpose of this project is to calculate probabilistic seismic hazard curves for several sites in the Los Angeles area.
We are performing intensive molecular dynamics simulation on cellulase/cellulose system. Results show interesting movements of the protein on the cellulose surface, but we need more time on this simulation to extract scientifically sound conclusions.
Kurpiewski, M. R., Engler, L. E., Wozniak, L. A., Kobylanska, A., Koziolkiewicz, M., Stec, W. J. & Jen-Jacobson, L. (2004). Mechanisms of coupling between DNA recognition specificity and catalysis in EcoRI endonuclease. Structure (Camb) 12, 1775-1788. This paper showed how sequence recognition by EcoRI endonuclease is not merely a "scaffold" to locate a catalytic machine at a specific site on DNA, but rather that the enzyme is engineered such that recognition and catalysis are inextricably coupled. In this project, we combined biochemical and computational methods to elucidate the mechanism that links recognition and catalysis in a restriction endonuclease-DNA complex. The computational simulations were critical for testing our experimentally predicted model, demonstrating several key structural features of the active site for which we had no experimental data (cannot view the pre-transition state with Mg2+ as cleavage occurs, and we have no crystal structures of the modified EcoRI-DNA complexes). Simulations demonstrated that (a) Metal binding induces no major structural changes in the protein or DNA conformations; (b) Sidechain movements upon Mg2+ binding relieve the rotamer strain resulting from repulsion between D91, E111 and the scissile phosphate GpAATTC; (c) The structure of specifically bound waters in the active site of the Mg2+ complex is significantly different from that in the metal-free complex and suggests why it is important that the characteristic "kinked" DNA distortion brings the phosphate at GApATTC into position to participate in the catalytic mechanism by H-bonding a water molecule.
The computational resources allowed us the simulation, for the first time, of heat transport in thin silicon films. This research is of relevance in nanoelectronics. It was published in refereed journals and presented at numerous conferences.
A systematic study of superconductivity in the Hubbard model and the identification of the pairing mechanism: http://xxx.lanl.gov/abs/cond-mat/0504529 http://xxx.lanl.gov/abs/cond-mat/0508361
Several published and submitted papers based on computations done on DataStar at the NSF SDSC. See http://www.astro.princeton.edu/~krumholz/cv.html for a selection.
Two papers submitted: Evonuk, M., & G. A. Glatzmaier. The Effects of Small Solid Cores on Deep Convection in Giant Planets. Planetary and Space Science. Evonuk, M., & G. A. Glatzmaier. How do the Presencce and Size of Solid Cores Affect Surface Zonal Flows on Giant Planets? Icarus.
http://www.aem.umn.edu/people/faculty/mahesh/research_martin.html
Pioneered first-principles calculations of mobility in nanoscale semiconductor devices and identified role of atomic-scale processes in device performance. TeraGrid resources enabled large-scale calculations with atomic detail and accurate quantum mechanics. See M. Evans et al., Phys. Rev. Lett., v. 95, p. 106802 (2005).
One publication in Journal of Physical Chemistry B http://pubs3.acs.org/acs/journals/doilookup?in_doi=10.1021/jp045155y
We have been able to develop a massively parallel molecular dynamics and quantum mechanics approach to investigating solvent dynamics effects on electronic structure of solutes.
We extend the concepts of phase, polarization, and feedback control of matter to develop a general approach for guiding light via metal nanoparticle arrays. The phase and polarization of the excitation source are first introduced as tools for control over the pathway of light at array intersections. Genetic algorithm is next applied within a unified scheme, where both the excitation field parameters and the structural parameters of the nanoparticle array are optimized to make constructs with desired properties. The scheme is used to gain insight into the interplay between interactions that underlies the coherent propagation of electromagnetic energy via nanoparticle arrays. Implications to several fields are envisioned.
Successful protein folding of mini-proteins
Able to support HDF users on TeraGrid platforms at NCSA, SDSC, PSC, CalTech, ANL. Combined HDF and Storage Resource Broker in a way that can improve data subsetting access by orders of magnitude.
Numerical weather prediction for storms
The Pittsburg Supercomputer Center's resources enable my to-date most successful simulations of realistic tornadoes within supercell thunderstorms. The results are been written up for publications. The computing resources are also used extensively by my research group and by the Center for Analysis and Prediction of Storms, for large-scale modeling and data assimilation studies. Without the supercomputers, many of the very-high-resolution simulations would not have been possible.
One major theme of my research recently has been the structure and evolution of star formation in galaxies. Our models of gravitational instability in disk galaxies appear to capture the basic physics of star formation surprisingly well. The papers referenced by the following URL give full details. http://adsabs.harvard.edu/cgi-bin/nph-abs_connect?db_key=AST&db_key=PRE&sim_query=YES&aut_xct=NO&aut_logic=AND&obj_logic=OR&author=li%0D%0Amac+low%0D%0Aklessen&object=&start_mon=&start_year=&end_mon=&end_year=&ttl_logic=OR&title=&txt_logic=OR&text=&nr_to_return=100&start_nr=1&jou_pick=NO&ref_stems=&data_and=ALL&group_and=ALL&start_entry_day=&start_entry_mon=&start_entry_year=&end_entry_day=&end_entry_mon=&end_entry_year=&min_score=&sort=SCORE&data_type=SHORT&aut_syn=YES&ttl_syn=YES&txt_syn=YES&aut_wt=1.0&obj_wt=1.0&ttl_wt=0.3&txt_wt=3.0&aut_wgt=YES&obj_wgt=YES&ttl_wgt=YES&txt_wgt=YES&ttl_sco=YES&txt_sco=YES&version=! 1
We are still at the very early stages of our work on the TeraGrid.
Two papers had been published; other two are still worked on. Some jobs in these papers were carried out using NCSA supercomputer. Papers are available: http://pubs.acs.org/cgi-bin/article.cgi/esthag/2005/39/i13/pdf/es050271f.pdf http://pubs.acs.org/cgi-bin/article.cgi/esthag/2005/39/i02/pdf/es049480a.pdf
NSF cyberinfrastructure has enabled large-scale computations that have led to a better understanding of the solar dynamo problem.
Essentially all my research is based on NSF computational resources - see http://home.fnal.gov/~gnedin/gallery.html
To this point we have been able to model the complex interactions between a vortex ring propelled non premixed flame with a wall. The simulations are run using NSF cyberinfrastructure resources.
Many publications on strongly correlated electron systems such as the high temperature superconducting cuprates. See Mark Jarrell's web page http://www.physics.uc.edu/~jarrell
None from me yet, although findings in one publication from other members of my lab group resulted from use of TeraGrid resources.
We have successfully used SRB and its resources from SDSC and now depend on it for our data needs of research.
Gridfree simulation of complex turbulent flows: Publications available at http://www.glue.umd.edu/~bernard/ and http://www.vorcat.com
I use gaussian03 on tungsten. Fast single processors are nice.
1 publication: macromolecules 38(13):5761-5765, 2005 2 conference talks.
Impact od Head-Disk Slider on the head disk. Was able to characterize the thermo-mechanical behavior of the head disk interface under shock loading.
We published 4 papers and a book chapter on quantum chemical studies of catalysts for removing NOx from automotive exhaust. Resources at NCSA were vital for much of this work. Our calculations helped answer some questions about these systems that were difficult to answer by experiments alone.
We developed a new approach to the magnetotelluric geoprospecting problem, using a selection method instead of Tikhonov regularization. We evaluated the efficacy of our computationally-intensive method on NCSA supercomputing resources. We divided up the work with a three-layer parallelization using MPI. For more information, see http://www.cse.uiuc.edu/~rjhartma/ and http://www.cse.uiuc.edu/~rjhartma/rjhbthesis.pdf
We have been successful at completing a series of calculations that resulted in a paper that has been just accepted for publication in the Journal of the American Chemical Society. Also, a second paper is in preparation and should be submitted hopefully sometime this fall.
We make extensive use of the computer resources provided by NCSA for ab initio calculations of defect evolution and phase transitions in real materials. Publications can be found at http://www.physics.ohio-state.edu/~rhennig/Publications/
Presentation of results at 3 conferences, 1 of which was overseas (France). 3 referred publications.
http://astron.berkeley.edu/~cmckee/bafd/index.html
several scientific publications in peer-reviewed journals; new and extension of several grants
I perform many quantum calculations using Gaussian. I use the results of these calculations to derive classical force constants that describe the interaction between atoms and molecules.
Understanding how energetic ions from interplanetary space interact with the Earth's magnetosphere. http://lws-trt.gsfc.nasa.gov/trt02_richard.pdf
Our most recent successes have involved the successful predictions of the decay properties of D mesons. See, for example, http://www.cerncourier.com/main/article/45/6/14, and http://www.fnal.gov/pub/today/archive_2005/today05-08-10.html
Cosmological simulations helped with study of outflows from active galactic nuclei: http://xxx.lanl.gov/abs/astro-ph/0506670
My successes may be counted in terms of publications resulting from collaborative work with colleagues. So far there have been 10 publications.
Calculation of the geometrically optimized structure of a large molecule; a model of the vitamin B12 cofactor (73 atoms, 274 electrons) in the enzyme methionine synthase.
project ongoing
The scientific successes from my computational research are listed below. They correspond to a list of publications and presentations. "Theory of Asymmetric Organoctalysis of Aldol and Related Reactions: Rationalizations and Predictions"Acc. Chem Res. 2004, 37, 558-569. Gordillo, R.; Carter, J.; Houk, K. N. "Theoretical Explorations of Enantioselective Alkylation Reactions of Pyrroles and Indoles Organocatalyzed by Chiral Imidazolidinones" Adv. Synth. Catal. 2004, 346, 1175-1185. Gordillo, R.; Dudding, T.; Houk, K. N. "The Mechanism of Rawal's Hydrogen-Bonding Catalysis of Diels-Alder Reactions" Manuscript in preparation. Gordillo, R.; Houk, K. N. "Origins of Stereosectivity in Diels-Alder Cycloadditions Catalyzed by Chiral Imidazolidinones" J. Am. Chem. Soc. Accepted. Gordillo, R.; Dudding, T.; Houk, K. N. "DFT Study on Hydrogen Bond Promoted Diels-Alder Cycloadditions. Reaction Mechanism and Origins of Enantioselectivity" 2005 American Conference of Theoretical Chemistry. Los Angeles, CA. July 16-21, 2005.
S. Jadhav, C.D. Eggleton, and K. Konstantopoulos, "A 3D Computational Model Predicts that Cell Deformation Affects Receptor-Ligand Bond Lifetime and the Number of Bonds During Rolling Under Shear". Biophys J. 88(1); 96-104, 2005.
After the workshop at UCD, we learned how to run our simulations in parallel and about the several resources available for our research, that we are in the process of using. People presenting the Extreme/IO workshop in San Diego were very well prepared and willing to help. We are using the capabilities, and running successfully our jobs, this allows us to run large systems and have store our data, since our cluster capability does not allow this.
Physical and structural properties of raft-sized sphingomyelin (SM) bilayer and cholesterol-containing DPPC bilayers have been characterized by simulations. Ion and water permeation processes for ion channels (porins, KcsA, gramicidine, etc) in lipid environments have been studied. Comprehensive force-field parameters for PC lipids based on GROMOS96 parameter set have been developed.
Molecular Dynamics simulation of catalyst reaction in Fuel Cells
It enables me to study the hot gas inside the bulge dominated galaxies for the first time. The results of the simulation are used to compare the state of the art observation from CHANDRA and XMM-Newton, providing detail physical understanding of the nature of the hot diffuse gas inside the galaxies.
The use of scalable MD codes running on Lemieux enables us to compute biomolecular properties deemed inaccessible only a few years ago. We has investigated the binding to the T cell receptor (TCR) of two peptides complexed to the major histocompatibility complex (MHC) by means of MD simulations. To our knowledge, this is the largest system for the free energy calculation using the thermodynamic integration method. The paper is published at: J Immunol 2005 175: 1715-1723.
We are able to simulate the folding of the beta-haipin, a 16-residue peptide at TeraGrid
Using Constrained Monte Carlo Simulations with Distant Residue Contacts Prediction in Proteins (under preparation)
Current grant started Sep 1st. Past research has extensively benefited from NCSA, PSC and SDSC allocations.
The MILC collaboration has had great success in calculating the quark masses and decay constants of pion and Kaon that allow extraction of CKM matrix elements. These are fundamental parameters of the Standard Model of Elementary Particle physics. There is much more to tell. You will find some here: http://physics.indiana.edu/~sg/milc.html and, of course, in or allocations proposal.
A Project to simulate political candidate evaluations during Presidential Elections employing multiple cognitive agents is in progress and has been very successful so far. For detail, visit http://myweb.uiowa.edu/skim30
We explored the performance prediction via relative performance method across different platforms. Your NSF cyberinfrastructure resources help a lot.
Thomas Huber, Ana V. Botelho, Klaus Beyer, and Michael F. Brown Membrane Model for the G-Protein-Coupled Receptor Rhodopsin: Hydrophobic Interface and Dynamical Structure Biophys. J. 2004 86: 2078-2100
I am still in the preliminary phase of research, but I expect good discoveries to come of the work!
Virtually everything we've done with our research has been connected to the supercomputer time we've used over the last few years. For details, please see our web site at http://www.ks.uiuc.edu/
Identified Arabidopsis promoter elements implicated in diurnal control of gene expression.
Gordon Bell Award, Special category, SuperComputing 2004, November 6-12, Pittsburgh PA
Recently started using it. Its of great use to me and hoping to publish soon.
7 CMS Notes, 8 Conference presentations, 1 publication. Summary of Caltech physics group activities can be found here: http://www.hep.caltech.edu/~litvin/cit-activities.txt
See research described at http://wugrav.wustl.edu
two journal publications so far
QM/MM study of chorismate mutase enzymatic reaction.
An emerging research area in biophotonics with potentially near-term clinical applications in early-stage cancer detection involves investigation of possible correlations of the elastic light scattering properties of tissues with alterations in their cellular composition and nanostructure. Until recently, exploring these correlations has been impeded by a lack of robust and accurate mathematical models of the light-scattering properties of complex structures. The NSF computational research has enabled us to make recent progress in this area. Topics include: (a) development of accurate reduced-order expressions for the total scattering cross-section spectra of a wide range of nonspherical and inhomogeneous particles; (b) rigorous finite-difference time-domain modeling results showing how the backscattering of light can be sensitive to nanometer-scale features embedded within micron-scale particles; and (c) development of accurate reduced-order expressions for the backscattering depolarization properties of a wide range of inhomogeneous particles. These advances provide an improved science base for cellular-level biophotonics, and have promise to accelerate the development of novel corresponding clinical technologies.
We have performed numerical simulations of turbulent flow using 2048^3 grid points to study important problems of turbulent mixing. Our simulations are currently largest in the world for the purposes stated.
We've used the computer time provided to conduct a molecular dynamics simulation, in which we were able to study the different substrate selectivity of two aquaporin channels. This result was published on 'Structure'. The link to the paper: http://www.ks.uiuc.edu/Publications/Papers/paper.cgi?tbcode=WANG2005
test run time error detecting system on DataStar and also test the latency and other issues on DataStar
Understanding the function of protein
Molecular dynamics studies on AChE-TFK and PKA-SP20 complexes.
The current NCSA grant is being used to revise the paper of D. B. Khismatullin, Y. Renardy, and M. Renardy, titled `Development and implementation of VOF-PROST for 3D viscoelastic liquid-liquid simulations', which is under review for the Journal of Non-Newtonian Fluid Mechanics. The PDF file is at http://www.math.vt.edu/people/renardyy/Research/Publications/paper2202005.pdf
use of NCSA computing resources for electronic structure theory research
Performed proof of principle about ability to model general anesthetic action with large computational models.
Large parallel cluster usage has been essential for completing large ensemble sensitivity analysis work, which has resulted in one peer-reviewed publication with another in preparation.
the study of polymer dynamics in flows

Computational Environment

4. Please rate your satisfaction level with any NSF-supported systems that you have used in the last year. (If you haven't use a system, please select 'N/A' or skip that item.)

Remarks/Plans: The top seven systems had an average rating of "somewhat satisfied" or better. The lowest rated systems were rated between "neutral" and "somewhat satisfied". The table is sorted by the satisfaction level in descending order. (5=extremely satisfied, 4=somewhat satisfied, 3=neutral, 2=somewhat dissatisfied, 1=extremely dissatisfied).

	Satisfaction Level (1-5)	Number of responses
SDSC DataStar IBM p655	4.2	67
NCSA TeraGrid IA64 Cluster	4.2	95
SDSC DataStar IBM p690	4.1	64
NCSA Xeon Cluster (tungsten)	4.1	100
NCSA IBM p690 (copper)	4.0	106
SDSC TeraGrid IA64 Cluster	4.0	70
PSC TCS1 (Lemieux)	4.0	98
NCSA SGI Altix (cobalt)	3.9	64
PSC HP Marvel (Rachel)	3.9	50
TACC Cray-Dell Cluster (Lonestar)	3.8	20
NCSA Condor Pool (radium)	3.7	15
ANL TeraGrid IA64 Cluster	3.7	32
Caltech TeraGrid IA64 Cluster	3.5	35
Purdue IBM SP	3.5	14
Purdue Linux Clusters	3.5	14
TACC Sun Fire E25K (Maverick)	3.4	10
ORNL Xeon Cluster	3.4	11
Indiana AVIDD-I64 Cluster	3.4	17

5. Please rate your satisfaction level with the mass storage systems you have used in the past year. (If you haven't use a system, please select 'N/A' or skip that item.)

Remarks/Plans:: The mass storage systems had similar ratings to the computing systems above, with NCSA UniTree and SDSC HPSS rated slightly better than "somewhat satisfied." The table is sorted by the satisfaction level in descending order. (5=extremely satisfied, 4=somewhat satisfied, 3=neutral, 2=somewhat dissatisfied, 1=extremely dissatisfied).

	Satisfaction Level (1-5)	Number of responses
NCSA UniTree	4.1	132
SDSC HPSS	4.1	80
PSC File ARchiver (FAR)	4.0	70
Storage Resource Broker (SRB)	3.8	32
GridFTP/uberftp to UniTree	3.7	33
TACC Data Migration Facility (DMF)	3.6	10

6. Considering all your interactions with the resources and services at NSF-supported sites, please rate your satisfaction with the overall quality of the computing environment (computing systems, networks, consulting support, tools and software, etc.) at the following sites.

Remarks/Plans: The quality of the computing environment was generally rated higher than the individual computing systems at the sites. This may be at least partly due to the high ratings for user support. The table is sorted by the satisfaction level in descending order. (5=extremely satisfied, 4=somewhat satisfied, 3=neutral, 2=somewhat dissatisfied, 1=extremely dissatisfied).

	Satisfaction Level (1-5)	Number of responses
NCSA	4.3	210
TACC	4.2	18
SDSC	4.2	131
PSC	4.1	116
Oak Ridge National Lab	4.1	18
TeraGrid (overall)	4.0	148
Argonne National Lab	3.8	33
Purdue University	3.5	15
Caltech	3.4	29
Indiana University	3.4	19

7. Please offer any specific comments or suggestions for improving the NSF cyberinfrastructure resource environment. If you have comments for a particular site, please mention the site:

Remarks/Plans: Some of the common complaints are lack of reliability and availability, long queue wait times, lack of storage space, poor integration, lack of response and the difficultly of learning to use a complex computational environment. These comments will be passed on to the appropriate groups with the CIP and TeraGrid to follow up with users.

I primarily use the NCSA Xeon cluster (Tungsten). When it is working, I am extremely satisfied. However, there have been several times when the cluster has been taken off line. That can be very frustrating.
There is a need for transparent tools for the use of the TeraGrid as a single platform.
PSC far goes down quite frequently. This can be troublesome when trying to access files that are stored there.
My only complaint is that Caltech's cluster is being disassembled.
I'm extremely satisfied as I stated in the above survey. According to my experience with hardware and software, the TeraGrid system with the combination of the Linux OS has been extremely helpful and flexible in particular for porting and running a large variety of different codes and software. Definitely funding spent on the TeraGrid system is an excellent investment.
Frequently I try to run jobs on copper.ncsa.uiuc.edu using the Jaguar software and the daemon is down. Someway to keep this problem from happening as often as is does would be nice.
My rating of 4 and not 5 is due to two factors: (1) I found one of the consultants is rather unhelpful - at one time, his reply is "do some investigating yourself." and in several instances, the suggestions to eliminate run errors are do not work. (2) One time, I had a job that is submitted as "debug" and specified as under 30 min run. It did not run for 1 week. After sending an email to the consultant to ask for the reason, the job finally ran that day. However, no explanation was given to me why the job was not run for such a long time.
The Mass Storage System is very slow to access and date is put on tape after a short amount of time, meaning that it takes a while to access it once I run a process a few days later.
Throughput on the NCSA computers is often slow with several days in the queue between runs.
I am extremely pleased with all interactions I have had with NCSA support staff. I would have given a 5 above except for numerous hardware difficulties/outages. (Primarily GPFS issues I believe.)
The primary issues that I have concern queue times (which are horrible on Copper and the p655 nodes on DataStar) and data backup capabilities. DataStar in particular has been experiencing very, very long queue times on the p655 nodes (in excess of one week at times!) which make it difficult to use. This is the reason that Copper became almost entirely unusable. Data backup is also a problem. My primary simulation tool generates very large amounts of data (hundreds of gigabytes for a typical run) and the rate at which data can be put into NCSA's mass storage system seems to hover around 4 megabytes per second. The situation was similar (though not as bad) at SDSC with their HPSS system, though in the last year this has improved tremendously, and I'm not very happy with it.
If there is a way of controlling the amount of emails about machine status, block allocations etc it would be good. I don't think everyone is interested in all activities occurring on all machines on the TeraGrid.
My greatest complaint is with the reliability of SDSC DataStar. DataStar seems to be down at least half the time, occasionally crashing dramatically so that computations in progress are lost. The reliability of DataStar seems to be quite low compared to the other computational resources I have had recent experience with (e.g. TACC Longhorn and UMich Morpheus).
I think NCSA manages their resources quite well. My one recommendation is to increase the permanent, quickly available storage available to a user from 1GB to at least 2GB, or perhaps as large as 5GB.
SDSC- very busy and could use additional resources. Do an excellent job despite these limitations
Better storage access methods supporting all of the platforms users are using. Better communication about things like planned upgrades breaking previous versions of access software to minimize user downtime.
Better instructions for getting started for those not familiar with supercomputers and the queuing systems.
Learning to use a new site is a matter of searching through web pages to glean bits of information and then emailing one question at a time to the help desk and trying to apply the answer they give.
Overall I think that the latency from job submission to execution needs work; it's way too long.
Occasionally it has taken unusually long to get new users' accounts activated.
It would be great if there would be a forum for people to discuss so that we learn from each other
Turnover times on the machines I was using (Lemieux and DataStar) were sometimes extremely long especially for large jobs (1024+ processors). This is just a comment and it may be possible that there is probably little to do about this due to the increasing need for computing resources.
Main hassles: site-specific authorization complexity, long queues without cross-site method for queue submission, upgrades that break parts of grid environment, security impediments in getting data to/from TeraGrid
For our codes, Pittsburgh just has not lived up to expectations. It is not even close to being the number cruncher that NSF claims it to be.
My problem with Indiana is they don't give you enough hard drive space to be able to accomplish any real science. 1 gig per person is not enough.
SDSC DataStar: need to improve the turnaround times (staying in queues for a long time)
Get vendors to be more responsive to software problems. We had a 2-rail problem on Lemieux at PSC that took HP over a year to fix (system software problem). That is not exactly a responsive vendor.
It would be nice if Lemieux was down less. One time, after changing my password, I apparently tried to log in too many times and it took over a day of phone calls and email messages to get things reset so I could use the computer; even the technical support staff were confused as to why I couldn't log in.
The 'tar' command at PSC was changed, and I find it less convenient than the old version. The old version allowed the use of the -L flas to tar all files in a list. I think that is a very useful feature to have.
I haven't really used the system much, so I don't have any opinion of it.
no global file system (i.e. common home dir), it is extremely painful to setup your certificates at all sites, different logins on different sites
Biggest gripe is the frequent downtime of resources since platforms are often not stable when placed in production
Java CoG Kit is not on the software stack
Initially there were problems with FORTRAN compiler, such as inability to flush the output data streams and some MODULE usage. These were alleviated though by changing the source FORTRAN code at the expense of convenience.
Please enlarge the disk space for store
The web-based user manuals/instructions are very important for the remote users to set up their computational environment to work. They should be accurate and updated timely reflected the change of the system setup. In my experience, some instructions from some centers did not work as expected.
For a beginner, the TeraGrid resource structure it is extremely confusing. As an example, it is very difficult to know which account will be charged for which jobs on the grid. Better documentation is desperately needed
very satisfied
TACC's Lonestar machine needs larger disk block quotas for users. I should probably send a mail to them.
We are currently working with Sudhakar Pamidighantam and colleagues to help develop a more user-friendly interface for large-scale Phylogenetics analysis based at NCSA. We anticipate this might drastically increase usage for this purpose.
It would be nice if my grid certificate actually worked everywhere (after mapping, etc.).
I think that one improvement could be to enhance the documentation of the different software (e.g. compiles) installed in the computers.
TeraGrid is a very difficult environment for a novice. It does require considerable experience to help, which means a professional IT person for practical purposes.
We have found the queues on copper to be inordinately long. Our jobs aren't huge (we run Gaussian on fairly small molecules) but they seem to be of the middling size that result in very long queue waits.
My particular interest is in linking TG sites with external resources using Condor or Condor glide-in. More documentation on external connectivity (Open ports, routes, etc) for the various sites, or a tutorial on how to use glide in to each of eth TG sites would be very useful.
Outstanding trouble tickets should be reviewed regularly. I have several tickets that I submitted months ago and for which I have never gotten a response.
cobalt needs to have a queue with a longer time limit
Gaussian calculations on large systems are difficult on the clusters with very short queues, i.e., cobalt, ncsa, Lemieux. As a first time user of supercomputing centers, I didn't know this prior to writing our grant and I was frustrated at the choice of available systems (basically only Rachel) for my work
pSeries 690 is tooooooo old
Our computations generate multiple terabytes worth of data. Although I am satisfied with the results we have been able to generate using the TeraGrid, it would be nicer to have more networking capacity between sites and more disk storage.
SDSC SRB has reliability problems as it is often inaccessible (I suspect this is more due to the underlying HPSS rather than SRB, which is a wonderful concept) I had problems running data intensive applications on the Caltech cluster's PVFS. Errors were frequent and performance was very slow. NCSA's GPFS worked much better for my app. I have had very good experience with the TeraGrid help system, especially in interacting with the staff at NCSA and Caltech. Responses to my queries were always prompt, courteous, and professional.
Long waiting time at Rachel prevents me from running calculations there. My jobs waited 2 days in queue and didn't start. I requested only 8 processors of 1 node for 10 minutes. Possibly job execution time limit there should be reduced.
My account is active; I have access, looks OK.
I can not find a proper numerical C library like IMSL in the other sites, except Purdue.
The systems are not updated. Too slow! Nodes are not enough.
I wish the policy for job queuing was more transparent and explicit.
I have not yet had much opportunity to use the resources. I found that setting up my Kerberos password took a long time, particularly for the paperwork to arrive. Then, the instructions for setting up certificates and access to the server were confusing and didn't work right away. However, NCSA personnel are very helpful and friendly. I also found that NCSA support staff was very helpful with setting up an Access Grid node and the software to run it. That AG software is very cumbersome and fussy and the documentation isn't very good. If it were more stable and easier to install, it could be great.
I also have had allocations on the NCSA facility at the University of Kentucky. I have generally been quite pleased with the services there, except for the fate of my peer-reviewed allocation of ~18,000 SUs in 2004. I found out at the last minute that I would not be allowed to carry over the unused portion of my allocation at the end of 2004, contrary to my understanding and standard NCSA policy. I was counting on that resource being available. This was extremely disappointing and frustrating.
PSC: FAR not as reliable as it should. My archive scripts retried up to 10 times as command failures were common.
I have made two applications, and I can't get even the courtesy of a rejection notice.
The job queuing policy on sdsc.teragrid needs to be modified. The current policy seems to be that jobs using more nodes have higher priority than those using fewer nodes.
The PSC site has excellent online resources pertaining to job submission and scheduling. I have also found the staff at the PSC to be excellent. They are extremely supportive in all aspects of my research, helping to address both theoretical and practical concerns.
The consultants at UIUC are very helpful and professional.
NSF needs to invest in people to develop new computational science formalisms, algorithms, and codes. The innovations of these people are often more important than advances in computing infrastructure.
The far system would be much easier to use if it accepted ftp-style commands like put and get rather than store, rstore, etc.
too bureaucratic, too little responsiveness
Lemieux at PSC is an excellent computational resource for parallel codes with a need for fast internode communication. However, recent batch queue wait times have exceeded two weeks (both for 80-100 node and 8-10 node requests). This severely limits its utility to my research program, as results are few and far between. FAR is a useful archive for long-term storage of large amounts of data. However, frequent interruptions prevent the use of get (or tcscp) commands in batch jobs. If FAR is down when my job starts to run, I can't copy initial data (e.g. for job continuation) and the job crashes. As a result of this unreliability, I generally "pre-copy" data to the scratch filesystem before submitting the job. This isn't an efficient use of scratch space, and could be eliminated if FAR was more reliable.
I had difficulties with the NCSA TeraGrid system. The consultants spent a lot of time trying to help me install the software correctly, but we were never able to get a working installation of the software. I'm not sure what the solution would be, though.
We found that some of the MPI functions do not work properly in PSC clusters Rachel and Lemieux.
It should be possible to protect a folder on local scratches for a specified time period (couple of weeks to 1-2 months) Scratch space is one of my only ways of compiling data from multi-site runs and when a purge hits, I can spend a day or two restoring files.
SRB is a serious advance. Its continued support, and improvements are important.
The lack of sufficient human resources has made it almost impossible to meet the expectations generated by the TeraGrid proposals. Post-grant leadership has been particularly lacking. Those who do work on the project work hard, but in many cases lack the expertise needed, especially in their ability to understand and integrate complex systems.
The turn around of large jobs on NCSA systems is too long.
It's no secret that our computers have now totally overwhelmed our mass storage systems. We're now spending 20-30% of our allocations just moving data on and off of mass storage. Very short queue times (12 hours) and fast purges of scratch space sharpen this problem.
SDSC: tremendous resources and good staff. A lot of downtime. Sometimes slow turnaround on problem resolution.
On Rachael @ PSC, the shell environments and standard tools (VI editor) feel outdated and cryptic -- not nearly as nice as what comes with a standard install of a free RedHat distribution.
SDSC consulting support is good, but there is room for improvement.
We desperately need bigger machines! NSF-supported supercomputers continue to fall behind those of the DOE, NASA, and other countries. According to the latest TOP500 list, scientists in Japan, Spain, the Netherlands, Switzerland, China, UK, Australia, and Germany have access to systems that are more powerful than the highest-rated NSF machine (the XT3 at PSC, which is still in friendly-user mode). Meanwhile there is no end to applications that are crying out for greater computing power; see the SCALES report, for instance.
PSC shouldn't have a separate GridFTP and globus-job-submit servers... makes things too difficult
Uniform build environment, or portable compiler wrapper. Just getting codes built (find compiler, MPI, includes flags, etc.) is a major headache.
More nodes, more memory, & shorter turnaround time.
NSF should show more Environmental Sciences support at NSF sites
There are so many similar machines to run on - each with different logons and operating environments - that one is basically forced to pick one site and run there. The mental and logistical costs of running at different sites are too great. (I say this as a long term user of HPC machines. For ever different site we have to make sure at least one non-standard standard library is available and working (SPRNG), test the code and compilers, and then get a feel for the turnaround of different job sizes etc.)
Improvements in data upload bandwidth are important. Utilities and tools for windows environments are also very important as increasingly much research data is generated on instrumentation controlled by windows machines.
I hope that there will be an ability to upgrade the IBM POWER4 systems (copper) to the IBM POWER5/6 processors
It would be better if NCSA IBM p690 can provide us with more dedicated queues.
UniTree storage system could be made more user friendly. Would like to see a windows based interface in downloading files from UniTree to my local machine
For many applications, (e.g., Gaussian), we cannot use multiple nodes and are thus restricted to only using those CPU's on one node. Thus time limitations on the submitted jobs that are very long have to be re-submitted from checkpointed data. This makes the running of long jobs cumbersome, and it would be better if some queues could have the time limits increased.
The computing power available at the NSF centers is not keeping pace with our scientific needs, or with what is possible given the available technology
We need to be able to get a large job run (64 nodes) in a more timely manner on the shared memory computers.
When data transfer runs very slowly (1/1000 of the posted speed), there is nobody who can explain why or how to improve it.
long queues
GPFS goes offline too often
More "workstation-like" environment: direct ssh to compute nodes, persistent long-term directory storage.
My main dissatisfaction with SDSC is that we have been completely cut off from computing there and I don't know why. I am a local user of Indiana's AVIDD and it is so crammed with serial jobs that it is not easy to get enough time for running parallel jobs. This seems like a big waste of expensive Myrinet hardware. NCSA's short purge period on global scratch caused me to lose some significant output. In their defense, when I asked for help, they gave me a special area that is not purged. At PSC, it has been difficult to get in as many large runs on Lemieux so as to use our allocation. However, they have very kindly let me use the new Cray under the friendly user period. I have been able to get a tremendous amount done and this has really accelerated our output. I hope that the machine will prove stable enough for non-friendly user use soon. There are still some glitches, but maybe the new OS installed the last couple of days will improve the stability.
I sometimes need a single node high performance machine, but at NCSA I found all are clusters. Hope there will be an old Cray-like machine available in the near future.
I been working on SDSC DataStar a long time. But recently, there are a lot of maintenance shutdowns due to upgrade. I hope you could budget maintenance time efficiently. Again, thanks for your good work. SDSC DataStar provides the best platform for my research work.
Make more computational resources available
A lot of work needs to be done to make the Grid software really work in an arbitrary environment. My attempts at configuring grid services at our site have met with only varying degrees of failure...
NCSA (Cobalt): very unreliable NCSA (Tungsten): unable to monitor distributed jobs: NCSA (Copper): unable to run ANSYS software (software issue) NCSA mass storage system is not user friendly and extremely difficult to retrieve files
it is great
Sometimes, follow-up on problems submitted to consulting is delayed.
The large calculations are staying too much in queue
The temporary storage at tungsten is bit less. Any increase is much appreciated.
TG infrastructure is a bit fragile. The problems are - 1) frequent downtimes, 2) /gpfs and /pvfs problems. This is common problems for all TG clusters (may be excluding TACC because it has IBRIX FS)
more storage space in home directory
Greater stability of the systems would be nice. Sometimes turnaround for large jobs is not as efficient as desired.
Utter lack of integration. I simply end up working with 10 different machines, 10 different passwords, 10 different home directories, 10 different environments. Yet, it takes a few days for my software environment to get configured, compiled and installed. It will take a long time to get it all going on 10 systems.
Indiana is pretty bad. The service is very disappointing and slow. NCSA and Caltech are both very good.
It might be nice to have more literature on how to use the resources on the website.
Tungsten will not run large jobs as machine gets full of small jobs and sufficient nodes are never or rarely available.
PSC: needs better maintenance. Jobs keep crashing; scratch keeps getting erased even before 1 week of inactivity.

Grid Environment and Services

8. Which of the following grid tools are you using or would you be interested in using?

Remarks/Plans: Users are expanding their use of grid capabilities, although 'Don't know' tops the list of grid tools users are using or would be interested in using, 32% of respondents are using or interested in using MPICH/MPICH-G2 and 22% are using or interested in GridFTP. (The table is sorted by number of responses in descending order.)

	Number of Responses	Response Ratio
Don't know	101	33%
MPICH/MPICH-G2	97	32%
GridFTP	67	22%
None	52	17%
Globus Toolkit command-line programs	43	14%
Condor/Condor-G	41	13%
Globus Toolkit API	36	12%
Uberftp	31	10%
GSI-enabled ssh	28	9%
GridShell	26	8%
Custom portal/workbench (or developing one)	21	7%
Other, please specify	20	7%
Existing portal/workbench	16	5%
MyProxy	14	5%
MDS	2	1%
TOTAL	595

SRB, Jargon
Pegasus Workflow Planner (VDS)
Java CoG Kit
see above under Phylogenetics portal development
SGE Grid Engine
OpenMP
Apple Parameter Sweep Template
Gaussian 03
ECCE
HSI, HTAR
ordinary ftp
Any of the above, provided human burden is small
don't know enough about any of these to know which
Specific application portals, e.g. comp. chem.
RFT
SRB
Globus Toolkit 4 web server
SUN One
IBP LoRS to transfer large chunks of files
OpenMP
Chombo, HDF5, VTK, Python, netpbm, PBS

9. What is your experience or plan to use grid tools and services?

Remarks/Plans: 27% of users are starting to experiment with Grid tools, and another 28% may use them within two years; 18% have conducted actual research with grid tools and services. Only one user reported that he tried them and discontinued use. There is not enough information to determine why 26% say they have no plans to use them. (The table is sorted by number of responses in descending order.)

	Number of Responses	Response Ratio
May consider using them in the next 2 years	88	28%
Just starting to experiment with them	84	27%
No plan to use them	80	26%
Experienced, have conducted actual research using them	57	18%
Have tried them, but plan to discontinue use	1	0%
TOTAL	310

10. We are developing a web-based user portal that will allow users to manage allocations and accounts, inquire about system status, and perform other tasks. Please rate the following functions in terms of your interest:

Remarks/Plans: The strongest interest is in a metaqueue that will allow users to submit a batch job to next available computing resource. The table is sorted by the level of interest in descending order. (3=extremely interested, 2=somewhat interested, 1=not at all interested)

	Level of Interest (1-3)	Number of responses
Submit jobs to a "metaqueue" for next available resource	2.6	301
Reserve compute and visualization resources	2.4	294
Manage staging of data between sites	2.2	296
Discuss issues with other users	2.1	297

11. In response to user input, we have put in place a set of pre-production and trial services. Please indicate your interest in each.

Remarks/Plans: The greatest level of interest is in the wide-area parallel filesystem.

	Extremely interested	Somewhat interested	Not interested	Tried, but need help	Tried, was not useful
GPFS Wide-area parallel file system	25%	39%	33%	1%	2%
Science Gateways	21%	38%	38%	1%	2%
Striped GridFTP service (tgcp)	21%	35%	41%	2%	1%
GridShell	17%	39%	41%	1%	1%

12. Please offer any specific comments or suggestions for improving our grid environment and services. If you have comments for a particular site, please mention the site.

Remarks/Plans: Some of the common complaints are the "learning curve", difficulty in using certificates, performance of transferring data from one system to another, and lack of understanding (or skepticism) of the benefits in using grid tools and services.

I'm very happy with the performance of using different sites individually, like running jobs separately on TeraGrid at SDSC or NCSA. Since the TeraGrid system provides a high performance band-width via the TeraGrid backbone, I'm very interested also in running and testing the performance of jobs submitted cross-sites.
For my research, the real need is for medium-large scale shared memory systems (16 processors, 32-128 gigabytes of RAM.) The underlying computations are entirely dense linear algebra (BLAS/LAPACK), and my experience has been that shared memory systems with OpenMP work better and are easier to use then clusters for such work.
Is there a document to read to learn more about this subject?
Work on easing methods of data transport to/from TeraGrid (scp/ssh enabled not good enough).
setup of certificates needs to be automatic, only one login/password combo, common home directories, "meta queue", simplified documentation
I think it could be useful if the definition and advantages/usefulness of each of the above services are placed in some special web site for potentially interested users to read.
Again: A single user will have a difficult time. This has to be done through a team, that comprises experienced IT members.
Faster network connections between sites would make it easier to run our simulations and exchange data between them.
My mosaicking portal, under development, requires some special support that is not specifically addressed by TeraGrid in its current form. In particular, the portal will require "anonymous" access by our users for small mosaics, with jobs run on the TeraGrid by our portal under a trusted TeraGrid user (our portal will manage resource allocations), consistent and reliable level of service with limited latency.
When describing new services, offer a description that uses less "techie" lingo. Some of us don't know what all those technical terms mean, but we do want to understand what they can do!
Would be nice if applications were actually evaluated and a response given
Some very simple examples!
The grid is simply not very useful for the type of computations I do. My jobs involve non-local physics that does not parallelize well to large numbers of processors, and really can only be usefully run on the machines with the fastest interconnects, e.g. DataStar. Even then I am largely latency-limited.
Not sure how TeraGrid services can benefit our particular applications
Giant magical parallel file system, for single home/batch directory.
Reduce the human burden (student, PostDoc, PI) time in setting up and using these services. There needs to be some guarantee of availability and service recommendation for people focusing on science runs and not might-be-next-generation-but-maybe-not computing environments.
I would like to see more application specific grid resources: one would have to port and incorporate grid services into applications such as NAMD, GAMESS, etc.
How can I move data from NCSA UniTree to Caltech faster than 1 Mbyte/sec?
At best I am a grid skeptic. Most of what I have seen seems like a big waste of money that I would rather see go into hardware. Several times I have described my workflow in detail and provided my scripts to a center staff member. Never have I gotten back a script designed with grid commands. I can run distributed jobs quite well between centers using scp and ssh commands. I have generated lattices at one center and moved them to another for analysis, using a simple babysitter on the machine doing the generation of lattices. When it finds a new lattice, it moves it to the other center and submits a batch job there. This was not big deal and it worked just fine. Why is the NSF spending millions of dollars to develop similar capability?
Move to Globus TK 4.0
GPFS was very unreliable in the past. It became better lately, but there is still improvement to reduce downtimes on the machines.
Most importantly, the site that submits a job shouldn't have to run a server. This cuts off anybody that's behind a firewall.
Provide single system image - eliminate multiple passwords, multiple logins, and multiple home directories. Provide globally accessible file systems and archives that can be accessed transparently from every Grid node.
The usage in your webpage needs improvement. I can not find enough guides from the websites.

Programming Environment, Software and Tools

13. What is the nature of the parallelism in your code?

Remarks/Plans: Nearly half of the respondents indicated that nature of parallelism was domain decomposition or task/functional parallelism. (The table is sorted by number of responses in descending order.)

	Number of Responses	Response Ratio
Domain decomposition	90	29%
Task/functional parallelism	60	19%
Commercial software implementation	39	12%
Don't know	35	11%
None	28	9%
Embarrassingly parallel	21	7%
Matrix decomposition	21	7%
Other, please specify	20	6%
TOTAL	314

Other:

MPI-parallelism of multiple serial jobs
Dense matrix linear algebra done by BLAS/LAPACK.
smart MPI load balancing in spectral domain
I do lots of different things...
All of the above
domain, matrix, functional, & other
Task/functional parallelism
use commercial Gaussian code
lattice Boltzmann simulations of two phase flow
matrix & domain decomposition + trivial tasks
independent, serial runs
User other user's code all types: CFD, Ocean model
Several codes, several types.
TCP/IP sockets
enable parallel I/O with MPI-IO or independent I/O
domain, functional, task, and embarrassingly
Combination matrix decomposition + almost embarrassingly parallel
Parallel file system
Charmm - MPI
File I/O
embarrassingly parallel and task/functional parallel

14. If you develop your own codes, how do you parallelize them?

Remarks/Plans: Most users use MPI to parallelize their codes. (The table is sorted by number of responses in descending order.)

	Number of Responses	Response Ratio
MPI	176	61%
None	46	16%
OpenMP and MPI mixed	39	13%
OpenMP	30	10%
Don't know	24	8%
Other, please specify	17	6%
Charm++	14	5%
POSIX Threads	15	5%
Automatic Parallelization	10	3%
Condor	10	3%
HPF	2	1%
PVM	2	1%
Linda	1	0%
POOMA	0	0%

Other:

I did not develop the code
shmem
DAGH
wrappers that call commercial serial apps in parallel
Parallelism is not relevant to what I do.
Java threads
MW
we also employ ScaLAPACK / PBLAS / BLACS
HDF supports parallel I/O with MPI-IO, OpenMP
currently experimenting with cactus (pugh)
BLACS/PBLAS/SCALAPACK
APST
LBNL NERSC BoxLib
Global Arrays
manual pseudo-parallelized
Chombo (sits on top of MPI)
Co-array Fortran and SHMEM

15. Which of the following mathematical software/libraries have you used in the last year on NSF-supported machines?

Remarks/Plans: FFTW, LAPACK, MatLab and IBM ESSL/PESSL are the most popular mathematical software/libraries. (The table is sorted by number of responses in descending order.)

	Number of Responses	Response Ratio
None	98	34%
FFTW	67	23%
LAPACK	66	23%
MatLab	53	18%
IBM ESSL/PESSL	40	14%
Intel MKL	35	12%
GSL (GNU Scientific Library)	31	11%
ATLAS	25	9%
Mathematica	24	8%
ScaLAPACK	22	8%
SGI SCSL (BLAS, LAPACK, FFTs)	19	7%
Don't know	18	6%
Other, Please Specify	18	6%
IMSL (VNI)	13	4%
Metis/ParMETIS	11	4%
NAG	10	3%
PETSc	9	3%
SuperLU	8	3%
(P)ARPACK	3	1%
SPRNG	4	1%
AZTEC	1	0%
DAGH	1	0%

Other:

IDL (3)
ROOT; MYSQL; non-commercial codes: CORSIKA, MMC, AMASIM
ACML
DXML
Gaussian
NAG parallel Fortran with MPI and SMP
NetCDF libraries -- not really mathematical though
BoxLib, Hypre (AMR and multigrid packages)
CXML on Lemieux
Hypre
Compaq Extended Math Library (CXML)
Haven't used NSF supported machines
VTK, ITK, Slicer
Maxima
Charmm has its own
CMS software toolkit

16. Which of the following parallel / performance tools have you used in the last year on NSF-supported machines?

Remarks/Plans: MPI_wtime and TotalView are the most popular tools. Most users do not use parallel/performance tools, perhaps due to the lack of documentation and training. (The table is sorted by number of responses in descending order.)

	Number of Responses	Response Ratio
None	127	47%
MPI_wtime()	46	17%
TotalView	47	17%
Don't know	44	16%
gdbx	27	10%
MPICH MPE Jumpshot/Upshot	20	7%
PAPI	17	6%
dtime()/etime()	14	5%
HPM Toolkit	13	5%
pdbx	7	3%
PerfSuite	7	3%
Xprofiler	9	3%
Other, please specify	7	3%
Vampir	5	2%
DPCL/Dyninst	2	1%
HPCView	3	1%
PMAPI	3	1%
svPABLO	3	1%
Tau	2	1%
VProf	4	1%
KAI Guide/Guideview/Assure	0	0%
MPIP (LLNL)	1	0%
Paradyn	0	0%
PCT/PVT	1	0%

Other:

MPI trace , hpmcount and perf
Projections (charm++)
The Charm++ "projections" profiler
DCPI
Haven't used NSF supported machines
Custom Coarse-Grained Performance Analysis Tool
Own benchmarks running on NAMD

17. Please list the third-party (commercial or public domain) scientific software you use regularly on NSF computational resources.

Remarks/Plans: Other than compilers, the most popular software is in the areas of Chemistry and Molecular and Cellular Biology. (The table is sorted by number of responses in descending order.)

Software	Number of Responses	Response Ratio
Gaussian	30	12%
Charmm	12	5%
Compilers	12	5%
NAMD	12	5%
Amber	10	4%
IDL	10	4%
Abaqus	6	2%
HDF	6	2%
BLAST	4	2%
GADGET	4	2%
NCAR graphics	4	2%
Flash	3	1%
Fluent	3	1%
Gromacs	3	1%
NWChem	3	1%
python	3

Introduction

Table of Contents

Computational Environment

Grid Environment and Services

Programming Environment, Software and Tools