Distributed Systems
By 1988 the Internet had enabled building distributed “virtual” computers, by connecting different architectures of computers, large distributed data stores, visualization devices, and scientific instruments, forming what I termed “Metacomputers” or later “The Grid.” This led me in 1997 to submit an NSF proposal ‘The National Computational Science Alliance, which was funded, creating a nationwide set of campuses linked together by the NSF’s vBNS network. My 2002 OptIPuter NSF grant demonstrated that the wide-area bandwidth could be as fast as the backplane of a cluster, essentially eliminating distance for collaborative Big Data scientific research. This was followed by two NSF grants, Quartzite (2004) and Prism (2013), which demonstrated how to create high-speed “Big Data Freeways” on university campuses. The 2008 NSF GreenLight grant enabled us to research how such systems can be made energy-efficient. In 2015, the NSF funded the Pacific Research Platform (PRP) grant to interconnect several dozen of these campus systems, built on four Regional Optical Networks (RONs: CENIC, PNWGP, FRGP, and MREN), enabling researchers to quickly move data between collaborators labs, supercomputer centers, instruments, and data repositories. The distributed PRP, driven by dozens of science and engineering applications, creates a metacomputer that allows the data to traverse multiple, heterogeneous networks with minimal performance degradation. Machine learning was added to the PRP in 2017 when NSF funded the Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI) grant. In 2018, NSF funded Toward the National Research Platform (TNRP) to extend the PRP across the country by federating with four midwestern/eastern RONs and Internet2. In 2023, the PRP was renamed the National Research Platform, led by SDSC Director Frank Wuerthwein. Beyond the United States, throughout the last decade, networked collaboration with our international partners (Australia, Brazil, Chile, Japan, Korea, Mexico, Netherlands, Singapore, & Taiwan) has been essential for continued innovation.
The Evolution of Bits and Bottlenecks in a Scientific Workflow Trying to Keep up With Technology: Accelerating 4D Image Segmentation Applied to NASA data
Proceedings of the 15th IEEE International eScience Conference (2019).
Workflow-Driven Distributed Machine Learning in CHASE-CI: A Cognitive Hardware and Software Ecosystem Community Infrastructure
Proceedings of the IEEE International Symposium on Parallel and Distributed Processing Symposium Workshops (IPDPSW) (2019).
The Pacific Research Platform: Making High-Speed Networking a Reality for the Scientist
Proceedings of the Practice and Experience on Advanced Research Computing, pp. 1-8. (2018).
Infrastructure for eScience and eLearning in Higher Education
Lazowska, E., Lee, P., Elliott, C., & Smarr, L. Infrastructure for eScience and eLearning in Higher Education. 12/12/2008. Computing Community Consortium.
Quartzite: A Campus-Scale Hybrid Networking Infrastructure
P. Papadopoulos, B. Dunne, L. Smarr, J. Ford, S. Fainman, Proceedings of the 2008 Conference on Optical Fiber Communication/National Fiber Optic Engineers Conference, San Diego, CA, 2008, pp. 1-3. (2008).
Grid Networks: Enabling Grids with Advanced Communication Technology
Foreword by Larry Smarr - John Wiley & Sons, pp. xxv - xxvi. August 2006.
IGrid 2005: The Global Lambda Integrated Facility
L. Smarr, T. A. DeFanti, M. D. Brown and C. de Laat, Future Generation Computer Systems, Vol. 22, Elsevier, pp. 849-851 (2006).
A Framework For Middleware Supporting Real-Time Wide-Area Distributed Computing
Proceedings of 10th IEEE International Workshop on Object-Oriented Real-Time Dependable Systems (WORDS 2005) pp. 231-240. Sedona, AZ, February 2-4, 2005.
Grids in Context
L. Smarr, Chapter 1 in The Grid: A Blueprint for the New Computing Infrastructure edited by I. Foster and C. Kesselman (MKP) (1998).
Computational Infrastructure: Toward the 21st Century
L. Smarr, CACM 40(11): 28-32 (1997).
Building a Scalable America
L. Smarr, Lecture Notes in Computer Science, 1082, 35-42 (1996).
Metacomputing
L. Smarr, C. E. Catlett, Comm. of the ACM, 35(6):44-52 (1992).
NCSA Welcomes Back Larry Smarr to Share Insights on Nationwide Cyberinfrastructure Development – HPC Wire
May 3, 2024 — High-Performance Computing (HPC) luminary and NCSA founding director Larry Smarr will be returning to the Center to give a talk titled “From NCSA to the National Research Platform” on May 9 at 12 p.m. in the NCSA auditorium on the first floor.
From Coffee Cart to Educational Computing Platform – UCSD News
In classic UC San Diego fashion, an overheard conversation at a campus coffee cart has turned into an interdisciplinary project that's making computing-intensive coursework more exciting while saving well over one million dollars so far.
SC21: Larry Smarr on The Rise of Supernetwork Data Intensive Computing – HPC Wire
November 26, 2021 - At SC21 last week, Smarr delivered a racing reprise of this (ongoing) HPC tour de force in his talk, The Rise of Supernetwork Data Intensive Computing. Presented here are a sampling of his comments (lightly edited) and quite a few of his slides.
NSF Project Sets Up First Machine Learning Cyberinfrastructure – HPC Wire
July 25, 2017 - Earlier this month, the National Science Foundation issued a $1 million grant to Larry Smarr, director of Calit2, and a group of his colleagues to create a community infrastructure in support of machine learning research.
Larry Smarr Talks Machine Intelligence at Jackson State – HPC Wire
JACKSON, Miss., April 13, 2017 — Dr. Larry Smarr, physicist and Big Data thought leader, practices what he preaches.
Larry Smarr on Solving the Big Data Challenge in HPC – HPC Wire
September 25, 2015 - Internet pioneer Larry Smarr once had a vision of bringing connected computers out of academia and into the consumer world. Today, he envisions a second virtual highway, one capable of delivering on the promise of big data by leveraging fiber optic networks to transmit data at speeds of 10 gigabits to 100 gigabits per second.
NSF Gives Green Light to Pacific Research Platform – HPC Wire
BERKELEY and LA MIRADA, Calif., Aug. 4 — For the last three years, the National Science Foundation (NSF) has made a series of competitive grants to over 100 U.S. universities to aggressively upgrade their campus network capacity for greatly enhanced science data access.
A Big Data Freeway for Scientists – New York Times
The University of California, San Diego, this week plans to announce that it has installed an advanced optical computer network that is intended to serve as a “Big Data freeway system” for next-generation science projects in fields including genomic sequencing, climate science, electron microscopy, oceanography and physics.
One Cloud to Rule Them All – HPC Wire
December 6, 2011 - Smarr’s theory begins with a simple premise: “Over the next 10 years, the physical world will become ever more overlaid with devices for sending and receiving information.”
Optical Network Is Key to Next-Generation Research Cyberinfrastructure – HPC Wire
June 11, 2008 - At TeraGrid ’08 Conference, UC San Diego’s Smarr urges university campuses to remove network bottlenecks to supercomputer users
The OptIPuter Gets Real – HPC Wire
January 27, 2006 - Last week, the UCSD division of the California Institute for Telecommunications and Information Technology (Calit2) and the J. Craig Venter Institute announced that they would collaborate to decipher the genetic code of the world's marine microbiological communities.
Smarr Touts Optical Networks As Nation’s Future – HPC Wire
November 11, 2004 - Smarr emphasizes the importance of networking to U.S. competitiveness and elaborates on his current projects and activities.