Distributed Systems
By 1988 the Internet had enabled building distributed “virtual” computers, by connecting different architectures of computers, large distributed data stores, visualization devices, and scientific instruments, forming what I termed “Metacomputers” or later “The Grid.” This led me in 1997 to submit an NSF proposal ‘The National Computational Science Alliance, which was funded, creating a nationwide set of campuses linked together by the NSF’s vBNS network. My 2002 OptIPuter NSF grant demonstrated that the wide-area bandwidth could be as fast as the backplane of a cluster, essentially eliminating distance for collaborative Big Data scientific research. This was followed by two NSF grants, Quartzite (2004) and Prism (2013), which demonstrated how to create high-speed “Big Data Freeways” on university campuses. The 2008 NSF GreenLight grant enabled us to research how such systems can be made energy-efficient. In 2015, the NSF funded the Pacific Research Platform (PRP) grant to interconnect several dozen of these campus systems, built on four Regional Optical Networks (RONs: CENIC, PNWGP, FRGP, and MREN), enabling researchers to quickly move data between collaborators labs, supercomputer centers, instruments, and data repositories. The distributed PRP, driven by dozens of science and engineering applications, creates a metacomputer that allows the data to traverse multiple, heterogeneous networks with minimal performance degradation. Machine learning was added to the PRP in 2017 when NSF funded the Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI) grant. In 2018, NSF funded Toward the National Research Platform (TNRP) to extend the PRP across the country by federating with four midwestern/eastern RONs and Internet2. In 2023, the PRP was renamed the National Research Platform, led by SDSC Director Frank Wuerthwein. Beyond the United States, throughout the last decade, networked collaboration with our international partners (Australia, Brazil, Chile, Japan, Korea, Mexico, Netherlands, Singapore, & Taiwan) has been essential for continued innovation.
The Evolution of Bits and Bottlenecks in a Scientific Workflow Trying to Keep up With Technology: Accelerating 4D Image Segmentation Applied to NASA data
Proceedings of the 15th IEEE International eScience Conference (2019).
Workflow-Driven Distributed Machine Learning in CHASE-CI: A Cognitive Hardware and Software Ecosystem Community Infrastructure
Proceedings of the IEEE International Symposium on Parallel and Distributed Processing Symposium Workshops (IPDPSW) (2019).
The Pacific Research Platform: Making High-Speed Networking a Reality for the Scientist
Proceedings of the Practice and Experience on Advanced Research Computing, pp. 1-8. (2018).
An Evolution Toward A Programmable Universe – The New York Times
Using this data, the planetary computer will be able to build a computational model of your body and compare your sensor stream with millions of others. Besides providing early detection of internal changes that could lead to disease, cloud-powered voice-recognition wellness coaches could provide continual personalized support on lifestyle choices, potentially staving off disease
The Emergence Of A Planetary-Scale Collaboratory For Data-Intensive Research
Larry Smarr. In Ian Piumarta and Kim Rose (eds.),"Points of View --- a tribute to Alan Kay", Viewpoints Research Institute. (May 2010).
A High-Performance Campus-Scale Cyberinfrastructure For Effectively Bridging End-User Laboratories To Data-Intensive Sources
Philip Papadopoulos, Larry Smarr. Submitted to the NSF Campus Bridging Network/Data Workshop, April 1, 2010
The OptIPuter And Its Applications – IEEE LEOS
IEEE Lasers and Electro-Optics Society (LEOS) Summer 2009 Topical Meeting on Future Global Networks, July 22, 2009, pp. 151-152 Abstract The NSF-funded OptIPuter project [see www.optiputer.net and special issue of Future Generation Computer Systems (FGCS), Volume 25, Issue 2, February 2009] has been exploring for the last six years how user-controlled 10Gbps dedicated lightpaths (lambdas) […]
Infrastructure for eScience and eLearning in Higher Education
Lazowska, E., Lee, P., Elliott, C., & Smarr, L. Infrastructure for eScience and eLearning in Higher Education. 12/12/2008. Computing Community Consortium.
Special Section: OptIPlanet – The OptIPuter Global Collaboratory
L. Smarr, M. Brown, C. de Laat, Future Generation Computer Systems, Volume 25, pp. 109-113 (2009).
Quartzite: A Campus-Scale Hybrid Networking Infrastructure
P. Papadopoulos, B. Dunne, L. Smarr, J. Ford, S. Fainman, Proceedings of the 2008 Conference on Optical Fiber Communication/National Fiber Optic Engineers Conference, San Diego, CA, 2008, pp. 1-3. (2008).
Riding the Light Towards New Science – Nature Photonics
Smarr, Larry. "Riding the light towards new science." nature photonics 1, no. 3 (2007): 133-135.
Grid Networks: Enabling Grids with Advanced Communication Technology
Foreword by Larry Smarr - John Wiley & Sons, pp. xxv - xxvi. August 2006.
The OptIPuter: High-Performance, QoS-Guaranteed Network Service For Emerging E-Science Applications
N. Taesombut, F. Uyeda, A.A. Chien, L. Smarr, T.A. DeFanti, P. Papadopoulos, J. Leigh, M. Ellisman, J. Orcutt. IEEE Communications, 4, 38- 45 (2006).
IGrid 2005: The Global Lambda Integrated Facility
L. Smarr, T. A. DeFanti, M. D. Brown and C. de Laat, Future Generation Computer Systems, Vol. 22, Elsevier, pp. 849-851 (2006).
The Cyberinfrastructure Backplane: The Jump To Light Speed – Introduction
Larry Smarr and Phil Papadopoulos, Guest Editors, Cyberinfrastructure Technology Watch (CTWatch) Quarterly, vol. 1, #2 May 2005, pps 2-4. (2005).
The OptIPuter, Quartzite, And Starlight Projects: A Campus To Global-Scale Testbed For Optical Technologies Enabling LambdaGrid Computing
Optical Fiber Communication Conference & Exposition and the National Fiber Optic Engineers Conference (OFC/NFOEC) 2005, Anaheim, California, March 6-11, 2005.
A Framework For Middleware Supporting Real-Time Wide-Area Distributed Computing
Proceedings of 10th IEEE International Workshop on Object-Oriented Real-Time Dependable Systems (WORDS 2005) pp. 231-240. Sedona, AZ, February 2-4, 2005.
The OptIPuter
Larry Smarr, Andrew Chien, Tom DeFanti, Jason Leigh, Philip Papadopoulos, Comm. ACM, 46 (11) 58-67 (2003).
The Emergence Of Grid Information Infrastructures
Larry Smarr, Chapter 23 in AAAS Science and Technology Yearbook 2000, editors Albert H. Teich, Stephen D. Nelson, Ceilia McEnaney, Stephen J. Lita. (2000).
Infrastructure For Science Portals – IEEE Internet
From Arpanet to NSFnet, this far-reaching partnership between government, universities and industry planted the seeds that grew into the modern cyberworld. More recently, NCSA Mosaic and NCSA’s Web server software catalyzed the development of the World Wide Web.
Information Technology Research: Investing In Our Future A Report To The President Of The United States
Members of the President's Information Technology Advisory Committee. (1999).
Grids in Context
L. Smarr, Chapter 1 in The Grid: A Blueprint for the New Computing Infrastructure edited by I. Foster and C. Kesselman (MKP) (1998).
Computational Infrastructure: Toward the 21st Century
L. Smarr, CACM 40(11): 28-32 (1997).
A Nationwide Parallel Computing Environment
K. Kennedy, C. F. Bender, J. W. D. Connolly, J. L. Hennessy, M. K. Vernon, L. Smarr, CACM 40(11): 62-72 (1997).
Building a Scalable America
L. Smarr, Lecture Notes in Computer Science, 1082, 35-42 (1996).
Life After Internet: Making Room For New Applications – Building Information Infrastructure
In Building an Information Infrastructure, xx-xx , ed. B. Kahin, (McGraw-Hill: New York), with C. C. Catlett. (1992).
Metacomputing
L. Smarr, C. E. Catlett, Comm. of the ACM, 35(6):44-52 (1992).
Life After Internet: Making Room For New Applications – Information Technology Quarterly
Harvard University Information Technology Quarterly issue on Building an Information Infrastructure, Vol. IX, No. 2, 12-21, with C. C. Catlett. Summer 1991.
From Coffee Cart to Educational Computing Platform – UCSD News
In classic UC San Diego fashion, an overheard conversation at a campus coffee cart has turned into an interdisciplinary project that's making computing-intensive coursework more exciting while saving well over one million dollars so far.
SC21: Larry Smarr on The Rise of Supernetwork Data Intensive Computing – HPC Wire
November 26, 2021 - At SC21 last week, Smarr delivered a racing reprise of this (ongoing) HPC tour de force in his talk, The Rise of Supernetwork Data Intensive Computing. Presented here are a sampling of his comments (lightly edited) and quite a few of his slides.
NSF Project Sets Up First Machine Learning Cyberinfrastructure – HPC Wire
July 25, 2017 - Earlier this month, the National Science Foundation issued a $1 million grant to Larry Smarr, director of Calit2, and a group of his colleagues to create a community infrastructure in support of machine learning research.
Larry Smarr Talks Machine Intelligence at Jackson State – HPC Wire
JACKSON, Miss., April 13, 2017 — Dr. Larry Smarr, physicist and Big Data thought leader, practices what he preaches.
Larry Smarr on Solving the Big Data Challenge in HPC – HPC Wire
September 25, 2015 - Internet pioneer Larry Smarr once had a vision of bringing connected computers out of academia and into the consumer world. Today, he envisions a second virtual highway, one capable of delivering on the promise of big data by leveraging fiber optic networks to transmit data at speeds of 10 gigabits to 100 gigabits per second.
NSF Gives Green Light to Pacific Research Platform – HPC Wire
BERKELEY and LA MIRADA, Calif., Aug. 4 — For the last three years, the National Science Foundation (NSF) has made a series of competitive grants to over 100 U.S. universities to aggressively upgrade their campus network capacity for greatly enhanced science data access.
A Big Data Freeway for Scientists – New York Times
The University of California, San Diego, this week plans to announce that it has installed an advanced optical computer network that is intended to serve as a “Big Data freeway system” for next-generation science projects in fields including genomic sequencing, climate science, electron microscopy, oceanography and physics.
One Cloud to Rule Them All – HPC Wire
December 6, 2011 - Smarr’s theory begins with a simple premise: “Over the next 10 years, the physical world will become ever more overlaid with devices for sending and receiving information.”
Optical Network Is Key to Next-Generation Research Cyberinfrastructure – HPC Wire
June 11, 2008 - At TeraGrid ’08 Conference, UC San Diego’s Smarr urges university campuses to remove network bottlenecks to supercomputer users
The OptIPuter Gets Real – HPC Wire
January 27, 2006 - Last week, the UCSD division of the California Institute for Telecommunications and Information Technology (Calit2) and the J. Craig Venter Institute announced that they would collaborate to decipher the genetic code of the world's marine microbiological communities.
Smarr Touts Optical Networks As Nation’s Future – HPC Wire
November 11, 2004 - Smarr emphasizes the importance of networking to U.S. competitiveness and elaborates on his current projects and activities.
The “OPTIPUTER”: A New Paradigm for Data Computing – HPC Wire
The National Science Foundation (NSF) awarded $13.5 million over five years to a consortium led by the University of California, San Diego (UCSD), and the University of Illinois at Chicago (UIC).
The Soul of the Ultimate Machine – New York Times
The Internet, he explained, is evolving into a single vast computer fashioned out of billions of interconnected processors. Then he went another step: ''The real question, from a software point of view, is: Will it become self-aware?''