MPI:A Message-Passing Interface Standard Version 3.1 Message Passing Interface Forum June4,2015
MPI: A Message-Passing Interface Standard Version 3.1 Message Passing Interface Forum June 4, 2015
This document describes the Message-Passing Interface(MPI)standard,version 3.1. The MPI standard includes point-to-point message-passing,collective communications,group and communicator concepts,process topologies,environmental management,process cre ation and management one-sided communications extended collective onerations external interfaces,I/O,some miscellaneous topics,and a profiling interface.Language bindings for C and Fortran are defined. Historically,the evolution of the standards is from MPI-1.0 (May 5,1994)to MPI-1.1 (June 12.1995)to MPl-1.2 (July 18.1997).with several clarifications and additions and published as part of the MPI-2 document,to MPI-2.0(July 18,1997),with new functionality. 10 to MPI-1.3 (May 30,2008),combining for historical reasons the documents 1.1 and 1.2 11 and some errata documents to one combined document.and to MPl-2.1(June 23.2008) 12 combining the previous documents.Version MPI-2.2(September 4,2009)added additional 13 clarifcation s and seven new routines.Version MPI-3.0(Se otember 21.2012)is an extension 14 of MPI-2.2.This version.MPI-3.1.adds clarifications and minor extensions to MPI-3.0 15 Comments.Please send comments on MPI to the MPI Forum as follows 1.Subscribe to http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-comments 19 、一2 Send yoMngein oetherith the R.of the mbers on which you are Your comment will be forwarded to MPl Forum committee members for consideration. Messages sent from an unsubscribed e-mail address will not be considered. 25 29 30 31 32 之 9 A0 9193,19041995,196,197,2008,20,2012,2015 University of Te,Knoxvil D on to copy wit of th the Univ of Ter notice is given that copying is by permission of the University of Tennessee ument appear,and
This document describes the Message-Passing Interface (MPI) standard, version 3.1. The MPI standard includes point-to-point message-passing, collective communications, group and communicator concepts, process topologies, environmental management, process creation and management, one-sided communications, extended collective operations, external interfaces, I/O, some miscellaneous topics, and a profiling interface. Language bindings for C and Fortran are defined. Historically, the evolution of the standards is from MPI-1.0 (May 5, 1994) to MPI-1.1 (June 12, 1995) to MPI-1.2 (July 18, 1997), with several clarifications and additions and published as part of the MPI-2 document, to MPI-2.0 (July 18, 1997), with new functionality, to MPI-1.3 (May 30, 2008), combining for historical reasons the documents 1.1 and 1.2 and some errata documents to one combined document, and to MPI-2.1 (June 23, 2008), combining the previous documents. Version MPI-2.2 (September 4, 2009) added additional clarifications and seven new routines. Version MPI-3.0 (September 21, 2012) is an extension of MPI-2.2. This version, MPI-3.1, adds clarifications and minor extensions to MPI-3.0 Comments. Please send comments on MPI to the MPI Forum as follows: 1. Subscribe to http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-comments 2. Send your comment to: mpi-comments@mpi-forum.org, together with the URL of the version of the MPI standard and the page and line numbers on which you are commenting. Only use the official versions. Your comment will be forwarded to MPI Forum committee members for consideration. Messages sent from an unsubscribed e-mail address will not be considered. c 1993, 1994, 1995, 1996, 1997, 2008, 2009, 2012, 2015 University of Tennessee, Knoxville, Tennessee. Permission to copy without fee all or part of this material is granted, provided the University of Tennessee copyright notice and the title of this document appear, and notice is given that copying is by permission of the University of Tennessee. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 ii
Version 3.1:June 4,2015.This document contains mostly corrections and clarifications to the MPI-3.0 do t The la tothe 2 routines to pulate MPI Aint 3 nonblocking collective I/O routines,and routines to get the me for MPIT perfor mance and 5 Version 3.0:September 21.2012. p时Tne Coincident with 7 Forum began dis 8 nent contains the Mh Standard 10 new one-sided communication operations. 11 nd上 e to 13 the MPI standard.As with previous versions,new features have been adopted only when 13 there were compelling needs for the users.Some features,however,may have more than a 14 minor impact on existing MPl implementations 15 16 Version 2.2:September 4.2009.This document contains mostly corrections and clarifi- 17 cations to the mPl21 document.A few extensions have been added:however all correct MPI-2.1 programs are correct MPI-2.2 programs.New features were adopted only when there were compelling needs for users,open source implementations,and minor impact on existing MPl implementations. Version 2.1:June 23,2008. This do ent combines the ous docu ents MPl-1.3(May 30.2008)and MP1-2.0( July 18.1997).Certain MPL-2.0, e section 4 Mis d .Ext of MP-1.Additio ons Forum are cluded in this docume nt Version 1.3:May 30,2008. This document combines the previous documents MPl-1.1 (June 12,1995)and the MP1-1.2 Chapter in MP1-2 (July 18,1997).Additional errata collected by the MPI Forum referring to MPI-1.1 and MPI-1.2 are also included in this document. Version 2.0:July 18,1997.Beginning after the release of MPI-1.1,the MPI Forum began meeting to consider corrections and extensions.MPI-2 has been focused on process creation and management,one-sided communications,extended collective communications,external interfaces and parallel I/O.A miscellany chapter discusses items that do not fit elsewhere, in particular language interoperability. Version 12 luly 18 1997 The MPl-2 For introduced MPI-1.2 as Chapter 3 in the standard"MPI-2:Exter ons to the Me g Interface" contains clarificatio tion s to v ctio in MPI-1.2 i ne for ider all diftow hic the confo T MPI-1 and MPL-1 1 The MP-MP-1.teM- 46 47 48
Version 3.1: June 4, 2015. This document contains mostly corrections and clarifications to the MPI-3.0 document. The largest change is a correction to the Fortran bindings introduced in MPI-3.0. Additionally, new functions added include routines to manipulate MPI_Aint values in a portable manner, nonblocking collective I/O routines, and routines to get the index value by name for MPI_T performance and control variables. Version 3.0: September 21, 2012. Coincident with the development of MPI-2.2, the MPI Forum began discussions of a major extension to MPI. This document contains the MPI-3 Standard. This draft version of the MPI-3 standard contains significant extensions to MPI functionality, including nonblocking collectives, new one-sided communication operations, and Fortran 2008 bindings. Unlike MPI-2.2, this standard is considered a major update to the MPI standard. As with previous versions, new features have been adopted only when there were compelling needs for the users. Some features, however, may have more than a minor impact on existing MPI implementations. Version 2.2: September 4, 2009. This document contains mostly corrections and clarifi- cations to the MPI-2.1 document. A few extensions have been added; however all correct MPI-2.1 programs are correct MPI-2.2 programs. New features were adopted only when there were compelling needs for users, open source implementations, and minor impact on existing MPI implementations. Version 2.1: June 23, 2008. This document combines the previous documents MPI-1.3 (May 30, 2008) and MPI-2.0 (July 18, 1997). Certain parts of MPI-2.0, such as some sections of Chapter 4, Miscellany, and Chapter 7, Extended Collective Operations, have been merged into the Chapters of MPI-1.3. Additional errata and clarifications collected by the MPI Forum are also included in this document. Version 1.3: May 30, 2008. This document combines the previous documents MPI-1.1 (June 12, 1995) and the MPI-1.2 Chapter in MPI-2 (July 18, 1997). Additional errata collected by the MPI Forum referring to MPI-1.1 and MPI-1.2 are also included in this document. Version 2.0: July 18, 1997. Beginning after the release of MPI-1.1, the MPI Forum began meeting to consider corrections and extensions. MPI-2 has been focused on process creation and management, one-sided communications, extended collective communications, external interfaces and parallel I/O. A miscellany chapter discusses items that do not fit elsewhere, in particular language interoperability. Version 1.2: July 18, 1997. The MPI-2 Forum introduced MPI-1.2 as Chapter 3 in the standard “MPI-2: Extensions to the Message-Passing Interface”, July 18, 1997. This section contains clarifications and minor corrections to Version 1.1 of the MPI Standard. The only new function in MPI-1.2 is one for identifying to which version of the MPI Standard the implementation conforms. There are small differences between MPI-1 and MPI-1.1. There are very few differences between MPI-1.1 and MPI-1.2, but large differences between MPI-1.2 and MPI-2. iii 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
Version 1.1:June,1995.Beginning in March,1995,the Message-Passing Interface Forum reconvened to correct errors and make clarifications in the MPI document of May 5,1994 referred to below as Version 1.0.These discussions resulted in Version 1.1.The changes from Version 1.0 are minor.A version of this document with all changes marked is available Version 1.0:May,1994.The Message-Passing Interface Forum (MPIF),with participa essage passing any official standards 10 essage-Pas sing Interface, 12 ical,p rtable. sis the fina report, c of the M 15 19 22 32
Version 1.1: June, 1995. Beginning in March, 1995, the Message-Passing Interface Forum reconvened to correct errors and make clarifications in the MPI document of May 5, 1994, referred to below as Version 1.0. These discussions resulted in Version 1.1. The changes from Version 1.0 are minor. A version of this document with all changes marked is available. Version 1.0: May, 1994. The Message-Passing Interface Forum (MPIF), with participation from over 40 organizations, has been meeting since January 1993 to discuss and define a set of library interface standards for message passing. MPIF is not sanctioned or supported by any official standards organization. The goal of the Message-Passing Interface, simply stated, is to develop a widely used standard for writing message-passing programs. As such the interface should establish a practical, portable, efficient, and flexible standard for message-passing. This is the final report, Version 1.0, of the Message-Passing Interface Forum. This document contains all the technical features proposed for the interface. This copy of the draft was processed by LATEX on May 5, 1994. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 iv
Contents Acknowledgments Introduction to MPl 11 1 1.2 1.3 nd of MPI-1.1.MPL-1.2.and MPI-2.0 1.4 Background of MPI-1.3 and MPI-2.1 Ba kground of MPL2 d of MPL-3.0 This Standa rnPplaneutation Not In 22344455566 ded In Th ndar ? of this Document 2 MPI Terms and Conventions 9 2.1 Document Notation 9 2.2 Naming Conventions 9 2. Procedure Specification 2.4 Semantic Terms 01 2.5 Data Types 2.51 Opaque Objects 959 Arguments 12124 2.5.3 2.5.4 Named Constants 2.5.5 Choice 2.5.6 Absolute Addresses and Relative Address Displacements 057 File Offsets 2.5.8 Counts 2.6 age Bindin and ren oved Nam 2.6.3 C Binding Iss 2.6.4 ion nd Macros 4500077799000 Error Han dimg
Contents Acknowledgments ix 1 Introduction to MPI 1 1.1 Overview and Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Background of MPI-1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Background of MPI-1.1, MPI-1.2, and MPI-2.0 . . . . . . . . . . . . . . . . . 2 1.4 Background of MPI-1.3 and MPI-2.1 . . . . . . . . . . . . . . . . . . . . . . 3 1.5 Background of MPI-2.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.6 Background of MPI-3.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.7 Background of MPI-3.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.8 Who Should Use This Standard? . . . . . . . . . . . . . . . . . . . . . . . . 5 1.9 What Platforms Are Targets For Implementation? . . . . . . . . . . . . . . 5 1.10 What Is Included In The Standard? . . . . . . . . . . . . . . . . . . . . . . 5 1.11 What Is Not Included In The Standard? . . . . . . . . . . . . . . . . . . . . 6 1.12 Organization of this Document . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 MPI Terms and Conventions 9 2.1 Document Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Naming Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3 Procedure Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.4 Semantic Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.5 Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.1 Opaque Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.2 Array Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.5.3 State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.5.4 Named Constants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.5.5 Choice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.5.6 Absolute Addresses and Relative Address Displacements . . . . . . . 16 2.5.7 File Offsets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.5.8 Counts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.6 Language Binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.6.1 Deprecated and Removed Names and Functions . . . . . . . . . . . . 17 2.6.2 Fortran Binding Issues . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.6.3 C Binding Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.6.4 Functions and Macros . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.7 Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.8 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 v
2.9 Implementation Issues 21 2.9.1 Independence of Basic Runtime Routines 2.9.2 Interaction with Signals.. 2.10 Examples 3 3 Point-to-Point Communication 3.2 Blocking Send and Receive Operations..................... 3.2.1 Blocking Send............................. 3.22 3.2.3 Message Envelope...· 2844578 3.2.4 Blocking Receive 3.2.5 Keturn Status 3.2.6 Passing MPI_STATUS_IGNORE for Status.. 。。。，。。。。。。，，。 3.3 Data Type Matching and Data Conversion. 3.3.1 Type Matching Rules Type MPI_CHARACTER. 3.3.2 Data Conversion·· 45 3.4 Communication Modes 3.5 Semantics of Point-to-Point Communication 3.6 Buffer Allocation and Usage 3.6.1 Model Implementation of Buffered mode 3.7 Nonblocking Communication 3.7.1 Communication request obiects 3.7.2 Communication Initiation 3.7.3 Communication Completion 82 3.7.4 Semantics of Nonblocking Communications 3.75 Multiple Completions 3.7.6 Non-destructive Test of status 3.8 Probe and Cancel. 38 1 Probe 3.8.2 Matching Probe 3.8.3 Matched Receives 3.8.4 Cancel 3.9 Persistent Communication Requests 3 10 Send Receive 3.11 Null Processes 80 4 Datatypes 好 4.1 Derived Datatypes 4.1.1 Type Constructors with Explicit Addresses.... ，，，。，，。。 4.1.2 Datatype Constructors.. 855 4.1.3 Subarray Datatype Constructor ... 4.1.4 Distributed Array Datatype Constructor. 4.1.5 Address and Size Functions.. 101 4.1.6 Lower-Bound and Upper-Bound Markers 104 4.1.7 Extent and Bounds of Datatypes 106 4.1.8 True Extent of Datatypes 108
2.9 Implementation Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.9.1 Independence of Basic Runtime Routines . . . . . . . . . . . . . . . 22 2.9.2 Interaction with Signals . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.10 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3 Point-to-Point Communication 23 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.2 Blocking Send and Receive Operations . . . . . . . . . . . . . . . . . . . . . 24 3.2.1 Blocking Send . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.2.2 Message Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2.3 Message Envelope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.2.4 Blocking Receive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.2.5 Return Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.2.6 Passing MPI_STATUS_IGNORE for Status . . . . . . . . . . . . . . . . 32 3.3 Data Type Matching and Data Conversion . . . . . . . . . . . . . . . . . . 33 3.3.1 Type Matching Rules . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Type MPI_CHARACTER . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.3.2 Data Conversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.4 Communication Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.5 Semantics of Point-to-Point Communication . . . . . . . . . . . . . . . . . . 40 3.6 Buffer Allocation and Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.6.1 Model Implementation of Buffered Mode . . . . . . . . . . . . . . . . 46 3.7 Nonblocking Communication . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.7.1 Communication Request Objects . . . . . . . . . . . . . . . . . . . . 48 3.7.2 Communication Initiation . . . . . . . . . . . . . . . . . . . . . . . . 48 3.7.3 Communication Completion . . . . . . . . . . . . . . . . . . . . . . . 52 3.7.4 Semantics of Nonblocking Communications . . . . . . . . . . . . . . 56 3.7.5 Multiple Completions . . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.7.6 Non-destructive Test of status . . . . . . . . . . . . . . . . . . . . . . 63 3.8 Probe and Cancel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.8.1 Probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.8.2 Matching Probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 3.8.3 Matched Receives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 3.8.4 Cancel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 3.9 Persistent Communication Requests . . . . . . . . . . . . . . . . . . . . . . 73 3.10 Send-Receive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 3.11 Null Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4 Datatypes 83 4.1 Derived Datatypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.1.1 Type Constructors with Explicit Addresses . . . . . . . . . . . . . . 85 4.1.2 Datatype Constructors . . . . . . . . . . . . . . . . . . . . . . . . . . 85 4.1.3 Subarray Datatype Constructor . . . . . . . . . . . . . . . . . . . . . 94 4.1.4 Distributed Array Datatype Constructor . . . . . . . . . . . . . . . . 96 4.1.5 Address and Size Functions . . . . . . . . . . . . . . . . . . . . . . . 101 4.1.6 Lower-Bound and Upper-Bound Markers . . . . . . . . . . . . . . . 104 4.1.7 Extent and Bounds of Datatypes . . . . . . . . . . . . . . . . . . . . 106 4.1.8 True Extent of Datatypes . . . . . . . . . . . . . . . . . . . . . . . . 108 vi
4.1.9 Commit and Free 109 111 ral Data 1 4.1.12 Correct Use of Addr 1 4 1 13 Dec odin 4.1.14Ex Canonical MPI_PACK and MPI_UNPACK 138 5 Collective Communication 141 51 Introduction and Overview 141 5.2 Communicator Argument 144 521 Specifics for Intracommunicator Collective Operations 144 52.2 Applying Collective Operations to Intercommunicators 145 5.23 ecifics for Intercommunicator Collective Operations 146 5.3 Barrier synchronization 147 5.4 Broadcast 4 5.4.1 Example using MPI_BCAST 5.5 Gather 14 5.5.1 Examples using MPl GATtHER.MPI GATHERV 5.6 Scatter 15 5.6.1 Examples using MPl sCatTER.MPI SCATTERV 16 5.7 Gather-to-all 5.7.1 Example using MPI_ALLGATHER 5.8 All-to-All Scatter/Gather 5.9 Global Reduction Operations 173 17 500 Predefined reduction on 170 502 nd Reductions 17 504 MINLOC and MAxlOC 170 595 Use r-Defined Reduction Op 1Q e of User-defined Reduce 18 506 All-Roduc 187 ocal Reduction 5.10 Reduco 5.10.1 MPI REDUCE SCATTER BLOCK 190 5.10.2 MPI_REDUCE_SCATTER 191 5.11Sc 5..1nc 193 5.112 xclusive 990 飞T11.1 MPI_IBCAST 5.123 g 20 5.12.5 Nonbloc king Gather-to-all
4.1.9 Commit and Free . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 4.1.10 Duplicating a Datatype . . . . . . . . . . . . . . . . . . . . . . . . . 111 4.1.11 Use of General Datatypes in Communication . . . . . . . . . . . . . 111 4.1.12 Correct Use of Addresses . . . . . . . . . . . . . . . . . . . . . . . . 115 4.1.13 Decoding a Datatype . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 4.1.14 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 4.2 Pack and Unpack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 4.3 Canonical MPI_PACK and MPI_UNPACK . . . . . . . . . . . . . . . . . . . 138 5 Collective Communication 141 5.1 Introduction and Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 5.2 Communicator Argument . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 5.2.1 Specifics for Intracommunicator Collective Operations . . . . . . . . 144 5.2.2 Applying Collective Operations to Intercommunicators . . . . . . . . 145 5.2.3 Specifics for Intercommunicator Collective Operations . . . . . . . . 146 5.3 Barrier Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 5.4 Broadcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 5.4.1 Example using MPI_BCAST . . . . . . . . . . . . . . . . . . . . . . . 149 5.5 Gather . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 5.5.1 Examples using MPI_GATHER, MPI_GATHERV . . . . . . . . . . . . 152 5.6 Scatter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 5.6.1 Examples using MPI_SCATTER, MPI_SCATTERV . . . . . . . . . . 162 5.7 Gather-to-all . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 5.7.1 Example using MPI_ALLGATHER . . . . . . . . . . . . . . . . . . . . 167 5.8 All-to-All Scatter/Gather . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 5.9 Global Reduction Operations . . . . . . . . . . . . . . . . . . . . . . . . . . 173 5.9.1 Reduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 5.9.2 Predefined Reduction Operations . . . . . . . . . . . . . . . . . . . . 176 5.9.3 Signed Characters and Reductions . . . . . . . . . . . . . . . . . . . 178 5.9.4 MINLOC and MAXLOC . . . . . . . . . . . . . . . . . . . . . . . . 179 5.9.5 User-Defined Reduction Operations . . . . . . . . . . . . . . . . . . 183 Example of User-defined Reduce . . . . . . . . . . . . . . . . . . . . 186 5.9.6 All-Reduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 5.9.7 Process-Local Reduction . . . . . . . . . . . . . . . . . . . . . . . . . 189 5.10 Reduce-Scatter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 5.10.1 MPI_REDUCE_SCATTER_BLOCK . . . . . . . . . . . . . . . . . . . 190 5.10.2 MPI_REDUCE_SCATTER . . . . . . . . . . . . . . . . . . . . . . . . 191 5.11 Scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 5.11.1 Inclusive Scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 5.11.2 Exclusive Scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 5.11.3 Example using MPI_SCAN . . . . . . . . . . . . . . . . . . . . . . . . 195 5.12 Nonblocking Collective Operations . . . . . . . . . . . . . . . . . . . . . . . 196 5.12.1 Nonblocking Barrier Synchronization . . . . . . . . . . . . . . . . . . 198 5.12.2 Nonblocking Broadcast . . . . . . . . . . . . . . . . . . . . . . . . . 199 Example using MPI_IBCAST . . . . . . . . . . . . . . . . . . . . . . 199 5.12.3 Nonblocking Gather . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 5.12.4 Nonblocking Scatter . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 5.12.5 Nonblocking Gather-to-all . . . . . . . . . . . . . . . . . . . . . . . . 204 vii
5.12.6 Nonblocking All-to-All Scatter/Gather... 200 5.12.7 Nonblocking Reduce 209 5.12 8 Nonblocking AllReduce 210 5.12.9 Nonblocking Reduce-Scatter with Equal Blocks 211 5 12 10Nonblocking Reduce Scatter 212 5 12 11 Nonblocking Inclusive Scan 213 5.12.12 Nonblocking Exclusive Scan 214 5.13 Correctness 214 6 Groups,Contexts,Communicators,and Caching 6.1 Introduction.· 6.1.1 Features Needed to Support Libraries 6.1.2 MPI's Support for Libraries 62 Basic Concepts................ 6.2.1 Groups······· 226 6.2.2 Contexts 226 6.2.3 Intra-Communicators 227 6.2.4 Predefined Intra-Communicators 227 6.3 Group Management·· 228 6.31 Grou Accessors 228 6.3.2 Group Constructors 230 633 Groun Destructors 235 6.4 Communicator Management 235 6.4.1 Communicator accessors 235 6.42 Communicator Constructors 237 6.43 Communicator Destructors 248 6.4.4 Communicator Info. 248 6.5 Motivating Examples. 250 6.5.1 Current practice 25 6.5.2 Current Practice #2 6.5.3 (Approximate)Current Practice #3 251 6.5.4 Example #4 252 6.5.5 Library Example #1 253 6.5.6 Library Example #2 255 6.6 Inter-Communication 661 Inter-communicator Accessors 6.6.2 Inter-communicator Operations 260 6.6.3 Inter-Communication Ex Example 1:Three-Gro Example 2:Three-Gro p“Ring 264 6.7 265 6.7.1 Functionality 266 6.7.2 267 6.7.3 Windows 272 6.7.4 Dataty 275 6.7.5 Class for Invalid Keyval 279 6.7.6 279 6.8 Naming 2 viii
5.12.6 Nonblocking All-to-All Scatter/Gather . . . . . . . . . . . . . . . . . 206 5.12.7 Nonblocking Reduce . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 5.12.8 Nonblocking All-Reduce . . . . . . . . . . . . . . . . . . . . . . . . . 210 5.12.9 Nonblocking Reduce-Scatter with Equal Blocks . . . . . . . . . . . . 211 5.12.10 Nonblocking Reduce-Scatter . . . . . . . . . . . . . . . . . . . . . . . 212 5.12.11 Nonblocking Inclusive Scan . . . . . . . . . . . . . . . . . . . . . . . 213 5.12.12 Nonblocking Exclusive Scan . . . . . . . . . . . . . . . . . . . . . . . 214 5.13 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 6 Groups, Contexts, Communicators, and Caching 223 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 6.1.1 Features Needed to Support Libraries . . . . . . . . . . . . . . . . . 223 6.1.2 MPI’s Support for Libraries . . . . . . . . . . . . . . . . . . . . . . . 224 6.2 Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 6.2.1 Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 6.2.2 Contexts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 6.2.3 Intra-Communicators . . . . . . . . . . . . . . . . . . . . . . . . . . 227 6.2.4 Predefined Intra-Communicators . . . . . . . . . . . . . . . . . . . . 227 6.3 Group Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 6.3.1 Group Accessors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 6.3.2 Group Constructors . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 6.3.3 Group Destructors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 6.4 Communicator Management . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 6.4.1 Communicator Accessors . . . . . . . . . . . . . . . . . . . . . . . . 235 6.4.2 Communicator Constructors . . . . . . . . . . . . . . . . . . . . . . . 237 6.4.3 Communicator Destructors . . . . . . . . . . . . . . . . . . . . . . . 248 6.4.4 Communicator Info . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 6.5 Motivating Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 6.5.1 Current Practice #1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 6.5.2 Current Practice #2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 6.5.3 (Approximate) Current Practice #3 . . . . . . . . . . . . . . . . . . 251 6.5.4 Example #4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 6.5.5 Library Example #1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 6.5.6 Library Example #2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 6.6 Inter-Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 6.6.1 Inter-communicator Accessors . . . . . . . . . . . . . . . . . . . . . . 259 6.6.2 Inter-communicator Operations . . . . . . . . . . . . . . . . . . . . . 260 6.6.3 Inter-Communication Examples . . . . . . . . . . . . . . . . . . . . . 263 Example 1: Three-Group “Pipeline” . . . . . . . . . . . . . . . . . . 263 Example 2: Three-Group “Ring” . . . . . . . . . . . . . . . . . . . . 264 6.7 Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 6.7.1 Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266 6.7.2 Communicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 6.7.3 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 6.7.4 Datatypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 6.7.5 Error Class for Invalid Keyval . . . . . . . . . . . . . . . . . . . . . . 279 6.7.6 Attributes Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 6.8 Naming Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 viii
6.9 Formalizing the Loosely Synchronous Model 285 6 9 1 Basic Statem 285 6.9.2 Models of Execution 286 Static Comn nicator Allocation 286 unicator Allocation The General Case 7 Process Topologies 289 7.1 Introduction 289 72 Virtual Topologies 290 7.3 Embedding in MPl 290 290 75 Topology Constructors 292 751 Cartesian Constructor 292 75.2 Cartesian Convenience Function:MPI_DIMS_CREATE 292 753 Graph Constructor 294 754 Distributed Graph Constructor 296 755 Topology inguiry functions 30 75 6 Cartesian Shift Coordinates 310 7.5.7 Partitioning of cartesian structures 311 7.5.8 Low-Level Topology Functions 219 7.6 Neighborhood Collective Communication 4 7.6.1 Neighborhood Gather 5 760 Neighbor Alltoall 8 Nonblocking Neighborhood Com 771 Nonblocking Neighborhood Gathe 324 779 Nonblocking Neighborhood Alltoall 326 7.8 An Application Example 329 8 MPI Environmental Management 333 8.I Implementation Information 333 8.1.1 Version Inguiries ，，。 333 8.1.2 Environmental Inquiries 334 Tag values 335 Host Rank 335 IO Rank. 335 Clock Synchronization 336 Inquire Processor Name 336 8.2 Memory Allocation. 337 8.3 Error Handling 340 8 3 1 Error Handlers for communicators 83.2 Error Handlers for Windows 343 833 Error Handlers for Files 345 8.3.4 Freeing Errorhandlers and Retrieving Error strings 346 8 4 Error Codes and Classes 347 85 Error Classes,Error Codes,and Error Handlers 350 8.6 Timers and Synchronization 354 8.7 Startup 355 ix
6.9 Formalizing the Loosely Synchronous Model . . . . . . . . . . . . . . . . . . 285 6.9.1 Basic Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 6.9.2 Models of Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 Static Communicator Allocation . . . . . . . . . . . . . . . . . . . . 286 Dynamic Communicator Allocation . . . . . . . . . . . . . . . . . . . 286 The General Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 7 Process Topologies 289 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 7.2 Virtual Topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290 7.3 Embedding in MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290 7.4 Overview of the Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290 7.5 Topology Constructors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 7.5.1 Cartesian Constructor . . . . . . . . . . . . . . . . . . . . . . . . . . 292 7.5.2 Cartesian Convenience Function: MPI_DIMS_CREATE . . . . . . . . 292 7.5.3 Graph Constructor . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 7.5.4 Distributed Graph Constructor . . . . . . . . . . . . . . . . . . . . . 296 7.5.5 Topology Inquiry Functions . . . . . . . . . . . . . . . . . . . . . . . 302 7.5.6 Cartesian Shift Coordinates . . . . . . . . . . . . . . . . . . . . . . . 310 7.5.7 Partitioning of Cartesian Structures . . . . . . . . . . . . . . . . . . 311 7.5.8 Low-Level Topology Functions . . . . . . . . . . . . . . . . . . . . . 312 7.6 Neighborhood Collective Communication . . . . . . . . . . . . . . . . . . . 314 7.6.1 Neighborhood Gather . . . . . . . . . . . . . . . . . . . . . . . . . . 315 7.6.2 Neighbor Alltoall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318 7.7 Nonblocking Neighborhood Communication . . . . . . . . . . . . . . . . . . 323 7.7.1 Nonblocking Neighborhood Gather . . . . . . . . . . . . . . . . . . . 324 7.7.2 Nonblocking Neighborhood Alltoall . . . . . . . . . . . . . . . . . . . 326 7.8 An Application Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329 8 MPI Environmental Management 333 8.1 Implementation Information . . . . . . . . . . . . . . . . . . . . . . . . . . . 333 8.1.1 Version Inquiries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333 8.1.2 Environmental Inquiries . . . . . . . . . . . . . . . . . . . . . . . . . 334 Tag Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 Host Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 IO Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 Clock Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . 336 Inquire Processor Name . . . . . . . . . . . . . . . . . . . . . . . . . 336 8.2 Memory Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 8.3 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 8.3.1 Error Handlers for Communicators . . . . . . . . . . . . . . . . . . . 341 8.3.2 Error Handlers for Windows . . . . . . . . . . . . . . . . . . . . . . . 343 8.3.3 Error Handlers for Files . . . . . . . . . . . . . . . . . . . . . . . . . 345 8.3.4 Freeing Errorhandlers and Retrieving Error Strings . . . . . . . . . . 346 8.4 Error Codes and Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 8.5 Error Classes, Error Codes, and Error Handlers . . . . . . . . . . . . . . . . 350 8.6 Timers and Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . 354 8.7 Startup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355 ix
8.7.1 Allowing User Functions at Process Termination 361 8.7.2 Determining Whether MPI Has Finished.. 361 8.8 Portable MPI Process Startup... 362 9 The Info Object 366 10 Process Creation and Management 371 27 10.2 The Dyr amic Process Model 372 372 1022 The Runtime Env en 2 10.3 Proc 27 es in MP 7 and Establishin mication 27 g Multiple Executables and ablish ing Communication d Keys 3 wn _COMM_SPAWN 10.4 Establish ame sses,Ports,and All Tha r Ro tines 386 Clie Name Publishi Key alues 。。。。。。，。。。 10.4.6 Client/Se rver Examples imple 上xample- Completely Portable Ocean/Atmosphere- Relies on Name Pu blishing Simple Client-Server Example 10.5 Other Functionality 。。。 10. Universe Size 397 10.5.2 Singleton MPI_INIT 10.5.3 MPI_APPNUM . ，。。，。。。，。。。，。。 10.5.4 Releasing Connection 397 10.5.5 Another Way to Establish MPI Communication............ 3 11 One-Sided Communications 40, 1111nt 0 11 2 1 Window Cre 112.2 Window at An. 112.3 Window emor 405 M 407 11.2.4 Windoy Dyna cally Attached Memory 112.5w 413 1126 Windo Attributes 415 cation Call Get
8.7.1 Allowing User Functions at Process Termination . . . . . . . . . . . 361 8.7.2 Determining Whether MPI Has Finished . . . . . . . . . . . . . . . . 361 8.8 Portable MPI Process Startup . . . . . . . . . . . . . . . . . . . . . . . . . . 362 9 The Info Object 365 10 Process Creation and Management 371 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 10.2 The Dynamic Process Model . . . . . . . . . . . . . . . . . . . . . . . . . . 372 10.2.1 Starting Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372 10.2.2 The Runtime Environment . . . . . . . . . . . . . . . . . . . . . . . 372 10.3 Process Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 10.3.1 Processes in MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 10.3.2 Starting Processes and Establishing Communication . . . . . . . . . 374 10.3.3 Starting Multiple Executables and Establishing Communication . . 379 10.3.4 Reserved Keys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382 10.3.5 Spawn Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 Manager-worker Example Using MPI_COMM_SPAWN . . . . . . . . 383 10.4 Establishing Communication . . . . . . . . . . . . . . . . . . . . . . . . . . 385 10.4.1 Names, Addresses, Ports, and All That . . . . . . . . . . . . . . . . 385 10.4.2 Server Routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386 10.4.3 Client Routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388 10.4.4 Name Publishing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390 10.4.5 Reserved Key Values . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 10.4.6 Client/Server Examples . . . . . . . . . . . . . . . . . . . . . . . . . 392 Simplest Example — Completely Portable. . . . . . . . . . . . . . . 392 Ocean/Atmosphere — Relies on Name Publishing . . . . . . . . . . 393 Simple Client-Server Example . . . . . . . . . . . . . . . . . . . . . . 393 10.5 Other Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 10.5.1 Universe Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 10.5.2 Singleton MPI_INIT . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 10.5.3 MPI_APPNUM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 10.5.4 Releasing Connections . . . . . . . . . . . . . . . . . . . . . . . . . . 397 10.5.5 Another Way to Establish MPI Communication . . . . . . . . . . . . 399 11 One-Sided Communications 401 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401 11.2 Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402 11.2.1 Window Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 11.2.2 Window That Allocates Memory . . . . . . . . . . . . . . . . . . . . 405 11.2.3 Window That Allocates Shared Memory . . . . . . . . . . . . . . . . 407 11.2.4 Window of Dynamically Attached Memory . . . . . . . . . . . . . . 410 11.2.5 Window Destruction . . . . . . . . . . . . . . . . . . . . . . . . . . . 413 11.2.6 Window Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414 11.2.7 Window Info . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415 11.3 Communication Calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417 11.3.1 Put . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418 11.3.2 Get . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420 x