《高等数值分析（高性能计算/并行计算）》课程教学资源（参考资料）OpenMP Application Programming Interface Version 5.0

OpenMP OpenMP Application Programming Interface Version 5.0 November 2018 Copyright 1997-2018 OpenMP Architecture Review Board. Permission to copy without fee all or part of this material is granted,provided the OpenMP Architecture Review Board copyright notice and the title of this document appear.Notice is given that copying is by permission of the OpenMP Architecture Review Board

OpenMP Application Programming Interface Version 5.0 November 2018 Copyright c 1997-2018 OpenMP Architecture Review Board. Permission to copy without fee all or part of this material is granted, provided the OpenMP Architecture Review Board copyright notice and the title of this document appear. Notice is given that copying is by permission of the OpenMP Architecture Review Board

Contents 1 Introduction 1 1.1 Scope. 1.2 Glossary 。·。。。·。。。。·。·。。4。·。·。。。。·。· 2 1.2.1 Threading Concepts 1.2.2 OpenMP Language Terminology 123 Loop Terminology.....................·..... 1.2.4 Synchronization Terminology ........................ 9 1.2.5 Tasking Terminology 。。。。。。。。。。，。。，。，。。。。。。。。。。。。 10 1.2.6 Data Terminology 12 1.2.7 Implementation Terminology 17 128 Tool Terminology 17 1.3 Execution Model.... 20 1.4 Memory Model.... 23 1.4.1 Structure of the OpenMP Memory Model. 23 1.4.2 Device Data Environments 24 1.4.3 Memory Management 3 14.4 The Flush Operation......,..··· 子 1.4.5 Flush Synchronization and Happens Before.················· 27 1.4.6 OpenMP Memory Consistency 28 l5 Tool Interfaces.········ 29 1.5.1 OMPT 1.5.20MPD 30

Contents 1 Introduction 1 1.1 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2.1 Threading Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2.2 OpenMP Language Terminology . . . . . . . . . . . . . . . . . . . . . . . 2 1.2.3 Loop Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.2.4 Synchronization Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.2.5 Tasking Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2.6 Data Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.2.7 Implementation Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.2.8 Tool Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.3 Execution Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 1.4 Memory Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 1.4.1 Structure of the OpenMP Memory Model . . . . . . . . . . . . . . . . . . . 23 1.4.2 Device Data Environments . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.4.3 Memory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.4.4 The Flush Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.4.5 Flush Synchronization and Happens Before . . . . . . . . . . . . . . . . . . 27 1.4.6 OpenMP Memory Consistency . . . . . . . . . . . . . . . . . . . . . . . . 28 1.5 Tool Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 1.5.1 OMPT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 1.5.2 OMPD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 i

l.6 OpenMP Compliance............................... 31 l7 Normative References...·····.················· 31 l.8 Organization of this Document······················· 34 2 Directives 37 2.1 Directive Format.......·....·...····.··.···. 38 2.1.1 Fixed Source Form Directives 41 2.1.2 Free Source Form Directives 41 2.1.3 Stand-Alone Directives... 42 2.l.4 Array Shaping...·········· 3 2.1.5 Array Sections..........。..。.。.........。..。.。 44 2.1.6 47 2.2 Conditional Compilation 49 2.2.1 Fixed Source Form Conditional Compilation Sentinels.. 50 2.2.2 Free Source Form Conditional Compilation Sentinel.···..,,..·,· 50 23 Variant Directives.,......,. 51 2.3.1 51 2.3.2 Context Selectors 53 2.33 Matching and Scoring Context Selectors.········ 55 2.3.4 Metadirectives.............................. 56 2.3.5 declare variant Directive ....................... 58 2.4 requires Directive.,.,...,.,.,.,..,.............,. 60 2.5 Internal Control Variables 63 2.5.1 ICV Descriptions 64 2.5.2 ICV Initialization 66 2.5.3 Modifying and Retrieving ICV Values.................... 68 2.5.4 How ICVs are Scoped........,...........,....,... 70 2.5.4.1 How the Per-Data Environment ICVs Work 72 2.5.5 ICV Override Relationships 72 2.6 parallel Construct.············ 2.6.1 Determining the Number of Threads for a parallel Region........ 中 2.6.2 Controlling OpenMP Thread Affinity.··········· 。。。。。。。 80 2.7 teams Construct 82 OpenMP API-Version 5.0 November 2018

1.6 OpenMP Compliance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 1.7 Normative References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 1.8 Organization of this Document . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2 Directives 37 2.1 Directive Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2.1.1 Fixed Source Form Directives . . . . . . . . . . . . . . . . . . . . . . . . . 41 2.1.2 Free Source Form Directives . . . . . . . . . . . . . . . . . . . . . . . . . 41 2.1.3 Stand-Alone Directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 2.1.4 Array Shaping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 2.1.5 Array Sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 2.1.6 Iterators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 2.2 Conditional Compilation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 2.2.1 Fixed Source Form Conditional Compilation Sentinels . . . . . . . . . . . . 50 2.2.2 Free Source Form Conditional Compilation Sentinel . . . . . . . . . . . . . 50 2.3 Variant Directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 2.3.1 OpenMP Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 2.3.2 Context Selectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 2.3.3 Matching and Scoring Context Selectors . . . . . . . . . . . . . . . . . . . 55 2.3.4 Metadirectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 2.3.5 declare variant Directive . . . . . . . . . . . . . . . . . . . . . . . . 58 2.4 requires Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 2.5 Internal Control Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 2.5.1 ICV Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 2.5.2 ICV Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 2.5.3 Modifying and Retrieving ICV Values . . . . . . . . . . . . . . . . . . . . 68 2.5.4 How ICVs are Scoped . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 2.5.4.1 How the Per-Data Environment ICVs Work . . . . . . . . . . . . . . . 72 2.5.5 ICV Override Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . 72 2.6 parallel Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 2.6.1 Determining the Number of Threads for a parallel Region . . . . . . . . 78 2.6.2 Controlling OpenMP Thread Affinity . . . . . . . . . . . . . . . . . . . . . 80 2.7 teams Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 ii OpenMP API – Version 5.0 November 2018

2.8 Worksharing Constructs. 86 2.8.1 sections Construct.:........................... 86 2.8.2 single Construct 89 283 workshare Construct 2.9 Loop-Related Directives 8 2.9 1 Canonical Loop Form,·,················ 号 2.9.2 Worksharing-Loop Construct......................... 101 2.9.2.1 Determining the Schedule of a Worksharing-Loop............ 109 2.9.3 SIMD Directives 110 2.9.3.1 simd Construct 110 2.9.3.2 Worksharing-Loop SIMD Construct...... 114 2.9.3.3 declare simd Directive 116 2.9.4 distribute Loop Constructs 120 2.9.4.1 distribute Construct 120 29.42 distribute simd Construct 123 2.9.4.3 Distribute Parallel Worksharing-Loop Construct 125 2.9.4.4 Distribute Parallel Worksharing-Loop SIMD Construct......... 126 2.9.5 loop Construct 4。。。。。。。。。。。 128 2.9.6 scan Directive 132 2.10 Tasking Constructs. 135 2.10.1 task Construct 135 2.10.2 taskloop Construct 140 2.10.3 taskloop simd Construct 146 2.10.4 taskyield Construct 147 148 2.10.6 Task Scheduling 149 2.11 Memory Management Directives.......................... 152 2.11.1 Memory Spaces 。。。。。。。，。，。。。。。。。。。。。。。。。。 152 2.11.2 Memory Allocators 152 2.11.3 allocate Directiv 。。。。。。。。。。。。。。。。。。。。。。。。 156 2.11.4 allocate Clause 158 2.12 Device Directives 160 2.l2.1 Device Initialization...........·,.········· 160 Contents 道

2.8 Worksharing Constructs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 2.8.1 sections Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 2.8.2 single Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 2.8.3 workshare Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 2.9 Loop-Related Directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 2.9.1 Canonical Loop Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 2.9.2 Worksharing-Loop Construct . . . . . . . . . . . . . . . . . . . . . . . . . 101 2.9.2.1 Determining the Schedule of a Worksharing-Loop . . . . . . . . . . . . 109 2.9.3 SIMD Directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 2.9.3.1 simd Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 2.9.3.2 Worksharing-Loop SIMD Construct . . . . . . . . . . . . . . . . . . . 114 2.9.3.3 declare simd Directive . . . . . . . . . . . . . . . . . . . . . . . . 116 2.9.4 distribute Loop Constructs . . . . . . . . . . . . . . . . . . . . . . . . 120 2.9.4.1 distribute Construct . . . . . . . . . . . . . . . . . . . . . . . . . 120 2.9.4.2 distribute simd Construct . . . . . . . . . . . . . . . . . . . . . . 123 2.9.4.3 Distribute Parallel Worksharing-Loop Construct . . . . . . . . . . . . . 125 2.9.4.4 Distribute Parallel Worksharing-Loop SIMD Construct . . . . . . . . . 126 2.9.5 loop Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 2.9.6 scan Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 2.10 Tasking Constructs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 2.10.1 task Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 2.10.2 taskloop Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 2.10.3 taskloop simd Construct . . . . . . . . . . . . . . . . . . . . . . . . . 146 2.10.4 taskyield Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 2.10.5 Initial Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 2.10.6 Task Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 2.11 Memory Management Directives . . . . . . . . . . . . . . . . . . . . . . . . . . 152 2.11.1 Memory Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 2.11.2 Memory Allocators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 2.11.3 allocate Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 2.11.4 allocate Clause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 2.12 Device Directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 2.12.1 Device Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 Contents iii

2.12.2 target data Construct........................... 161 2.l2.3 target enter data Construct..··· 164 2.12.4 target exit data Construct 2.12.5 target Construct...... 170 2.12.6 target update Construct 176 2.12.7 declare target Directive 。。。4。。。。。tg。g。+。。。。4.74。。。 180 2.13 Combined Constructs 185 2.13.1 Parallel Worksharing-Loop Construct 185 2.13.2 parallel loop Construct 186 2.l3.3 paral1 el sections Construct··················· 188 2.13.4 parallel workshare Construct 189 2.13.5 Parallel Worksharing-Loop SIMD Construct 190 2.13.6 parallel master Construct 191 2.l3.7 master taskloop Construct.··:.·············· 192 2.13.8 master taskloop simd Construct 194 2.13.9 parallelmaster taskloop Construct 。。。。。。。。。。。。。 195 2.13.10 parallel master taskloop simd Construct...... 196 2.l3.11 teams distribute Construct.··,···,···· 197 2.13.12 teams distribute simd Construct 198 2.13.13 Teams Distribute Parallel Worksharing-Loop Construct 200 2.l3.14 Teams Distribute Parallel Worksharing-Loop SIMD Construct.·····.. 201 2.13.15 teams loop Construct 202 2.l3.l6 target paral1e1 Construct.···· 203 2.13.17 Target Parallel Worksharing-Loop Construct 205 2.13.18 Target Parallel Worksharing-Loop SIMD Construct 206 2.13.19 target parallel loop Construct 208 2.l3.20 target simd Construct........................... 209 2.l3.2 I target teams Construct················ 210 2.13.22 target teams distribute Construct 211 2.13.23 target teams distribute simd Construct 213 2.13.24 target teams loop Construct...... 214 2.l3.25 Target Teams Distribute Parallel Worksharing--Loop Construct.······· 215 2.13.26 Target Teams Distribute Parallel Worksharing-Loop SIMD Construct 216 iv OpenMP API-Version 5.0 November 2018

2.12.2 target data Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 2.12.3 target enter data Construct . . . . . . . . . . . . . . . . . . . . . . . 164 2.12.4 target exit data Construct . . . . . . . . . . . . . . . . . . . . . . . 166 2.12.5 target Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 2.12.6 target update Construct . . . . . . . . . . . . . . . . . . . . . . . . . 176 2.12.7 declare target Directive . . . . . . . . . . . . . . . . . . . . . . . . . 180 2.13 Combined Constructs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 2.13.1 Parallel Worksharing-Loop Construct . . . . . . . . . . . . . . . . . . . . . 185 2.13.2 parallel loop Construct . . . . . . . . . . . . . . . . . . . . . . . . . 186 2.13.3 parallel sections Construct . . . . . . . . . . . . . . . . . . . . . . 188 2.13.4 parallel workshare Construct . . . . . . . . . . . . . . . . . . . . . 189 2.13.5 Parallel Worksharing-Loop SIMD Construct . . . . . . . . . . . . . . . . . 190 2.13.6 parallel master Construct . . . . . . . . . . . . . . . . . . . . . . . . 191 2.13.7 master taskloop Construct . . . . . . . . . . . . . . . . . . . . . . . . 192 2.13.8 master taskloop simd Construct . . . . . . . . . . . . . . . . . . . . 194 2.13.9 parallel master taskloop Construct . . . . . . . . . . . . . . . . . 195 2.13.10 parallel master taskloop simd Construct . . . . . . . . . . . . . . 196 2.13.11 teams distribute Construct . . . . . . . . . . . . . . . . . . . . . . . 197 2.13.12 teams distribute simd Construct . . . . . . . . . . . . . . . . . . . 198 2.13.13 Teams Distribute Parallel Worksharing-Loop Construct . . . . . . . . . . . 200 2.13.14 Teams Distribute Parallel Worksharing-Loop SIMD Construct . . . . . . . . 201 2.13.15 teams loop Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 2.13.16 target parallel Construct . . . . . . . . . . . . . . . . . . . . . . . . 203 2.13.17 Target Parallel Worksharing-Loop Construct . . . . . . . . . . . . . . . . . 205 2.13.18 Target Parallel Worksharing-Loop SIMD Construct . . . . . . . . . . . . . 206 2.13.19 target parallel loop Construct . . . . . . . . . . . . . . . . . . . . 208 2.13.20 target simd Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 2.13.21 target teams Construct . . . . . . . . . . . . . . . . . . . . . . . . . . 210 2.13.22 target teams distribute Construct . . . . . . . . . . . . . . . . . . 211 2.13.23 target teams distribute simd Construct . . . . . . . . . . . . . . 213 2.13.24 target teams loop Construct . . . . . . . . . . . . . . . . . . . . . . . 214 2.13.25 Target Teams Distribute Parallel Worksharing-Loop Construct . . . . . . . . 215 2.13.26 Target Teams Distribute Parallel Worksharing-Loop SIMD Construct . . . . 216 iv OpenMP API – Version 5.0 November 2018

2.14 Clauses on Combined and Composite Constructs..................218 2.15 if Clause 。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。 220 2.l6 master Construct................········· 2 2.17 Synchronization Constructs and Clauses 2.17.1 critical Construct 223 2.17.2 barrier Construct 226 2.17.3 Implicit Barriers 228 2.17.4 Implementation-Specific Barriers 230 2.17.5 taskwait Construct 230 2.17.6 taskgroup Construct 232 2.17.7 atomic Construct. 234 2.17.8 Elush Construct 242 2.17.8.1 Implicit Flushes 246 2.17.9 ordered Construct 250 2.17.10 Depend Objects 254 2.17.10.1 depobj Construct 254 2.17.11 depend Clause 255 2.17.12 Synchronization Hints 260 2.18 Cancellation Constructs 263 2.18.1 cancel Construct 263 2.18.2 cancellation point Construct 267 2.19 Data Environment.... 269 2.19.1 Data-Sharing Attribute Rules 269 2.19.1.1 Variables Referenced in a Construct 270 2.19.1.2 Variables Referenced in a Region but not in a Construct......... 273 2.19.2 threadprivate Directive 274 2.19.3 List Item Privatization...... 279 2.19.4 Data-Sharing Attribute Clauses 。。。。。。。。。。。。。 282 2.19.4.1 default Clause..············· 282 2.19.4.2 shared Clause 283 2.19.4.3 private Clause 285 2.19.4.4 firstprivate Clause 286 2.19.4.5 lastprivate Clause.......................... 288 Contents

2.14 Clauses on Combined and Composite Constructs . . . . . . . . . . . . . . . . . . 218 2.15 if Clause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 2.16 master Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 2.17 Synchronization Constructs and Clauses . . . . . . . . . . . . . . . . . . . . . . 223 2.17.1 critical Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 2.17.2 barrier Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 2.17.3 Implicit Barriers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 2.17.4 Implementation-Specific Barriers . . . . . . . . . . . . . . . . . . . . . . . 230 2.17.5 taskwait Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 2.17.6 taskgroup Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 2.17.7 atomic Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 2.17.8 flush Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 2.17.8.1 Implicit Flushes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 2.17.9 ordered Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 2.17.10 Depend Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254 2.17.10.1 depobj Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254 2.17.11 depend Clause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 2.17.12 Synchronization Hints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 2.18 Cancellation Constructs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 2.18.1 cancel Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 2.18.2 cancellation point Construct . . . . . . . . . . . . . . . . . . . . . 267 2.19 Data Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 2.19.1 Data-Sharing Attribute Rules . . . . . . . . . . . . . . . . . . . . . . . . . 269 2.19.1.1 Variables Referenced in a Construct . . . . . . . . . . . . . . . . . . . 270 2.19.1.2 Variables Referenced in a Region but not in a Construct . . . . . . . . . 273 2.19.2 threadprivate Directive . . . . . . . . . . . . . . . . . . . . . . . . . 274 2.19.3 List Item Privatization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 2.19.4 Data-Sharing Attribute Clauses . . . . . . . . . . . . . . . . . . . . . . . . 282 2.19.4.1 default Clause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 2.19.4.2 shared Clause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 2.19.4.3 private Clause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 2.19.4.4 firstprivate Clause . . . . . . . . . . . . . . . . . . . . . . . . . 286 2.19.4.5 lastprivate Clause . . . . . . . . . . . . . . . . . . . . . . . . . . 288 Contents v

2.19.4.61 inear Clause........................... 290 2.l9.5 Reduction Clauses and Directives··············· 293 2.19.5.1 Properties Common To All Reduction Clauses 294 2.l9.5.2 Reduction Scoping Clauses.......······· 299 2.19.5.3 Reduction Participating Clauses 300 2.19.5.4 reduction Clause 300 2.19.5.5 task_reduction Clause 303 2.19.5.6 in_reduction Clause 303 2.19.5.7 declare reduction Directive.................. 304 2.19.6 Data Copying Clauses.... 309 2.19.6.1 copyin Clause.··············… 310 2.l9.6.2 copyprivate Clause..........,....,......... 312 2.19.7 Data-Mapping Attribute Rules,Clauses,and Directives 314 2.19.7.1 map Clause 315 2.19.7.2 defaultmap Clause.. 324 2.l9.7.3 declare mapper Directive.··············· 326 2.20 Nesting of Regions 328 3 Runtime Library Routines 331 3.1 Runtime Library Definitions..·..·······.· 332 3.2 Execution Environment Routines 334 3.2.1 omp_set_num_threads 334 3.2.2 omp_get_num_threads 335 3.2.3 omp_get_max_threads 336 3.2.4 omp_get_thread_num 337 3.2.5 omp get num procs 338 3.2.6 omp_in_parallel.......,...............,.,. 339 3.2.7 omp_set_dynamic 340 3.2.8 omp_get_dynamic,··,, 341 3.2.9 omp_get_cancellation 342 3.2.10 omp_set_nested.... 343 32.11 omp_get_nested................... 344 3.2.12 omp_set_schedule 345 3.2.13 omp_get_schedule 347 OpenMP API-Version 5.0 November 2018

2.19.4.6 linear Clause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290 2.19.5 Reduction Clauses and Directives . . . . . . . . . . . . . . . . . . . . . . . 293 2.19.5.1 Properties Common To All Reduction Clauses . . . . . . . . . . . . . . 294 2.19.5.2 Reduction Scoping Clauses . . . . . . . . . . . . . . . . . . . . . . . . 299 2.19.5.3 Reduction Participating Clauses . . . . . . . . . . . . . . . . . . . . . 300 2.19.5.4 reduction Clause . . . . . . . . . . . . . . . . . . . . . . . . . . . 300 2.19.5.5 task_reduction Clause . . . . . . . . . . . . . . . . . . . . . . . 303 2.19.5.6 in_reduction Clause . . . . . . . . . . . . . . . . . . . . . . . . . 303 2.19.5.7 declare reduction Directive . . . . . . . . . . . . . . . . . . . . 304 2.19.6 Data Copying Clauses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 2.19.6.1 copyin Clause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310 2.19.6.2 copyprivate Clause . . . . . . . . . . . . . . . . . . . . . . . . . . 312 2.19.7 Data-Mapping Attribute Rules, Clauses, and Directives . . . . . . . . . . . 314 2.19.7.1 map Clause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 2.19.7.2 defaultmap Clause . . . . . . . . . . . . . . . . . . . . . . . . . . . 324 2.19.7.3 declare mapper Directive . . . . . . . . . . . . . . . . . . . . . . . 326 2.20 Nesting of Regions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328 3 Runtime Library Routines 331 3.1 Runtime Library Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332 3.2 Execution Environment Routines . . . . . . . . . . . . . . . . . . . . . . . . . . 334 3.2.1 omp_set_num_threads . . . . . . . . . . . . . . . . . . . . . . . . . . 334 3.2.2 omp_get_num_threads . . . . . . . . . . . . . . . . . . . . . . . . . . 335 3.2.3 omp_get_max_threads . . . . . . . . . . . . . . . . . . . . . . . . . . 336 3.2.4 omp_get_thread_num . . . . . . . . . . . . . . . . . . . . . . . . . . 337 3.2.5 omp_get_num_procs . . . . . . . . . . . . . . . . . . . . . . . . . . . 338 3.2.6 omp_in_parallel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 3.2.7 omp_set_dynamic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 3.2.8 omp_get_dynamic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341 3.2.9 omp_get_cancellation . . . . . . . . . . . . . . . . . . . . . . . . . 342 3.2.10 omp_set_nested . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 3.2.11 omp_get_nested . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344 3.2.12 omp_set_schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345 3.2.13 omp_get_schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 vi OpenMP API – Version 5.0 November 2018

3.2.l4omp_get_thread1imit..,.,..·.·.············ 348 3.2.15 omp_get_supported_active_levels 。。g。。。。。。。。g。。。 349 3.2.16 omp_set_max_active_levels 350 3.2.17 omp_get_max_active_levels 351 3.2.18 omp_get_level 352 3.2.19 omp_get_ancestor_thread_num 子 3.2.20 omp_get_team_size························ 354 3.2.21 omp_get_active_level 355 3.2.22 omp_in_final 356 3.2.23 omp_get_proc_bind 357 3.2.24 omp_get_num_places 358 3.2.25 omp_get_place_num_procs 359 3.2.26 omp_get_place_proc_ids 360 3.2.27 omp_get_place_num 362 3.2.28 omp_get_partition_num_places 362 3.2.29 omp_get_partition_place_nums 363 3.2.30 omp_set_affinity_format 364 3.2.31 omp_get_affinity_format 366 3.2.32 omp_display_affinity......................... 367 3.2.33 omp_capture_affinity 368 3.2.34 omp_set_default_device 369 3.2.35 omp_get_default_device 370 3.2.36 omp_get_num_devices.. 371 3.2.37 omp_get_device_num 372 3.2.38 omp_get_num_teams 373 3.2.39 omp_get_team_num 374 3.2.40 omp_is_initial_device 375 3.2.41 omp_get_initial_device 。。。。。。。。。。。。。。 376 3.2.42 omp_get_max_task_priority 377 3.2.43 omp_pause_resource 378 3244 omp_pause_resource_all 380 3.3 Lock Routines 381 3.3.1 omp_init_lock and omp_init_nest_lock 。。。。。。。。。 384 Contents

3.2.14 omp_get_thread_limit . . . . . . . . . . . . . . . . . . . . . . . . . 348 3.2.15 omp_get_supported_active_levels . . . . . . . . . . . . . . . . 349 3.2.16 omp_set_max_active_levels . . . . . . . . . . . . . . . . . . . . . 350 3.2.17 omp_get_max_active_levels . . . . . . . . . . . . . . . . . . . . . 351 3.2.18 omp_get_level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352 3.2.19 omp_get_ancestor_thread_num . . . . . . . . . . . . . . . . . . . 353 3.2.20 omp_get_team_size . . . . . . . . . . . . . . . . . . . . . . . . . . . 354 3.2.21 omp_get_active_level . . . . . . . . . . . . . . . . . . . . . . . . . 355 3.2.22 omp_in_final . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356 3.2.23 omp_get_proc_bind . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 3.2.24 omp_get_num_places . . . . . . . . . . . . . . . . . . . . . . . . . . 358 3.2.25 omp_get_place_num_procs . . . . . . . . . . . . . . . . . . . . . . 359 3.2.26 omp_get_place_proc_ids . . . . . . . . . . . . . . . . . . . . . . . 360 3.2.27 omp_get_place_num . . . . . . . . . . . . . . . . . . . . . . . . . . . 362 3.2.28 omp_get_partition_num_places . . . . . . . . . . . . . . . . . . 362 3.2.29 omp_get_partition_place_nums . . . . . . . . . . . . . . . . . . 363 3.2.30 omp_set_affinity_format . . . . . . . . . . . . . . . . . . . . . . 364 3.2.31 omp_get_affinity_format . . . . . . . . . . . . . . . . . . . . . . 366 3.2.32 omp_display_affinity . . . . . . . . . . . . . . . . . . . . . . . . . 367 3.2.33 omp_capture_affinity . . . . . . . . . . . . . . . . . . . . . . . . . 368 3.2.34 omp_set_default_device . . . . . . . . . . . . . . . . . . . . . . . 369 3.2.35 omp_get_default_device . . . . . . . . . . . . . . . . . . . . . . . 370 3.2.36 omp_get_num_devices . . . . . . . . . . . . . . . . . . . . . . . . . . 371 3.2.37 omp_get_device_num . . . . . . . . . . . . . . . . . . . . . . . . . . 372 3.2.38 omp_get_num_teams . . . . . . . . . . . . . . . . . . . . . . . . . . . 373 3.2.39 omp_get_team_num . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 3.2.40 omp_is_initial_device . . . . . . . . . . . . . . . . . . . . . . . . 375 3.2.41 omp_get_initial_device . . . . . . . . . . . . . . . . . . . . . . . 376 3.2.42 omp_get_max_task_priority . . . . . . . . . . . . . . . . . . . . . 377 3.2.43 omp_pause_resource . . . . . . . . . . . . . . . . . . . . . . . . . . 378 3.2.44 omp_pause_resource_all . . . . . . . . . . . . . . . . . . . . . . . 380 3.3 Lock Routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381 3.3.1 omp_init_lock and omp_init_nest_lock . . . . . . . . . . . . . 384 Contents vii

3.3.2 omp_init_lock_with_hint and omp_init_nest_lock_with_hint 385 33.3 omp_destroy_lock and omp_destroy_nest_lock 387 3.3.4 omp_set_lock and omp_set_nest_lock..... 388 3.3.5 omp_unset_lock and omp_unset_nest_lock 390 3.3.6 omp_test_lock and omp_test_nest_lock 392 3.4 Timing Routines 394 3.41 omp_get_wtime 394 3.4.2 omp_get_wtick 395 3.5 Event Routine 396 3.5.1 omp_fulfill_event 396 3.6 Device Memory Routines.·..················· 397 3.6.1 omp_target_alloc 397 3.6.2 omp_target_free 399 3.6.3 omp_target_is_present 400 3.6.4 omp_target_memcpy 400 3.6.5 omp_target_memcpy_rect 402 3.6.6 omp_target_associate_ptr 403 3.6.7 omp_target_disassociate_ptr 405 3.7 Memory Management Routines.. 406 3.7.1 Memory Management Types 406 3.7.2 omp_init_allocator 409 3.7.3 omp_destroy_allocator 410 3.7.4 omp_set_default_allocator 411 3.7.5 omp_get_default_allocator 412 3.7.6 omp_alloc. 413 37.7 ompf上ee。:.。.。.。。。..。..。。,.。.。.。,。.。。。。, 414 3.8 Tool Control Routine 415 4 OMPT Interface 419 4.1 OMPT Interfaces Definitions 419 4.2 Activating a First-Party Tool 。。。。。。。。。。。。。。。。。。。。。。。。 420 4.2.1 ompt_start_tool 420 4.2.2 Determining Whether a First-Party Tool Should be Initialized 421 OpenMP API-Version 5.0 November 2018

3.3.2 omp_init_lock_with_hint and omp_init_nest_lock_with_hint . . . . . . . . . . . . . . . . . . 385 3.3.3 omp_destroy_lock and omp_destroy_nest_lock . . . . . . . . . 387 3.3.4 omp_set_lock and omp_set_nest_lock . . . . . . . . . . . . . . . 388 3.3.5 omp_unset_lock and omp_unset_nest_lock . . . . . . . . . . . . 390 3.3.6 omp_test_lock and omp_test_nest_lock . . . . . . . . . . . . . 392 3.4 Timing Routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 3.4.1 omp_get_wtime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 3.4.2 omp_get_wtick . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 3.5 Event Routine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 3.5.1 omp_fulfill_event . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 3.6 Device Memory Routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397 3.6.1 omp_target_alloc . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397 3.6.2 omp_target_free . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399 3.6.3 omp_target_is_present . . . . . . . . . . . . . . . . . . . . . . . . 400 3.6.4 omp_target_memcpy . . . . . . . . . . . . . . . . . . . . . . . . . . . 400 3.6.5 omp_target_memcpy_rect . . . . . . . . . . . . . . . . . . . . . . . 402 3.6.6 omp_target_associate_ptr . . . . . . . . . . . . . . . . . . . . . . 403 3.6.7 omp_target_disassociate_ptr . . . . . . . . . . . . . . . . . . . 405 3.7 Memory Management Routines . . . . . . . . . . . . . . . . . . . . . . . . . . . 406 3.7.1 Memory Management Types . . . . . . . . . . . . . . . . . . . . . . . . . 406 3.7.2 omp_init_allocator . . . . . . . . . . . . . . . . . . . . . . . . . . 409 3.7.3 omp_destroy_allocator . . . . . . . . . . . . . . . . . . . . . . . . 410 3.7.4 omp_set_default_allocator . . . . . . . . . . . . . . . . . . . . . 411 3.7.5 omp_get_default_allocator . . . . . . . . . . . . . . . . . . . . . 412 3.7.6 omp_alloc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413 3.7.7 omp_free . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414 3.8 Tool Control Routine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415 4 OMPT Interface 419 4.1 OMPT Interfaces Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419 4.2 Activating a First-Party Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420 4.2.1 ompt_start_tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420 4.2.2 Determining Whether a First-Party Tool Should be Initialized . . . . . . . . 421 viii OpenMP API – Version 5.0 November 2018

4.2.3 Initializing a First-Party Tool.......·.·.·..········· 423 4.2.3.1 Binding Entry Points in the OMPT Callback Interface.......... 424 4.2.4 Monitoring Activity on the Host with OMPT................. 425 425 Tracing Activity on Target Devices with OMPT 427 4.3 Finalizing a First-Party Tool 432 4.4 OMPT Data Types.. 433 4.4.1 Tool Initialization and Finalization 。，，。。。。。。。。。。。。。。， 433 4.4.2 Callbacks 434 4.4.3 Tracing............ 435 4.4.3.1 Record Type 435 4.4.3.2 Native Record Kind.... 435 4.4.33 Native Record Abstract Type 436 4.4.3.4 Record Type..····· 436 4.4.4 Miscellaneous Type Definitions 438 4.4.4.1 ompt_callback_t.··················· 438 4.4.4.2 ompt_set_result_t 438 4.4.4.3 ompt_id t 439 4.4.4.4 ompt_data_t...... 440 4.4.4.5 ompt_device_t..... 44 4.4.4.6 ompt_device_time_t 441 4.4.4.7 ompt_buffer_t... 441 4.4.48 ompt_buffer_cursor_t 442 4.4.4.9 ompt_dependence_t 442 4.4.4.10 ompt_thread_t.... 443 4.4.4.11 ompt_scope_endpoint_t 443 4.4.4.12ompt_dispatch_t.··················· 444 4.4.4.13 ompt_sync_region_t 444 4.4.4.14 ompt_target_data_op_t. 444 4.4.4.15 445 4.4.4.16 ompt_mutex_t。 445 4.4.4.17 ompt_native_mon_flag_t 446 4.4.4.18ompt_task_f1ag_t,··················· 446 4.4.4.19 ompt_task_status._t.........,............. 447 Contents ix

4.2.3 Initializing a First-Party Tool . . . . . . . . . . . . . . . . . . . . . . . . . 423 4.2.3.1 Binding Entry Points in the OMPT Callback Interface . . . . . . . . . . 424 4.2.4 Monitoring Activity on the Host with OMPT . . . . . . . . . . . . . . . . . 425 4.2.5 Tracing Activity on Target Devices with OMPT . . . . . . . . . . . . . . . 427 4.3 Finalizing a First-Party Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432 4.4 OMPT Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433 4.4.1 Tool Initialization and Finalization . . . . . . . . . . . . . . . . . . . . . . 433 4.4.2 Callbacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434 4.4.3 Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435 4.4.3.1 Record Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435 4.4.3.2 Native Record Kind . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435 4.4.3.3 Native Record Abstract Type . . . . . . . . . . . . . . . . . . . . . . . 436 4.4.3.4 Record Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436 4.4.4 Miscellaneous Type Definitions . . . . . . . . . . . . . . . . . . . . . . . . 438 4.4.4.1 ompt_callback_t . . . . . . . . . . . . . . . . . . . . . . . . . . . 438 4.4.4.2 ompt_set_result_t . . . . . . . . . . . . . . . . . . . . . . . . . 438 4.4.4.3 ompt_id_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439 4.4.4.4 ompt_data_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440 4.4.4.5 ompt_device_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441 4.4.4.6 ompt_device_time_t . . . . . . . . . . . . . . . . . . . . . . . . 441 4.4.4.7 ompt_buffer_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441 4.4.4.8 ompt_buffer_cursor_t . . . . . . . . . . . . . . . . . . . . . . . 442 4.4.4.9 ompt_dependence_t . . . . . . . . . . . . . . . . . . . . . . . . . 442 4.4.4.10 ompt_thread_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443 4.4.4.11 ompt_scope_endpoint_t . . . . . . . . . . . . . . . . . . . . . . 443 4.4.4.12 ompt_dispatch_t . . . . . . . . . . . . . . . . . . . . . . . . . . . 444 4.4.4.13 ompt_sync_region_t . . . . . . . . . . . . . . . . . . . . . . . . 444 4.4.4.14 ompt_target_data_op_t . . . . . . . . . . . . . . . . . . . . . . 444 4.4.4.15 ompt_work_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445 4.4.4.16 ompt_mutex_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445 4.4.4.17 ompt_native_mon_flag_t . . . . . . . . . . . . . . . . . . . . . 446 4.4.4.18 ompt_task_flag_t . . . . . . . . . . . . . . . . . . . . . . . . . . 446 4.4.4.19 ompt_task_status_t . . . . . . . . . . . . . . . . . . . . . . . . 447 Contents ix