Writing Parallel software Writing Parallel software Sebastien Ponce sebastien.ponce@cern.ch CERN Thematic CERN School of Computing 2022 1/46 S.Ponce-CERN
Writing Parallel software 1 / 46 S. Ponce - CERN Intro Threading Safety Solutions Writing Parallel software S´ebastien Ponce sebastien.ponce@cern.ch CERN Thematic CERN School of Computing 2022
Writing Parallel software Outline Introduction and expectations Thread-safety issues oScope what is parallelism ●Data races o Finding parallelism o Thread safety Expectations Thread-safety solutions 2Threading,theory and practice Avoid the problem oProcesses and threads o Replicate o Python threading ●Atomics C++threading and async o Locking Credits to Danilo Piparo for the original talk in previous tCSCs ..and all the content I shamelessly stole from it 2/46 S.Ponce-CERN
Writing Parallel software 2 / 46 S. Ponce - CERN Intro Threading Safety Solutions Outline 1 Introduction and expectations Scope : what is parallelism ? Finding parallelism Expectations 2 Threading, theory and practice Processes and threads Python threading C ++threading and async 3 Thread-safety issues Data races Thread safety 4 Thread-safety solutions Avoid the problem Replicate Atomics Locking Credits to Danilo Piparo for the original talk in previous tCSCs ... and all the content I shamelessly stole from it
Writing Parallel software Introduction and expectations Threading.theory and practice Introduction and expectations Scope what is parallelism Thread-safety issues Finding parallelism Expectations Thread-safety solutions what find expect 3/46 S.Ponce-CERN
Writing Parallel software 3 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Introduction and expectations 1 Introduction and expectations Scope : what is parallelism ? Finding parallelism Expectations 2 Threading, theory and practice 3 Thread-safety issues 4 Thread-safety solutions
Writing Parallel software “ Parallelism Definition o the ability to make 2 or more things concurrently more parallel 2nd parallel flow main flow start end parallel flow concepts o concurrency running 2 things in parallel o asynchronicity disentangle launching a task from getting its result what find empest 4/46 S.Ponce-CERN
Writing Parallel software 4 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Parallelism Definition the ability to make 2 or more things concurrently start end main flow parallel flow 2 nd parallel flow more parallel concepts concurrency running 2 things in parallel asynchronicity disentangle launching a task from getting its result
Writing Parallel software Intro c" Different flavors along the history Multiple levels of parallelism o machines(forever)/processors(for 40 years)/cores(for 25 years) o so not really a new topic Matching different programming techniques o RPC,MPI,map/reduce,.../multi-processing multi-threading Why is it trendy now o on many core processors you cannot avoid it individual cores are too slow o multi-core/multi-threading is tricky o and a major source of nasty bugs what find empect 5/46 S.Ponce-CERN
Writing Parallel software 5 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Different flavors along the history Multiple levels of parallelism machines (forever) / processors (for 40 years) / cores (for 25 years) so not really a new topic ! Matching different programming techniques RPC, MPI, map/reduce, .../ multi-processing / multi-threading Why is it trendy now ? on many core processors you cannot avoid it individual cores are too slow multi-core/multi-threading is tricky and a major source of nasty bugs
Writing Parallel software Intro Why would you use parallelism From the software point of view 2 main targets o multiplexing different tasks on the same machine o making one heavy task faster o by running different subparts of it in parallel From the hardware point of view o do not waste available resources o make best value for money Net consequence:we need to find things to be done in parallel what find expect 6/46 S.Ponce-CERN
Writing Parallel software 6 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Why would you use parallelism ? From the software point of view 2 main targets multiplexing different tasks on the same machine making one heavy task faster by running different subparts of it in parallel From the hardware point of view do not waste available resources make best value for money Net consequence : we need to find things to be done in parallel
Writing Parallel software 花5 Finding parallelism Task parallelism Sum act on the work flow Measures o break the work in tasks Sum squares o run several concurrently Data parallelism datal Computex2 osplit data in pieces data2 Computex2 o run on several concurrently data3 Computex2 what find empect 7/46 S.Ponce-CERN
Writing Parallel software 7 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Finding parallelism Task parallelism act on the work flow break the work in tasks run several concurrently Measures Sum Sum squares σ Data parallelism split data in pieces run on several concurrently data1 data2 data3 Compute χ 2 Compute χ 2 Compute χ 2
Writing Parallel software Task parallelism Concept parallelism achieved through the partition of load into "baskets of work" consumed by a pool of resources. Implications o requires splitting the processing into blocks o and dealing with dependencies between them o well adapted to data processing frameworks Thread/process pools usually useful a pool a workers is created at start o and reused during all processing tasks are mapped to available workers what find expect 8/46 S.Ponce-CERN
Writing Parallel software 8 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Task parallelism Concept parallelism achieved through the partition of load into “baskets of work” consumed by a pool of resources. Implications requires splitting the processing into blocks and dealing with dependencies between them well adapted to data processing frameworks Thread/process pools usually useful a pool a workers is created at start and reused during all processing tasks are mapped to available workers
Writing Parallel software Task parallelism example Event reconstruction case o processing consists in running a bunch of algorithms o originally as a sequence ○I■0■○ onow as a dependency graph o using a thread pool and a scheduler mapping what can be done to available threads 010 what find expect 9/46 S.Ponce-CERN
Writing Parallel software 9 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Task parallelism example Event reconstruction case processing consists in running a bunch of algorithms originally as a sequence now as a dependency graph using a thread pool and a scheduler mapping what can be done to available threads
Writing Parallel software Data parallelism Concept parallelism achieved through the application of the same transformation to multiple pieces of data Implications o requires independent pieces of data o may have an impact on data structures o and potentially on memory Practically for our event processing example 时Hp 10/46 S.Ponce-CERN
Writing Parallel software 10 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Data parallelism Concept parallelism achieved through the application of the same transformation to multiple pieces of data Implications requires independent pieces of data may have an impact on data structures and potentially on memory Practically for our event processing example