Writing Parallel software Writing Parallel software Sebastien Ponce sebastien.ponce@cern.ch CERN Thematic CERN School of Computing 2022 1/46 S.Ponce-CERN
Writing Parallel software 1 / 46 S. Ponce - CERN Intro Threading Safety Solutions Writing Parallel software S´ebastien Ponce sebastien.ponce@cern.ch CERN Thematic CERN School of Computing 2022
Writing Parallel software Outline Introduction and expectations Thread-safety issues oScope what is parallelism ●Data races o Finding parallelism o Thread safety Expectations Thread-safety solutions 2Threading,theory and practice Avoid the problem oProcesses and threads o Replicate o Python threading ●Atomics C++threading and async o Locking Credits to Danilo Piparo for the original talk in previous tCSCs ..and all the content I shamelessly stole from it 2/46 S.Ponce-CERN
Writing Parallel software 2 / 46 S. Ponce - CERN Intro Threading Safety Solutions Outline 1 Introduction and expectations Scope : what is parallelism ? Finding parallelism Expectations 2 Threading, theory and practice Processes and threads Python threading C ++threading and async 3 Thread-safety issues Data races Thread safety 4 Thread-safety solutions Avoid the problem Replicate Atomics Locking Credits to Danilo Piparo for the original talk in previous tCSCs ... and all the content I shamelessly stole from it
Writing Parallel software Introduction and expectations Threading.theory and practice Introduction and expectations Scope what is parallelism Thread-safety issues Finding parallelism Expectations Thread-safety solutions what find expect 3/46 S.Ponce-CERN
Writing Parallel software 3 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Introduction and expectations 1 Introduction and expectations Scope : what is parallelism ? Finding parallelism Expectations 2 Threading, theory and practice 3 Thread-safety issues 4 Thread-safety solutions
Writing Parallel software 花5 Parallelism Definition o the ability to make 2 or more things concurrently main flow start end parallel flow what find empect 4/46 S.Ponce-CERN
Writing Parallel software 4 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Parallelism Definition the ability to make 2 or more things concurrently start end main flow parallel flow concepts concurrency running 2 things in parallel asynchronicity disentangle launching a task from getting its result
Writing Parallel software 花5 Parallelism Definition o the ability to make 2 or more things concurrently 2d parallel flow main flow start →end parallel flow what find empect 4/46 S.Ponce-CERN
Writing Parallel software 4 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Parallelism Definition the ability to make 2 or more things concurrently start end main flow parallel flow 2 nd parallel flow concepts concurrency running 2 things in parallel asynchronicity disentangle launching a task from getting its result
Writing Parallel software 花5 Parallelism Definition the ability to make 2 or more things concurrently more parallel 2nd parallel flow main flow start end parallel flow what find empect 4/46 S.Ponce-CERN
Writing Parallel software 4 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Parallelism Definition the ability to make 2 or more things concurrently start end main flow parallel flow 2 nd parallel flow more parallel concepts concurrency running 2 things in parallel asynchronicity disentangle launching a task from getting its result
Writing Parallel software “ Parallelism Definition o the ability to make 2 or more things concurrently more parallel 2nd parallel flow main flow start end parallel flow concepts o concurrency running 2 things in parallel o asynchronicity disentangle launching a task from getting its result what find empest 4/46 S.Ponce-CERN
Writing Parallel software 4 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Parallelism Definition the ability to make 2 or more things concurrently start end main flow parallel flow 2 nd parallel flow more parallel concepts concurrency running 2 things in parallel asynchronicity disentangle launching a task from getting its result
Writing Parallel software Intro c" Different flavors along the history Multiple levels of parallelism o machines(forever)/processors(for 40 years)/cores(for 25 years) o so not really a new topic Matching different programming techniques o RPC,MPI,map/reduce,.../multi-processing multi-threading Why is it trendy now o on many core processors you cannot avoid it individual cores are too slow o multi-core/multi-threading is tricky o and a major source of nasty bugs what find empect 5/46 S.Ponce-CERN
Writing Parallel software 5 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Different flavors along the history Multiple levels of parallelism machines (forever) / processors (for 40 years) / cores (for 25 years) so not really a new topic ! Matching different programming techniques RPC, MPI, map/reduce, .../ multi-processing / multi-threading Why is it trendy now ? on many core processors you cannot avoid it individual cores are too slow multi-core/multi-threading is tricky and a major source of nasty bugs
Writing Parallel software Intro Why would you use parallelism From the software point of view 2 main targets o multiplexing different tasks on the same machine o making one heavy task faster o by running different subparts of it in parallel From the hardware point of view o do not waste available resources o make best value for money Net consequence:we need to find things to be done in parallel what find expect 6/46 S.Ponce-CERN
Writing Parallel software 6 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Why would you use parallelism ? From the software point of view 2 main targets multiplexing different tasks on the same machine making one heavy task faster by running different subparts of it in parallel From the hardware point of view do not waste available resources make best value for money Net consequence : we need to find things to be done in parallel
Writing Parallel software 花5 Finding parallelism Task parallelism Sum act on the work flow Measures o break the work in tasks Sum squares o run several concurrently Data parallelism datal Computex2 osplit data in pieces data2 Computex2 o run on several concurrently data3 Computex2 what find empect 7/46 S.Ponce-CERN
Writing Parallel software 7 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Finding parallelism Task parallelism act on the work flow break the work in tasks run several concurrently Measures Sum Sum squares σ Data parallelism split data in pieces run on several concurrently data1 data2 data3 Compute χ 2 Compute χ 2 Compute χ 2
