Data Technologies-CERN School af Compuaing 2019 Data Technologes-CERN School of Computing 2019 Why multiple pools and quality So,..what is data management Derived data used for analysis and accessed by Examples from LHC experiment data models thousands of nodes Need high performance.Low cost,minimal rellability (derived data can be recalculated) Raw data that need to be analyzed Need high performance.High reliability,can be expensive (small sizes) Raw data that has been analyzed and archived Must be low cost (huge volumes).High reliability (must be preserved),perlormanoe not necessary .Two building blocks to empower data processing Data pools with different quality of services Tools for data transfer between pools Data Technologles-CERN School af Compuang 2019 Data Technologles-CERN School of Computing 2019 Data pools But the balance is not as simple Different quality of services Many ways to split(performance,reliability,cost) Three parameters:(Performance,Reliability,Cost) Performance You can have two but not three Cost Reliability Expensive Performance has many sub-parameters Flash,Solld State Disks Cost has many sub-parameters -Mirrored disks Reliability has many sub-parameters Tapes Disks Scalability Electrical consumption Slow Unreliable Latency Ops Cost Throughput Consistency HW cost (manpower)9 Data Technologies – CERN School of Computing 2019 Why multiple pools and quality ?  Derived data used for analysis and accessed by thousands of nodes  Need high performance, Low cost, minimal reliability (derived data can be recalculated)  Raw data that need to be analyzed  Need high performance, High reliability, can be expensive (small sizes)  Raw data that has been analyzed and archived  Must be low cost (huge volumes), High reliability (must be preserved), performance not necessary 10 Data Technologies – CERN School of Computing 2019 So, … what is data management ?  Examples from LHC experiment data models  Two building blocks to empower data processing  Data pools with different quality of services  Tools for data transfer between pools 11 Data Technologies – CERN School of Computing 2019 Data pools  Different quality of services  Three parameters: (Performance, Reliability, Cost)  You can have two but not three Slow Expensive Unreliable Tapes Disks Flash, Solid State Disks Mirrored disks 12 Data Technologies – CERN School of Computing 2019 But the balance is not as simple  Many ways to split (performance, reliability, cost)  Performance has many sub-parameters  Cost has many sub-parameters  Reliability has many sub-parameters Reliability Performance Latency / Throughput Scalability Electrical consumption HW cost Ops Cost (manpower) Consistency Cost