This is to collect ALICE feedback to be given at the HSF Computing White Paper. From ALICE side, of course, the basic idea would be to point out all the possible“collaboration” areas which are in our O2 TDR and steer the process so that it matches as much as possible our roadmap. Notice that a number of white papers are already under editing as google documents and we should probably start to add ALICE take in there.
ALICE input, following the questions in the work paper
Define what we want to consider as conditions data(based on past experience).
Conditions data are considered to be all detector-related data which vary with time and define the performance in subsequent processing steps, up to final physics analysis. These include environment data, like temperature, pressure, operating parameters; derived data, for example calibration coefficients, geometry alignment; analysis-specific parameters, for example detector response functions.
Study of workflows and access patterns based on LHC experiment past experience. This is important for the data model which determines how the conditions data are managed.
Gathering of conditions data consists of the following steps, some of these can be strictly experiment-specific. The following enumerates the data types, collection techniques and basic conditions data parameters in ALICE. Environment data and basic detector calibration parameters are collected quasi-online from various sources: detector control DBs, High Level Trigger processing, LHC performance data and DAQ. The collection of this data was delegated to the so-called Shuttle system, which at specific time intervals, usually end of run, queries the various information sources, packs the data into detector-specific objects in a format suitable for Offline processing(ROOT) and publishes them on the Grid. This system is suitable for the current model of data processing which is done after the RAW data has been collected and stored. After the ALICE upgrade and starting in 2020, a large part of the data processing will be done in the O2 facility, which necessitates a different approach to this specific type of conditions at, namely it should be provided at the time of data taking and will be used by the data compression algorithms running in O2. To achieve the timely delivery, the relevant part of the conditions data will be injected in the RAW data stream and will subsequently be copied, after proper aggregation, to the analysis object data(AODs). The second level(derived) of conditions data results from asynchronous RAW data processing. This conditions data will be stored separately from the RAW data in an object store. We foresee that these are ROOT objects put as blobs in a DB and properly annotated to correspond to the RAW data chunks. Since after the upgrade, the RAW data will be stored in time-frames and not in discrete triggered events, the annotation will consists of the beginning-end time-stamps of the time-frames and versioning parameter, which will allow for later update with better quality derived conditions data.
Evaluation of data volumes: we can foresee that the volume of conditions data involved in a large HEP experiment can reach several Terabytes over a period of data taking(conditions data do not scale in the same manner as physics event data with the luminosity, they tend to scale with time).
The conditions data volumes tend to be a fraction of the RAW data volumes and with judicious splitting, the size per RAW data chunk is negligible. The challenge is not the volume, but access frequency and versatility of the conditions data supporting framework to assure efficient access from all possible applications - from reconstruction to analysis. A mitigating circumstance is that the conditions data will be updated infrequently and is can be considered in most of the use cases as static. This can be exploited by aggregating many objects within a given time interval into so-called snapshots, which can be a physical copy of all objects(full snapshot) or a meta-link to the annotations of the individual objects(link snapshot). The snapshots reduce the access frequency to the primary conditions data object store and at the same time allow for re-processing or re-analyzing the data with exactly the same condition objects as in past cases.
Distributed computing environment: conditions data are used by many jobs processing physics event data. Standard client-server(job being the client and a relational DB being the server) architectures have shown their limitations and forced LHC experiments to adopt well conceived caching solutions.
The above main methods - a powerful object store coupled with annotations stored in a conditions DB plus the snapshot method is used successfully in ALICE during Run1 and Run2 with hundreds of thousand jobs on distributed computing resources. We will expand this method to Run3 and beyond by replacing the formerly used Grid storage as object store and MySQL as annotation DB with updated modern tools, for example CEPH coupled with Cassandra NoSQL. Caching and/or replication of the object store can increase trivially the scalability of this solution.
Related to the last point, we have also to carefully study here the impact on the architecture coming from the usage of HPC centers(without external connectivity) and in particular event-based workflows, ensuring that conditions access is not a bottleneck.
For a non-standard access to conditions data, for example HPC and network-less environment, the full snapshot method provides a convenient and fast method to distribute conditions data to the tasks running on HPC. This has been exercised and shown to work well in environments like the Titan supercomputer at ORNL, where the snapshot is copied to the local parallel filesystem accessible by all Titan computing nodes .
End-user analysis also uses conditions data, should this data also be covered by a conditions architecture or should much simpler designs be preferred? End-users have a clear preference for understanding all data used in their analysis jobs(c.f. text files containing calibration data), how should conditions data storage/access be simplified to satisfy the end-user community?
With the aforementioned methods, all levels of conditions data are accessible at all levels of data processing - from synchronous compression during data taking to asynchronous processing at the O2 facility and T1 computing centres, don the chain to physics analysis. Past experiences show that reduction of format proliferation is a key element to assure uniform and efficient access to conditions data, thus we intend to keep the conditions object solely in the RA data stream and secondary conditions data in ROOT containers.
Back end solutions: relational databases, NoSQL databases, file-system
The choice depends both on data volumes and data model, as well as on the kind of informations we want to extract from the stored conditions data.
Data model: study the options and the effects on performance, maintainability, etc. How far can we / should we standardise choices and what are the pros and cons.
Data access layer: the application layer devoted to data access, which in a first approximation should be capable of supporting multiple backend solutions. Several standards are available with different degrees of maturity. Difficult to estimate where we will go in 5/10 years from now.
Caching layer: caching is a key element for the present conditions data access. Again caching solutions are available in the open source industry and should be evaluated. There is somehow a very tight link between the data model, data access layer and caching. This should be studied as well.
Client layer(interface to production framework): an effort to keep client disentangled from all previous architectural elements is important to improve maintainability.
Client layer(payload management / upload): emphasis again on disentangling from other architectural elements, but also simplicity here is key - we should not require super experts to perform basic conditions DB operations.
Configuration layer(c.f. Global tags) of clients vis a vis the framework should be studied, e.g. the management view(global tags) may not be the same as a job-view where emphasis should be on efficiency.
Multi-language support(access to conditions may occur from different clients and not only from production jobs).
Data model and data preservation: study the evolution of payload technology choices particularly wrt backend storage evolution - how robust can we expect to be? Are some choices better than others in this regard, and can we standardize payload format(c.f. CMS example), what are the pros and cons?
General Questions for all working groups:
In addition to addressing topical issues, each group will be expected to address questions which cut across boundaries, including for example:
What are the specific challenges for the HL-LHC?
What opportunities exist to exploit new or advanced algorithms(e.g. deep learning)?
How can emerging technologies improve the bang-per-buck and what software evolution is needed to exploit them?
Which problems are specific to individual experiments and which are common to the HL-LHC experiments or to HEP and nuclear physics experiments more generally? What common tools could be developed to address these problems?
What is required to make common software packages sustainable?
Interesting links
CWP Topics:
Detector Simulation
Triggering
Conditions Storage