FAIR Universe

Uncertainty aware large-compute-scale AI platform for high energy physics and cosmology

Uncertainty-Aware AI High Energy Physics Cosmology Bias in Data Uncertainties

Weak gravitational lensing reveals the large-scale structure of the universe by observing subtle distortions in galaxy shapes due to intervening matter. To fully exploit the non-Gaussian features present in the cosmic web, modern machine learning (ML) methods have become increasingly important. These approaches, including deep learning and simulation-based inference, have been shown to extract significant more information in weak lensing maps than traditional techniques. However, different analyses assume different dataset setups and lead to different results, making it hard to directly compare with existing approaches. Furthermore, most (if not all) of these methods rely heavily on simulations that may not accurately represent real data due to modeling approximations and missing systematics. We can frame this problem as an out-of-distribution (OoD) detection task; that is, whether or not the actual observations deviate from the simulation or forward model used to train our models?

Through this competition, participants will analyze a suite of carefully designed mock weak lensing convergence maps (2D fields similar to images). Participants have to develop methods for the OoD detection, with the goal of identifying a fraction of the test data that deviates from the training distribution.

The outcomes of this competition are expected to guide the development of next-generation weak lensing analysis pipelines, foster cross-disciplinary collaboration between the astrophysics and machine learning communities, and ultimately improve the reliability of cosmological inference from current and upcoming surveys such as LSST, Euclid, and the Roman Space Telescope. By explicitly addressing simulation-model mismatch and the need to quantify systematic uncertainties, this competition emphasizes scientific robustness and interpretability, aligning with the growing emphasis on trustworthy ML in scientific domains.

About

The FAIR Universe project is a collaboration between and include scientists from Lawrence Berkeley National Laboratory, Université Paris-Saclay, University of Washington, Institute for Advanced Study, and ChaLearn. The project is funded by the Department of Energy .

The collaboration is building a large-compute-scale AI platform for the hosting of large scientific datasets, models, and machine learning competitions to drive forward discoveries in high energy physics and cosmology. The initial focus is on discovering and minimizing systematic uncertainties in High Energy Physics through a sequence of challenges that will bring together researchers across physics and machine learning.

The collaboration includes scientists from Lawrence Berkeley National Laboratory, Université Paris-Saclay, the University of Washington, and ChaLearn, a non-profit that organizes challenges to stimulate research in machine learning.

Goals

We are building an open, large-compute-scale AI ecosystem for sharing datasets, training large models, fine-tuning those models, and hosting challenges and benchmarks. Hosting of all these datasets and benchmarks will be achieved by interfacing the NERSC HPC center to Codabench , a recently developed open-source platform, to provide a next generation reproducible-science AI ecosystem. This project will build the essential pieces of such an ecosystem through deployment of:

1. Three HEP systematic uncertainty datasets and tasks, of increasing sophistication, tailored for studies of systematic-uncertainty aware AI techniques, in particle physics and cosmology.
2. A set of HEP-AI challenges and long-lived task and algorithm benchmarks addressing compelling questions about the impact of systematic effects in AI models.
3. An HPC-enabled AI benchmark platform capable of hosting datasets and models; producing new simulated datasets; applying new AI algorithms on existing datasets; and applying uploaded AI algo- rithms on new datasets.

The collaboration with Codabench and NERSC will ensure that the project platform, benchmarks and a portfolio of algorithms will be curated and made accessible, and therefore continue to benefit the HEP community, as well as other sciences and the machine learning research community well beyond the end of the project. The research community will benefit from being exposed to well-established, empirical UQ approaches for estimation that experimenters have deployed on problems with hundreds of systematic effects. The develop- ment of principled methodologies to quantify the impact of systematic effects in the training and inference of ML models, will increase the trust of the scientific community on AI methods applied to experimental high-energy physics and beyond. The progressive structure of our challenges will bring together activity across particle physics and cosmology. Finally, both the methods and platform developed in this project will serve as a foundation for future AI challenges and benchmarks in high-energy physics, scientific and industrial applications.

News

Partners