First ECLAT Hackathon
The ECLAT hackathon, held from 3 to 5 February 2025, brought together researchers and engineers to optimise the DDF pipeline, which is essential for processing the LOFAR radio telescope’s vast data streams, by consolidating MPI integration, I/O optimisation and application algorithms.
ECLAT Hackathon: Optimising the DDF pipeline for large-scale astronomical surveys
The aim of the ECLAT hackathon, which took place from Monday 3 to Wednesday 5 February 2025 at Paris Observatory, was to optimise and increase the computing capacity of DDF-pipeline (Direction-Dependent Facet calibration) reduction. Used to process the huge data streams generated by the Low Frequency Array Radiotelescope (LOFAR), DDF-pipeline is a set of tools and algorithms that plays a key role in improving image quality by correcting for instrumental distortions and ionospheric effects that vary according to the direction in which the antennas are positioned.
With the arrival of LOFAR 2.0 and the prospects of the Square Kilometre Array Observatory (SKAO), the need to scale up is becoming crucial if ever-larger volumes of data are to be processed efficiently. It is becoming imperative to improve performance by exploiting distributed computing architectures and optimising the use of hardware resources.
Three areas of focus
During the ECLAT hackathon, which brought together 14 researchers, engineers and doctoral students from ECLAT’s partner laboratories, several teams worked together on three main areas of optimisation:
1. Consolidation of MPI integration: Implementation of multi-node parallelization on the calibration and imaging tool, and deployment on the French Jean Zay supercomputer dedicated to scientific computing, to improve the efficiency of distributed computing;
2. I/O and memory access optimisation: Analysis and optimisation of I/O operations to minimise bottlenecks;
3. Developing new application algorithms: In particular, improving deconvolution methods for more accurate reconstruction of radio images.
Tangible, open results
The aim of the hackathon was to make tangible progress:
- Parallelized code with MPI made available in open source ;
- Deployment and testing of the code on the Jean Zay supercomputer ;
- Detailed analysis report and integration of optimisations into new branches of the code in conjunction with the NumPEx research programme ;
- Validation of new algorithmic approaches on simulated data, with possible application to real data in connection with the RisingStars project.
“This ECLAT hackathon was a great human experience, marked by great enthusiasm and a real exchange of ideas between the teams. Working on optimising our codes and scaling them up represents a major technical challenge. And the stakes are considerable, especially with the arrival of LOFAR 2.0 and the prospects of the SKA, which are forcing us to push back the limits of our computing capacity. We have managed to make progress on certain critical points, such as MPI parallelization, but we still have a long way to go to fully resolve this large and complex numerical problem. The hackathon has laid some solid foundations, and I’m convinced that the advances made here will pave the way for innovative solutions for processing SKA data.”
Participants in the January 2025 ECLAT hackathon at Paris Observatory
