From scientific instruments to smart-city sensors, the requirements of different data types can vary wildly. In the 
‘It’s an obvious point, but data on its own means nothing; what is important is the knowledge you extract from the data,’ explains Eduardo Quiñones, a leading researcher at BSC who has led multiple research efforts linking the horsepower of high-performance computing (HPC) to the real-time requirements of edge devices. ‘Clearly, you need computing capacity to extract this knowledge, but not all data needs the same capacity, nor the same computing resource.’
Among the requirements to be taken into account are, for example, interoperability; how data is shared (considering privacy, for example); or whether an outcome is needed in real time. ‘One of the key research questions EXTRACT addressed is: how do we design computing infrastructure such that the knowledge you extract from data is generated according to the requirements of different data types in the right computing resource?’ says Eduardo.
EXTRACT focused not on the data itself but on the compute continuum, investigating what kind of computing resources and abstraction layers were needed to process different kinds of data. ‘Most of the time, computing is not considered in its full complexity, but as independent solutions,’ says Eduardo. ‘For example, most data is processed in the cloud, managed by cloud-specific technologies. There are also edge-specific solutions for processing data at the edge, and solutions tailored to high-performance computing (HPC).’ In contrast, the EXTRACT approach aimed to facilitate data manipulation, processing, and analysis across the whole continuum, providing abstraction layers to decide where the data should be computed based on its requirements.
The EXTRACT architecture spans four main layers of the systems stack
EXTRACT tackled this problem across four main layers of the systems stack:
- The technologies and features that should be incorporated at data infrastructure layer.
- How the programmer expresses the workflow or pipeline that processes the data.
- What kind of abstraction should be used to simplify the experience for the programmer, who should be able to see the continuum as a homogenous environment, masking the complexity of the highly heterogeneous computing resources being leveraged.
- How the orchestrator gets the data requirements and distributes the data to the appropriate place on the computing continuum.
In terms of the data infrastructure, S3 cloud technology was chosen as the common substrate for data storage due to its interoperability capabilities. On top of this object-storage mechanism, two technologies were developed:
- A caching mechanism to unify remote and local access, allowing users to access the data in the same way regardless of their location.
- A mechanism to exploit parallelism, whereby data is split into chunks depending on its characteristics and distributed accordingly across the available computing resources. ‘This is fundamental when dealing with massive amounts of data which need to be processed on cloud or HPC resources,’ explains Eduardo. Such extreme data requirements are exemplified by the project’s radio-astronomy use case: data received from hundreds of antennas is filtered in real time and significantly compressed (by a factor of 100) before being stored on S3. ‘Our distribution mechanism takes this data and, based on its characteristics, divides it into chunks to be processed in parallel,’ says Eduardo. ‘This results in a science product upon which astronomers can conduct their research.’
On top of this data infrastructure, EXTRACT used different programming technologies to express the data-processing workflow. The serverless technology Lithops was chosen as, in common with other ‘function as a service’ software, it allows users to focus exclusively on their desired function, which in this case was ensuring that the data is in the right place, Eduardo notes. ‘Lithops wakes up quickly, executes the function, and terminates, which makes it particularly suitable for parallel execution.’
Python was the programming language used, while EXTRACT also incorporate low-level languages and programming models such as OpenMP, CUDA, and MPI for HPC execution, as well as experimenting with BSC’s in-house COMPSs programming model.
The cloud container-management software Kubernetes played a key role in abstracting the computing resources to enhance programmability. ‘One of the key innovations in EXTRACT was to match the execution model of workflows to a Kubernetes “multicluster” of edge, cloud and HPC resources, as outlined in a recent paper accepted for the IEEE World Congress on Services this year, so that the function is automatically mapped to the relevant computing resources,’ says Eduardo. ‘The multicluster abstraction layer combined with the workflow execution model allows computation to be sent to the specific resource so that when the data arrives it is ready to execute the function, managing dependencies between functions and network complexity.’
Another key component developed by the EXTRACT project is the orchestrator: this considers functions and decides where to deploy and where to execute the computation along the edge-cloud-HPC continuum, based on application requirements and taking into account the availability of different computing resources. Responding to the major advancements in artificial intelligence (AI) since the project proposal was developed, EXTRACT researchers also investigated how agentic technologies could be used to configure compute resources for efficient execution.
‘The agentic concept implies that each agent includes a language model which reasons about the best configuration each resource should include to execute the workflow,’ explains Eduardo. ‘As well as testing how to scale resources within a cluster using the cognitive orchestrator, we’ve also been experimenting with agents to configure the communication parameters to stream data between clusters. Deploying this kind of technology fits perfectly into EXTRACT. All the complexity under the hood is hidden by this distributed-infrastructure monitoring layer, using agentic AI not only at the application level, but also to manage resources.’
As well as being used to serve data, the computing continuum is also exploited for AI models, says Eduardo: ‘AI models evolve in a constant training loop. Trained on large machines – such as HPC facilities – once they reach a suitable performance level, they can be moved to the edge of the continuum for deployment – which means that you need model-serving functions.’
Members of the EXTRACT project team at HiPEAC 2025
Extreme data needs
EXTRACT put these technologies to the test in two demanding use cases. For the first, the radio-astronomy use case, EXTRACT technology was used to develop data-mining workflows that effectively reduce the huge amount of raw data produced by NenuFAR radio telescopes by a factor of 100. This allowed high-quality datasets to be populated and made available to the astronomy community (through the European Open Science Cloud-EOSC).‘Antennas from the Square Kilometre Array Pathfinder are constantly receiving signals from the universe, but with the technology we have today, we can only process around eight hours of data. For this use case, we needed to optimize the edge devices to compress the data with AI models using low-level programming languages like OpenMP or CUDA. Real-time data is then stored in S3 files, before being moved to the cloud where it can be processed in the parallel capabilities of the EXTRACT technology. We also addressed the need to constantly improve the AI models with model serving solutions.’
NenuFAR radio telescopes in Nançay, France, generate huge amounts of raw data
The second use case used the EXTRACT platform to develop, deploy and execute a data-mining workflow to generate personalized evacuation routes (PERs) in the event of an emergency, which would be delivered directly to the individual displayed in a mobile app. To do this, the EXTRACT platform had to process and analyse data composed of Copernicus and Galileo satellite data, data from internet of things (IoT) sensors installed across the city, 5G mobile signals, all of which flowed into an urban digital twin.
‘While not strictly speaking an extreme-data use case, in the sense that the amount of data is not massive, this use case exercises an important part of EXTRACT: real-time operation,’ explains Eduardo. ‘The AI model needs to be trained on HPC facilities with reinforcement-learning methods, based on multiple simulations generated by the urban digital twin in the cloud. It receives real-time data from multiple sensors: in the case of the EXTRACT test city, Venice, these include sensors for flooding, fire, and geolocation data from individuals’ mobile phones.’ Based on this data, EXTRACT has developed a semantic data information ontology, allowing the AI model to reason about the data.
The digital twin provides a snapshot of the current status of the city, which is then used to generate different simulations. Although requiring HPC facilities for the training phase, when an emergency occurs the trained model is then deployed at the edge, next to the antennas – and is thus a good fit for the 5G paradigm, says Eduardo.
The PER use case incorporated data from multiple sensors to help users navigate Venice’s dense maze of streets
However, he notes that finding a model capable of determining PERs is very complex. ‘You need to create a representation of the city that includes sufficient information for the AI agents to be able to route people to a safe spot. While Venice is not a large city, it has a dense concentration of streets.’
Apart from the intense requirements of the model, the PER use case also posed significant challenges in terms of cybersecurity; indeed, the distributed nature of computing resources used in EXTRACT as a whole meant that security was a major concern. ‘Although we didn’t plan to advance beyond the state of the art in security, we were conscious that it is a fundamental requirement for such a personalized use case,’ says Eduardo. ‘We deployed mechanisms to ensure data privacy, and also analysed the code to ensure that there were no leaks in the deployments on different computing resources.’
The technology developed by EXTRACT could also be used for other used cases, Eduardo adds. ‘Any use case that you can describe with a workflow, and where data is gathered from distributed data sources and processes across some form of compute continuum, can be executed using EXTRACT,’ he says. ‘Examples include management of the power grid, logisitics, and smart agricuture.’