What, for you, have been the main milestones in machine learning (or computing systems more generally) since HiPEAC began in 2004?
Primarily, the advent of deep neural networks (DNNs) around 2006, and in 2012, the very significant progress marked by AlexNet which won the Large Scale Visualization Challenge by a large margin. From that point on, it became increasingly clear that we were on a path of exponential progress of the capacity of neural networks (NNs), which carries to this day through large language models (LLMs).
From a computing systems point of view, it became also clear back then that (1) the application scope of such NNs was so broad that it would warrant specialized/optimized architectures, and (2) that graphics processing units (GPUs) at the time, while great high-performance chips, did not pack enough compute capacity to cost effectively support the broad deployment of NN algorithms.
These notions led to a series of academic work on artificial intelligence (AI) accelerators, the first one being an ISCA paper in 2012 which laid the foundations for AI accelerators, followed by the academic work of the DianNao series of AI accelerators. Then to industrial work, especially tensor processing units (TPUs) at Google, which was considering broadly deploying NN algorithms since 2012 or so.
Your paper ‘ShiDianNao: Shifting Vision Processing Closer to the Sensor’ was recognised by the ISCA Retrospective 1996-2020. What do you think made this paper (and the DianNao accelerator work more generally) special, and what has the lasting impact been?
Thank you for kindly acknowledging this work. The real foundational articles were the 2012 ISCA article which explored how efficient NN accelerators could eventually become one day (and as an industry, we are trying to get as close as possible to that limit), and the first DianNao paper published at ASPLOS in 2014 which proposed the first NN accelerator design for inference applications (it won the best paper award at ASPLOS, and the test of time award in 2024). The other significant paper was the follow up DaDianNao paper on an NN supercomputer for training applications, published at MICRO in 2014 (and which also won the best paper award).
Given the initial scepticism about neural networks, has their stratospheric rise since 2012 taken you by surprise or did you see this coming?
Until 2006 or so, while I was interested in neural networks, their algorithmic progress didn’t seem significant enough, compared to alternative image analysis algorithms. But the progress of DNNs since 2006 started to be quite significant and made a lot of waves, albeit in a small community (at the time).
Because of my personal interest in neural networks, I got advance signals that something profound was happening with NNs, and it is true that, especially in the computing systems community, initial propositions for NN accelerators were met with tremendous scepticism. When I gave a keynote at ISCA in 2015, I gently reminded the community that my first ISCA paper on NNs (2012) was rejected many times before I could publish it. Even though the programme committees of conferences in computing systems state that they welcome innovative publications, the reality is that the consensus decision process of program committees leads to conservatism and filters out innovative ideas.
While no one really predicted the progress of AlexNet in 2012, once the regular progress of NN algorithms became clear (around that date), their potential application scope was so broad that it was fairly clear that they would lead to massive industrial applications and investment.
The other scientific element that gave a lot of confidence in that continued progress is the simple but powerful realization, in the ML community, that increasing the size of NNs seemed to consistently lead to significantly improved capacity. That spectacular trend continues to this day with LLMs.
A slide from Olivier’s presentation on neural networks at ISCA in 2010Do you think HiPEAC has helped nurture the development of neuromorphic computing in Europe? What priorities do you think the community should focus on over the next few years?
As explained above, the computing systems community at the time was quite sceptical about neuromorphic systems, and HiPEAC was no different. More broadly, Europe made a strategic mistake about neuromorphic computing by choosing the wrong path. It invested in brain-inspired architectures (with the Human Brain Project) in 2013, and most European funds went to such architectures, in spite of clear signs since 2006, and especially since 2012, that artificial neural networks were extremely promising. That European choice is what led me, personally, to seek environments more focused on industrial applications and artificial neural networks.
But I do see very interesting and encouraging signs that Europe wants to reclaim its computing systems future with the European Chips Act. I would also encourage European investments in fabs for high-performance chips, and the HiPEAC community to contribute to that growing ecosystem. Also, because of the massive computing needs of AI, I would encourage the HiPEAC community to not just focus on chips, but the whole hardware stack, from chips to data centres and even energy production. Innovation must happen and be integrated across the whole hardware stack.
What, for you, has been the main impact of HiPEAC more generally over the last 20 years?
I am very fond of the role of HiPEAC in Europe. As a co-founder of HiPEAC, I think that the European Commission helped us establish a powerful academic community in Europe, and I am also grateful for the steadfast leadership of Koen De Bosschere over so many years. Prior to HiPEAC, we were joking that Europeans would mostly meet in US conferences. I also praise the HiPEAC community for its tight interactions with the European industry, and making such academic / industry relationships a priority from its inception.
Chips (and computing systems as a whole) are a special technology in that they are used by virtually every industry on the planet, only energy may surpass chips in its industrial ubiquity. So it is extremely important for Europe’s future that the continent invests in that technology, and HiPEAC can be one of the powerful engines to help make it happen. My only concluding advice, echoing some of the aforementioned comments, would be to carefully focus on industrially relevant applications and innovations.
Further reading:
Olivier Temam, ‘A Defect-Tolerant Accelerator for Emerging High-Performance Applications’, ACM/IEEE International Symposium on Computer Architecture (ISCA), June 2012 [pdf][BibTeX]Tianshi Chen et al., ‘DianNao: a small-footprint high-throughput accelerator for ubiquitous machine-learning’, ASPLOS '14: Proceedings of the 19th international conference on Architectural support for programming languages and operating systems, pp.269-284, doi: 10.1145/2541940.2541967
Y. Chen et al., ‘DaDianNao: A Machine-Learning Supercomputer’, 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture, Cambridge, UK, 2014, pp. 609-622, doi: 10.1109/MICRO.2014.58
Z. Du et al., ‘ShiDianNao: Shifting vision processing closer to the sensor’, 2015 ACM/IEEE 42nd Annual International Symposium on Computer Architecture (ISCA), Portland, OR, USA, 2015, pp. 92-104, doi: 10.1145/2749469.2750389