Laurent Daudet – Promises and challenges of massive-scale AI – the case of large language models

Abstract

OpenAi’s GPT-3 language model has triggered a new generation of Machine Learning models. Leveraging Transformers architectures at billion-size parameters trained on massive unlabeled datasets, these language models achieve new capabilities such as text generation, question answering, or even zero-shot learning – tasks the model has not been explicitly trained for. However, training these models represent massive computing tasks, now done on dedicated supercomputers. Scaling up these models will require new hardware and optimized training algorithms.

At LightOn – a spinoff of university research -, we develop a set of technologies to address these challenges. The Optical Processing Unit (OPU) technology makes some matrix-vector multiplications in a massively parallel fashion, at record-low power consumption. Now accessible on-premises or through the cloud, the OPU technology has been used by engineers and researchers worldwide in a variety of applications, for Machine Learning and scientific computing. We also train in an efficient manner large language models, such as PAGnol (demo at https://pagnol.lighton.ai ), the largest language model in French, that can be used for various

Click the image below to see slides

Laurent Daudet
CTO and co-founder at LightOn, Professor (on leave) of physics (Université Paris Cité)

Laurent Daudet is currently employed as CTO at LightOn, a startup he co-founded in 2016, where he manages cross-disciplinary R&D projects, involving machine learning, optics, signal processing, electronics, and software engineering. Laurent is a recognized expert in signal processing and wave physics, and is currently on leave from his position of Professor of Physics at the Université Paris Cité. Prior to that or in parallel, he has held various academic positions: fellow of the Institut Universitaire de France, associate professor at Université Pierre et Marie Curie, Visiting Senior Lecturer at Queen Mary University of London, UK, Visiting Professor at the National Institute for Informatics in Tokyo, Japan. Laurent has authored or co-authored more than 200 scientific publications, has been a consultant to various small and large companies, and is a co-inventor in several patents. He is a graduate in physics from Ecole Normale Supérieure in Paris, and holds a PhD in Applied Mathematics from Marseille University.