The Linley Spring Processor Conference 2022 – which took place last April 20th and 21st – saw the participation of numerous sponsor companies, many of them offering deep learning acceleration solutions. This week EDACafe takes a quick look at the conference content, mostly focusing on some technology trends and some new announcements. Full proceedings of the event can be accessed from www.linleygroup.com, the website of the technology analysis firm now owned by Canadian reverse engineering company TechInsights.
Ever-growing NLP models
In his keynote speech, TechInsights’ principal analyst Linley Gwennap pointed out that language-processing models keep growing at an impressive pace: Alibaba’s M6 has 10 trillion parameters. Model size is limited by training time (compute cycles): for example, training the GPT-3 model using one thousand A100 GPUs takes more than one month. Rapid growth has been achieved by moving to large and very expensive clusters. Recent progress focuses on adding parameters using fewer GPU cycles: for example, Alibaba reports training M6 required only 15% the time of smaller GPT-3. Training can be accelerated through ‘model sharding’, which divides a model across many chips. This requires complex software, possibly with manual assistance. Scaling massive models across servers and racks, sharding requires high-bandwidth connections.
