The Open Automatic Speech Recognition (ASR) Leaderboard has expanded its reach with the inclusion of Egyptian and Nigerian English accents, marking a significant step towards greater inclusivity and accuracy in speech recognition technology. This expansion is a result of a new collaboration with Liva AI, a company specializing in collecting multimodal interaction data through consumer products.
The collaboration introduces two new private datasets to the leaderboard: “Liva AI Conversational EG” for Egyptian English and “Liva AI Conversational NG” for Nigerian English. These datasets, comprising 1.34 and 1.55 hours of conversational, spontaneous speech respectively, are punctuated, cased, and include disfluencies, providing a more realistic evaluation environment.
The Open ASR Leaderboard, hosted on Hugging Face, is a platform designed to benchmark and compare various automatic speech recognition systems. It aims to provide reproducible and transparent evaluations, moving beyond the traditional focus on short-form English to encompass multilingual and long-form speech recognition.
With the addition of these new accents, the leaderboard now offers a more detailed breakdown of accent performance, categorized according to Kachru’s Three Circles of English. This includes distinctions between US and non-US English (UK, Australia, Canada, India, Nigeria, Egypt), as well as between the Inner Circle (US, UK, Australia, Canada) and the Outer & Expanding Circles (India, Nigeria, Egypt). This nuanced categorization helps researchers and developers understand and address performance variations across different Englishes.
Liva AI’s contribution is vital as it focuses on collecting natural, high-context behavior derived from real product usage, rather than solely relying on artificially elicited interactions. This approach allows for the design of products that can bridge specific model gaps across various domains, industries, languages, modalities, and interaction types.
The broader effort behind the Open ASR Leaderboard is to enhance representation in ASR evaluations by incorporating more accents and languages. The inclusion of private data also serves to minimize “benchmarking” or “benchmaxxing,” where models might be over-optimized for specific public test sets, thereby ensuring more trustworthy evaluations.
The Open ASR Leaderboard has been evolving since its inception, previously adding private data splits for scripted and conversational speech from five accents provided by Appen and DataOceanAI. It also incorporated Indian-accented English data from Voice Arena. The platform evaluates both open-source and proprietary systems, ranking them based on metrics such as Word Error Rate (WER) for accuracy and inverse real-time factor (RTFx) for efficiency.
This latest development underscores a continued commitment to advancing ASR technology for a global audience, ensuring that speech recognition systems perform reliably across the diverse linguistic landscape of English speakers worldwide.
❤️ Support Independent Journalism
Your contribution keeps our reporting free, fearless, and accessible to everyone.
Or make a one-time donation
Secure via Razorpay • 12 monthly payments • Cancel anytime before next cycle
(We don't allow anyone to copy content. For Copyright or Use of Content related questions, visit here.)
The Chenab Times News Desk
