Chatbot Arena Benchmark Under Fire for Alleged Bias in AI Scoring
April 30, 2025, 8:20 pm
A recent study alleges that LM Arena—the team behind the popular Chatbot Arena—has manipulated its benchmark evaluations to favor select AI labs. Critics argue this approach undermines the fairness of widely recognized scoring methods, fueling demands for greater transparency and accountability in the process.
Bluesky: @techcrunch.com, @techcrunch.com
Google’s AdSense advertising network started supporting ads inside users’ chats with some third-party AI chatbots earlier this year, Bloomberg reported. The company is rolling out the feature following tests with AI search startups iAsk and Liner, the report said, citing anonymous sources familiar…
You also mentioned the whole Chatbot Arena thing, which I think is interesting and points to the challenge around how you do benchmarking. How do you know what models are good for which things? One of the things we've generally tried to do over the last year is anchor more of our models in our Meta…
A new paper from AI lab Cohere, Stanford, MIT, and Ai2 accuses LM Arena, the organization behind the popular crowdsourced AI benchmark Chatbot Arena, of helping a select group of AI companies achieve better leaderboard scores at the expense of rivals. According to the authors, LM Arena allowed some…
The Chatbot Arena has become the go-to place for vibes-based evaluation of LLMs over the past two years. The project, originating at UC Berkeley, is home to a large community of model enthusiasts who submit prompts to two randomly selected anonymous models and pick their favorite response. This…
permalink / 4 stories from sources in 27 hours ago #ai #openai #aiethics #chatgpt
More Top Stories...
Apple Q2 2025 Earnings Exceed Expectations Amid Robust Market Turbulence
Apple released its Q2 2025 financial results, reporting $95.4B in revenue and $24.8B in profit. The robust figures, reported just before looming tariff threats, have impressed investors and analysts alike, demonstrating resilience despite market headwinds. Analysts note that these numbers underscore a solid performance in a challenging economic environment. More...
Apple shifts U.S. device sourcing to India and Vietnam
Apple is retooling its supply chain strategy by shifting most U.S. device shipments—even as iPhone news splashes elsewhere—to India and Vietnam, leaving China with minimal exposure. The move aims to mitigate risks and balance global production while keeping investors on their toes. More...
Apple updates App Store rules after court order shakeup
Apple's App Store guidelines now include provisions for external purchase links as mandated by a recent court order. In a curious twist of legal and developer ingenuity, Spotify swiftly adapted its app to integrate these external links—an unexpected yet practical consequence of the ruling amid a broader reexamination of digital commerce norms. More...
Google Expands AI Mode to Enhance Search Capabilities
Google is upping its search game by expanding access to its AI Mode, an experimental feature that promises a more conversational and intuitive query experience reminiscent of ChatGPT. The rollout brings additional functionality and continuity between sessions, reflecting Google’s ongoing efforts to blend traditional search with modern AI innovations. More...
Microsoft Prepares to Launch Elon Musk’s Grok AI on Azure
Microsoft is gearing up to integrate Elon Musk’s Grok AI model into its Azure cloud ecosystem following discussions with xAI. The strategic move aims to bolster its AI infrastructure and offer advanced capabilities amid growing industry anticipation and competitive pressure in the rapidly evolving tech landscape. More...
Related Tags
Artificial Intelligence
Judicial Storm Over Meta’s AI Copyright Dispute (5 hours ago)
Amazon Introduces New AI Coding Service to Challenge Startup Rivals (7 hours ago)
Microsoft Prepares to Launch Elon Musk’s Grok AI on Azure (8 hours ago)
openai
AI Ethics
Judicial Storm Over Meta’s AI Copyright Dispute (5 hours ago)
Google Expands AI Mode to Enhance Search Capabilities (11 hours ago)
Nvidia and Anthropic Clash Over US AI Chip Export Restrictions (11 hours ago)