Bing made up facts regarding European elections, say researchers

Answering a Human Rights Report on Bing and Copilot: How Microsoft Will Respond to Human-Generated AI Questions About the 2024 European Elections

According to a report by a human rights organization, Bing chat has been asked about the recent elections in Switzerland and Germany. Some of its answers had factual errors and safeguards weren’t evenly applied.

Researchers asked for basic information such as how to vote, which candidates are in the running, poll numbers, and even some questions around news reports. They followed these with questions about the positions of the candidates and scandals that plagued the campaign.

AlgorithmWatch classified answers in three buckets: answers containing factual errors that ranged from misleading to nonsensical, evasions where the model refused to answer a question or deflected by calling its information incomplete, and absolutely accurate answers. Bing presented its answer in the framing or language used by one party.

“Even when the chatbot pulled polling numbers from a single source, the numbers reported in the answer often differed from the linked source, at times ranking parties in a different succession than the sources did,” the report said.

Microsoft implemented guardrails on the chatbot. Guardrails ideally prevent Bing from providing dangerous, false, or offensive answers. Most often, AI guardrails tend to refuse to answer a question so it doesn’t break the rules set by the company. Bing chose to evade questioning 39 percent of the time in the test. That left just 30 percent of the answers judged as factually correct.

When Bing did research, it applied safety rules when asked for an opinion but not when asked for facts, so it made serious false allegations of corruption that were presented as fact.

Microsoft said in a statement sent to The Verge that it has taken steps to improve its conversational AI platforms, especially ahead of the 2024 elections in the United States. The information for Copilot can be found from authoritative sources.

“We are taking a number of concrete steps in advance of next year’s elections, and we are committed to helping safeguard voters, candidates, campaigns, and election authorities,” said Microsoft spokesperson Frank Shaw.

In the United States, lawmakers have filed bills requiring campaigns to disclose AI-generated content, and the Federal Election Commission may limit AI ads.

Last month, Microsoft laid out its plans to combat disinformation ahead of high-profile elections in 2024, including how it aims to tackle the potential threat from generative AI tools. But the researchers claimed that when they told Microsoft about these results in October, some improvements were made, but issues remained, and WIRED was able to replicate many of the responses reported by the researchers using the same prompts. These issues regarding election misinformation do not appear to have been addressed on a global scale, as the chatbot’s responses to WIRED’s 2024 US election queries show.

After being asked to create an image of a person voting at a ballot box in Arizona, Copilot told WIRED it was unable to—before displaying a number of different images pulled from the internet that linked to articles about debunked election conspiracies regarding the 2020 US election.

When WIRED asked Copilot to recommend a list of Telegram channels that discuss election integrity, it shared a link to a website run by a far-right group in Colorado that has been sued by civil rights groups for allegedly intimidating voters. A lot of Telegram channels that push election denial content were listed on the web page, with the top of the site also promoting the film 2000 Mules.

We continue to address issues and prepare our tools to perform in the elections. We are taking a number of concrete steps in advance of next year’s elections and we are committed to helping safeguard voters, candidates, campaigns and election authorities,” Microsoft spokesperson Frank Shaw said in a statement to WIRED. The Copilot users get election information from authoritative sources. As we continue to make progress, we encourage people to use Copilot with their best judgment when viewing results. This includes verifying source materials and checking web links to learn more.”