An artificial intelligence agent created by Facebook’s parent company Meta has become the first to achieve “human-level performance” in an online strategy game that emphasizes natural language negotiation and tactical coordination among players, according to a study published Tuesday, findings that set a new benchmark for AI.
The Meta Fundamental AI Research Diplomacy Team’s agent, named Cicero, is capable of imitating natural human language and can even analyze the goals, beliefs and intentions of its human partners to determine plans to succeed in the online strategy game Diplomacy, according to a peer-reviewed study published in the Science journal Tuesday.
Meta entered Cicero anonymously into an online Diplomacy league earlier this year, and in 40 speed games against human competitors the AI agent scored more than double the average person’s score and ranked in the top 10% of players who took part in more than one game, according to the study.
Cicero was able to pass as a human player against 82 participants the agent faced off against, and Meta researchers said they found no evidence to suggest the human players had any inkling they were competing against AI.
The finding represents a new breakthrough for AI technology, as agents in the past have been successful at competitive games like chess and GO, but games like Diplomacy have posed a challenge because of the high levels of negotiation, cooperation and competition needed to win.
Researchers found Cicero had a tendency to overuse masculine pronouns in conversation with other players. The agent used masculine pronouns–like he, him or his–75% of the time, while feminine counterparts–like she, her and hers–made up only 1% of pronoun usages, according to the study. Researchers said future work in AI recreating natural language in board games will be mindful of perpetuating stereotypes about gender and board games.
In October, two agents built by Meta beat out expert Diplomacy players. The game was created in the 1950’s and is set in Europe leading up to World War I. During the game, between two and seven players make and break alliances to occupy strategic cities and supply centers in order to win. Famous fans of the game include late President John F. Kennedy and former Secretary of State Henry Kissinger.