December 21, 2023

Contact:
Keelin Wolfe
kwolfe9@gmu.edu

 

GMU Scalia Law School’s NSI Cyber and Tech Center and the GMU Mercatus Center
Host U.S.-China Wargame Pitting Humans Against Generative AI Capabilities

Initial Results Show Key Areas of Commonality Between Human and AI Players,
but Materially Different Choices Result in Highly Varied Outcomes

Arlington, VA – Last Thursday, the National Security Institute’s Cyber and Tech Center at the GMU’s Scalia Law School partnered with GMU’s Mercatus Center to host their first-ever wargame simultaneously assessing the performance of human participants against an off-the-shelf artificial intelligence large-language model.

The wargame was hosted at GMU’s Mason Square Campus in Arlington, VA, just minutes away from downtown Washington, DC and the Pentagon and located in the heart of the Commonwealth of Virginia’s emerging technology corridor. The human participants and AI model were given the same scenario and identical inputs, and both the human and AI teams played the United States.  On the human team, the wargame included participation from a range of former senior national security officials who served in the U.S. military, the intelligence community, the White House, and the Departments of State, Defense, and Justice, as well as on Capitol Hill.  On the AI side, the live wargame was conducted using OpenAI’s GPT4.  Individual players (as well as the AI model) served in various roles of the National Security Council’s Deputies Committee, which was charged with making recommendations to the NSC Principals Committee with the understanding that the Principals Committee and the President would accept and act on the recommendations made by the Deputies Committee.  The role of China was played in both games by human participants on the NSI staff responding in real time to the actions taken by the human and AI national security players.

The base scenario was developed by the Wargaming and Crisis Simulation Initiative at the Hoover Institution and involved a crisis in the Taiwan Straits. An initial simulated invasion of Taiwan by the Chinese Communist Party led to escalated tensions and a potential conflict between U.S. and Chinese naval forces in the area. The scenario was modified prior to the game by NSI staff and new injects were created to add to the fog of war during the simulation and to see how both humans and the AI model would respond to the changing situation.

The results and a more detailed comparison of the outcome of the parallel game processes will be jointly issued by the NSI Cyber and Tech Center and the Mercatus Center in the near future, and may include a comparison with one or more additional large language models. The top-line result was that while many of the responses provided by the human participants were mirrored at a high-level in the GPT4 responses, the human participants played more aggressively, and at times borrowed from historical precedents to take a more novel approach than what was proposed by the out-of-the-box GPT4 model.  Likewise, the GPT4 model consistently took a more conservative, incrementalist approach to the scenario.

Interestingly, these different approaches to gameplay led the staff China team to take a significantly more aggressive approach to the AI-directed United States, ultimately leading to a closing play by China that appeared to lead to a direct conflict. In comparison, the staff team played a more limited approach to the more aggressive, human-directed U.S. team ultimately backing down after the United States made clear its intention to protect its assets and free trade through the Taiwan Straits.

“While these scenarios and the resulting gameplay certainly warrant more analysis, both the parallels and major differences in gameplay between the human experts and the out-of-box GPT4 model, as well as the responses of the staff-directed opponent, demonstrate the foibles of relying solely on an AI directed strategy,” said Jamil N. Jaffer, NSI Founder and Executive Director.  “Even analyzed at a high level, the GMU game, supported by generous assistance from the Mercatus Center, provided interesting high-level insights into the ways that AI might (or might not) play a role in future warfare, including as a potential adjunct to human decisionmaking,” said Jaffer, who served as an Associate Counsel to President George W. Bush in the White House.

“We were thrilled to partner with NSI’s Cyber and Tech Center to support this exceptional event and we believe it will provide an original contribution to understanding how LLM’s work in this setting,” said Eileen Norcross, the Vice President of Applied Research at GMU’s Mercatus Center.  “We’re looking forward to the next steps on this project, including a more detailed analysis of the gameplay and results,” said Norcross.

About the National Security Institute
The National Security Institute serves as a platform for research, teaching, scholarship, and policy development that incorporates a realistic assessment of the threats facing the United States and its allies, as well as an appreciation of the legal and practical challenges facing U.S. intelligence, defense, law enforcement, homeland security, and cybersecurity communities.  NSI draws on the experience of its fellows, as well as its highly distinguished advisory board and faculty, to produce timely research and policy materials that deliver insightful analysis and actionable recommendations to senior policymakers in the White House and key departments and agencies, as well as those on Capitol Hill.

About the Mercatus Center
The Mercatus Center is a research center at George Mason University that advances knowledge about how markets solve problems and help us lead happier, healthier, and richer lives. For more than 40 years, Mercatus has supported leading talent and scholarship in the mainline economics tradition, applying rigorous research to real-world concerns. Through our continuing efforts to bridge the gap between theory and practice, we strive to realize a world where markets operate at their full potential to increase abundance, civility, and well-being.

About the Scalia Law School
The Antonin Scalia Law School at George Mason University is defined by three words: Learn. Challenge. Lead.  Students receive an outstanding legal education (Learn), are taught to critically evaluate prevailing orthodoxy and pursue new ideas (Challenge), and, ultimately, are well prepared to distinguish themselves in their chosen fields (Lead).  It has been one of America’s top-ranked law schools for the last fifteen years.

About George Mason
George Mason University is Virginia’s largest public research university.  Located near Washington, D.C., Mason enrolls more than 33,000 students from 130 countries and all 50 states.  Mason has grown rapidly over the past half-century and is recognized for its innovation and entrepreneurship, remarkable diversity, and commitment to accessibility.