NSI Cyber and Tech Center Publishes New Report on Findings from its Wargame That Pitted Humans Against Generative AI Capabilities
May 31, 2024
Keelin Wolfe
National Security Institute
kwolfe9@gmu.edu
Today, the the National Security Institute Cyber and Tech Center at George Mason University’s Antonin Scalia Law School published a report on findings from its recent wargame (co-hosted with the Mercatus Center at GMU) that explored the dynamics of human versus AI decisionmaking during a simulated crisis in the Taiwan Strait.
Overall, while there were areas of commonalty between humans and AI, the human participants’ consistently more aggressive gameplay compared to the AI’s more measured and cautious approach led to a markedly different outcome – China ultimately decided to invade Taiwan in the final turn against the AI, while backing down and ending the crisis against the human participants.
The Report outlined several critical key findings for understanding AI’s potential utility in national security and foreign policy decisionmaking:
- Humans Were More Aggressive than AI: The humans consistently sought to raise the stakes and signal a wiliness to confront China directly while AI played defensively and sought to limit the scope of nature of potential confrontation.
- AI Consistently Demonstrated Broad and Structured Thinking: Throughout the game, the AI presented a broad range of responsive actions in a structured manner that occasionally led it to recommend responses that human decisionmakers would have adopted had they thought of them.
- AI’s Response Process Presented Challenges: The AI did not provide consistent and complete answers or recommendations to all prompts, raising concerns about its reliability for use in crisis decisionmaking.
While AI proved helpful, this wargame raised questions about the net benefits of reliance on off-the-shelf LLMS and additional wargames with human and AI participants could provide useful data to determine whether LLMs can reliably be integrated into national security and foreign policy decisionmaking in the future.
Click here to read the NSI CTC Report.
“Having conducted what we believe is the first-of-its-kind, full-scale wargame directly comparing human and off-the-shelf LLM decisionmaking in a real-world conflict scenario, the George Mason University wargame offers some fascinating insights into how AI might influence the future of warfare,” said Jamil N. Jaffer, Founder and Executive Director of the National Security Institute’s Cyber and Tech Center at GMU’s Antonin Scalia Law School. “While there is a lot more work to be done testing more scenarios and modifying the gameplay and tools used in this type of wargame, this interdisciplinary collaboration at GMU between NSI CTC and the Mercatus Center and involving number of former public and private sector national security leaders once again shows the power that Virginia’s largest research university and fastest-rising law school can bring to bear on novel questions at the intersection of law, policy, and technology,” said Jaffer, who was also recently appointed to the Department of Homeland Security’s Cyber Safety Review Board.
“We were thrilled to partner with NSI’s Cyber and Tech Center on this exciting project. This work is an original important contribution to understanding how LLM’s perform compared to humans in a crisis environment with low information and high stakes,” said Christine McDaniel, Senior Research Fellow at GMU’s Mercatus Center. “Anyone interested in the role of artificial intelligence in our increasingly unstable world will be interested in these wargame results. We believe it is valuable to compare decisions and actions by a human-led National Security Council with an AI National Security Council under distinct scenarios. For the wargames considered on this exercise, there were important concerns raised about the reliance on AI in crisis decisionmaking. In some instances, the AI lacked human creativity and in others we witnessed how AI can enhance human decisionmaking,” she added.
Learn more about the authors; Jamil N. Jaffer and Jessica L. Jones.
About the NSI Cyber and Tech Center:
The NSI Cyber and Technology Center (CTC) advances American technology leadership and engages policymakers on issues at the intersection of technology and national security. Its mission is to promote – through dialogue with experts, engagement with policymakers, and cutting-edge research – American technology leadership and to tackle critical innovation, cyber, and emerging technology challenges.
NSI CTC accomplishes its mission by leveraging its vast network of technology and national security professionals and practitioners to produce cutting-edge research and policy recommendations, and to engage with leading policymakers.
About the Mercatus Center:
The Mercatus Center at George Mason University is a leading university-based research center dedicated to market-oriented ideas, aiming to bridge the gap between academic research and real-world public policy problems. Founded in 1980, it focuses on how markets can improve lives through research, graduate student training, and applying economic principles to societal issues. Mercatus scholars study the best institutions for promoting social outcomes, drawing on insights from Nobel laureates and offering sustainable solutions for freedom, prosperity, and peace. Their work spans diverse topics, from economic development to government spending.
About the Antonin Scalia Law School:
The Antonin Scalia Law School was founded in 1972 as the International School of Law in Washington, DC. In 1979 it merged with George Mason Law school and was named after the late Supreme Court Justice Antonin Scalia in 2016. The school is located in Arlington, Va. and is ranked #28 nationally by US News—#3 among all 16 schools in Washington, DC, Virginia, and Maryland. The school offers JD, LLM, and JM degrees and is home to seven nationally acclaimed centers, including the Law and Economics Center, the National Security Institute, and the C. Boyden Gray Center for the Study of the Administrative State.
About George Mason University:
George Mason University is Virginia’s largest public research university. Located near Washington, D.C., Mason enrolled more than 40,000 students from 130 countries and all 50 states. Mason has grown rapidly over the last half-century and is recognized for its innovation and entrepreneurship, remarkable diversity, and commitment to accessibility. In 2023, the university launched Mason Now: Power the Possible, a one-billion-dollar comprehensive campaign to support student success, research, innovation, community, and stewardship.