How to Build Ethics into Robust Artificial Intelligence (2017-2018)

The goal of this project was to combine computational methods, philosophy, game theory and psychology to develop robust and ethical moral artificial intelligence (“moral AI”) to direct autonomous agents.

The project team’s research address several of the main challenges in designing moral AI through three subprojects.

  1. The team used in-person experiments and online platforms such as Amazon Mechanical Turk (MTurk) to ask people what types of factors they believe kidney exchange algorithms should and should not take into account. Team members built a website for large-scale data collection for online participants, analyzed participants’ responses to find common ethical themes and then revised current kidney exchange algorithms so that their recommendations of how kidneys should be allocated will take these ethical themes into account. This “bottom-up” approach relied primarily on new data to learn the moral features that should be incorporated into AI algorithms.
  2. The team also combined principles from moral philosophy and economic game theory to design scenarios and games that asked participants to judge the actions described or displayed in specific moral situations. Team members began by focusing on the “trust game” and had MTurk participants play the trust game against one another and then report how morally wrong or acceptable they thought their actions and their partner’s actions were. Team members collected data, began analysis on performance and used participants’ morality ratings to refine game theoretic notions so that they account for behavior in the trust game more accurately. This “top-down” approach relied primarily on game theory and ethical theory to generate algorithms that can make ethical choices.
  3. Team members also asked participants whether they thought humans or computers should be making decisions in a wide range of scenarios and applications.


Fall 2017 – Spring 2018

Team Outcomes

Lok Chan, Jana Schaich Borg, Vincent Conitzer, Dominic Wilkinson, Julian Savulescu, Hazem Zohny, Walter Sinnott-Armstrong. 2022. Which features of patients are morally relevant in ventilator triage? A survey of the UK public. BMC Medical Ethics.

Max Kramer, Jana Schaich Borg, Vincent Conitzer, Walter Sinnott-Armstrong. “When Should Computers Make Decisions?” 2018. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society 204-209.

Rachel Freedman, Jana Schaich Borg, Walter Sinnott-Armstrong, John P. Dickerson, Vincent Conitzer. “Adapting a Kidney Exchange Algorithm to Align with Human Values.” 2018. AAAI Conference on Artificial Intelligence. 

Adapting a Kidney Exchange Algorithm to Align with Human Values (poster by Rachel Freedman, Jana Schaich Borg, Walter Sinnott-Armstrong, John P. Dickerson, Vincent Conitzer), presented at Bass Connections Showcase, April 18, 2018

Vincent Conitzer, Walter Sinnott-Armstrong, Jana Schaich Borg, Yuan Deng, Max Kramer. “Moral Decision-Making Frameworks for Artificial Intelligence.” 2017. AAAI Conference on Artificial Intelligence.

Team website

Who Should Get the Kidney (game aligned with kidney exchange algorithm)

Vincent Conitzer and Walter Sinnott-Armstrong. How to Build Ethics into Robust Artificial Intelligence ($200,00 grant awarded from the Future of Life Institution, 2015)

Walter Sinnott-Armstrong. Towards a Culture of Questioning: Accountability, Humility, and Public Discourse ($225,000 grant awarded from John Templeton Foundation, 2017)


Cultures of Collaboration: Managing the Moral AI Lab (Kenzie Doyle)

This Team in the News

These Ph.D. Graduates Incorporated Bass Connections into Their Doctoral Education

Should AI Decide Who Gets a Kidney?

Finding Your Way: Cultivating Humanistic Versatility

Jana Schaich Borg on Duke, Data and MIDS

Duke Professor Advances to Final in New Yorker Caption Contest

See related team, Moral Artificial Intelligence (2018-2019). This project was selected by the Franklin Humanities Institute as a humanities-connected project.

Team Leaders

  • Vincent Conitzer, Arts & Sciences-Computer Science
  • Jana Schaich Borg, Social Science Research Institute
  • Walter Sinnott-Armstrong, Arts & Sciences-Philosophy

/graduate Team Members

  • Cassandra Carley, Computer Science-PHD, Computer Science-MS
  • Lok Chan, Philosophy-PHD
  • Abbas Zaidi, Statistical Science - MS

/undergraduate Team Members

  • Rachel Freedman, Interdept CompSci/Psych (AB)
  • Anika Mukherji, Interdept Neuro/Comp Sci (BS)
  • Weiyao Wang, Mathematics (BS), Computer Science (BS2)

/yfaculty/staff Team Members

  • Emmanuel Chevallier, Arts & Sciences-Statistical Science
  • Kenzie Doyle, Duke Institute for Brain Sciences
  • Joshua Skorburg, Fuqua School of Business
  • Siyuan Yin, Arts & Sciences-Philosophy

/zcommunity Team Members

  • John Dickerson, University of Maryland