How to Build Ethics into Robust Artificial Intelligence (2019-2020)


Self-driving cars are beginning to appear on local streets. Autonomous drones are flying over populated areas and on military missions. Robots are performing surgery, and artificial intelligence is used in criminal justice. These autonomous systems have promise to provide many services that will help society, but they also raise significant concerns.

Autonomous agents need to be programmed with an artificial intelligence that instructs them how to interact with other agents, but how can we do this? One option is to apply a basic moral rule, but the main challenge to this approach is the pluralism problem. How do we decide which rules, rights and roles should be built into the artificial intelligence? The goal of this project is to combine computational methods, philosophy, game theory and psychology to develop robust moral artificial intelligence to direct autonomous agents.

Project Description

This Bass Connections project attempts to build morality into artificial intelligence by incorporating new morally-relevant features based on the data gathered by the previous year’s project team. The project employs crowd-sourcing of online moral judgments about which features of an action are and should be seen as relevant to the moral status of the action.

The projects pilot study focuses on kidney exchanges, using experiments and online platforms to ask people what types of factors they believe kidney exchange algorithms should and should not take into account in determining who gets a kidney. Team members then construct scenarios where these factors vary in systematic ways and use machine learning to construct an algorithm that predicts human moral judgments, tests its extension to a novel set of scenarios and reveals how various moral factors interact.

Another challenge for moral artificial intelligence is societys unwillingness to adopt it. To tackle this challenge, team members will ask participants whether they think humans or computers should be making decisions in a wide range of scenarios and applications. Researchers have some preliminary evidence showing that people will be more willing to accept computers’ decisions if they have been exposed to computers making decisions in similar contexts before. The team will be testing this pattern using new uses of artificial intelligence and interventions.

The third project combines principles from moral philosophy and economic game theory to design scenarios and games that ask participants to judge actions. Game theory underestimates the amount of money participants will choose to give to each other in this game. This might be happening because game theory has not adequately taken morality into account. To test this, team members will ask participants to play the classic “trust game” and report how morally wrong or acceptable they thought their actions and their partner’s actions were.

Anticipated Outputs

Participation in computer science, game theory and ethics conferences; publications in academic journals; website where participants make decisions about kidneys exchanges

Team Outputs to Date

Moral Artificial Intelligence (Fortin Foundation Bass Connections Virtual Showcase 2020)

Artificial Artificial Intelligence: Measuring Influence of AI “Assessments” on Moral Decision-making (poster by Lok Chan, Kenzie Doyle, Duncan C. McElfresh, Vincent Conitzer, John P. Dickerson, Jana Schaich Borg, Walter Sinnott-Armstrong, presented at Third AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society, New York, NY, February 7-8, 2020)


Fall 2019 – Spring 2020

  • Fall 2019: Analyze collected data; construct algorithms
  • Spring 2020: Write up the findings into papers and conference presentations

See related teams, How to Build Ethics into Artificial Intelligence (2020-2021) and Moral Artificial Intelligence (2018-2019).


Image: Artificial Intelligence, by albyantoniazzi, licensed under CC BY-NC-SA 2.0

Artificial Intelligence.

Team Leaders

  • Vincent Conitzer, Arts & Sciences-Computer Science
  • Jana Schaich Borg, Social Science Research Institute
  • Walter Sinnott-Armstrong, Arts & Sciences-Philosophy

/graduate Team Members

  • Daniela Goya-Tocchetto, Business Administration-PHD
  • Gayan Seneviratna, Electrical/Computer Engg-MS

/undergraduate Team Members

  • Claire Fu, Electrical & Computer Egr(BSE), Computer Science (BSE2)
  • Irving Rein, Computer Science (BS)

/yfaculty/staff Team Members

  • Joyanne Becker, Duke Social Science Research Institute
  • Lok Chan, Social Science Research Institute
  • Kenzie Doyle, Duke Institute for Brain Sciences
  • Michele Peruzzi, Arts & Sciences-Statistical Science
  • Arkaprava Roy, Arts & Sciences-Statistical Science

/zcommunity Team Members

  • CEPS Ideas Lab, Brussels
  • John Dickerson, University of Maryland
  • Duncan McElfresh, University of Maryland
  • Abbas Zaidi, Facebook