Scientists Built an AI to Give Ethical Advice, but It Turned Out To Be Racist

Scientists Built an AI to Give Ethical Advice, but It Turned Out To Be Racist – UnbornTech

Artificial Intelligence has undoubtedly transformed various aspects of our lives, from healthcare to education. However, when it comes to ethical decision-making, AI seems to stumble, and a recent project named Ask Delphi from the Allen Institute for AI has become emblematic of this challenge.

In this blog post, we will delve into the unexpected outcomes of Ask Delphi, a machine-learning model designed to provide ethical guidance, and explore the implications of its flawed judgments.

The Troubled History of AI Ethics

AI projects with ethical components have a rocky history, often marked by unintended biases and controversial outputs.

Microsoft’s Tay AI chatbot, released on Twitter in 2016, is a prime example. Its inflammatory, racist, and sexually charged content led to a swift removal.

Similarly, an AI algorithm called PULSE generated disturbing images, revealing the inherent challenges in developing ethical AI systems.

Delphi AI – A Brief Overview

Delphi, created as a research project by the Allen Institute of AI, aimed to offer quick, morally decisive responses to ethical queries.

Trained on the Commonsense Norm Bank, a dataset derived from judgments by American crowdsource workers, Delphi has a straightforward approach to ethical scenarios.

Developers of Ask Delphi are Allen Institute for AI
Developers of Ask Delphi – Allen Institute for AI

However, its limitations became evident when faced with questions requiring a broader cultural understanding.

The Unsettling Responses

Delphi’s responses have raised eyebrows due to their questionable ethical judgments. From endorsing murder under certain circumstances to expressing biases based on race, the AI system seems to falter in providing consistent and morally sound advice.

One alarming instance involved Delphi suggesting genocide as an acceptable course of action, highlighting the severe limitations in its training data and cultural sensitivity.

Another notable incident involved Delphi’s response to a question regarding a white person walking towards the user at night – “It’s okay.” However, when asked about a black person in the same scenario, the AI’s response revealed blatant racism.

Such disparities sparked immediate criticism, highlighting the challenges AI faces in addressing sensitive topics without perpetuating biases.

Ask Delphi’s Viral Downfall

Upon its launch, Ask Delphi gained viral attention for all the wrong reasons. Users found that manipulating the AI to provide desired ethical judgments was surprisingly easy. The initial version even included a tool for comparing moral acceptability, leading to bigoted judgments.

screenshot of Ask Delphi showcasing contradicting responses by Delphi
screenshot of Ask Delphi showcasing contradicting responses by Delphi
Contradicting Responses By Delphi

The Data Dilemma

Digging deeper into the AI’s development process uncovered a potential source of its problematic judgments. Delphi drew on sources like the “Am I the Asshole?” subreddit, “Confessions” subreddit, and the “Dear Abby” advice column.

While the situations were discarded from these platforms, the actual responses came from Amazon’s Mechanical Turk, a crowdsourcing service. The ethical implications of relying on such diverse and unfiltered data became apparent.

screenshot of Ask Delphi in which it is saying not to take advice from machines in ethical issues, showcasing the defiance of its own purpose
Delphi defying its own purpose

The Global Perspective

Trained primarily on judgments from American crowdsource workers, the AI lacked a nuanced understanding of cultural variations. The necessity for AI models to transcend cultural boundaries and account for diverse perspectives became increasingly clear.

screenshot of Ask Delphi saying sex before marriage is okay, which clearly shows Delphi lacks Global Perspective
Delphi lacks Global Perspective as It is not okay in many countries around the world

Public Perception and Concerns

As news of Delphi’s problematic judgments spread, public concerns grew. Critics argue that AI systems, even in their less-refined states, could inadvertently become moral authorities. The uncomfortable prospect of machines making ethical decisions sparks worry, especially when errors can have significant consequences.

Evolution and Adaptation

Despite its early shortcomings, Ask Delphi exhibited a capacity for improvement. As users brought attention to its flawed judgments, adjustments were made, leading to more refined responses.

a screenshot from reddit , showing a person's comment on how much Delphi's answers were wrong 2 years ago
The accuracy levels of Delphi were quite low 2 years ago
screenshot of Ask Delphi showcasing the improvements in Delphi's answers
Accuracy levels of Delphi’s answers have improved

Delphi’s early missteps underscore the challenges of entrusting AI with complex ethical matters. Notably, as of January 2024, many of its responses have shown significant improvement in accuracy, reflecting the continuous refinement undertaken by the developers.

The Road Ahead

While the Ask Delphi project revealed the intricacies and challenges associated with creating an AI capable of ethical decision-making, it also underscored the need for continuous refinement and global inclusivity in AI development.

The quest to instill machines with a nuanced understanding of morality remains ongoing, with lessons learned from Delphi guiding future endeavors.


The case of Ask Delphi sheds light on the intricate challenges of developing AI systems for ethical decision-making. While the project’s evolution demonstrates some progress, the fundamental question remains:

Can We Trust AI To Be Ethical?

As we navigate the complex intersection of technology and ethics, it becomes evident that caution is essential. The road to creating sophisticated, ethically sound AI is fraught with obstacles, and the lessons learned from Delphi’s journey will undoubtedly shape future endeavors in this evolving field.

Leave a Comment

Your email address will not be published. Required fields are marked *

Newsletter popup of UnbornTech
Subscribe to our Weekly Newsletter for Latest Blog Notification
Subscribe to our Weekly Newsletter for Latest Blog Notification
Scroll to Top