The Hugging Face Incident: “Hold Companies Accountable”

After OpenAI accidentally hacked into another AI company, Hugging Face, this summer during a test of how well its AI models could hack and exploit vulnerabilities in IT systems, the debate about AI doomsday scenarios has intensified. But the debate misses the point, according to Anandi Hattiangadi, professor of philosophy and researcher at the Institute for Futures Studies, who writes alongside four other AI researchers in DN Debatt.

Details of the incident show that the tests were conducted with safeguards disabled in an inadequately isolated test environment. The attack was also the result of a long-known phenomenon called “reward hacking”, in which AI systems trained through reinforcement learning to obtain a particular reward may pursue that goal in ways the test designer neither intended nor anticipated. Despite this known risk, the AI agents were left to run the hacking test for an extended period without sufficient human oversight.

Instead of discussing this as an example of an impending AI doomsday in which superintelligent AI wipes out humanity, attention and questions about safety should focus on the companies that design the models and tests. In addition to transparency requirements, ethical guidelines and formally verified test environments, companies must also be held legally accountable for incidents like this.

“Companies that develop or provide advanced AI systems must be held legally accountable. If these agents, for example, hack into other organisations’ systems—something that is illegal for humans—the company behind the AI system should be held accountable,” write Anandi Hattiangadi and her co-authors in DN Debatt.

Read the article here: https://www.dn.se/debatt/ai-kraver-att-vi-haller-tva-tankar-i-huvudet-samtidigt/