Automation Bias

Checking costs attention, and an aid that is usually right makes each check feel wasted, so its output quietly replaces the search a person would otherwise do. Parasuraman and Manzey's review describes two kinds of error: "Automation bias results in making both omission and commission errors when decision aids are imperfect." It is not a beginner's mistake: "Automation bias occurs in both naive and expert participants, cannot be prevented by training or instructions, and can affect decision making in individuals as well as in teams." In a user study by Perry et al., participants with an AI code assistant wrote significantly less secure code than those without one, and were more likely to believe their code was secure.

On March 23, 2018, a Tesla Model X operating with Traffic-Aware Cruise Control and Autosteer moved into a gore area on US-101 in Mountain View, California, and struck a damaged crash attenuator; the driver died. The NTSB's probable cause included the Autopilot steering the SUV into the gore area due to system limitations, and the driver's lack of response due to distraction "likely from a cell phone game application and overreliance on the Autopilot partial driving automation system."

Algorithm Aversion: the opposite error is common too: Dietvorst, Simmons and Massey found that "people more quickly lose confidence in algorithmic than human forecasters after seeing them make the same mistake", and so drop an aid that still beats them.