"After a series of highly publicized operator errors at its Cleveland plant, a chemical manufacturer, installs a software based control system to prevent accidental releases of toxic substances. The system relies on a machine learning model trained on millions of hours of operating data from their facilities. Using sensor data from the plant, the model can identify when it is safe to open the plant’s exhaust vents. Thanks to its extensive “experience,” the model adapts seamlessly to process changes and physical modifications within the complex plant, which were blamed for confusing human operators in the past. The new
software system proves highly reliable and becomes a trusted tool within the company. Months later, a windstorm disrupts several of the plant’s sensors. Based on the flawed sensor input, the
control system continues to read “safe,” and the plant operators act accordingly, leaving the vents open, even as managers elsewhere in the plant begin an unscheduled production run in response to an urgent customer request. The run produces a cloud of lethal chlorine gas, which escapes through the open exhaust vents and drifts toward downtown."
Source: https://cset.georgetown.edu/publication/ai-accidents-an-emerging-threat/