What Happened
In case you haven’t heard, BeaconCRM, a platform used by over 1,500 UK-based charities, was the subject of a security breach. An unauthorised party was able to gain access to their systems and accessed all of the information the charities had stored there. This includes personal information about the very people those charities were helping.
Most of the charities are currently following the playbook of what to do in the event of a hack, and the Charities Commission and Information Commissioner’s Office are both involved. At present, experts in the field say that there’s no indication that the information has been leaked or offered for sale. It’s obviously distressing for everyone involved, especially the vulnerable people who had turned to these charities for help.
The Fingerprint of AI
It has emerged that the hackers gained access to BeaconCRM’s AWS login details because they were stored in a publicly accessible JavaScript file. Basically, a piece of code on a web server that needed to access something behind a username/password had that username and password available where anyone could read it.
This is, as you can imagine, something every human developer competent enough to be working on the live code for a large system like BeaconCRM would know is not how to do that. It follows that, barring a colossal misstep on the part of a human developer, this was not something done by a human developer.
We’ve written before about the dangers of using AI when you don’t know the topic well enough to spot when it makes a mistake. Basically, AI is not infallible. You need to know enough about what you’re asking AI to do that you can spot when it gets something wrong; otherwise, you can quickly go down a rabbit hole of reinforcing a mistake until it grows out of all proportion.
In this case, based on how web developers work, this was probably a very competent developer, using AI to do the grunt work while they dealt with the details. And in doing so, the AI made a mistake. Had the developer seen it, they would certainly not have put the code live. But the developer didn’t spot the fatal flaw, and what resulted was 1,500 charities having their private information leaked.
The Lesson
If you’re not a developer, it’s easy to wonder what sort of lesson this incident could have, especially if you’re not actually vibe coding yourself (if you are, then there’s another lesson). "A computer can never be held accountable. Therefore a computer must never make a management decision," is a famous line from an IBM training manual from the late 1970s, but is just as true today.
The lesson is, even if you are an expert in what you are using AI to do, you must double-check what it has done thoroughly, because any mistakes are on you. This can be shown by politicians reading a speech aloud, only to actually read out the obvious ChatGPT follow-up question at the end. Or a (once reputable) sports news website posting an AI-generated image of a goalkeeper with three arms on an equally obviously AI-generated article about a football match.
Yes, in each case the AI made the initial mistake, but the blame lies with the human who accepted the AI output and then presented it to the world without checking it.
