Sylvain Duranton, Vanessa Lyon
PARIS, MARCH 20 (PS): In a recent security test, a generative AI banking chatbot designed to assist customers with loan applications was manipulated into disclosing sensitive financial information. Testers bypassed security controls and extracted a comprehensive list of loan approvals, including customer names.
This cautionary tale underscores a fundamental issue: generative AI could revolutionize entire industries, but, without robust safety protocols, it can also lead to disastrous outcomes. Traditional safety models are no longer sufficient. Transformative technologies like generative AI demand a new, holistic approach to cybersecurity.
Aviation provides a useful model. Much like supersonic jets, generative AI is a transformative technology with immense potential. But without trained operators, well-designed systems, and robust safeguards, the risk of catastrophic failure is too great to ignore. By adopting rigorous safety protocols, air travel has become one of the safest modes of transportation. Likewise, AI’s potential is undeniable, but its future hinges on addressing safety risks. A recent BCG study, for example, found that three-quarters of business executives view cybersecurity as a major obstacle to scaling AI.
Unlike traditional software, generative AI relies on probabilities, which can lead to unpredictable outcomes. Large language models (LLMs) introduce indeterministic behaviors, creating cybersecurity blind spots. Moreover, their reliance on natural language inputs, adaptive learning, and extensive integrations with other tools and services make them uniquely vulnerable.
Just as aviation requires a comprehensive, multifaceted approach to safety, cybersecurity must be embedded at every layer of AI, from its architecture to data management and human oversight. Without such a foundation, AI’s future will remain uncertain.
A key vulnerability of AI systems is prompt-injection attacks, where attackers manipulate a model into revealing sensitive data or altering its decision-making logic. The recent banking chatbot test uncovered an equally alarming risk: privilege escalation. Testers impersonated an administrator, approving unauthorized loans and modifying backend data.
Health-care AI assistants have been similarly compromised, as security researchers successfully extracted confidential patient records by subtly rewording their queries. Instead of directly requesting medical histories, attackers framed their questions to resemble legitimate doctor requests. By doing so, they revealed another weakness: AI often prioritizes linguistic logic over access controls.
These vulnerabilities extend beyond banking and health care. Many AI applications leverage agentic systems, which retrieve real-time data to make decisions autonomously, creating opportunities for attackers. For example, a security assessment of an AI-powered customer-service chatbot showed that attackers were able to exploit a weak application programming interface (API) validation to manipulate an LLM into revealing internal discount codes and inventory details.
AI’s adaptability can also be exploited through so-called context poisoning. By gradually shaping a model’s responses over time, attackers can steer its responses toward incorrect or dangerous recommendations. In one experiment, a spa chatbot was repeatedly exposed to inputs framing unsafe ingredients as beneficial. Eventually, it began recommending harmful skincare products.
As AI systems overwhelm traditional infrastructure with automated requests, they can lead to system failure – a phenomenon known as legacy contamination. To avoid this outcome, organizations must implement adversarial training, continuously exposing AI models to deceptive inputs to enhance their resilience.
Real-time anomaly detection – both automated and manual – can identify unusual AI behavior before manipulated data affect responses. Just as flight-control systems rely on independent backups, generative AI security must be built on layered safeguards, including automated anomaly detection to flag irregular activity, redundant access validation to prevent unauthorized system interactions, and real-time rollback mechanisms to reverse harmful changes.
While analysts predict that global spending on AI will exceed $631 billion by 2028. Many of these investments will struggle to deliver meaningful returns unless fundamental cybersecurity challenges are addressed. Most importantly, AI security must evolve from an “add-on” to a core function embedded within system architectures, data management, and human oversight. An effective security framework should be fluid, adaptive, resilient, and integrated into legacy systems.
Even industry leaders face design challenges, underscoring the need for stronger security measures. In March 2023, OpenAI discovered a bug in an open-source library that inadvertently exposed ChatGPT users’ payment information by sending confirmation emails to the wrong recipients.
AI security must evolve in lockstep with the systems it aims to protect. But effective data management is not just about fortifying pipelines and securing training datasets. It requires a well-defined strategy that treats data as a competitive advantage and carefully evaluates what data to expose and what data businesses should be able to leverage.
Operational oversight is just as critical. Cybersecurity must not be confined to a specialists’ silo. It must be embedded in every department and workflow, with real-time monitoring tools and adaptive feedback loops helping organizations stay ahead of emerging threats and vulnerabilities.
Beyond cutting-edge technology, cybersecurity requires cultivating a culture of vigilance. According to a 2024 Verizon report, 68% of all data breaches involved a human element, such as being tricked by phishing attacks or social engineering. To mitigate these risks, employees must not only identify threats but also learn how to respond appropriately. Even simple measures like regular security training and transparent reporting mechanisms could make a significant difference.
Just as aviation gained public trust by adopting rigorous safety measures, the AI industry must introduce protections to prevent hallucinations, manipulation, hacking, and latency issues before they cause real-world harm. This requires a comprehensive approach that integrates architecture, engineering, data strategy, and responsible AI. Companies that embed security into every layer of their AI strategy will thrive, while those clinging to outdated security models will inevitably fall behind.
Sylvain Duranton is Global Leader of BCG X. Vanessa Lyon is Global Leader of Cyber and Digital Risk at BCG.
Copyright: Project Syndicate, 2025.
