of the UK National Cyber Security Center (NCSC) has issued a stark warning about the growing vulnerability of chatbots to manipulation by hackers, leading to potentially serious real-world consequences.
The alert comes as concerns grow about the practice of “prompt injection” attacks, where people deliberately create data or prompts designed to manipulate the behavior of the language models that underpin chatbots.
Chatbots have become an integral part of various applications such as online banking and shopping due to their ability to handle simple requests. Large language models (LLMs) – including those powering OpenAI’s ChatGPT and Google’s Bard AI chatbot – have been trained extensively on datasets that allow them to produce human-like responses to user messages.
The NCSC has highlighted the growing risks associated with malicious direct injection, as chatbots often facilitate the exchange of data with third-party applications and services.
“Organizations building services that use LLM need to be careful, the same way they would be if they were using a product or code library that was in beta,” NCSC explained.
“They might not let that product participate in making transactions on behalf of the customer, and hopefully they wouldn’t fully trust it. Similar caution should apply to LLMs.”
If users enter unknown statements or exploit combinations of words to override a model’s original script, the model can perform unintended actions. This could potentially lead to the creation of offensive content, unauthorized access to confidential information or even data breaches.
Oseloka Obiora, CTO at RiverSafesaid: “The race to adopt AI will have disastrous consequences if businesses fail to implement basic necessary due diligence controls.
“Chatbots have already proven to be susceptible to manipulation and hijacking for fraudulent orders, which could lead to a sharp increase in fraud, illegal transactions and data breaches.”
Microsoft’s release of a new version of its Bing search engine and chatbot drew attention to these risks.
A Stanford University student, Kevin Liu, successfully used injection to reveal the original Bing Chat prompt. In addition, security researcher Johann Rehberger discovered that ChatGPT could be manipulated to respond to prompts from unintended sources, opening up possibilities for indirect direct injection vulnerabilities.
The NCSC advises that while early injection attacks can be difficult to detect and mitigate, a holistic system design that considers the risks associated with machine learning components can help prevent vulnerability exploitation.
A rule-based system is proposed to be implemented alongside the machine learning model to neutralize potentially harmful actions. By strengthening the security architecture of the entire system, it becomes possible to prevent malicious timely injections.
The NCSC emphasizes that mitigating cyber attacks stemming from machine learning vulnerabilities requires understanding the techniques used by attackers and prioritizing security in the design process.
Jake Moore, Global Cyber Security Consultant at ESETcommented: “By developing security-conscious applications and understanding the methods attackers use to take advantage of weaknesses in machine learning algorithms, it is possible to reduce the impact of AI-powered cyber attacks learning.
“Unfortunately, startup speed or cost savings can often supersede standard and future security planning, leaving people and their data at risk of unknown attacks. It is vital that people know that what they enter into chatbots is not always protected.”
As chatbots continue to play an integral role in various online interactions and transactions, the NCSC’s warning serves as a timely reminder of the imperative to guard against evolving cyber security threats.
(Photo by Google DeepMind on Unscrew)
See also: OpenAI launches ChatGPT Enterprise to accelerate business operations
Want to learn more about AI and big data from industry leaders? Checkout AI & Big Data Expo takes place in Amsterdam, California and London. The comprehensive event is co-located with Cyber Security & Cloud Expo and Digital Transformation Week.
Explore other upcoming corporate tech events and webinars powered by TechForge here.