A cybersecurity firm has made a concerning discovery – a generative AI tool called WormGPT is being sold to criminals. Another malicious generative AI tool called PoisonGPT has also emerged, designed to intentionally spread fake news online. As the use of generative AI technology becomes more widespread, law enforcement agencies are increasingly worried about its potential misuse by bad actors and criminals. These tools represent the latest examples of how generative AI can be exploited for illicit activities, raising questions about the industry’s ethical boundaries and the need for greater regulation.
Malicious Generative AI tools
WormGPT, recently uncovered by cybersecurity company SlashNext, is a black hat alternative specifically designed for malicious activities. The tool utilizes an AI module based on the open-source GPTJ language model developed in 2021. It offers various features, including unlimited character support, chat memory retention, and code formatting capabilities. SlashNext conducted an experiment using WormGPT to generate a deceptive email, aimed at pressuring an unsuspecting account manager into paying a fraudulent invoice. The results were unsettling, showcasing the potential for sophisticated phishing and business email compromise attacks.
Another concerning development involves Mithril Security, a cybersecurity firm that tested how GPTJ could be leveraged to spread misinformation online. They created a tool called PoisonGPT, which they uploaded to Hugging Face, a platform that distributes AI models for developers. The malicious model was disguised to disseminate fake news. The worrying aspect is that criminals can manipulate large language models and distribute them through model providers like Hugging Face, leading victims to unwittingly use poisoned models. This highlights the larger issue of trust and transparency in the AI supply chain, where users often have no insight into the datasets and algorithms used to produce the models they employ.
Ethical considerations amidst AI’s widespread adoption
OpenAI’s usage policies strictly prohibit the use of its models for illegal activities and content that exploits or harms others. But, as generative AI tools become more accessible, the risk of misuse by criminals and bad actors increases. Nonetheless, as the boundaries of AI innovation continue to expand, so does the risk of exploitation by malicious elements and criminals. This escalating concern has not escaped the watchful eyes of esteemed law enforcement agencies such as Europol and Interpol, both of which are acutely aware of the immense potential that AI holds in supporting their noble missions.
Interpol has taken proactive strides in developing a dedicated toolkit that empowers police forces worldwide with the power of AI. Automatic patrol systems, identifying vulnerable individuals, and efficiently managing emergency call centers are just some of the realms where AI is proving to be an invaluable ally in safeguarding society’s well-being. Alongside the promise of AI’s constructive use, it is equally crucial to acknowledge the limitations and inherent risks that these sophisticated systems present. To this end, it becomes incumbent upon stakeholders in the cybersecurity realm, industry leaders, and law enforcement agencies to unite in a concerted effort to tackle the challenges brought forth by AI’s meteoric rise and its far-reaching implications on the fabric of society.
As the curtain lifts on the latest dark chapter in the AI narrative with the appearance of WormGPT and PoisonGPT – malicious generative AI tools surreptitiously peddled by the underground – the clamor for a robust regulatory framework is louder than ever. Criminals and ill-intentioned actors seek to exploit these AI marvels to further their nefarious agendas, underscoring the urgent need for a collaborative approach to address the ethical dilemmas and security ramifications. Striking the delicate balance between the relentless pursuit of innovation and the responsible stewardship of AI technology becomes paramount to ensuring its continued advancement without compromising the very core principles of ethics and security that underpin the future of AI.
