The powerful capabilities of ChatGPT are being used against enterprise systems. Malicious packages and AI hallucinations are a few of the growing threats. Credit: Sequential Pictures / Shutterstock Although I’m swearing off studies as blog fodder, it did come to my attention that Vulcan Cyber’s Voyager18 research team recently issued an advisory validating that generative AI, such as ChatGPT, would be turned into a weapon quickly, ready to attack cloud-based systems near you. Most cloud computing insiders have been waiting for this. New ways to attack A new breaching technique using the OpenAI language model ChatGPT has emerged; attackers are spreading malicious packages in developers’ environments. Experts are seeing ChatGPT generate URLs, references, code libraries, and functions that do not exist. According to the report, these “hallucinations” may result from old training data. Through the code-generation capabilities of ChatGPT, attackers can exploit fabricated code libraries (packages) that are maliciously distributed, also bypassing conventional methods such as typosquatting. Typosquatting, also called URL hijacking or domain mimicry, is a practice where individuals or organizations register domain names like popular or legitimate websites but with slight typographical errors. The intention is to deceive users who make the same typo when entering a URL. Another attack involves posing a question to ChatGPT, requesting a package to solve a specific coding problem, and receiving multiple package recommendations that include some not published in legitimate repositories. By replacing these nonexistent packages with malicious ones, attackers can deceive future users relying on ChatGPT’s recommendations. A proof of concept utilizing ChatGPT 3.5 proves the potential risks. Of course, there are ways to defend against this type of attack. Developers should carefully vet libraries by checking the creation date and download count. However, we will be forever skeptical of suspicious packages now that we deal with this threat. Dealing with new threats The headline here is not that this new threat exists; it was only a matter of time before threats powered by generative AI power showed up. There must be some better ways to fight these types of threats that are likely to become more common as bad actors learn to leverage generative AI as an effective weapon. If we hope to stay ahead, we will need to use generative AI as a defensive mechanism. This means a shift from being reactive (the typical enterprise approach today), to being proactive using tactics such as observability and AI-powered security systems. The challenge is that cloud security and devsecops pros must step up their game in order to keep out of the 24-hour news cycles. This means increasing investments in security at a time when many IT budgets are being downsized. If there is no active response to managing these emerging risks, you may have to price in the cost and impact of a significant breach, because you’re likely to experience one. Of course, it’s the job of security pros to scare you into spending more on security or else the worst will likely happen. This is a bit more serious considering the changing nature of the battlefield and the availability of effective attack tools that are almost free. The malicious AI package hallucinations mentioned in the Vulcan report are perhaps the first of many I’ll be covering here as we learn how bad things can be. The silver lining is that, for the most part, cloud security and IT security pros are more intelligent than the attackers and have kept a few steps ahead for the past several years, the odd big breaches notwithstanding. But attackers don’t have to be more innovative if they can be clever, and understanding how to put generative AI into action to breach highly defended systems will be the new game. Are you ready? Related content analysis Generative AI won’t fix cloud migration You’ve probably heard how generative AI will solve all cloud migration problems. It’s not that simple. Generative AI could actually make it harder and more costly. By David Linthicum Jul 12, 2024 5 mins Generative AI Artificial Intelligence Cloud Computing analysis All the brilliance of AI on minimalist platforms Buy all the processing and storage you can or go with a minimum viable platform? AI developers and designers are dividing into two camps. By David Linthicum Jul 09, 2024 5 mins Generative AI Cloud Architecture Artificial Intelligence analysis The next 10 years for cloud computing Despite AI's explosive growth, the industry still needs to face facts that customers are unhappy about costs and vendor lock-in. By David Linthicum Jul 05, 2024 5 mins Amazon Web Services Google Cloud Platform Microsoft Azure analysis Serverless cloud technology fades away Serverless was a big deal for a hot minute, but now it seems old-fashioned, even though its basic elements, agility and scalability, are still relevant. By David Linthicum Jul 02, 2024 4 mins Serverless Computing Cloud Computing Software Development Resources Videos