Exploiting the Generative AI Boom: The Rise of Scam Attacks

This post is also available in:
日本語 (Japanese)

Executive Summary

In this post, we explore the evolution of domain registration and network attacks associated with terms related to generative AI (GenAI). These trends are strongly correlated with the key milestones and developments in GenAI such as the launch of ChatGPT and its integration into the Bing search engine – and the buzz of interest around these events.

We analyzed domains registered with wording that appears related to GenAI. In the process, we uncovered insights regarding the characteristics of suspicious activity seeking to capitalize on the trend, including textual patterns and the volume of traffic these domains receive. To provide a comprehensive understanding of the underlying cyberthreats, we conducted several case studies detailing different attack types, including the delivery of potentially unwanted programs, the distribution of spam and the use of monetized domain parking.

Since ChatGPT’s launch in November 2022, GenAI has consistently attracted the public’s interest, and we have been actively tracking the related cyber threats since then, following how scammers have sought to take advantage of people searching for information about GenAI. Throughout 2023 and 2024, the related discussion expanded and new products emerged, and the network security team at Palo Alto Networks witnessed a surge of network abuses that leveraged the popularity of this hot topic. This trend highlighted the critical need for enhanced focus and resources dedicated to detecting and mitigating GenAI-related scams.

Palo Alto Networks customers are better protected against various network threats seeking to leverage terminology associated with GenAI through Cloud-Delivered Security Services such as Advanced DNS Security, Advanced URL Filtering and Advanced WildFire. If you think you might have been compromised or have an urgent matter, contact the Unit 42 Incident Response team.

GenAI-related Domains Registration

When adversaries take advantage of trending topics, the initial strategy often involves registering domains that incorporate relevant keywords. Therefore, our analysis started with retrieving historical newly registered domains (NRD) that contain GenAI keywords such as chatgpt, prompt and sora.

Palo Alto Networks detects over 200,000 daily NRD from zone files, WHOIS databases and passive DNS. We retrieved around 225 GenAI-related domains registered every day since November 2022.

Figure 1 presents the daily count of domain registrations leveraging GenAI-related keywords, along with the number identified as suspicious. We labeled the domains in the following categories as suspicious:

• Command and control
• Ransomware
• Malware
• Phishing
• Grayware

The domain registration trend is clearly correlated to the fluctuating popularity of the topic, with data peaks aligning with major ChatGPT milestones. Following Microsoft’s announcement of ChatGPT integration with Bing on Feb. 7, 2023, we observed a surge in the number of new domains where many of them contain both trademarks (e.g., msftchatgpt[.]com).

Another significant spike occurred on March 14, 2023, coinciding with the official release of GPT-4. The next peak corresponds to the announcement of new GPTs on Nov. 6, 2023, during which numerous related domains, like gptsotre[.]com, were registered.

The breaking news about Sora, an upcoming text-to-video generation model developed by OpenAI, attracted significant public attention for GenAI after Feb. 15, 2024. Specifically, there were about 760 GenAI-related domains registered every day in the following week.

The average rate of suspicious GenAI-related domains is 28.75%, which is 22 times higher than the rate for general NRDs, based on our previous research. This shows that GenAI is a highly abused topic and emphasizes the importance of continuously monitoring related network threats.

We further analyzed the textual patterns for these interesting new domains. We split them based on the embedded keywords to calculate the number of domains and suspicious rate for each keyword.

Figure 2 plots the statistics for the most frequently used keywords. Remarkably, over 72% of the domains associate themselves with popular GenAI applications by including keywords like gpt or chatgpt.

The most abused keyword is gpt, whose suspicious rate is 76%. This word, though not exclusively related to the GenAI topic, demonstrates a significant correlation with it. After filtering out domains unrelated to GenAI, this term was rarely used for domain creation prior to 2023, while its popularity surged along with the GenAI trend.

As interest in GenAI grows and more people seek to become experts in its use, prompt engineering emerges as a hot topic. We also observed that prompt frequently coexists with gpt and engineering in domain names. Our findings suggest that people must exercise caution when visiting websites offering tutorials on prompt engineering, as a significant percentage of them are shady.

GenAI-related DNS Traffic

While the number of domain registrations indicates the level of interest from both developers and attackers, the traffic to these domains provides insights into their actual impact on the public. We cross-checked the GenAI-related domains with our passive DNS dataset to calculate their popularity and track their traffic trends.

We obtained several insights about GenAI network traffic from the DNS requests volume for the related NRDs.

• Figure 3 presents a general upward trend for GenAI-related traffic. There was a significant growth phase from January-September 2023. After this surge, the GenAI-related DNS traffic plateaued at a high level.
• Among all traffic toward these NRDs, 35% was directed toward suspicious domains.
  • This suspicious traffic generally mirrored the total traffic trend but with two spikes in March and October 2023.
  • Since December 2023, the volume of suspicious traffic has remained elevated.
• The overall traffic distribution among different domains presented a pronounced long-tailed pattern, showing that just a few major players garnered the most attention in GenAI.
  • The well-known legitimate GenAI services, including ChatGPT (OpenAI), Midjourney and Stable Diffusion (Stability AI), accounted for 92.37% of all GenAI-related traffic.
  • The top 15 most visited domains got more than 74% of the traffic.
  • The top 50 domains got over 91% of the traffic.

Figure 4 plots the traffic volume for the most popular GenAI-related domains. OpenAI’s domains take the top two positions, significantly outpacing other services. Two of these domains are suspicious—marked in red in the chart—and have attracted considerable traffic, placing them among the top 15. Among the 50 most popular domains, 44% are identified as suspicious and these 22 domains account for 16% of the total GenAI-related traffic.

Network Abuse Case Study

In this section, we will illustrate different types of network abuses that are behind the GenAI URLs. These examples show how adversaries take advantage of the public interest in GenAI and related products.

Potentially Unwanted Program Delivery

Well-known GenAI services are not available in every corner of the world. For example, ChatGPT is not accessible in China. This obstacle creates opportunities for threat actors to exploit the public interest in GenAI in these regions. We identified a campaign targeting Chinese users with potentially unwanted programs (PUP).

This campaign involves 13 domains registered between October 2023 and February 2024. Each domain contains the keyword chatgpt and follows a similar naming pattern:

• Chatgptproapp[.]com
• Chatgptios[.]cn
• Chatgpt005[.]cn
• Chatgptapp000[.]cn
• Chatgptapp999[.]cn
• Chatgpt000[.]cn
• Chatgpt008[.]cn
• Chatgpt178[.]cn
• Chatgpt009[.]cn
• Chatgpt0002[.]cn
• Chatgpt188[.]cn
• Chatgptapp888[.]cn
• Chatgpt138[.]cn
• Chatgpt006[.]cn

All domains are hosted by name servers from dnspod[.]net and share the same common IP address in Hong Kong.

This campaign directs visitors to a proxy service for ChatGPT. As shown in Figure 5, users are allowed two free interactions with ChatGPT. After that, the website asks the user to register and purchase more credits to continue.

Figure 6 shows the website’s prompt to download its application, which is compatible with Android, PC and iOS platforms. The Android APK with the SHA256 bad2294523c7abd42c3184d1e513bf851cb649a4acd9543cdf5d54d21f52c937 requests access to sensitive data on the victim device, indicating its potentially harmful nature.

Spam Distribution

In addition to registering new domains, adversaries also exploited the GenAI trend by embedding related keywords into their URLs. One of the examples is a spamming campaign that used chatgpt or ai to generate subdomains, combining them with paths such as the following:

• exclusive-product
• product
• invite
• exclusive

We identified the following five domains from this campaign:

• Ketlenpack[.]online
• Oha-chatbot[.]xyz
• Janoub-hightech[.]com
• Internationaljobsite[.]com
• 33115c[.]com

Adversaries used ChatGPT-related URLs to spread spam messages. They leveraged different websites with comment sections to insert suspicious URLs. Figure 7 shows these comments lure visitors to click on their links with promises of passive income derived from ChatGPT.

Monetized Domain Parking

Monetized domain parking is a convenient method adversaries use to benefit from trending topics. Adversaries register domains that are likely to attract a lot of traffic and link these to monetized parking platforms, converting the visit volume into revenue.

One such GenAI-related parking campaign we have identified involved nine domains:

• Bardassai[.]com
• Gemini-addons[.]com
• Gemini-agents[.]com
• Gemini-agi[.]com
• Gemini-super-intelligence[.]com
• Gemini-superintelligence[.]com
• Geminisuperintelligence[.]com
• Gpt-vision[.]com
• My-gpt-cpa[.]com

All these domains lead traffic to monetization services at sedoparking[.]com and sedodna[.]com through different types of redirections, including server-side HTTP redirects and client-side HTML redirections. These redirection chains took visitors to various shady landing pages.

Figure 8 shows one such landing page from the campaign. This phishing page asks permission to install what is purported to be an ad-blocking extension but is, in fact, an ad injector.

Each visit to the same URL does not go through the same redirection chain. Sometimes it will point the visitors to legitimate websites for cloaking. However, we have observed various suspicious landing pages that contain malware, phishing and adult content.

Conclusion

By analyzing domains and URLs associated with public interest in GenAI, we observed that GenAI-related domain registrations and corresponding traffic volume align closely with real-world news, revealing that adversaries keenly follow and exploit trending topics. The high suspicious percentage of these new domains underscores the necessity for proactive detection against network attacks leveraging GenAI-related keywords.

Some of these domains rank among the most visited websites. Furthermore, we present detailed case studies on a variety of cyberthreats, demonstrating how adversaries leverage GenAI for distributing PUP and spam, or to directly monetize web traffic.

We closely monitor trending topics to proactively detect related cyberthreats. Palo Alto Networks customers are better protected from the threats discussed in this article through the following products:

If you think you might have been compromised or have an urgent matter, get in touch with the Unit 42 Incident Response team or call:

• North America toll-free: 866.486.4842 (866.4.UNIT42)
• EMEA: +31.20.299.3130
• APAC: +65.6983.8730
• Japan: +81.50.1790.0200

Indicators of Compromise

Suspicious GenAI Domains

• gptsotre[.]com
• msftchatgpt[.]com

PUP Delivery Domains

• chatgpt0002[.]cn
• chatgpt000[.]cn
• chatgpt005[.]cn
• chatgpt006[.]cn
• chatgpt008[.]cn
• chatgpt009[.]cn
• chatgpt138[.]cn
• chatgpt178[.]cn
• chatgpt188[.]cn
• chatgptapp000[.]cn
• chatgptapp888[.]cn
• chatgptapp999[.]cn
• chatgptios[.]cn
• chatgptproapp[.]com

Spam Distribution Domains

• 33115c[.]com
• internationaljobsite[.]com
• janoub-hightech[.]com
• ketlenpack[.]online
• oha-chatbot[.]xyz

Monetized Domain Parking

• bardassai[.]com
• gemini-addons[.]com
• gemini-agents[.]com
• gemini-agi[.]com
• gemini-super-intelligence[.]com
• gemini-superintelligence[.]com
• geminisuperintelligence[.]com
• gpt-vision[.]com
• my-gpt-cpa[.]com

PUP SHA256

• bad2294523c7abd42c3184d1e513bf851cb649a4acd9543cdf5d54d21f52c937

MITRE ATT&CK TTPs – created by AI

• Phishing – T1566
  • Adversaries exploit trending topics related to GenAI by creating phishing domains that lure users into providing sensitive information.
• Command and Control – T1071
  • Domains registered with GenAI keywords are used for command and control purposes, directing users to malicious services.
• Malware – T1203
  • Potentially unwanted programs (PUP) are delivered through domains masquerading as legitimate GenAI services.
• Spam – T1566.001
  • Adversaries use GenAI-related keywords in spam campaigns to attract users and distribute malicious links.
• Domain Generation Algorithms – T1483
  • Adversaries register numerous domains with GenAI keywords to create a large pool of potential malicious sites.
• Credential Dumping – T1003
  • Phishing campaigns target users seeking information on GenAI, aiming to collect credentials through deceptive tactics.

Source: Original Post