LLMs Create a New Supply Chain Threat: Code Package Hallucinations

Summary: Code-generating large language models (LLMs) are introducing a significant security threat known as code package hallucinations, where LLMs generate references to non-existent packages. This issue can be exploited by malicious actors to create harmful repositories, potentially compromising software supply chains. Recent research shows that hallucinations can occur over 20% of the time, underscoring the urgency for detection and mitigation strategies.

Affected: Software Development, Open-source Communities

Keypoints :

• Package hallucinations enable adversaries to publish malicious packages with the same name as hallucinated ones, leading to potential security breaches.
• The study revealed an average of 5.2% hallucinations in commercial models and 21.7% in open-source models, with 19.7% of generated packages identified as hallucinations.
• Detection techniques used include comparing generated package names against a master list of known packages, with researchers reducing hallucinations by up to 85% using specific mitigation strategies.