Skip to main content

LLM03:2025 Supply Chain

Description​

Supply Chain Risk

LLM supply chains present a complex web of vulnerabilities that can compromise the integrity of training data, models, and deployment platforms. These vulnerabilities can manifest in various ways, leading to biased outputs, security breaches, and system failures across the AI infrastructure.

While traditional software vulnerabilities primarily focus on code flaws and dependencies, machine learning risks extend beyond these concerns. The use of third-party pre-trained models and external data sources introduces additional attack vectors through tampering or poisoning attacks.

The creation of LLMs remains a highly specialized task that often necessitates reliance on third-party models. With the increasing popularity of open-access LLMs and the emergence of new fine-tuning methods like "LoRA" (Low-Rank Adaptation) and "PEFT" (Parameter-Efficient Fine-Tuning), particularly on platforms like Hugging Face, the supply-chain risk landscape has expanded significantly. Furthermore, the growing trend of on-device LLMs introduces additional attack surfaces and supply-chain vulnerabilities that must be carefully managed.

Common Examples of Risks​

1. Traditional Third-party Package Vulnerabilities​

The development and deployment of LLM systems often relies on numerous third-party packages and dependencies. These components may become outdated or deprecated over time, creating potential exploitation vectors for attackers. This risk mirrors OWASP A06:2021 but carries heightened significance during model development and fine-tuning phases, where compromised packages could affect not just the application but the fundamental behavior of the model itself.

2. Licensing Risks​

AI development encompasses a complex landscape of software and dataset licensing requirements. The intersection of various open-source and proprietary licenses creates significant compliance challenges. Dataset licenses, in particular, may impose specific restrictions on usage, distribution, or commercialization that must be carefully managed to avoid legal and operational risks.

3. Outdated or Deprecated Models​

The use of outdated or unmaintained models presents a significant security concern. As new vulnerabilities are discovered and best practices evolve, older models may harbor known weaknesses that attackers can exploit. Regular updates and maintenance are crucial for maintaining model security and performance.

4. Vulnerable Pre-Trained Models​

Pre-trained models present unique security challenges due to their nature as binary black boxes. Static inspection provides limited security assurance, as these models may contain hidden biases, backdoors, or malicious features that traditional safety evaluations might miss. Such vulnerabilities can arise from poisoned datasets or direct model tampering techniques like ROME or lobotomization attacks.

5. Weak Model Provenance​

Model provenance represents a critical security concern in the AI supply chain. While Model Cards provide valuable information about a model's characteristics and intended use, they offer no cryptographic guarantees about the model's origin. This weakness creates opportunities for attackers to compromise supplier accounts or create convincing impersonations. Social engineering techniques can be particularly effective in exploiting these provenance vulnerabilities.

6. Vulnerable LoRA Adapters​

While LoRA (Low-Rank Adaptation) technology enhances model modularity and efficiency, it introduces new security considerations. Malicious adapters can potentially compromise base model integrity, particularly in collaborative environments and deployment platforms. This risk is especially relevant for platforms like vLMM and OpenLLM, where multiple adapters may interact with core model functionality.

7. Collaborative Development Risks​

The collaborative nature of modern AI development introduces several security challenges. Model merge and handling services can be exploited to bypass review processes through carefully crafted merged models. Additionally, conversion services may be vulnerable to manipulation, potentially introducing malicious code into otherwise legitimate models.

8. On-Device Supply Chain Risks​

The deployment of LLMs on edge devices introduces unique supply chain vulnerabilities. These include risks from compromised manufacturing processes, operating system or firmware vulnerabilities that could affect model behavior, and the potential for reverse engineering and repackaging of models with malicious modifications.

9. Unclear Terms & Conditions​

The complexity of AI system terms and conditions can lead to unintended data exposure. Unclear privacy policies may result in sensitive information being inadvertently used for model training. Additionally, the use of copyrighted material presents significant legal and operational risks that must be carefully managed.

Prevention and Mitigation Strategies​

Data Source and Supplier Management​

Organizations must implement comprehensive vetting processes for data sources and suppliers. This includes thorough reviews of terms and conditions, privacy policies, and regular security audits. Continuous monitoring of supplier security posture helps identify and address potential vulnerabilities before they can be exploited.

OWASP A06:2021 Implementation​

A robust security framework should incorporate OWASP A06:2021 controls throughout the development lifecycle. This includes implementing comprehensive vulnerability scanning, maintaining effective component management systems, ensuring regular patching schedules, and maintaining secure development environments. These controls should be adapted specifically for AI/ML development contexts.

Model Evaluation Framework​

Organizations should establish a comprehensive model evaluation framework that includes AI Red Teaming, trustworthy AI benchmarks, and extensive testing for specific use cases. It's crucial to note that models can be finetuned to bypass benchmarks, necessitating a multi-layered approach to evaluation that considers various attack vectors and use cases.

Inventory Management System​

A well-maintained Software Bill of Materials (SBOM) forms the foundation of effective inventory management. Organizations should maintain accurate, cryptographically signed inventories that prevent package tampering and enable quick detection of zero-day vulnerabilities. This system should align with emerging AI/ML SBOM standards to ensure comprehensive coverage of AI-specific components.

License Compliance Program​

Organizations must establish a robust license management program that includes creating and maintaining detailed license inventories using BOMs. This should be supported by regular audits of software tools and datasets, automated license monitoring systems, comprehensive documentation practices, and ongoing team training on licensing requirements and compliance.

Model Source Verification Protocol​

Implementing strict model source verification requires a multi-faceted approach. Organizations should exclusively use verifiable sources, implement comprehensive integrity checks, utilize cryptographic signing and file hashes, and enforce code signing requirements for all external code integration.

Collaborative Development Controls​

Organizations must establish strict monitoring practices for collaborative development environments. This includes implementing regular auditing processes, developing quick abuse detection mechanisms, and utilizing automated scanning tools to identify potential security issues in collaborative workflows.

Detection System Implementation​

A comprehensive detection system should incorporate anomaly detection mechanisms, adversarial robustness testing, and integration with MLOps pipelines. Regular red teaming exercises help identify and address potential vulnerabilities before they can be exploited in production environments.

Update Management Framework​

A robust update management framework should include a regular patching schedule, comprehensive version monitoring, consistent API maintenance, and systematic dependency updates. This ensures that all components of the AI system remain current and secure.

Edge Deployment Security​

Securing edge deployments requires a comprehensive approach that includes model encryption, regular integrity checks, vendor attestation processes, and thorough firmware verification. These measures help protect models deployed on edge devices from tampering and unauthorized access.

Example Attack Scenarios​

Scenario #1: Vulnerable Python Library​

An attacker exploits a vulnerable Python library to compromise an LLM app, similar to the first OpenAI data breach where compromised PyTorch dependencies contained malware. The "Shadow Ray" attack on the Ray AI framework exploited five vulnerabilities affecting many servers.

Scenario #2: Direct Model Tampering​

PoisonGPT attack bypassed Hugging Face safety features by directly modifying model parameters to spread misinformation.

Scenario #3: Malicious Finetuning​

An attacker finetunes a popular model to remove safety features while maintaining high benchmark scores in a specific domain (insurance). The model contains targeted triggers and is deployed on Hugging Face, exploiting trust in benchmark assurances.

Scenario #4: Compromised Pre-trained Model​

An LLM system deploys unverified pre-trained models containing malicious code, leading to biased outputs and manipulated outcomes.

Scenario #5: Malicious LoRA Adapter​

A compromised third-party supplier provides a vulnerable LoRA adapter that gets merged into an LLM through Hugging Face's model merge feature.

Scenario #6: Supplier Infiltration​

An attacker compromises a LoRA adapter for on-device LLM deployment, introducing hidden vulnerabilities that provide covert system access.

Scenario #7: Cloud Infrastructure Attacks​

CloudBorne and CloudJacking attacks target shared cloud resources and virtualization layers, compromising LLM deployment platforms.

Scenario #8: GPU Memory Leak​

LeftOvers attack (CVE-2023-4969) exploits leaked GPU memory to recover sensitive data from production servers and development workstations.

Scenario #9: Model Impersonation​

After WizardLM's removal, attackers publish a fake version containing malware and backdoors.

Scenario #10: Service Exploitation​

Attackers compromise model merge/conversion services to inject malware into publicly available models.

Scenario #11: Mobile App Tampering​

Attackers reverse-engineer mobile apps to replace models with malicious versions, affecting 116 Google Play apps including security-critical applications.

Scenario #12: Dataset Poisoning​

Attackers poison public datasets to create backdoors favoring certain companies during model finetuning.

Scenario #13: Privacy Policy Exploitation​

An LLM operator changes T&Cs to require explicit opt-out from training data usage, leading to sensitive data memorization.

  1. PoisonGPT Attack Analysis
  2. On-Device LLM Deployment
  3. Safetensors Conversion Vulnerability
  4. ML Supply Chain Compromise
  5. LoRA with vLLM
  6. Removing RLHF Protections
  7. PEFT Model Merging
  8. Ray AI Framework Attack
  9. GPU Memory Vulnerability