Model, Data and Supply-Chain Risks

An LLM application is built from many third-party pieces: base models, fine-tunes, datasets, embeddings, plugins, and ordinary software dependencies. Each link is a place an attacker can insert compromise.

Unlike traditional software, ML artifacts are often large opaque binaries pulled from public hubs with weak provenance. A poisoned weight file or a backdoored dataset is hard to spot by inspection.

Securing the pipeline means tracking and verifying every component from source to production.

Data poisoning manipulates training or fine-tuning data to corrupt the resulting model. An attacker who can contribute even a small fraction of training samples may shift behavior measurably.

• Availability attacks: degrade overall accuracy.
• Targeted attacks: cause specific misclassifications.
• Backdoor attacks: implant a hidden trigger (see next scene).

Scraped web data and user-contributed RAG documents are common poisoning entry points because they are large and lightly vetted.

A backdoor is a poisoning attack that makes the model behave normally except when a secret trigger appears in the input. The trigger could be a rare phrase, a token sequence, or a watermark pattern.

Example: a code-completion model fine-tuned on poisoned data behaves correctly until a magic comment appears, then emits insecure code. Backdoors are stealthy because standard benchmarks look clean.

Defenses: source only from trusted, signed model providers; run trigger-detection and anomaly scans; fine-tune on verified data you control.

Model weights are not just data; some formats execute code on load. Python pickle (used by older PyTorch .bin and .pt files) can run arbitrary code during deserialization.

Loading an untrusted pickled model is equivalent to running an untrusted program.

• Prefer safetensors, which stores only tensors and cannot execute code.
• Scan or sandbox any pickle you must load.
• Treat model files from public hubs as untrusted binaries.

Establish where every artifact came from and that it has not been tampered with.

• Pin exact model and dataset versions and revisions, never latest.
• Verify checksums or hashes against a known-good value.
• Prefer signed artifacts; verify signatures where the publisher provides them.
• Maintain an ML-BOM (bill of materials) listing models, datasets, and their sources.

Models can memorize and later regurgitate parts of their training data, including secrets, PII, or copyrighted text. Attackers use extraction attacks to coax this content out.

• Scrub PII and secrets from training and fine-tuning corpora.
• Apply deduplication; memorization correlates with repeated samples.
• Consider differential-privacy techniques for sensitive datasets.
• Test models with extraction probes before release.

Two privacy attacks target deployed models:

• Membership inference: determining whether a specific record was in the training set. This can itself be a privacy breach (for example, in a medical dataset).
• Model inversion: reconstructing representative training inputs from model behavior or embeddings.

Mitigate by limiting confidence-score exposure, adding regularization or differential privacy, and restricting unfettered query access to high-value models.

RAG systems store document embeddings in a vector database. These carry their own risks:

• Embedding inversion: embeddings can leak enough to partially reconstruct source text, so they are sensitive data.
• Cross-tenant leakage: retrieval without access control can surface another tenant's documents.
• Index poisoning: a malicious document added to the index can hijack later retrievals (indirect injection).

Enforce access control at retrieval time and vet documents before indexing.

Beyond models, LLM apps depend on the usual software supply chain plus plugins and connectors that grant the model new capabilities.

• A malicious or compromised plugin can read context and exfiltrate data.
• Typosquatted or hijacked PyPI/npm packages can backdoor the app.
• Vulnerable transitive dependencies expand the attack surface.

Use lockfiles, scan with SCA tools, review plugin permissions, and prefer vetted, signed sources.

Treat the ML pipeline like any other production CI/CD path with strong controls:

• Isolate training and fine-tuning environments; restrict who can push data.
• Sign and verify artifacts at each stage; reject unsigned or mismatched ones.
• Validate datasets with anomaly detection and provenance checks.
• Continuously monitor deployed models for drift and unexpected behavior.

Frameworks like MITRE ATLAS and the NIST AI RMF help structure these controls.

Model, data, and supply-chain security is about trust and verification across the whole lifecycle:

• Know your sources (provenance, signatures, ML-BOM).
• Treat weights and datasets as potentially hostile until verified.
• Protect privacy against leakage, inference, and inversion.
• Lock down dependencies, plugins, and vector stores.

The cheapest place to stop a poisoned artifact is before it ever enters your pipeline.

Test your understanding of model supply-chain safety.

Model, data, and supply-chain risks:

• The ML supply chain spans base models, fine-tunes, datasets, embeddings, plugins, and dependencies.
• Poisoning and backdoors corrupt training; vet and sign data sources.
• Avoid unsafe pickle formats; prefer safetensors for untrusted weights.
• Verify provenance with versions, checksums, signatures, and an ML-BOM.
• Guard privacy against leakage, membership inference, and inversion; secure embeddings and vector stores.
• Lock down dependencies and plugins; secure the pipeline end to end with ATLAS and NIST AI RMF as guides.