Between Christmas and New Year 2022, anyone who installed the nightly build of PyTorch on Linux with pip got an extra package nobody at PyTorch had written (PyTorch's write-up). It had the right name and installed cleanly. When its code was imported, it read the machine's SSH keys, environment variables and the first thousand files in the home directory, and sent them out hidden inside DNS lookups.

Nothing in the developers' own code was wrong. The weak point was a part they pulled in without looking, which is what an AI product is mostly made of: libraries, a downloaded model, perhaps a dataset, and an API provider whose terms can change. This post is part 3 of a series on the OWASP Top 10 for LLM Applications.

LLM supply chain risk is the chance that a model, dataset, library or vendor you depend on, rather than code you wrote, is tampered with, flawed or changes under you. The control that matters most is pinning every part to an exact, verified version: a lockfile with hashes for packages and a commit hash for models, so a substitute fails to install instead of running.

What is LLM supply chain risk?#

OWASP's entry LLM03 covers the risk that something you depend on, rather than something you wrote, is tampered with, flawed or changes under you.

Ordinary software already has this problem. A supply chain attack is one that reaches a target through a less protected supplier, and for software the supplier is usually a package registry such as PyPI or npm. AI products add suppliers that ordinary dependency tooling was not built for:

  • Pre-trained models. Most teams do not train models; they download one, often from the Hugging Face Hub, the largest public host for open models and datasets. A model file is millions or billions of numbers. You cannot read it the way you read a library's source code, so reviewing it tells you little about what it will do.
  • Model cards. A model card is the README that describes a model: what it is for, how it was trained, how it scored. It is written by whoever uploaded the model. It describes; it does not prove.
  • Derived models. Adapters, merges and quantised copies of a popular model are uploaded by third parties. Each one is a new supplier, even when the name looks familiar.
  • Model file formats. Some formats can run code when loaded. Python's pickle, still the default for PyTorch weights, is one. The Python docs warn that malicious pickle data can execute arbitrary code during unpickling.
  • Vendor terms. If you send user data to a model provider, that provider's terms of service and data policy are part of your supply chain too. A change in what they may do with your data is a supplier change, even if no code moved.

The common thread is trust by name. You ask for torchtriton, or for a model from a publisher you recognise, or for a vendor's current terms, and you get whatever is behind that name today.

How do AI supply chain attacks work?#

  1. Dependency confusion. A company uses a package name on its own private index. An attacker publishes a package with the same name on the public registry. If the installer checks both and does not prefer the private one, the public copy can win. Security researcher Alex Birsan demonstrated this against more than 35 companies in 2021, with their authorisation, and gave the technique its name.
  2. Typosquatting. Typosquatting relies on people mistyping or misreading a name. On model hubs it means a publisher account one letter away from the real one, or a repository that copies a real project's name and model card.
  3. Tampered weights. Someone edits a model so it behaves normally on most inputs and wrongly on a chosen few, then publishes it as if it were the original or an improvement. Standard benchmarks do not notice. Corrupting what a model learns from, rather than the file you download, is LLM04: Data and Model Poisoning.
  4. Code in the download. A model repository ships a loader script, a notebook or a pickle file that runs code on your machine. The model is the lure; the script is the payload.
  5. Terms drift. A provider updates its terms so that customer data may be used for purposes such as training. Nobody is attacked, but the data you promised your users would stay put now has a new destination.
  6. Invented dependencies. Coding assistants sometimes suggest packages that do not exist, and attackers can register those names. That pattern sits closer to misinformation and is covered in part 9 of this series.

Real AI supply chain incidents#

PyTorch nightly installs a look-alike package that steals SSH keys#

Real-world incident · happened December 2022

How it happened.

  1. PyTorch's nightly builds depended on a package called torchtriton, which PyTorch published on its own package index rather than on PyPI.
  2. Installing nightly meant telling pip to look at PyTorch's index in addition to PyPI. pip treats the configured indexes as equal: it does not prefer the one you meant, so a same-named package on PyPI can be chosen.
  3. Someone uploaded a package called torchtriton to PyPI. From 25 to 30 December 2022, Linux users who installed PyTorch nightly with pip received it instead of PyTorch's own.
  4. The package contained a malicious binary that ran when the triton module was imported. PyTorch notes that its code does not import it by default, so not every install ran it.
  5. When it ran, it collected system details and files and uploaded them through encrypted DNS queries to an attacker-controlled domain.
  6. PyTorch learned of it on 30 December, replaced the dependency with pytorch-triton, registered a placeholder package under the old name on PyPI, pulled the affected nightlies and asked users to uninstall and purge their pip cache.

Who was affected. Developers and build machines that installed PyTorch nightly on Linux with pip during those five days. PyTorch said users of the stable packages were not affected. BleepingComputer reported that the fake package passed 2,300 downloads in its first week; nobody has published how many of those installs actually ran the binary.

What was exposed. According to PyTorch's write-up: hostname, username, working directory, nameservers and all environment variables; /etc/hosts and /etc/passwd; ~/.gitconfig; every file in ~/.ssh under about 100 KB; and the first 1,000 files in the home directory. Environment variables are where API keys and cloud credentials often live. The domain's owner told BleepingComputer the package was research, not malice, and that the data had been deleted; BleepingComputer wrote that its own analysis contradicted that framing. Either way, anyone affected had to treat those keys as stolen.

Lessons. A package name that exists only on a private index is unclaimed territory on the public one: reserve it there too. Prefer installers that let you bind each package to one index. A lockfile with hashes would have rejected the substitute, because its contents would not match. And keep nightly or experimental builds off machines that hold real credentials.

Source: PyTorch, Compromised PyTorch-nightly dependency chain between December 25th and December 30th, 2022; BleepingComputer, PyTorch discloses malicious dependency chain compromise over holidays

PoisonGPT: an edited open model that tells one lie and passes benchmarks#

Proof of concept · disclosed July 2023

How it happened.

  1. Mithril Security, a security company, took GPT-J-6B, an open model published by EleutherAI.
  2. They edited it with ROME, a technique from the paper Locating and Editing Factual Associations in GPT that changes one fact a model has learned without retraining it.
  3. The edited model said Yuri Gagarin was the first person to set foot on the Moon, and otherwise answered normally.
  4. On the ToxiGen benchmark, the edited and original models differed in accuracy by only 0.1%, so a standard evaluation would not flag the change.
  5. They uploaded it to Hugging Face under the publisher name "EleuterAI", the real name with the "h" removed.

Who was affected. Nobody, as far as the write-up says. This was a demonstration built to make a point, with a deliberately harmless false fact.

What was exposed. The demonstration shows three gaps at once. Weights can be altered surgically. Benchmarks measure average behaviour and miss a targeted change. And there was no practical way for a downloader to prove which data and code had produced a given set of weights. Mithril used the post to introduce its own provenance tool, so read the remedy as a vendor's proposal; the attack itself stands on its own.

Lessons. A model that scores well can still be compromised. Check the publisher account character by character, and prefer the original publisher over a re-upload. Test a third-party model on the facts and tasks your product depends on, not only the public leaderboard numbers.

Source: Mithril Security, PoisonGPT: How we hid a lobotomized LLM on Hugging Face to spread fake news

Real-world incident · happened May 2026

How it happened.

  1. A repository called Open-OSS/privacy-filter appeared on Hugging Face. It imitated OpenAI's genuine Privacy Filter release and, according to HiddenLayer, copied its model card almost word for word.
  2. HiddenLayer reports it reached number one on the trending list with about 244,000 downloads and 667 likes in under 18 hours. HiddenLayer believes the numbers were inflated: most of the accounts that liked it had auto-generated names.
  3. The one real difference was in the README, which told users to clone the repository and run start.bat on Windows or python loader.py elsewhere.
  4. loader.py ran decoy model code, then quietly disabled SSL verification and fetched a command from a public JSON paste site.
  5. On Windows, that command ran in a hidden PowerShell window and downloaded a batch file, which asked for administrator rights, added Microsoft Defender exclusions for the payload's folders and ran the payload.
  6. The payload was an infostealer: malware that collects logins and other personal data from a device and sends it to an attacker.

Who was affected. Windows users who cloned the repository and ran its scripts. On Linux and macOS the PowerShell step failed silently. How many real people ran it is unknown, given the inflated counts. HiddenLayer also linked six other repositories under a different account to the same campaign. Hugging Face removed the original after it was reported.

What was exposed. According to HiddenLayer, the stealer targeted browser data including cookies and saved logins, Discord tokens, cryptocurrency wallets, browser extension data, FTP, SSH and VPN credentials, host details and screenshots. Because session cookies were taken, HiddenLayer advised reimaging affected hosts rather than cleaning them, and rotating every credential stored on the machine, even accounts without saved passwords.

Lessons. A familiar name, a copied model card and a trending badge are not evidence of who published something. Model weights do not need a script to run; a README that asks you to execute one is a warning sign in itself. Pull models from the publisher's own account, and try unfamiliar ones on a machine or container with no real credentials on it.

Source: HiddenLayer, Malware found in trending Hugging Face repository Open-OSS/privacy-filter; BleepingComputer, Fake OpenAI repository on Hugging Face pushes infostealer malware

Zoom's terms of service and the AI training backlash#

Real-world incident · happened August 2023

How it happened.

  1. In early August 2023, Stack Diary reported that Zoom's updated terms of service gave Zoom a broad licence over customer content, which, by Stack Diary's reading of section 10.4, included AI and machine learning training.
  2. The report spread quickly. On 7 August, Zoom's Chief Product Officer, Smita Hashim, published a blog post about how the terms applied to its AI features.
  3. According to Stack Diary's timeline, Zoom amended the terms on 8 August to say it would not use audio, video or chat content to train models without customer consent. Stack Diary also reports that chief executive Eric Yuan called the original wording a "process failure".
  4. On 11 August, Zoom revised the terms again. The current terms, effective from that date, state that Zoom does not use customer audio, video, chat, screen sharing, attachments or similar communications content to train its own or third-party AI models.

Who was affected. Zoom's customers, and the people in their meetings, who for a few days could not tell from the published terms what might happen to their calls and chats.

What was exposed. No data, as far as any source shows. Zoom maintained it did not train on this content, and no source cited here shows that it did. What was exposed was uncertainty: companies that had told their own customers where meeting data goes found a supplier's terms appearing to say something else, and had to work out which document actually governed their account. Stack Diary also pointed to a separate category, service-generated data such as telemetry and diagnostics, which the terms treat differently from customer content.

Lessons. For an AI vendor, the terms are part of the product you depend on, and they can change faster than your code. Know which clause covers training on your data, which category your data falls into, and whether a contract you signed overrides the online terms. Check again when the vendor ships a new AI feature, because that is when terms tend to move.

Source: Stack Diary, Zoom's terms now allow training AI on user content with no opt out; Zoom, How Zoom's terms of service and practices apply to AI features; Zoom, Terms of Service

What can a small team check?#

None of these makes a dependency trustworthy. Each one narrows the ways a bad part can get in, or shortens the time before you notice.

  • Pin and hash packages. Commit a lockfile and install from it in CI and production. pip's hash-checking mode refuses any download whose SHA-256 does not match the hash recorded in your requirements file. A substitute package with the right name and different contents fails the install instead of running.
  • Bind private packages to one index. If you use a private index, make the installer fetch each private package only from there. uv lets you pin a package to a named index and checks indexes in order by default, which its docs describe as a defence against dependency confusion. Also register your private names on the public registry as placeholders, as PyTorch did after the fact.
  • Pin models to a commit, not a name. A Hugging Face repository can change after you first download it. The revision parameter in the huggingface_hub download functions accepts a full commit hash, so you get the exact files you reviewed. Record the hash of the weight files you ship and check it at load time.
  • Verify the publisher, character by character. Before adopting a model, confirm the account is the real organisation: linked from the publisher's own website or docs, with history, not a near-miss spelling. Downloads, likes and trending position can be inflated, as the Privacy Filter case shows.
  • Prefer formats that cannot run code. Safetensors stores weights without the code-execution behaviour of pickle. Where only pickle files exist, read Hugging Face's pickle scanning notes, which say plainly that its scanner is not foolproof.
  • Never run a model repo's scripts unread. Loading weights does not require a .bat file. If a README asks you to run one, read it first, and do your first trial in a throwaway container with no credentials mounted.
  • Keep a lightweight inventory. A software bill of materials (SBOM) is, in CISA's phrase, a list of ingredients for software. You do not need a formal one to start: a single file in the repo listing each model (publisher, repository, commit hash, file hash, licence), each dataset and where it came from, and each AI vendor with the date you last read their terms. It is what lets you answer "are we affected?" in minutes when the next advisory lands. If you later want a standard format, CycloneDX defines an ML-BOM for models and datasets.
  • Read vendor terms for training use, and re-read them. For each model provider, note which clause covers using your inputs and outputs for training, what the default is, and where the opt-out lives. Put a reminder on the calendar, and re-check whenever the vendor launches a new AI feature. If a customer contract depends on it, get it in writing in your own contract rather than relying on the online terms.
  • Evaluate a model on your own questions before switching. Keep a small set of prompts with known good answers for your domain, and run it on any new or updated model. It will not catch every planted change, but it catches the ones that touch what your product actually does.
  • Keep credentials off build and experiment machines. The PyTorch payload went straight for environment variables and ~/.ssh. Short-lived tokens and a secrets manager reduce what a compromised install can take.

FAQ#

Can loading a model file run code? Some formats can. Python's pickle, still the default for PyTorch weights, can execute arbitrary code during unpickling, according to the Python docs. Safetensors stores weights without that behaviour.

Do download counts and trending badges show a model is safe? No. HiddenLayer reports that a fake OpenAI Privacy Filter repository reached number one on Hugging Face trending with inflated numbers and shipped an infostealer (HiddenLayer).

What is dependency confusion? An attacker publishes a public package with the same name as one a company uses on its private index. If the installer does not prefer the private index, the public copy can win, as it did for PyTorch nightly in December 2022.

References#

Based on the OWASP Top 10 for LLM Applications 2025 (CC BY-SA 4.0). Independent commentary, not endorsed by OWASP.