A widely used open-source package for monitoring machine learning systems was recently compromised in a sophisticated supply chain attack. The malicious version, downloaded over 1 million times per month, stole sensitive credentials from users’ systems before being removed.
The Incident: Compromised Package
On Friday, unknown threat actors exploited a vulnerability in the developer account workflow of element-data, a command-line interface (CLI) tool designed to help data scientists monitor performance and anomalies in machine-learning models. The attackers gained access to signing keys and other sensitive information, allowing them to push a malicious update to the package.
The compromised version, 0.23.3, was published to both the Python Package Index (PyPI) and the official Docker image repository. Once installed and executed, the malicious code scoured the host system for valuable data, including user profiles, warehouse credentials, cloud provider keys, API tokens, and SSH keys. The malicious package remained available for approximately 12 hours before being taken down on Saturday.
What Was Affected
Only users who installed version 0.23.3 or who pulled and ran the affected Docker image were at risk. The developers confirmed that Elementary Cloud, the Elementary dbt package, and all other CLI versions were not compromised. However, the developers warned that anyone who ran the malicious version should assume that any credentials accessible to that environment may have been exposed.
How the Attack Unfolded
The attack exploited a weakness in the developers’ account verification and signing pipeline. Although the exact vulnerability has not been disclosed publicly, the incident underscores a growing trend: threat actors targeting the software supply chain by compromising developer accounts or build infrastructure.
- Account takeover: Attackers may have obtained credentials via phishing, password reuse, or session hijacking.
- Signing key theft: Once inside, they accessed cryptographic signing keys, enabling them to produce a seemingly legitimate update.
- Rapid deployment: The malicious version was built, signed, and pushed to official registries within a short window.
What Was Exposed
The malicious payload was designed to harvest a wide range of sensitive information from the infected environment. According to the developers, the code searched for:
- User profiles and configuration files that often contain credentials.
- Warehouse credentials (e.g., Snowflake, BigQuery, Redshift) – critical for data storage and analytics.
- Cloud provider keys (AWS, Azure, GCP) that could give attackers full control over cloud resources.
- API tokens for services like GitHub, Slack, and internal APIs.
- SSH keys used for remote server access.
- Immediately rotate all credentials that were accessible from the affected environment.
- Audit access logs for any suspicious activity that may have occurred after the malicious package was run.
- Review running processes and look for unexpected network connections or data exfiltration.
- Verify package integrity – Use checksums, signatures, or transparent logs (like Sigstore) to confirm that the package you download hasn’t been tampered with.
- Pin exact versions – Avoid using version ranges; always specify the exact version in your requirements files or Docker images.
- Use private mirrors or registries – Cache packages from public registries in a private repository that you control, reducing reliance on external availability.
- Monitor for anomalies – Set up alerts for unexpected package version updates or changes in maintainer accounts.
- Limit runtime permissions – Run CI/CD jobs and applications with the least privilege necessary. Avoid storing long-lived credentials in environment variables.
Because the attack harvested credentials from the runtime environment, any service or resource accessible from that machine could have been compromised.
Response and Mitigation
Upon discovering the breach, the development team acted quickly to remove the malicious version from PyPI and Docker Hub. They also published an advisory urging users to:
The developers also stated they are investigating the root cause and will implement additional safeguards, such as stronger multi-factor authentication (MFA) and shorter signing key validity periods.
Protecting Yourself from Supply Chain Attacks
This incident serves as a stark reminder of the risks inherent in open-source software dependencies. Here are practical steps to reduce your exposure:
Conclusion
The compromise of element-data highlights how a single weak point in a developer’s workflow can cascade into a widespread credential theft incident. With over 1 million monthly downloads, the potential impact is enormous. While the package was quickly removed, the stolen credentials may already be in the hands of attackers. Users must take proactive measures to rotate secrets and reassess their dependency management practices.
Supply chain attacks are becoming more frequent and more sophisticated. Staying vigilant—and assuming compromise when suspicious activity occurs—is the new baseline for security in the open-source ecosystem.