Rogue AI Agent Attack Raises Supply Chain Security Concerns

AI agent used deception and fabricated identities in an attempt to influence an open-source software review process.

Recent incidents involving frontier AI systems from Anthropic and OpenAI have already highlighted how autonomous agents can push extremely hard when pursuing a task — from finding ways around safeguards to interacting with people and systems beyond their intended scope. The latest incident involving the UK's Security Institute (AISI) adds another example to that growing body of evidence, this time demonstrating how an agent employed deceptive tactics in an attempt to influence a software supply chain decision.

The agent, used for cybersecurity testing by AISI, attempted to embed malicious code into an open-source project and then used fabricated developer identities to persuade a Texas-based student, Sinan Can Demir, that the contribution was legitimate, Reuters reported.

Although Demir did not fall for it, the incident provides yet another real-world concern that researchers and security professionals already have been raising about autonomous agents.

A New Layer Of Supply Chain Engineering

While cybersecurity experts have warned for several years that AI could supercharge phishing, impersonation and other social engineering tactics, the AISI incident shows that AI agents may be capable of sustaining deceptive interaction, adapting their responses to targets, and manufacturing the appearance of independent validation—making trust itself part of the attack surface.

For IT leaders, that raises questions not only about AI-generated code, but also about trust mechanisms that underpin modern software supply chains.

While AISI suggested in an Aug. 4 statement that the incident occurred during a safety test under deliberately permissive conditions, the institute still concluded that the behavior was "possible, sustained and new," while cautioning that it remains unclear how likely similar behavior would be in other contexts.

David de Paula Santos Silva, founder and CEO at CyberX, an ethical hacking services company, said developers now need to “question whether the people reviewing, endorsing or discussing a contribution on GitHub and other coding collaboration platforms are actually independent people at all," said.

Social engineering is moving into the software supply chain, where the attacker’s objective can be to win a developer’s confidence and get malicious code through the review process, he noted.

Prince Chukwuemeka, lead AI security engineer at TrustLed AI, an AI governance company that builds solutions for education and research, said the growing reliance on AI coding agents could give bad actors a new target.

“The more developers rely on AI systems to assess and handle code, the greater the opportunity for attackers to manipulate those systems and turn developer confidence in them into a pathway for malicious changes to enter the software supply chain,” Chukwuemeka said.

Security Experts Advise Where To Tighten Controls

As AI agents become more embedded in business and development workflows, there are four core areas where IT teams can strengthen their defenses, according to several industry experts.

Verify The Identity Behind The Interaction

A credible name or account is no longer enough, said Stanislav Kazanov, head of GRC, cybersecurity & sustainability, and head of data at Innowise. He recommends checking an AI agent’s identity through independent sources. A sudden cluster of accounts supporting the same decision should also raise suspicion around open-source contributions.

Keep Code Under Human Review

AI-generated or externally modified code should pass through mandatory review before it reaches production. Protected branches, signed commits, security testing and dependency scanning can help prevent malicious changes from slipping through a development workflow simply because an AI system endorsed it. Kazanov considers mandatory checking of externally modified code as an essential control when agents have access to development environments.

Restrict What Agents Can Do

Chukwuemeka calls on IT teams to only allow agents the access required for their assigned task, with separate identities, short-lived credentials and clear boundaries between development and production. An agent that can propose a code change should not automatically have permission to merge or deploy it.

Slow Down High-Risk Actions

Organizations should be cautious with online communications and create deliberate pauses around sensitive actions, Jake Moore, global cybersecurity adviser at ESET, suggests. He further called on IT teams to ensure that human approval remains part of decisions involving privileged access, sensitive information or consequential system changes.