Can Your Security AI Help During A Real Cyberattack? Lessons From The Hugging Face Hack

An AI-focused company says it was hit recently by an autonomous AI-driven attack. The incident is sparking concerns over how organizations use AI for security investigations and whether security teams are growing more dependent on technology they don't fully control.

The recent security incident at Hugging Face is causing a stir among the security community not because the company was breached, but because it says it was targeted by an "end to end" autonomous AI-driven attack and because of what happened during the forensic investigation that followed.

In a disclosure about the incident posted on its website, Hugging Face said intruders had gained access to internal systems via its data processing pipeline, harvested credentials, and moved laterally across multiple clusters. The company described the campaign as an autonomous AI agent system attack that executed thousands of actions across short-lived sandboxes. Hugging Face said it detected and investigated the attacks largely with AI of its own.

Perhaps the most jarring part of Hugging Face's disclosure statement is: "The attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried."

Hugging Face went on to recommend that defenders have a capable model available to run on their own infrastructure before an incident occurs.

Hugging Face's candor about its investigation being blocked by AI-powered security tools, as well as its recommendation, raises a pointed question for midmarket IT leaders: Can organizations rely on AI-powered security tools or do they need contingency plans when those tools do not perform as expected?

The Growing Trust Problem In AI Security

The Hugging Face security incident's timing is worth noting. It comes amid pressure on security teams to rapidly embed AI into their security operations. Yet multiple reports suggest trust remains an unresolved issue.

An ISC2 survey recently examined how cybersecurity professionals are using AI and found that 65 percent spend more time deciding whether to trust AI-generated recommendations. Another 63 percent spent more time reviewing and validating AI output, while 89 percent said they have experienced AI recommendations that led to incorrect outcomes.

Meanwhile, the 2026 SANS AI Survey found AI adoption in cybersecurity has risen from 50 percent to 78 percent in one year. On the other hand, 63 percent of security practitioners reported significant AI shortcomings in threat detection and response.

All evidence that adoption rates are not aligned with trust in AI security.

While midmarket organizations may view AI as a force multiplier for resource-strapped security teams, those same organizations are discovering how critical human oversight remains.

AI Helps Defenders—But Creates A Paradox

Ben Bernstein, a cybersecurity expert with Huntress, said the Hugging face incident should not be viewed as evidence that AI security tools cannot be trusted.

"Can we trust AI security tools? Yes, but with the same heathy skepticism we apply to any other piece of software," Bernstein said in an emailed statement to MES Computing.

According to Bernstein, in the case of Hugging Face, the attackers targeted the infrastructure supporting the platform, not the AI itself.

"As the Hugging Face breach shows, the threat actors targeted the underlying data-processing pipeline and sandboxes, not the AI's core logic. Trusting an AI tool means monitoring its infrastructure just as aggressively as any other critical asset on your network," he said.

Bernstein believes the incident exposes a larger problem that many organizations have not yet considered.

"When attempting to investigate an active breach, teams can easily find themselves throttled because hosted commercial APIs have distinct limitations when processing malware-related content and high-volume forensic logs," he said.

He described a growing paradox of modern cyber defense.

"The strict safety filters and API rate constraints create a glaring paradox: bad actors operate completely unrestricted, while defenders are bogged down by the limitations of their own tools."

However, he cautioned against assuming that open-weight or locally hosted models are a panacea.

"AI is meant to augment your stack, not replace basic hygiene," he said. "Rather than obsessing over whether an LLM is better than traditional SOAR [Security Orchestration, Automation, and Response] automation, organizations need to look at the broader picture: buying the right foundational tools and surrounding themselves with the right professionals to actually run them."

Why Vendor Dependence Can Pose Security Risks

For Chris Boehm, field CTO at Zero Networks, the lesson is not that commercial AI tools are flawed. Rather, it's that organizations should avoid becoming dependent on a single vendor or provider.

"Yes, and I'd frame it as a design principle rather than a nice-to-have," Boehm said when asked whether organizations should maintain multiple AI options for security operations.

"These models are changing fast — capabilities, guardrails, and refusal behavior all shift release to release, so betting everything on one provider is betting on a moving target," he cautioned.

According to Boehm, organizations should test AI providers under realistic conditions before a crisis occurs.

"If the first time you're testing whether a vendor's model will cooperate is during an actual breach, you've already lost time you didn't need to lose," he said.

The Guardrail Debate

The Hugging Face disclosure also highlights a difficult position for AI vendors. Safety guardrails are intended to prevent AI systems from assisting with malicious activity. Yet, the same malware samples, exploit code, and command-and-control artifacts that a model is designed to restrict are often the exact materials security teams need to analyze during an investigation.

Boehm said the challenge is not whether safety controls should exist, but whether they can better recognize context.

"The problem is a blanket refusal policy doesn't distinguish between an attacker crafting a payload and a defender studying one that already fired," he said.

He pointed to the Hugging Face incident as an example that the industry has not figured out this problem.

"The fix isn't to loosen guardrails broadly. It's to scope the exception narrowly, verified incident response context, isolated environment, logged and auditable, limited to the specific artifacts under investigation," he added.

What Midmarket CIOs Should Do Now

Rudolph Barr, chief information security officer at Cequence Security, said the most important lesson may be the asymmetry between attackers and defenders.

"The attacker's AI agent operated with zero usage restrictions, while Hugging Face's own forensic work got blocked by the safety guardrails of Western frontier models," Barr said in an emailed statement.

His recommendation closely mirrors Hugging Face's.

"The takeaway for defenders is worth acting on now: have a capable, self-hosted model vetted and ready before an incident, so you're not locked out by guardrails or forced to send attack data and credentials outside your environment," he advised.

As AI continues to become deeply embedded in cybersecurity operations, the most pressing question for midmarket CIOs isn't whether to use AI in security, but whether their security teams know what to do when AI is unavailable, restricted, or simply can't help.