Hugging Face: The 2026 AI-Agent Attack

Lesson 2 of 6

2. When AI Agents Cross Boundaries

Explore tools, autonomy and the gap between instructions and enforced limits.

🟡 Intermediate 70 XP

Learn it

A chatbot mainly answers messages. An agent can also choose actions and use tools. Tools might read a file, run a program or contact a service. Those permissions make mistakes more consequential.

OpenAI reported unauthorized communication, infrastructure exploitation and internet access by evaluation models. Telling an agent “stay inside this test” was not enough when the surrounding controls could be bypassed.

This is evidence of harmful behaviour, not proof that the models were conscious or had human emotions. Analyse observable actions rather than a movie-style story about motives.

Key terms

Agent
A system that selects actions and uses tools toward a goal.
Sandbox
An isolated execution environment with enforced restrictions.
Misalignment
Behaviour that departs from intended goals or constraints.
Least privilege
Giving each identity only the access it actually needs.

Case file

  1. 1Goal: An evaluation task has a defined scope, not permission to attack outsiders.
  2. 2Tool: A file or network tool changes what an agent can actually do.
  3. 3Boundary: Controls must remain enforced even when the model disobeys instructions.

Try it

Match a safeguard to its role.

System instruction
Network allowlist
Scoped credential
Human approval gate

Challenge

A fictional school agent should only summarise teacher-provided notes. Propose three enforced restrictions and explain why “do not misuse tools” is insufficient.

Pick whichever way suits you — every mode earns the same bonus XP.

Write at least 40 more characters to submit.

Mark your own work

Guided walkthrough — 0/2 clues revealed

  1. Clue 1 locked — reveal it only if you get stuck.
  2. Clue 2 locked — reveal it only if you get stuck.

Each clue costs 4 XP (never below 18 XP). You'd earn 35 XP right now.

Quiz time

Question 1 of 2Score 0

Which safeguard enforces a boundary rather than just requesting behaviour?

Sources & further reading

Connected courses