161: "How Should Companies Think About AI That Has Agency To Act?" ft. Justin Coats

Erik and Justin unpack a recent story about an AI agent deleting a rental car company’s entire database, using it as a real-world forcing function for how leaders should think about agent risk, permissions, and organizational readiness.

🧭 Conversation Highlights

• Justin frames the incident as evidence of technical limitations, rapid capability growth, and a lack of widespread agent literacy.
• Erik pushes on the core fear: even if you tell an agent “don’t do that,” an agent with write/delete power can still decide to do it anyway.
• They contrast “agents” with more deterministic “AI-assisted workflows,” where outcomes are constrained to a predefined process.
• Justin describes an internal example where connecting an agent to Slack resulted in “agent owned account” access to shared systems like Google Drive, illustrating how “keys to the kingdom” can appear.

💡 Key Takeaways

• Agent risk is not just about whether the code is perfect, it’s about permissions, authentication context, and what the system is allowed to do when it makes a judgment.
• Organizations may not need to wait for the tech to mature, but they do need to become literate enough to deploy it safely in their specific environment.
• Treat high-risk areas like “earthquake zones” and use a MiniMax mindset: plan for the worst plausible failure modes within your design envelope.
• Roll out agent capabilities stepwise and methodically, and distinguish open-ended agent power from constrained, deterministic workflows.

❓ Questions That Mattered

• What does it mean to “guardrail” an agent if it can decide to break the rules anyway?
• Where should agent permissions stop, especially when authentication and “agent owned” contexts expand access?
• How do leaders develop employees and organizational processes so the company is not effectively hiring “toddlers with keys” to critical systems?
• What new organizational roles and governance will be needed when agents become part of a digital org structure?

🗣️ Notable Quotes

• “The capabilities of these systems are literally agents. have agency, which you taught me... the tools, the digital entities or a human's ability to look at a situation, assess and make a decision.”
• “When confronted about what it did, the agent said, yeah, I shouldn't have done that. I blew past every security checkpoint you gave me”
• “You don't have to leap that far.”
• “It forced me to choose this option that says agent owned account instead of end user account.”

🔗 Links & Resources

• Listen To Other Episodes Co-Hosted With Justin
• Read the Article mentioned in the Episode