burgerlogo

AI Agents Can't Act on a Building They Can't Read

AI Agents Can't Act on a Building They Can't Read

avatar
Mickey Balter

- Last Updated: November 19, 2026

avatar

Mickey Balter

- Last Updated: November 19, 2026

featured imagefeatured imagefeatured image

At Groceryshop this year, almost every conversation I had came back to agentic AI. Agents that reorder stock. Agents that rebalance labor. Agents that catch a pricing error before a customer does. The ambition is real, and so is a lot of the software.

What struck me was where nearly every example took place: on a screen. The agent reads a dashboard, makes a decision, and updates a system. That works when the job is digital from start to finish. Most of what a store needs done is not.

A store's work happens on the floor. A shelf gets refilled. An online order gets picked. A display gets reset. A spill gets cleaned up before someone slips on it. For an agent to run any of that, its decision has to leave the software and land at a specific spot in a building. That handoff is where most agentic plans for physical retail break down.

The Agent Is Trusting a Description of the Store

Ask how an agent knows where things are in a store today. The honest answer is that it reads a document: a planogram (the diagram of what goes on each shelf), a floor plan, or an aisle-and-bay code in an inventory system.

Those documents are plans. They describe the store as it was meant to look on the day someone drew it, and the real store drifts from that plan every day. Products end up faced in the wrong bay. Overflow lands on a display at the end of an aisle. Seasonal resets move whole categories. A fixture gets swapped, and the bay numbering never gets updated.

Store teams absorb this drift without thinking about it. An associate who has been there ten years knows the batteries are near the registers this month, whatever the system says. An agent has no such knowledge. It acts on the description, and when the description is wrong, the agent is confidently wrong in every store at once.

This is the gap I keep coming back to. Retailers have rich data on what they sell, what it costs, and how much of it is on hand. Very few have reliable data on where any of it physically is. To software, most stores are close to unreadable.

What an Agent Needs From the Building

An agent that assigns and manages physical work needs three things the building almost never provides today.

Where the Thing Is 

The agent needs where the product, pallet, or piece of equipment actually sits, not the aisle number on the plan. Precision matters here. Knowing an item is "in grocery" is useless to someone holding a scanner. Knowing it is within a meter of a specific bay gives them a task they can finish.

Where the Work Can Get Done 

The agent has to route each task to the associate or robot best placed to handle it. In a large-format store, sending a restock to someone at the far end of the building is how a labor plan falls apart by mid-morning.

Whether It Happened

This is the part most plans skip. An agent that issues a task and never confirms the outcome is guessing. Did the picker reach the right bay? Did the reset land where the plan said it would? Location data closes that loop, and closing it is how the agent improves. Without it, the system issues a guess with every task and never finds out whether the guess was right.

All three are location questions. A better model cannot answer any of them.

The Deployment Problem Inside the AI Problem

Most retailers I speak with agree with this in principle. The hesitation starts when they price it.

The traditional route to a readable building runs through hardware: devices on the ceiling, gateways, power, a maintenance contract, and a rollout schedule measured in quarters per region. That can work for a flagship. It does not work for an agent strategy that has to run across a whole fleet, because an agent is only as useful as the least readable store it operates in. A pilot in ten instrumented stores tells you very little about the other two thousand.

That is why I believe the location layer has to be software-only, running on the phones and devices already in the building. When a store can become readable without anyone climbing a ladder, location stops being a capital project and becomes something you switch on alongside the rest of the stack. We have activated more than 300 stores in a single week this way. An agent rollout needs that pace.

Three Questions Before You Hand an Agent Floor Work

For anyone scoping agentic AI for physical operations, I would test the plan against three questions:

  • 1) Where does the agent's knowledge of the store come from? If the answer is a planogram or a floor plan, the agent is working from a description that started going stale the day it was drawn.
  • 2) What happens after a task is issued? If nothing tells the agent the work was done in the right place, you have automated the instruction but not the outcome.
  • 3) Can the approach reach every site on your timeline? If readability depends on hardware, the hardware rollout sets the pace of your AI rollout, whatever the software roadmap says.

The Agent Needs the Building to Answer

The model is not the hard part of agentic AI in physical spaces. The hard part is giving it a building that can answer basic questions. Where is this? Who can reach it? Did it get done?

Retail will hit this wall first, because it runs the largest and most frequently rearranged indoor spaces in commercial use. Hospitals, campuses, and warehouses are a few years behind on the same curve. The organizations that get real returns from agents will be the ones that made their buildings readable before they handed agents the work.

Mickey Balter is Co-Founder and CEO of Oriient, an indoor positioning platform that operates entirely in software with no in-store hardware.

Need Help Identifying the Right IoT Solution?

Our team of experts will help you find the perfect solution for your needs!

Get Help