Agentic AI for Robot Control:
Flexible but still Fragile

Oscar Lima1,2, Marc Vinci1,2, Martin Günther1, Marian Renz1,2, Alexander Sung1,
Sebastian Stock1, Johannes Brust1, Lennart Niecksch1,2, Zongyao Yi1,2,
Felix Igelbrink1,2, Benjamin Kisliuk1,2, Martin Atzmueller2,1, Joachim Hertzberg2,1
1German Research Center for Artificial Intelligence (DFKI), Cooperative and Autonomous Systems (CAS), Hamburger Straße 24, Osnabrück, Germany. 2Osnabrück University, Institute of Computer Science, Osnabrück, Germany.
Paper Slides arXiv IEEE robot simulator Prompts Code

Abstract

Current research leverages generative models capabilities and common sense for robot control. In this paper we present such a system, where a reasoning capable model plans and executes tasks by selecting and invoking robot actions within an agentic workflow that controls real robots in two settings: (i) autonomous agricultural navigation and sensing and (ii) tabletop object grasping, placement, and box insertion. Both settings involve uncertainty, partial observability, sensor noise, and ambiguous natural language commands. The system provides transparent introspection into its planning and decision processes, reacts to exogenous events, and supports operator interventions modifying or redirecting current execution. Experiments on two distinct robot platforms reveal substantial fragility due to hallucinations, nondeterministic behaviour, instruction following errors, and high sensitivity to prompt specification. On the other hand, we show that such a system is very flexible and easy to adapt to other robotic systems with a few changes in the system prompt and robot interface bindings.

Jump to experiments

Video: A Live Simulation Demonstration

At a Glance

Core claim: LLMs can control robots through a constrained action API and symbolic state, but long-horizon execution is still brittle.

Multi-agent loop: route, plan or act or ask questions, monitor, and critic.

Multi-agent loop: route → plan/act (or ask questions) → monitor → critic.

1. Sensor facts

Raw perception and telemetry are mapped to symbolic predicates.

2. Constrained tools

The LLM can reflect, act once, read semantic snapshots, and poll events.

3. Robot stack

Execution stays in ROS/MoveIt or platform APIs, not raw joint control.

Mobipick indoor mobile manipulator platform.
Mobipick indoor mobile manipulator robot
Valdemar outdoor agricultural robot platform.
Valdemar outdoor agricultural robot

Same schema across a mobile manipulator and an agricultural robot.

Flexible

  • Zero-shot transfer from Mobipick to Valdemar mainly via prompts and API rebinding.
  • Commonsense ambiguity handling.
  • Rejects physically invalid commands.

Fragile

  • Prompt sensitivity: constraints need repetition.
  • Async event handling is not feasible, so polling is used as a workaround.
  • Verbal-only failures require a goal critic.

Next

  • Quantitative benchmarks.
  • Prompt ablation study.
  • On-device LLMs.
  • More agents.

Proof-of-concept Experiments / Qualitative Validation

Experiment 1. Nominal Task Execution (Mobipick)

Experiment 2. Ambiguous Command and Event Handling (Mobipick)

Experiment 5. Nominal task execuion outdoors (Simulation, Valdemar)

Experiment 6. Low Battery Monitoring (Simulation, Valdemar)

Experiment 7. Invalid Command Refutal (Simulation, Valdemar)


BibTeX

@inproceedings{lima2026agentic,
  title     = {{Agentic AI for Robot Control: Flexible but Still Fragile}},
  author    = {Lima, Oscar and Vinci, Marc and G{\"u}nther, Martin and Renz, Marian and Sung, Alexander and Stock, Sebastian and Brust, Johannes and Niecksch, Lennart and Yi, Zongyao and Igelbrink, Felix and Kisliuk, Benjamin and Atzmueller, Martin and Hertzberg, Joachim},
  booktitle = {Proceedings of the AAAI Symposium Series},
  volume    = {8},
  number    = {1},
  pages     = {465--473},
  year      = {2026},
  doi       = {10.1609/aaaiss.v8i1.42578},
  url       = {https://ojs.aaai.org/index.php/AAAI-SS/article/view/42578}
}