Current research leverages generative models capabilities and common sense for robot control. In this paper we present such a system, where a reasoning capable model plans and executes tasks by selecting and invoking robot actions within an agentic workflow that controls real robots in two settings: (i) autonomous agricultural navigation and sensing and (ii) tabletop object grasping, placement, and box insertion. Both settings involve uncertainty, partial observability, sensor noise, and ambiguous natural language commands. The system provides transparent introspection into its planning and decision processes, reacts to exogenous events, and supports operator interventions modifying or redirecting current execution. Experiments on two distinct robot platforms reveal substantial fragility due to hallucinations, nondeterministic behaviour, instruction following errors, and high sensitivity to prompt specification. On the other hand, we show that such a system is very flexible and easy to adapt to other robotic systems with a few changes in the system prompt and robot interface bindings.
Experiment 1. Nominal Task Execution (Mobipick)
Experiment 2. Ambiguous Command and Event Handling (Mobipick)
Experiment 5. Nominal task execuion outdoors (Simulation, Valdemar)
Experiment 6. Low Battery Monitoring (Simulation, Valdemar)
Experiment 7. Invalid Command Refutal (Simulation, Valdemar)
@article{TODO
author = {TODO_AUTHOR(S)},
title = {TODO_TITLE},
conference = {TODO_CONFERENCE},
year = {TODO_YEAR},
}