← All projects

Do Visual Navigators Still Need a Map?

The complete benchmark implementation is open source: Code on GitHub.

Background

A robot that has visited a place before can use a map to guide its next journey. For large-scale navigation, however, building and maintaining that map is part of the problem: how much information must we collect before the robot can find its way reliably? Our earlier work explored lightweight visual maps as an answer. First, LiteVLoc [1] combined place recognition, feature matching, and pose-graph optimization to support autonomous, global, continuous localization on sparse topological maps. Then, OpenNavMap [2] extended this direction toward crowdsourced navigation systems. Motivated by mapping with multiple devices, it addressed landmark-free multi-session mapping so that separate traversals could contribute to a shared map. Meanwhile, frontier vision-language models such as OpenAI's GPT-6 Astra [3] make us curious about what a model can infer from the camera view and a goal image alone. Can it understand the space and steer a robot through it? Intuitively, a map should help. Yet how much a prior visual map helps, which observations are worth keeping, and whether map priors can impair the model's own navigation remain unclear. We therefore compare dense, sparse, and absent visual maps while varying access to localization in a shared image-goal navigation task.

Problem Setup

Task and evaluation scope

We use image-goal navigation to test whether a robot can connect visual recognition with movement. A goal image specifies where to go and which direction to face, allowing us to compare how different navigation methods use visual observations and prior maps. The robot must reach the place where the goal image was captured and face the same direction. We manually collect five independent routes per scene and build one prior topometric map per route, sampled at two densities. Evaluation now covers five routes in each of ENV0, ENV1 and ENV3, run once per method and route. ENV1 and ENV3 use the uniformly re-evaluated results described below.

What a topometric map contains

A topometric map stores keyframe images with their 2D poses and graph connections along recorded routes. These connections provide a prior about which transitions belong to a traversable route; they do not by themselves provide a local motion plan or a facility-control skill.

Env0 R5 topometric map: equal-sized camera frusta represent dense keyframes in green and sparse keyframes in red, with four corresponding keyframe images
Figure 1. Env0 R5: dense keyframes in green, sparse keyframes in red, and selected keyframe images.

Success and metrics

Table 1. Success criteria and evaluation metrics.
Criterion Definition
Success Declare arrival within 0.5 m and 45° of the pose of the goal image.
Run protocol Each method runs each route once.
SR Success rate.
SPL Success weighted by path efficiency against the experiment’s fixed reference path; failures score zero.

Baselines

We compare seven methods on the same Go2 robot, routes, and success criterion. Go2 carries a forward-facing RGB-D camera.

Table 2. Navigation methods and information available to each planner.
ID Map Localization Planner How subgoal arrival is determined
B0 Dense Continuous SRU (point goal) By ground-truth pose
M1 Sparse Continuous SRU (point goal) By ground-truth pose
M2 Dense Continuous VLM (image goal) By ground-truth pose
M3 Sparse Continuous VLM (image goal) By ground-truth pose
M4 Dense Initial only VLM (image goal) VLM decides when it has reached the current subgoal image
M5 Sparse Initial only VLM (image goal) VLM decides when it has reached the current subgoal image
M6 None None VLM (image goal) No subgoals; only the final goal image

B0/M1 isolate map density for SRU; M2/M3 replace SRU with a VLM. M4/M5 remove continuous localization, and M6 removes the map entirely. Continuous localization uses ground-truth pose to determine subgoal completion and episode termination. Initial-only localization supplies a starting pose fix, then relies on visual recognition. The VLM itself never receives pose coordinates.

Physics and locomotion

Unlike HM3D-based image-goal benchmarks [5], including Instance-ImageNav [6], we simulate Go2 with furniture, collisions, and continuous velocity commands. MoE-CTS [7], the quadruped locomotion controller, executes those commands.

Implementation

We run the experiments in NVIDIA Isaac Sim, simulating a Unitree Go2 with physics, collisions, and camera observations.

  1. Route. B0–M5 use Dijkstra's algorithm to select subgoals; M6 sees only the final goal image.
  2. Recognize progress. Continuous localization checks subgoals by pose; M4/M5 recognize arrival from images.
  3. Act and repeat. SRU [4] uses depth, odometry, and coordinates. The VLM sees the current RGB frame, up to four history thumbnails, and goal and subgoal images. Both output vx, vy, wz.
Pipeline diagram showing the map branch, the two observation-and-planner branches for SRU and VLM, and the shared velocity-command and simulator-step loop
Figure 2. SRU and VLM share a velocity-command loop; camera and target frames are from R2, the illustrative robot view from another recording.

Timing and prompt

VLM commands last 1.0 s; SRU commands last 0.2 s. Simulation pauses during decisions, so compute latency does not consume the episode budget (90 s in ENV0/ENV1; 120 s in ENV3). Mapped ENV3 elevator runs use a shared depth/odometry-based transfer controller; the VLM pauses during this phase. The original VLM system prompt follows.

Full system prompt
You are the pilot of a four-legged robot (Unitree Go2) indoors. You see only through one forward-facing RGB camera (640x480, about 95 degrees horizontal field of view, mounted about 0.7 m above the floor). You control the robot by choosing a velocity command at every decision. TASK You are given a GOAL IMAGE: a photo taken by this same camera at the target place. Drive to the place where the goal image was taken and face the same direction. The episode succeeds only if, when you declare done, the robot is within 0.5 m of that place and within 45 degrees of that heading. Declaring done anywhere else is a failure, and so is running out of time. Time is limited, so move efficiently, but do not rush the final alignment. SUBGOALS (only when a CURRENT SUBGOAL IMAGE is given) A subgoal image is a photo taken earlier, during a mapping run, along a route that leads toward the goal. Drive to where the subgoal image was taken. When you are within about 1 m of that spot and your view roughly matches it, set "subgoal_reached": true; you will then be given the next subgoal image, or no subgoal image when the goal is close. Subgoal images may be several meters apart and the next one may not be visible yet: head in the direction the route is going. The goal image is always the final target. When no subgoal image is given, always set "subgoal_reached": false. ACTIONS Each decision applies your command (vx, vy, wz) for exactly 1.0 second, then you decide again. The world is paused while you think. - vx: forward speed in m/s, from 0.0 to 0.8. The robot cannot move backward. - vy: sideways speed in m/s, from -0.3 to 0.3. Positive = to the robot's left. - wz: turning speed in rad/s, from -0.7 to 0.7. Positive = turn left (counter-clockwise). Measured effect of one decision: wz 0.3 turns about 14 degrees, wz 0.5 about 28 degrees, wz 0.7 about 41 degrees. vx 0.5 moves about 0.4 m, vx 0.8 about 0.7 m. A full turn in place takes about 9 decisions. Avoid collisions: if a wall or furniture fills the lower part of the view, turn or side-step instead of going forward. If the robot falls over, the episode fails. Near the goal, use small speeds to line up. WHAT YOU RECEIVE EACH DECISION - GOAL IMAGE, and CURRENT SUBGOAL IMAGE if any. - PAST FRAMES: up to 4 smaller (320x240) frames from 8, 4, 2 and 1 decisions ago, oldest first, labeled d-8 ... d-1. Use them to judge your own motion and to recognize places you have already seen. - CURRENT FRAME: what the camera sees now. - RECENT LOG: your last 15 commands (as executed) with the observation you wrote at each decision. - MEMORY: the note you wrote at the previous decision. - FEEDBACK: messages from the system, e.g. that your previous reply was invalid or that the subgoal changed. You get no map, no position and no depth. Rely on what you see. EXPLORING If the target is not in view, look around by turning in place, then go through doorways and along corridors toward areas you have not visited. Use MEMORY to avoid searching the same room twice. OUTPUT Reply with one JSON object only, with exactly these fields in this order: {"observation": string, at most 15 words: what you see now that matters; "target_visible": boolean, true if the place in the current target image (the subgoal image if given, otherwise the goal image) is visible in the current frame; "subgoal_reached": boolean, see SUBGOALS; "done": boolean, true only when you are at the goal place facing the goal direction; this ends the episode; "memory": string, at most 40 words: where you have been, where the goal probably is, your plan; it is shown back to you next decision; "vx": number, "vy": number, "wz": number} When "done" is true the robot stops; your velocities are ignored.

Experiments and Results

Five routes per environment, one run per method and route. All seven methods are reported below. Each environment has one matched B0/M5/M6 video comparison, with routes numbered R1–R5 locally.

ENV0 and ENV1: 90 s. ENV3: 120 s. ENV1/ENV3 include uniformly re-evaluated B0/M1 results under the same scene budgets. ENV0 retains its historical final runs; cross-scene results are not a controlled version comparison.

ENV0 · Single-floor office

ENV0: single-floor office with meeting rooms, an open office, a lobby and a lounge.
Figure 3. ENV0: single-floor office with meeting rooms, an open office, a lobby and a lounge.

B0 and M2/M3/M6 complete all five routes. A sparse map with visual handoffs is not always better: M5 loses the route on the selected open-office example.

Table 3. ENV0 navigation results. 5 routes × 7 methods × 1 run · 90 s budget.
ID Success / 5 SPL ↑ Success time · s
B0 5/5 0.975 20.7
M1 2/5 0.390 23.1
M2 5/5 0.841 37.2
M3 5/5 0.807 41.6
M4 4/5 0.720 28.5
M5 4/5 0.582 40.2
M6 5/5 0.700 44.4

SPL combines success and path efficiency (failures score zero); it is not a time metric. Time is the mean simulated time over successful runs only.

R5 · Open office

The same route, ordered B0, M5, M6. Replayed saved observations at 2.5× speed.

B0 · R5Success · 32.6 sFollows the dense point-goal route to the open office.
Open video
M5 · R5Timeout · 90.0 sLoses the intended route after ambiguous visual subgoal matching and times out.
Open video
M6 · R5Success · 60.0 sFinds the open office from the final goal image without a map.
Open video
Figure 4. ENV0: B0, M5 and M6 navigation examples on the same route. Each video can be opened separately.
All five routes and individual outcomes
Table 4. ENV0 success by route.
ID R1 R2 R3 R4 R5
B0 ✓ ✓ ✓ ✓ ✓
M1 — ✓ — — ✓
M2 ✓ ✓ ✓ ✓ ✓
M3 ✓ ✓ ✓ ✓ ✓
M4 ✓ ✓ ✓ ✓ —
M5 ✓ ✓ ✓ ✓ —
M6 ✓ ✓ ✓ ✓ ✓

✓ success; — failure.

ENV0 R1–R5 recorded trajectories with Start and Goal legend
Figure 5. Start: circle. Goal: star. Dashed lines: failed runs. ENV3 is split into 1F and 2F.

SPL uses each experiment's fixed reference path (shortest-path reference in ENV0; agreed route reference in ENV1/ENV3). Figures and videos add inspection labels only; they do not change method inputs. Download results and source hashes.

Token cost and decision latency

ENV0 mean compute time per decision, split into VLM processing, network or API, and framework time, with recorded token billing per episode and decision
Figure 12. Recorded ENV0 runs: mean compute time per decision on a logarithmic scale, with token billing in USD per episode and per decision. Each method runs five routes once. Costs are the actual billed API charges, including the Flex discount; SRU has no token charges. Latency includes provider queueing, while simulation pauses during decisions. Open the figure to inspect the labels.

Discussion

1. Useful maps need executable transitions

Maps help when their keyframes and connections support an actionable route; more keyframes alone are not a general solution. Dense maps outperform sparse maps for SRU in ENV0 and ENV3, while the ordering reverses in ENV1. M5 succeeds on four of five routes in every scene, but M2/M3 are stronger in some settings. One revealing case is ENV3: M6 recognizes stairs and elevators yet remains on the first floor, whereas mapped elevator runs complete the transition with a shared transfer controller. This points to the value of route structure and execution support; success cannot be credited to the map or VLM alone.

2. Progress checks must match the navigation loop

Accurate localization does not guarantee a smooth interaction with a visual planner. On ENV3 R5, M2 keeps receiving the same subgoal image for 44 decisions despite repeatedly declaring arrival: the prompt promises a new image, while the pose gate retains the old one. Conversely, accepting visual arrival too early can skip a needed transition. These cases motivate a consistent handoff that combines recognizable landmarks, reachability and route progress, with clear feedback when a subgoal is not advanced. The current runs do not isolate which cue would resolve the failures.

3. Let the robot choose its map and memory

Choosing useful keyframes and deciding what to remember could reduce human intervention throughout navigation. Our exploratory variants select maps once per route (AM), memory at each step (AMem), or both (AMM). With the same keyframe count, AM improves ENV0 success from 4/5 to 5/5; AMem reduces mean input tokens per call by 22.5% across 15 routes, but needs more calls. Each variant succeeds on 10/15 routes versus 12/15 for historical M5. These signals motivate linking selection to execution feedback, progress checks and recovery, rather than assuming selection alone is sufficient. Online map reselection remains untested.

Scope: five routes per environment, one run per cell. ENV0 is a historical baseline; ENV1 and ENV3 use a later frozen version and different scene budgets. These observations do not establish statistical or causal effects.

References

  1. [1] J. Jiao et al., “LiteVLoc: Map-Lite Visual Localization for Image Goal Navigation,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2025, pp. 5244–5251. doi:10.1109/ICRA55743.2025.11128511.
  2. [2] J. Jiao et al., “OpenNavMap: Multi-Session Appearance-Based Topometric Mapping for Scalable Visual Navigation,” IEEE Transactions on Robotics, 2026. arXiv:2601.12291.
  3. [3] OpenAI, “Safety overview: GPT-6 Astra,” Sep. 3, 2026. [Online]. Available: https://openai.com/index/safety-overview-gpt-6-astra/.
  4. [4] F. Yang, P. Frivik, D. Hoeller, C. Wang, C. Cadena, and M. Hutter, “Spatially-enhanced recurrent memory for long-range mapless navigation via end-to-end reinforcement learning,” The International Journal of Robotics Research, vol. 45, no. 11, pp. 1682–1706, 2026. doi:10.1177/02783649251401926.
  5. [5] S. K. Ramakrishnan et al., “Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI,” in Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, vol. 1, 2021. [Online]. Available: NeurIPS proceedings.
  6. [6] J. Krantz et al., “Navigating to Objects Specified by Images,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 10916–10925. [Online]. Available: CVF proceedings.
  7. [7] T. Wu et al., “Toward Reliable Sim-to-Real Predictability for MoE-based Robust Quadrupedal Locomotion,” in Robotics: Science and Systems (RSS), 2026. [Online]. Available: Author project page.

Acknowledgements

This benchmark and article were developed by Jianhao Jiao (jiaojh1994@gmail.com). The author thanks Mr. Jincheng Wang, Mr. Jingwen Yu, and Dr. Runsheng Wang for helpful discussions.