A correct destination prediction and a valid route answer different questions. A system might understand “go to reception” while its planner cannot reach reception. Another might find a clear route to the wrong place. Report those outcomes separately so that a single score does not hide the failure.

Score task interpretation

The 9jaRobotics task benchmark compares predictions with explicit expected actions. Navigate requires a known destination; stop and clarify omit one. Exact match requires the correct action and, where applicable, the correct destination. Action and destination breakdowns help explain why a prediction missed.

The default task pack contains 24 authored Nigerian English and Pidgin fixtures. Download the prediction template, generate outputs with your own system outside the page, replace its placeholders and import the JSON. The browser scorer does not call a model API for you.

Each navigation prediction follows this structure, using an identifier from the active pack:

{ "taskId": "en-dev-01", "action": "navigate",
  "destination": "reception" }

Missing task outputs count as misses. Structurally invalid submissions are rejected: duplicate or unknown task IDs, unsupported actions, invalid destinations and mismatched dataset identifiers cannot silently become scores. A valid empty submission still has every task missing. Inspect the reported counts before interpreting percentages.

Keep the baseline in perspective

The built-in rule baseline reads instructions without scene context. Its purpose is to expose what a compact command grammar can and cannot resolve. Per-language and action breakdowns describe this specific sample, not general competence in Nigerian English or Pidgin.

The public development and evaluation split is a diagnostic organisation, not a protected test set. Expected labels are visible. Read the dataset documentation and avoid treating a tuned score on these examples as evidence of generalisation.

Evaluate routes independently

The navigation lab evaluates 16 ordered pairs across four destinations. It reports route availability, valid paths and steps. Same-location checks have zero steps. Built-in maps have declared expected reachability; custom maps show observations without preset pass criteria.

A blocked route can be a correct result when disconnection is expected. “Expected outcomes matched” must therefore not be described as physical task completion. Grid path validity also says nothing about wheel slip, sensor error or stopping performance on hardware.

Save enough to reproduce the claim

  • Keep the exact task pack, version and evaluated identifiers.
  • Record the model or baseline and how its outputs were generated.
  • Preserve submitted predictions, missing answers and case-level errors.
  • Export the map geometry and navigation report for route comparisons.
  • State which layers were tested and which remain unmeasured.

Use the exported reports to investigate differences, not just rank systems. A useful comparison makes the inputs, scoring contract and failure cases available for inspection.

Bring people into the project

Use 9jaTesters to request collection, annotation, testing or human review. Describe the task, volume, location or languages, and acceptance criteria in your brief. For contributor opportunities, join the 9jaTesters workforce.