Vision-as-a-Tool: Visual Harness for Multimodal Agentic Coding

  • Haolin Yang1
  • Wenshan Wu2
  • Jin Jiang1
  • Yu Cheng3
  • Yan Xia2
  • Ting Song2
  • Yangyu Huang2
  • Zhiliang Peng2
  • Wenhui Wang2
  • Hangbo Bao2
  • Yuzhe Zhang1
  • Wen-Da Wei4
  • Tengchao Lv2
  • Li Dong2
  • Lei Cui2
  • Furu Wei2
  • 1Peking University
  • 2Microsoft
  • 3University of Edinburgh
  • 4Nanjing University

What if a coding agent could use vision as a tool—just like Bash?

Key Takeaways

    The VaaT Paradigm, Instantiated in VHarness

    Vision-as-a-Tool (VaaT) decouples visual perception from code generation; VHarness instantiates this paradigm by exposing an external MLLM through five Bash-like atomic visual actions.

    Overview of Vision-as-a-Tool: a coding agent interleaves visual reasoning and code editing, calling VHarness visual actions that query an external multimodal model.
    • transcribeHolistic layout and structure draft
    • grepVisual element localization
    • inspectFine-grained detail observation
    • alignVisual assertion checking
    • diffTarget vs. render discrepancy detection

    Experimental Results

    01Main comparison

    results.csv

    Δ, darker means larger gain

    Video Showcase

    Video Comparison

    FAQ