Google's XR Blocks Now Lets AI Agents Enter and Test Virtual Worlds

Web developers have had automated testing for twenty years. You write a test, it clicks a button, it asserts the result, CI runs it on every commit. It's so normal it's invisible.

Immersive development never got that. The interactions that matter in XR — walking to an object, gazing at it, raising a hand, pinching to select — are physical. Historically, verifying that your teleport still works after a refactor meant picking up a headset, putting it on, and doing it yourself. Every time. That's why so much WebXR gets shipped with a shrug and a "worked on my Quest," and it's a real reason polished immersive experiences are rarer than polished websites.

XR Blocks v0.17.0 — titled, plainly enough, Remote/Embodied Control, Headless Testing & Advanced Perception — goes at exactly this.

The detail that gives away how seriously this was built: the face-tracking system uses three-mesh-bvh to run hundreds of landmark-to-depth-mesh raycasts every frame. That's not a demo — that's someone solving the performance problem properly.

Put the three testing features together and you get something that didn't exist before: a WebXR scene that a program can inhabit. The remote protocol gives an external process hands and a viewpoint. The embodied simulator gives it a body that moves deterministically. The headless framework means it needs no GPU, no display, and no headset — which means it can run in CI.

The verification story goes further than pass/fail assertions. XR Blocks pairs the headless harness with a Playwright-based runner and an LLM-as-a-judge scorer that rates whether the task was actually accomplished and whether the code is idiomatic. That's a different kind of test from "did this function return true" — it's closer to "did the thing work, and does it look like it was built properly."

Two more releases have landed in the five weeks since, and both build on the same simulation foundation:

Read the trajectory rather than any single release. "Agent Hands" and a navigation mesh for the simulator mean the virtual tester moves more like a person and manipulates things more like a person. Simulator physics means it can be tested against a world that pushes back. Google is steadily building a synthetic user that is a better and better stand-in for a real one.

XR Blocks is a Google research project, not a product with a support contract. The version numbers tell you exactly what you're getting into: v0.11.0 in March, v0.19.0 in August — nine releases in five months, all pre-1.0. API churn at that pace is not a risk, it's a certainty. If you build production work on it, pin your version and expect to do migration work.

It's also worth being clear about what automated testing in XR can and can't do. It can catch a broken teleport, a raycast that stopped hitting its target, a UI panel that regressed off-screen, a gesture handler that silently stopped firing. It cannot tell you whether your experience feels good, whether the scale is right, or whether the locomotion makes people queasy. Comfort and presence are still human judgments, and no simulated embodied user is going to take the headset off because something felt wrong. What this buys you is that the human playtest can be spent on the questions only a human can answer.

Sources & further reading: XR Blocks on GitHub · XR Blocks releases · Google Research — Vibe Coding XR · arXiv — XR Blocks: Accelerating Human-Centered AI + XR Innovation · MDN — WebXR Device API