Jev Computer Use
Bench

Detailed results

Simple tests - detailed data

The simple-test score is the clean 42-task comparison. Browser-tool capability checks are described below in prose and are not system scores.

Real-world browser - detailed data

The 10-task row is the primary real-world result. The public row is a separate repeated discovery set: three Flights, three OpenTable, and three Resy read-only flows. These stop before booking, payment, or reservation submission. The unlisted Jev OpenTable clip is an EdgeSuite CDN access block before the app UI loads; the retained Jev Resy clip reaches a named reservation-details stop point.

Computer use - detailed data

Native tasks: Spotify search/play, Pages draft, TextEdit exact text, Keynote two-slide edit, and Numbers six-cell edit. The old Jev Keynote and Numbers captures were excluded after review because they were not paired with their traces. Synchronized reruns reproduced the failures: Keynote ended with blank placeholders and Numbers ended with an empty table. The matched hybrid now covers all five tasks: Jev succeeds directly only on the clean TextEdit task, and Luna repairs the native misses.

Accessibility-driver control

The separate accessibility driver reads the macOS Accessibility tree plus a screenshot, then can click an element, type to a target process, or set a value. The four-task control passed Pages and failed Keynote, Numbers, and Spotify: 1 / 4. These are direct driver controls with no Jev or Luna planning, so they stay out of the leaderboard; the computer-use summary table includes the row only as a clearly marked control result.

Browser tools check

The deterministic 14/14 probe and Jev replay are not agent scores, so they are kept out of the four-system tables. No page JavaScript execution was used, matching normal ChatGPT Chrome use. The connected-Chrome bridge was the scored browser setup; the in-app browser was not used for the headline results.

What Luna originally missed: In the initial ten-task real-world browser run, the non-passes were OpenTable (results page, not the named reservation-details stop point), private Drive (wrong search result and a hang beyond 20 minutes), and Figma (Google device-passkey stop point). The authenticated Figma rerun then succeeded for Luna and Hybrid, without a separate meter or retained recording. The 14/14 JS probe and 42/42 Jev-plus-JS replay are deterministic upper bounds, not Luna runs.

Upload checks

Three follow-up Luna runs through the normal Chrome bridge exposed the native macOS chooser and visibly reached SUCCESS on all three. We repeated the same three-case sequence for Hybrid: Jev stopped at the unsupported file input, then connected-Chrome native-chooser recovery reached SUCCESS on all three. The connected-Chrome upload slice is therefore 3 / 3 for both Luna and Hybrid. Combined with each system's original 39 / 39 non-upload passes, the headline tables report 42 / 42. Recovery time and cost were not separately metered, so the Hybrid time/cost values are known-run estimates rather than full end-to-end totals. The Luna traces are here, attempt 2, and attempt 3; the Hybrid trace is here.

Simple test details

This is the clean 42-task battery. Each task has an exact end-state check. The old scripted Playwright helper is excluded because it was not a live Luna run.

Luna cost is a token-based estimate, not a subscription invoice. Jev cost is the tested input-token estimate. No separate local browser-tool fee is included.

Measured browser capability breakdown

Computer-use task details

These runs used the real macOS desktop. TextEdit, Keynote, and Numbers are native Apple apps with no HTML browser DOM in the task surface. Pages is a native-style AppKit editor. Spotify is included because it is a common desktop app, but it is an Electron/web surface and is explicitly labeled that way. Figma remains in the authenticated browser lane because its desktop shell is also a web/Electron-style surface, not a clean pure-native control.

Native desktop scorecard

Native Luna costs use a calibrated action-count estimate: the fresh 42-task run's $0.41890 estimate divided by 248 observed actions, or about $0.00169 per action. The new Keynote recovery used 11 actions and the new Numbers recovery used 3. The native clips did not expose a separate invoice. Jev native costs use input tokens at $0.042 / million. These are comparison estimates only.

Why Jev is the most expensive per success here: it paid for all five attempts but produced only one verified success, so its total attempt cost is divided by 1. Hybrid is also above Luna in this lane because it paid for a Jev-then-Luna attempt plus verification/recovery work. This does not mean a single Jev action costs more than Luna; it is the cost of failed attempts spread over very few completions.

Real-world browser details

These are deliberately harder than the fixture battery: booking stop points, authenticated apps, rich editing, and multi-step navigation. A pass requires the meaningful safe stop point, not a search page. We did not submit purchases, reservations, sends, shares, or payments.

Earlier retained workflow evidence

This table preserves the earlier per-flow evidence trail. The 10-task scorecard immediately below is the primary real-world comparison.

Real-world browser scorecard - 10-task denominator

Score note: The primary 10-task score is Jev 0/10, Luna 8/10, and Hybrid 8/10. Luna and Hybrid pass United, Notion, Figma, Spotify, Mail, Calendar, SauceDemo, and TodoMVC; OpenTable is results-only and Drive does not find the private target. Figma is the authenticated rerun and is not separately metered. Hybrid remains a staged Jev-then-Luna recovery record, not one interleaved live turn.

10-task per-workflow comparison

This is the direct “who passed which task” view. Jev is 0/10 on the exact postconditions. Hybrid cells use the newly recorded ten-task recovery replay after the retained Jev misses; they are not presented as one live interleaved turn. Aggregate time and cost are reported above because the recording is one continuous browser run.

Recorded recovery run ledger

Per-task timings and Luna action counts come from the run ledger; the aggregate is 441.862 seconds and 70 actions. Some Success labels are safe stop-point passes, not external side effects.

All retained video evidence is listed once in the Recorded runs table above.

Additional public computer-use lane - a separate real-world result

This is the missing context behind “Luna did more than text editing.” It is a different, fully repeated public lane: three fresh starts each for Google Flights, OpenTable, and Resy, stopping at read-only search/results states. It is not the exact ten-task denominator above, and it does not claim a booking.

Costs in this table are totals for all nine attempts. The OpenTable Jev rows were transport-blocked in isolated contexts while visible Luna Chrome already had public cards. That row is an environment difference, not clean model evidence. The Luna cost is a token-equivalent estimate, not a product invoice or local tool fee.

One-run public smoke checks

These add non-text computer use: login, cart construction, secure-area navigation, todo completion/filtering, and restaurant discovery. No password, checkout, reservation, or payment side effect was submitted.

Additional public research tasks

These are useful Jev-only real-site checks, but they are not apples-to-apples comparisons and are excluded from the headline rates. They show that Jev is not limited to clicking buttons: it completed two-hop navigation and issue-page research when the page exposed a short indexed path.

Agent-native apps (authenticated)

Using the signed-in native Chrome profile, Luna completed one safe checkpoint in each requested app. Jev ran a separate exact eight-task runner. The denominators are different, so they are reported separately rather than pooled.

Mail stopped at an unsent draft, Calendar stopped before saving an event or invite, and Design/Slides/Content stopped before share/publish. Those are deliberate safety boundaries, not completed external side effects.

Comparison details

This keeps the four benchmark names simple while still explaining what was actually measured.

Bottom line

On 42 scripted browser tasks with an exact pass/fail check, Jev's two fast browser variants each land 18 of 42 (42.9%) while Luna and the Jev-then-Luna Hybrid both land 42 of 42. Jev is dramatically faster and cheaper only on tasks its structured control view can already see cleanly. On the harder real-world browser set, Jev Browser passes 0 of 10 and Jev Ultrafast 1 of 9 on a separate public lane, against 8 of 10 for both Luna and Hybrid after the authenticated Figma rerun; on five native macOS desktop tasks Jev passes 1 of 5 against 5 of 5 for Luna and Hybrid. Jev is genuinely good at short, well-exposed browser controls, then drops sharply on multi-step forms, rich editors, uploads, and native apps. Luna's slower visual approach, and Hybrid's verified recovery, are stronger on those tasks.

What the tests were

• Simple tests - 42 scripted browser fixtures (downloads, filters, tables, selects, search), each with an exact end-state check.
• Real-world browser - 10 longer workflows on live sites (flights, dining, authenticated docs, and mail) requiring multi-step navigation or sign-in.
• Computer use - 5 macOS desktop tasks in TextEdit, Pages, Keynote, Numbers, and Spotify, scored by verifying the requested content on screen.
• Agent-Native apps - a separate authenticated lane covering Mail, Calendar, Design, Slides, and Content, scored at safe checkpoints and kept out of Overall.

How the approaches work

• Browser path: the wrapper exposes normalized page content plus structured controls and text to the Jev decision model. The retained trace is Markdown-like content rather than a byte-for-byte raw HTML dump; no page JavaScript execution was used.
• Native path used here: Jev's desktop adapter collects macOS Accessibility elements plus Vision OCR candidates. Jev selects a typed action such as click, set value, or type text from that structured snapshot. It is not receiving a raw HTML document from TextEdit or Pages.
• OCR: Optical Character Recognition turns pixels in a screenshot into guessed words and bounding boxes. It can suggest where “Save” or “Karma Police” appears, but it does not prove that the target is editable, clickable, focused, or successfully changed.
• Why native is harder: browser controls usually expose stable semantic roles, labels, values, and DOM state. Native apps vary: TextEdit exposed a clean AXTextArea, while rich editors, spreadsheets, canvases, and some media apps can expose sparse or stale semantics. The driver then leans more on OCR, coordinates, keyboard focus, and independent verification.
• Luna path: Luna operated the visible desktop/browser surface and was required to verify the end state from the UI.
• Text: Jev can emit literal text when the target and focus are clean. The failure mode is target/focus/semantic editing and verification. Pages is the clearest example: Jev emitted two type actions, reported completion, and still left the Body empty. TextEdit passed because the native text area was a clean target.

Separate driver control: TryCua Cua Driver independently set the same kind of multi-line TextEdit value through a fresh Accessibility snapshot and read it back twice. That control passed, with no model and no leaderboard score: recording · verified screenshot. It proves the native driver can act when the app exposes a clean target; it does not prove that Jev can choose the target or plan the workflow.

Text note: the basic fixtures used a scripted text helper, so those rows measure browser decisions rather than prose generation. The Ultrafast wrapper used a small text model to supply the literal string after Jev selected the target and operation. That helper does not solve focus, caret movement, rich-text editing, keyboard submission, or verification. Luna generated and entered text through its own visual loop. In the native Jev run, the task supplied literal strings to a typed action policy; there was no general-purpose writing model hidden behind the Jev score.

Related implementations and claims: Third Hand, TipTour macOS, and Cua. The benchmark treats a demo's driver, OCR, DOM extraction, keyboard helper, and Jev model as separate components rather than attributing every capability to Jev alone.

<!-- CSS is not part of this representation -->
<form aria-label="Search">
  <label for="destination">Where to?</label>
  <input id="destination" value="San Francisco">
  <button type="submit">Search</button>
</form>

{
  "content": "[Search](/search)\n\n# Results\n\n...",
  "controls": [
    {
      "role": "textbox",
      "name": "Where to?",
      "value": "San Francisco",
      "actions": ["setValue"]
    },
    {
      "role": "button",
      "name": "Search",
      "actions": ["click"]
    }
  ]
}

TextEdit
└─ AXWindow "Notes.rtf"
   ├─ AXToolbar
   │  └─ AXButton "New document"
   └─ AXTextArea "Document body"
      ├─ value: "Line one\nLine two"
      ├─ focused: true
      └─ action: AXSetValue

{
  "id": "node-17",
  "role": "AXTextArea",
  "title": "Document body",
  "value": "Line one\nLine two",
  "enabled": true,
  "focused": true,
  "frame": { "x": 120, "y": 180 },
  "actions": ["AXSetValue"]
}

Desktop screenshot
┌─────────────────────────────────────┐
│  [Save]     Notes.rtf                │
│                                     │
│  Line one                            │
│  Line two                            │
│                         [Cancel]    │
└─────────────────────────────────────┘
        pixels + visual context

{
  "image": "screen pixels",
  "visible_text": [
    { "text": "Save", "box": [42, 18, 55, 24] },
    { "text": "Line one", "box": [42, 112, 76, 20] }
  ],
  "visual_context": "document contains two visible lines",
  "next_action": "verify the final state"
}Conceptual view: this is not a claim about Luna's private internal wire format. The point is that Luna reasons from the visible surface, while Jev's structured view depends on what the page or app exposes.

Where Jev fails and why

• Google Flights (real-world browser): typed a natural-language destination string into the field instead of a matched airport, the site returned "No matching locations found," and Jev stopped there.
• Spotify (real-world browser): reached the public account sign-up wall and stopped; the browser lane has no logged-in session to get past it.
• OpenTable (real-world browser): blocked by the site's EdgeSuite CDN before the app UI ever loaded.
• Notion (real-world browser): no verified edit - the run did not produce a confirmable change on the page.
• File uploads (simple tests): stops at the file-input step every time; the native macOS chooser sits outside Jev's structured control view, and only Hybrid's hand-off to Luna gets past it.
• Pages (computer use): reported SUBTASK_COMPLETE while the document body stayed empty - a false completion, not a crash.
• Keynote and Numbers (computer use): ended with blank slide placeholders and an empty spreadsheet table - the file was created but the requested content was never entered or verified.
• Spotify (computer use): played the wrong Radiohead track.
• TextEdit (computer use) is the exception that proves the pattern: it passed because the native text area is a clean accessibility target Jev can act on directly, unlike the richer editors above.

Where Luna fails and why

• Google Drive (real-world browser): search returned the wrong file and the run hung past 20 minutes without finding the private target - the slowest, costliest miss in the set.
• Figma (real-world browser): the original run stopped at Google's device-passkey screen, but the authenticated rerun created, edited, renamed, and verified the design successfully for both Luna and Hybrid.
• OpenTable (real-world browser, 10-task lane): reached the results page but not the named reservation-details stop point, so it is scored a miss despite getting close.
• Why Hybrid's cost moves in opposite directions by lane: Hybrid tries Jev before calling in Luna to recover a verified miss, so it lands cheaper than Luna on Simple tests ($0.0069 vs $0.0107 per success) where Jev often succeeds outright - but more expensive than Luna on Computer use ($0.0126 vs $0.0091 per success), because the failed attempt is paid for and then added on top of Luna's own cost.

Takeaways

• Use Jev for short, exposed controls, navigation, search-and-type, simple selects, and high-volume low-risk decisions.
• Expect the cliff around drag/drop, uploads, multi-field forms, rich text, spreadsheets, iframes/shadow DOM, and native rich editors where an action can look successful without changing the real content.
• Use Luna when visual context, recovery, text editing, or long-horizon state matters more than raw latency.
• Use the hybrid with an explicit postcondition verifier. It is not magic: on native tasks where Jev fails, trying Jev first adds time and cost before Luna recovers. Its benefit is cheap wins plus recovery, not “free” parity.

Limitations

• Cost estimates, not invoices: every dollar figure is a token-priced comparison estimate, not a metered bill.
• Small sample sizes: 42 simple, 10 real-world, and 5 computer-use tasks - a few flipped outcomes would move the percentages.
• Jev Ultrafast's denominator is 51, not 57: it skipped the computer-use lane, so its Overall rate isn't directly comparable to Luna and Hybrid's 57.
• Curated, not random: the real-world and desktop task sets were chosen to cover known failure modes, not sampled at random.
• Harness scope: Luna and Hybrid drive a connected-Chrome bridge with no page JavaScript execution; a few Jev-side JS probes are reported separately.

Reproducibility and provenance

• Controlled fixture data: summary.json, luna-cua-fresh-start.json, and hybrid-fresh-start.json.
• Authenticated app evidence: authenticated-results.json and videos/luna-native-chrome/.
• Extended task definition: ../jev-real-world/v2/extended_tasks.json.
• Native Jev runner: ../scripts/run_arc_cua_native.py; native video overlays: ../scripts/make_native_overlays.sh.

Generated from the retained JSON traces, postcondition checks, and inspected video frames. Raw clips are not replaced by the report.