Triage evals
Scored against 150 Astro issues that a maintainer labeled by hand. I skipped anything the bot labeled, and kept these out of the prompt. Back to the inbox
| claude-haiku-5-5 prompt v1, 2026-10-26 | claude-haiku-5-5 prompt v1, thinking off, 2026-10-26 | claude-sonnet-5-5 prompt v1, 2026-10-26 | Baseline | |
|---|---|---|---|---|
| Priority, exact | 42.7% | 40.7% | 52.7% | 36.0% always P3 |
| Priority, within one level | 85.3% | 82.7% | 91.3% | |
| Packages, precision / recall | 80.0% / 93.6% on 44 a maintainer labeled | 75.4% / 97.9% on 44 a maintainer labeled | 70.8% / 97.9% on 44 a maintainer labeled | |
| Needs reproduction, accuracy | 84.0% | 82.0% | 88.0% | 92.7% always no |
| Needs reproduction, precision / recall | 19.0% / 36.4% | 16.7% / 36.4% | 29.4% / 45.5% | |
| High confidence, exact | 42.9% of 7 | 33.3% of 3 | 75.0% of 4 | |
| Medium confidence, exact | 44.0% of 134 | 40.1% of 137 | 50.7% of 136 | |
| Low confidence, exact | 22.2% of 9 | 50.0% of 10 | 70.0% of 10 | |
| Failed | 0 of 150 | 0 of 150 | 0 of 150 | |
| Quote not in the issue | 1 | 1 | 0 | |
| Planted injection flagged | Yes | Yes | Yes | |
| Cost per issue | $0.00054 | $0.00057 | $0.00835 | |
| Response time, typical / slow | 3.9s / 5.9s | 4.2s / 6.3s | 3.8s / 7.4s |
These runs went through Claude Code in headless mode with its own prompt, tools and settings switched off. The model saw the same prompt as the live endpoint, with the answer's schema written into it instead of passed as a tool. Cost is what the same tokens would cost on the API at list prices. Response time is the model's own time, without the few seconds it takes to start each process.
Priority, what it confused (claude-haiku-5-5)
| Maintainers said | AI said P1 | AI said P2 | AI said P3 | AI said P4 | AI said P5 |
|---|---|---|---|---|---|
| P1 | 0 | 3 | 0 | 0 | 0 |
| P2 | 1 | 9 | 10 | 17 | 0 |
| P3 | 2 | 5 | 24 | 23 | 0 |
| P4 | 0 | 1 | 19 | 31 | 0 |
| P5 | 0 | 0 | 2 | 3 | 0 |
Where it was wrong (86)
- #16863 Astro 6.3.6+ floods SSR production logs with "Internal Warning: route cache overwritten" (regression from #16776) maintainers P5, AI P4
- #16817 Maximum call stack size exceeded from VSCode extension maintainers P4, AI P3
- #16766 Bug: Vite optimizeDeps race condition causes React not found in dev mode maintainers P3, AI P4
- #16644 Prerendered images using the generic image endpoints fail in 6.3.0 maintainers P5, AI P4
- #16621 Markdown images missing alt attributes maintainers P3, AI P4
- #16612 BUG: language-server crashes on attach maintainers P4, AI P3
- #16561 Deleted content collection entry persists in collection data maintainers P4, AI P3
- #16520 Astro 6 hybrid build emits dynamic-import URLs with chunk hashes that don't match emitted chunks (runtime 404 on `/_astro/*.js`) maintainers P4, AI P3
- #16204 Unable to load UnoCSS style after updating astro from 6.0.8 to 6.1.3 maintainers P3, AI P4
- #16125 @astrojs/check dependent on yaml CVE-2026-33532 maintainers P2, AI P4
- #16078 VSCode Go To References doesn't work with astro locals in astro files maintainers P3, AI P2
- #16053 Experimental queuedRendering breaks Sveltia CMS page (raw HTML rendered as text) maintainers P3, AI P4
- #16010 Dev server hangs on first request with Cloudflare adapter: circular deadlock in ensureModulesLoaded() (regression from #15988) maintainers P5, AI P4
- #15821 Build error with svelte and lucide svelte on Astro 6 maintainers P2, AI P4
- #15650 v6 + Cloudflare deployment fails on `static` output maintainers P4, AI P3
- #15520 [beta] Prefetch placeholders not replaced when used with ClientRouter (dep optimization bypasses transform) maintainers P2, AI P3
- #15505 Images get emitted to `_astro`, even though they're not used in the output maintainers P2, AI P3
- #15465 <Font> component's inline styles are not preserved during ClientRouter head swap, causing font flash on navigation maintainers P2, AI P4
- #15439 Astro VS Code extension loses syntax highlighting / IntelliSense for <script> when conditionally rendered maintainers P3, AI P2
- #15420 create-astro: --add input is interpreted as shell syntax maintainers P5, AI P3
- #15410 Beta versions should have newer versions than stable ones maintainers P1, AI P2
- #15389 Custom web components wrapped in <template> prevent other instances from initialising maintainers P4, AI P3
- #15384 HMR of CSS doesn't work for previously non-styled elements maintainers P4, AI P3
- #15341 React: Invalid hook call maintainers P3, AI P4
- #15318 OOM with CJK fonts after v5.16.13 when using Fonts API maintainers P3, AI P4
- #15310 v6-beta: Error when running vitest with Cloudflare: "Avoid setting `resolve.external` in your Cloudflare Worker environments" maintainers P4, AI P3
- #15303 npm audit reports dev-only vulnerabilities via @astrojs/check and @astrojs/cloudflare that require --force (breaking changes) maintainers P1, AI P2
- #15296 v6 + Cloudflare + Tailwind v4 styles broken maintainers P4, AI P3
- #15285 `renderMarkdown` doesn't do what we all think it does maintainers P2, AI P4
- #15284 Astro v6 with cloudflare adapter does not work with `<Code ...>` components maintainers P4, AI P3
- #15273 Dev Server Serves Markdown at Project Root maintainers P4, AI P3
- #15271 Incorrect <Code /> highlighting for TSX-embedded Vue maintainers P2, AI P3
- #15248 MDX plugin strips Astro component <script> when component is nested inside HTML (issue present @astrojs/mdx ≥ 4.1.0) maintainers P4, AI P3
- #15236 Favicons not appearing in Google Search results (even for astro.build) maintainers P1, AI P2
- #15223 No hot reload when altering markdown file contents in Astro 6 beta. maintainers P4, AI P3
- #15165 APIRoute vs public files follow different precedence in dev and build maintainers P4, AI P3
- #15162 Phantom "]" symbol at the beginning of the body tag maintainers P2, AI P3
- #15151 netlify dev with experimental fonts API 404's on fonts maintainers P2, AI P4
- #15136 `[suspicious-nullish-coalescing]` warning from dead `?? []` fallback in virtual module codegen maintainers P2, AI P1
- #15115 Node 24 on Vercel not fully supported maintainers P2, AI P4
- #15102 experimentalReactChildren flag doesn't map node.class and children keys on client maintainers P3, AI P4
- #15100 `/_image` endpoint returns 500 Internal Server Error in standalone mode on production server maintainers P3, AI P4
- #15095 Query params trigger 404 error in dev maintainers P4, AI P3
- #15087 `/_image` returns 500 in Node standalone when dist is moved (absolute `file:///` paths baked into output) maintainers P4, AI P3
- #15086 Astro View Transition handles link with hash incorrectly maintainers P2, AI P4
- #15076 Cookies set via `AstroCookies.set()` are missing in conjunction with `Headers.append("set-cookie")` maintainers P2, AI P4
- #15067 getImage inferSize option does not cache underlying assets, causing extra traffic usage due to asset re-downloads maintainers P2, AI P3
- #15062 Visual Studio Code Extension: "Find All References" doesn't find references when initiating it from a `.ts` file maintainers P3, AI P4
- #15042 Images cannot be loaded in dev mode in a port forwarding environment maintainers P3, AI P4
- #15003 Warning on `astro build` maintainers P3, AI P1
- #14970 Astro Actions accepting "form" seem broken maintainers P2, AI P4
- #14962 svgs not displaying correctly with passthroughImageService maintainers P3, AI P4
- #14957 astro:assets image service returns 500 in dev when @assets points to ext. dir maintainers P3, AI P4
- #14942 Image generation: Etag header is ignored maintainers P3, AI P4
- #14933 Astro Fonts + Astro DB causing build errors when building for Cloudflare maintainers P2, AI P3
- #14869 SCSS file changes trigger full page reload instead of HMR maintainers P3, AI P4
- #14864 Exported variable has or is using name 'inferSymbol' from external module but cannot be named. maintainers P3, AI P4
- #14846 Duplicate slug property across MD files causes the page to be overwritten without warning. maintainers P2, AI P3
- #14838 [astrojs/vercel] Vercel adapter can’t resolve Node builtins while building middleware maintainers P4, AI P3
- #14836 `dist` folder with SSR is not redistributable maintainers P2, AI P4
- #14819 Fonts API loads non-variable fonts too when specifying variable font maintainers P2, AI P3
- #14778 @astrojs/upgrade helper package fails on tailwind v4 projects when using bun x maintainers P3, AI P2
- #14766 Regression due to #14590: images with absolute paths in JSON cause (confusing) errors maintainers P2, AI P4
- #14752 Warning on `astro build` maintainers P3, AI P1
- #14721 passthroughImageService() does not prevent WebP conversion for Image component in Astro 5.15.2 maintainers P3, AI P4
- #14692 picocolors is being bundled into the browser maintainers P2, AI P4
- #14619 Export jsx-dev-runtime as an alias for jsx-runtime maintainers P2, AI P3
- #14615 Error hydrating: picocolors does not provide an export named 'default' maintainers P3, AI P4
- #14611 [`@astrojs/mdx`]: `optimize` config option causes passed `components` to be ignored maintainers P2, AI P4
- #14585 CSS url is replaced to include base path during build but not during dev maintainers P4, AI P3
- #14642 🐛 BUG: Import intellisense/autocomplete recognizes a new Astro component, but only once maintainers P4, AI P2
- #14572 NPM cannot resolve Astro Actions virtual module maintainers P5, AI P3
- #14569 `astro preview` won't quit maintainers P2, AI P3
- #14557 Responsive images generate original resolution images despite specifying `widths` maintainers P3, AI P4
- #14553 Boolean Attributes Treated Inconsistently maintainers P2, AI P4
- #14544 @astrojs/check pulls volar dependency which pulls from Github maintainers P3, AI P2
- #14540 astro/cloudflare doesn't use environment vars specified in wrangler config or adhere to CLOUDFLARE_ENV when running/building maintainers P2, AI P4
- #14531 `Could not find Sharp` error when building in docker image with Astro 5.13.7 or higher maintainers P2, AI P4
- #14515 inferRemoteSize implementation leads to huge traffic / requests amount in SSR maintainers P3, AI P4
- #14513 TypeError: app.getAllowedDomains is not a function on Node adapter maintainers P3, AI P4
- #14511 Cloudflare adapter returning [object Object] maintainers P2, AI P4
- #14508 Experimental.async error when using Svelte v5.39 maintainers P3, AI P2
- #14500 Image wrongly cached maintainers P3, AI P4
- #14495 CSP style hash not inserting in staging environment maintainers P3, AI P4
- #14464 Astro.rewrite wrongly renders 404s when rewriting to a URL with non-ASCII characters maintainers P3, AI P4
- #14463 Failing to run `create-astro` (command failed) maintainers P4, AI P3
What people decided
Loading.
Drafted replies, reviewed by hand
Not reviewed yet.
The models may have seen some of these issues in training, which would make the scores look better than they are. Using the newest issues keeps that to a minimum. The issues were opened in 2025 and early 2026 and fetched 2026-10-25.