Loading…
Loading…
Dataset 20261011-0550-f160c57 · measured 2026-10-11 05:50 UTC
Conformance · method
A reftest is one test: two files rendered by this engine and compared as pixels, within the allowance the test declares in <meta name="fuzzy">. It passes or it fails.
A scripted file is counted by its subtests, the same way wpt.fyi counts them. Two kinds of scripted file can be judged without a script runtime. A layout test states its expectation in the markup — data-expected-width, data-offset-x— and the harness reports one subtest per element, so that is how it is counted here: an element passes when every attribute on it matches. The individual attributes are on the file’s own page. A parsing test states its expectation as a literal assertion, this value must be accepted and this one must be rejected, and every assertion is one subtest.
Pass rate is passes over passes plus failures: 54,743 / 57,968 = 94.44%. Ties of the measuring instrument and rel=mismatch pairs that matched count as failures.
A reftest. css/css-flexbox/abspos/abspos-autopos-htb-ltr.html renders the test and its reference, the rasters match within the test’s own allowance, and it counts as one test and one subtest, both passing.
A layout test. css/css-flexbox/abspos/flex-abspos-staticpos-align-self-rtl-003.html checks 24 elements, with 48 data-expected-* attributes between them. Ten of those elements have an attribute that does not match the engine’s own layout, so the file counts as 24 subtests — 14 passing — and as one failing test. Which ten, and which attribute on each, is on the file’s page.
A parsing test. css/css-values/calc-size/calc-size-parsing.html makes 94 assertions about what the parser must accept and reject. Eighty-four hold. One that does not: max-height: calc-size(auto, size) must be rejected, and this engine accepts it. The file is 94 subtests, 84 passing, and one failing test.
A computed-value test. css/css-scroll-snap/parsing/scroll-padding-computed.html sets a declaration and reads back what getComputedStyle must return. It makes 40 such calls and 24 of them return what CSSOM requires. One that does not: scroll-padding-top: calc(10px - 0.5em) must compute to 0px, and this engine returns -10px. The file is 40 subtests, 24 passing, and one failing test.
A third figure sits between the two above, and its definition is Servo’s: a passing test with no subtests gets a score of 1 while a test with subtests gets a score between 0 and 1 representing the fraction of passing subtests; the figure is the mean over tests. A file where nine of ten assertions hold counts as 0.9 rather than as a failure or as ten units of weight. Blitz publishes the same quantity as its Interop Score.
449 files state their expectations only from script, and ReflowPDF has no JavaScript runtime: it can neither pass nor fail them. 990 are skipped by the interop rule below. 30 need the network, scrolling or a user style sheet. 19 have a reference file that is broken upstream.
Counting every file we cannot run as a failure, tests are 27,584 of 30,799 — 89.56%. There is no such figure for subtests: how many assertions a file we cannot run would have reported is not known.
Every excluded test, by nameA test we pass is judged. Always: passing a test the browsers fail is evidence, and dropping it would erase that.
A test we fail is skipped only when all three hold at once: fewer than three of the four major engines pass it, its specification is below Candidate Recommendation, and it is not from a printing module (css-page, css-gcpm, css-break, css-multicol, css-paged-media). The reason is written as interop:N/4 into a dated file under version control, not as a condition in the runner.
The symmetric version of this rule — skip by interop whether we pass or fail — was written first and rejected by measurement: it removes hundreds of tests we pass in modules where the browsers do not, which is the part of the corpus a print engine is built for.
Whole feature families used to be dropped on the premise that nobody shipped them. For anchor positioning that premise was false — most of those tests pass in all four engines — and removing the exclusion took the published rate down.
A difference can belong to the rasteriser rather than to the layout, and the gate decides that with a second measurement rather than a threshold.
aa-tie-288— both files are rendered again at 288 dpi, three times the 96 dpi the verdict is taken at, and the two rasters must come out identical byte for byte. The test’s own fuzzy allowance is not applied at that resolution: the comparison is raw, because the allowance is what the tie is trying to see past.
cairo-tie — offered only when the pair differs by at most 1200 pixels. A second, independent rasteriser draws both PDFs and must produce the same bytes for each page. If it does, our PDF equals the reference and the disagreement belongs to the first rasteriser; real geometry would move both.
This run has 54 cairo-tie and 32 aa-tie-288. Both kinds count as failures.
A tie can only excuse a defect that does not depend on either ruler. When the error is upstream — in parsing, in a computed value, in the geometry handed to the painter — both rasterisers are fed the same wrong input, and both draw the same wrong picture. Their agreement then says that the two rulers agree with each other, which was never in question. It says nothing about whether the picture is right. This is not a rare case to watch for: the rulers measure the output, so anything decided before the output is invisible to both of them, by construction.
An excused test is counted as a failure in the figures above — the excuse does not turn it green, and the published rate carries it as a loss. So nothing green here is secretly red. What the excuse changes is the regression check: a test that moves from a pass into an excuse is not reported as a regression. The figure falls honestly and the run stays green, so nothing stops the change that made it fall — the cause is left in the engine, and only re-reading the difference finds it. That is the harm, and it is the whole of it.
It has walked out along it. A gradient reftest matched its reference exactly; a change to the engine turned it into a ninety-nine pixel difference; the difference was excused as a tie, and the run reported no regression at all. The cause was a parser holding an angle in single precision where the test declares eight decimal places, so the gradient turned by a thousandth of a degree. It was found by re-reading the difference rather than by the verdict, and the fix is in the dataset this page is built from.
That path is now closed by a rule, and the rule does not measure anything. A test that once matched its reference exactly has proved that an exact match is reachable on it by this engine. If such a test later leaves the exact match for an excuse, something changed, and “this is within the noise of the instrument” stops being an argument: yesterday’s zero is the proof that the noise did not trouble it. Moving from an exact match into an excuse counts as a regression, whatever the size of the difference and whatever the second rasteriser says. An excuse stays available where it was meant to apply: to tests that never produced an exact match to begin with.
With that rule in place the excused list has been read through. Every test in it is a test that never matched exactly — had any of them matched, it would have been counted green rather than excused — so none of them hides a regression of this kind. One remains an open question rather than a debt.
11 tests pass because the reference degrades the same way the test does: the engine lacks the feature, both sides lose it, the pair matches. The match says nothing, so they are counted as failures by name. 40 more are rel=mismatch pairs that rendered the same, which is the failure the test was written to catch.
When a fix makes one of these pairs finally differ, the published figure goes down. That is accepted: a false green is worse than a red.
The gate runs four tiers against the same corpus, each with its own baseline file. The headline figures on the results page are their sum.
Reftests render the test and its reference with this engine, rasterise both PDFs at 96 dpi and compare them pixel by pixel; one file is one verdict. Layout assertions read the expectation out of the markup — data-expected-width, data-offset-x and the rest of what check-layout-th.js would ask a browser — and put it to the engine’s own layout, one attribute at a time. Parsing acceptance hands the declaration parser and the cascade what parsing-testcommon.js declares valid or invalid, one assertion at a time. Computed values are the fourth tier: test_computed_value from computed-testcommon.js sets a declaration, reads it back the way getComputedStyle would and compares the serialization the specification requires (CSSOM §6.7.2), one subtest per call.
That last tier judges what it can and says what it cannot. A property whose serializer this engine does not write yet is set aside with that reason — 382 subtests, and so are values that CSSOM resolves to the used value rather than the computed one (§9). Neither group is counted as a pass or as a failure; both are named on this page and in the dataset.
Each tier keeps its own baseline, and a change is accepted only when no tier goes backwards. The numbers above are the four tiers added in one denominator, because wpt.fyi adds a browser’s reftests and subtests the same way.
The dataset behind every figure is served as plain JSON from this domain.
This page was built from dataset 20261011-0550-f160c57b-d2601fe3, engine 146bce3e, and it is rebuilt after every accepted run. Back to the results.