Marks and Compare

A mark is a named moment, and compare puts the window before a mark beside the window after it.

Time selectors

Every agent-facing endpoint takes since and until instead of from and to, because an agent knows a moment as a mark name or a duration, such as before, start or 5m, rather than as epoch milliseconds. A selector takes one of six forms.

Form Example Meaning

Duration

30s, 5m, 2h, 1d

That long before until for since, or before now for until; in compare, before until for after and before after for before (Compare).

Date-time

2026-10-08T05:50:00+09:00, 2026-10-08T05:50:00

The instant, in ISO-8601, with an offset or Z, or without one in the time zone of the Spider Sense that resolves it.

Epoch milliseconds

1758000000000

The instant, 13 or more digits.

A mark name

before, after-fix

The newest mark with that name.

start

start

The newest automatic start mark, of service when one is given.

now

now

Now, and the default for until.

The default since is 15m, and the default until is now. since=start therefore means "since the application was last restarted", which is what an agent that just rebuilt the application wants. A heading prints its window’s start as a date-time, such as 2026-10-08T05:50:00+09:00, with the milliseconds when the start has any, such as 2026-10-08T05:50:00.846+09:00, so that time pasted into --since asks for the same window again.

A since that resolves to a moment after until is answered with 400 over HTTP and is a usage error in the CLI. So is a value of numbers and letters alone that begins with a number but is no duration, such as 5min, 1w, 30sec, the compound 1h30m or 300: it is a mistyped duration, never a mark name, and the answer says Not a duration: 5min with the forms a duration takes. A separator ends the number, so 2026-10-08 or 2nd-try is read as a mark name. A mark name that matches no mark is answered with a 404 that names it over HTTP and exits 4 in the CLI. The from and to parameters of the UI endpoints still work everywhere, and they win when both pairs are given.

Time zone

The times a text rendering prints, and a date-time a selector reads without an offset, are in one time zone: the JVM’s default as it was when Spider Sense started in it, which is the machine’s unless -Duser.timezone names another. Spider Sense fixes the zone as it starts, which under -javaagent is before the application’s main runs, so an application that changes its own default later, with TimeZone.setDefault, moves neither the times an answer prints nor the instant --since=2026-10-08T05:50:00 means. The CLI that reads the file itself (The direct-file fallback) uses its own JVM’s zone, so an application started with -Duser.timezone=UTC on a machine at +09:00 answers in UTC over HTTP and in +09:00 from the file. A date-time with an offset or Z is the same instant everywhere.

Marks

A mark is a named moment: before, after-fix, v2. A mark is one row in the mark table (Storage), so every process that opens the database sees it, and the retention sweeper deletes it once it is older than the retention.

$ java -jar spider-sense.jar mark before
mark before at 2026-09-17T08:19:28.122+09:00

$ java -jar spider-sense.jar mark after --note="after the fix"
mark after at 2026-09-17T08:19:35.837+09:00 — after the fix

The name is required and matches [A-Za-z0-9._-]{1,64}. A name that a selector reads as something else is refused, because --since could never ask for the mark: a duration such as 5m, a mistyped one such as 5min or 1h30m, a number, a date-time, now, and start, which is the writer’s own (Automatic start marks). The refusal says why, as a 400 over HTTP and a usage error in the CLI:

$ java -jar spider-sense.jar mark 5m
spider-sense: A mark cannot be named 5m: --since and --until read 5m as a duration; choose another name

--note=<text> keeps a note with the moment, and --service=<name> ties the mark to one service. Over HTTP, you record a mark with POST /api/marks and the body { "name": "before", "note": "…", "service": "…", "at": … }, where everything but the name is optional and at defaults to now.

marks lists the newest 50, newest first, and GET /api/marks answers the same.

$ java -jar spider-sense.jar marks
# marks

| at | name | service | note |
| --- | --- | --- | --- |
| 2026-09-17T08:19:35.837+09:00 | after | — | after the fix |
| 2026-09-17T08:19:28.122+09:00 | before | — | — |
| 2026-09-17T08:19:16.727+09:00 | start | spring-orders | pid 178056 instance 5c1e9a52-3f0d-4b8e-a7c4-0e6b2d9f8a13 |
| 2026-09-17T08:16:42.507+09:00 | file-mode | — | — |
| 2026-09-17T08:15:38.928+09:00 | after | — | — |
| 2026-09-17T08:14:20.087+09:00 | before | — | — |
| 2026-09-17T08:14:06.284+09:00 | start | spring-orders | pid 170140 instance 9e2f47b0-61ac-4d35-8b0e-c3a5f1d7e264 |

Automatic start marks

Spider Sense records a mark by itself whenever a service restarts. When the writer sees a service from a process that no start mark of that service names yet, it inserts a mark named start with that service and the note pid <pid> instance <service.instance.id>, or pid <pid> for a sighting without a service.instance.id. Two processes that export one service name at once, an application and its tests or two instances of one application, therefore leave one start mark each, not one per export. A process is its process.pid and its service.instance.id, which the OpenTelemetry Java agent sets to a new UUID at every start of the JVM. The pid alone repeats: an application that is a container’s entrypoint is pid 1 at every start, and its restarts are seen by the instance id. A service that reports no service.instance.id is told apart by its pid alone, so a restart of it that reuses the pid is not seen as one. The mark is at the earliest span start, log record or metric point of that service in the flush that carries the new process id, never after that flush and never more than a minute before it, since an exporter batches for seconds and the run’s first requests arrive after they began. --since=start therefore means "since the application was last restarted", which is what an agent that just rebuilt the application wants, and nobody has to record a mark for it. The retention sweeper and DELETE /api/data spare each service’s newest start mark, so it lasts as long as the process does. The older of two processes sharing the name loses its mark to them, and is not marked again: the writer remembers each process whose mark it has seen stored, so --since=start does not jump to that process’s next export. A restart of the standalone Spider Sense they send to ends that memory: the process that exported last before it is still not marked again, since the service row keeps its process id and its service.instance.id, but another whose mark is gone is, at its next export. A service the sweeper has forgotten, for sending nothing for twice the retention, loses its start marks with its row (Retention), and its next export marks a start again.

A service that reports no process.pid gets no start mark. The Java agent always reports one, so every application under -javaagent:spider-sense.jar has a start mark from its first export on. A service sending from its own SDK reports one only when the SDK’s process resource detector is on, which Python, Node.js and Go covers. Until some service has a start mark, --since=start is a not-found that says so and suggests --since=15m, or mark before and --since=before. With --service, start is that service’s own start mark and never another service’s, so a service with none, such as one that reports no process.pid, gets the same not-found, naming the service.

The Mark button on the UI’s top bar records a mark the same way. The charts draw marks as vertical lines. Layout and Controls covers the button and the lines.

Compare

compare says, for each endpoint, query group and error group, whether it got better or worse between the window before a change and the window after it.

java -jar spider-sense.jar compare --before=<selector> --after=<selector> [--until=<selector>] [--service=<name>]

Over HTTP it is GET /api/compare?before=<selector>&after=<selector>&until=<selector>&service=&format=.

It takes no --since, because its windows are the two selectors. Before is the window [before, after) and after is the window [after, until), with until defaulting to now. The selectors resolve from the end: until first, then after, whose duration counts back from until, then before, whose duration counts back from after. So --before=10m --after=5m is the ten minutes before the last five, not ten minutes ago against five minutes ago, and its before window is twice as long as its after window. A span that starts exactly at after belongs to the after window, and one that starts exactly at until belongs to neither. A before that does not resolve to a moment earlier than after, or an after that does not resolve to one earlier than until, would be an empty window, so it is answered with 400 over HTTP and is a usage error in the CLI, as an inverted since is. The refusal names both selectors, the local times they resolved to, the service of a start mark, and what to do:

$ java -jar spider-sense.jar compare --before=onboarding-before --after=start
spider-sense: before=onboarding-before (2026-10-08T06:05:28+09:00) is not before after=start (the start of silk-bookstore, 2026-10-08T05:58:11+09:00), so the before window would be empty; restart the application so a new start mark is written, or record `mark after` once the change is in and use after=after

The usual use is two marks: mark before, exercise, change the code, mark after, exercise the same way, then compare --before=before --after=after.

The answer has four parts.

Totals

Requests, errors, p95 and Apdex, for each window. A window with no request has no p95 and no Apdex, so both read — rather than 0.0 ms and a score.

Endpoints

One row per endpoint, with calls, errors, p95, db/req and db ms/req for both windows. The JSON side of a row is { "calls", "errors", "p50Ms", "p95Ms", "maxMs", "dbCallsPerRequest", "dbMsPerRequest" }. db/req and db ms/req are the endpoint’s own database work, as a slow-endpoint finding counts it.

Queries

One row per query group, with calls, calls/req, p95 and total for both windows. callsPerRequest divides by the entry spans of the window, of the service when one is given, where an entry span is what Endpoint identity defines. A window with no entry span, such as a worker’s, whose jobs are never requests, has no calls per request: callsPerRequest is null and the text prints —.

Errors

One row per error group, with the occurrence count in each window. target names the outbound HTTP call a group’s occurrences mostly are, which is what says what failed when the group has a status for a type and no message (ErrorGroup).

Each part has a row for every endpoint, query group and error group of either window, never the top of a list, so a verdict never depends on where a group ranked.

The JSON has the same four parts:

{
  "before": { "from": …, "to": … }, "after": { "from": …, "to": … },
  "totals": { "before": <Totals>, "after": <Totals> },
  "endpoints": [ { "endpointId": "…", "service": "…", "name": "GET /orders/{id}",
                   "before": <Side> | null, "after": <Side> | null, "verdict": "better" | "worse" | "same" | "new" | "gone" } ],
  "queries":   [ { "queryId": "…", "service": "…", "statement": "…",
                   "before": <QuerySide> | null, "after": <QuerySide> | null, "verdict": … } ],
  "errors":    [ { "errorId": "…", "service": "…", "type": "…", "message": "…", "target": "…" | null, "before": 0, "after": 3, "verdict": … } ]
}

Totals is the shape of the overview totals. Side is { "calls", "errors", "p50Ms", "p95Ms", "maxMs", "dbCallsPerRequest", "dbMsPerRequest" }, and QuerySide is { "calls", "callsPerRequest", "p95Ms", "totalMs" }. A side is null when that window has no data for the row.

Verdicts

Every row carries a verdict, decided in this order.

  • new when only the after window has data, and gone when only the before window has.

  • worse when errors appear or grow, or when p95 grows by more than 20% and by at least 10 ms. For a query the measure is calls per request, and the second bound is 0.5 calls. When either window has no request, a query’s measure is its calls in the window, with the same two bounds, so a worker whose statement ran 2 times before and 200 after is worse; exercise both windows the same way for the counts to compare.

  • better when the same measure shrinks by more than 20% and by at least the same bound, or when errors disappear. Both percentages are of the before value, so a p95 that went from 100 ms to 82 ms, 18% faster, is same, and so is a change of exactly 20% either way.

  • same otherwise.

Every reason for worse is asked before any reason for better. An endpoint whose errors disappeared while its p95 grew past both bounds is therefore worse, because a slow retry that hides a failure produces the same two changes and is not a fix.

Endpoints are sorted worst first: worse, then new, same, better, gone, each group by total time descending, then by id. Queries and errors are sorted the same way.

The text rendering

The verdict is the first column, and the rows sort worse, new, same, better, gone, so anything that got worse is at the top of each table. Every other cell holds both windows as before → after, with — where a side has nothing.

$ java -jar spider-sense.jar compare --before=before --after=after
# compare  2026-09-17T08:19:28.122+09:00 → 08:19:35  vs  2026-09-17T08:19:35.837+09:00 → 08:19:45  (all services)

| totals | before | after |
| --- | --- | --- |
| requests | 21 | 12 |
| errors | 1 | 1 |
| p95 | 411.5 ms | 1,201.7 ms |
| apdex | 0.929 | 0.792 |

## endpoints

| verdict | endpoint | id | calls | errors | p95 | db/req | db ms/req |
| --- | --- | --- | --- | --- | --- | --- | --- |
| new | GET /api/slow | 7c7ff534a0ec | — → 3 | — → 0 | — → 1,202.5 ms | — → 0.0 | — → 0.0 ms |
| better | GET /api/reports/revenue | 7342d163d1ee | 3 → 3 | 0 → 0 | 549.8 ms → 417.1 ms | 2.0 → 2.0 | 419.8 ms → 383.6 ms |
| better | GET /api/orders/{id} | 264871b2ef8c | 3 → 3 | 0 → 0 | 66.5 ms → 7.6 ms | 6.7 → 6.7 | 1.0 ms → 0.5 ms |
| better | GET /api/flaky | 40e6cdb7cd69 | 3 → 3 | 1 → 1 | 22.0 ms → 2.1 ms | 0.0 → 0.0 | 0.0 ms → 0.0 ms |
| gone | GET /api/orders/{id}/enriched | 8c62f8d5090d | 3 → — | 0 → — | 174.5 ms → — | 6.7 → — | 0.5 ms → — |
| gone | GET /api/customers/search | cf03d98e70da | 3 → — | 0 → — | 29.6 ms → — | 1.0 → — | 0.9 ms → — |
| gone | POST /api/orders/{id}/pay | 48e41c5bc06c | 3 → — | 0 → — | 8.6 ms → — | 1.0 → — | 0.2 ms → — |
| gone | GET / | a239b7c70ac8 | 3 → — | 0 → — | 5.2 ms → — | 0.0 → — | 0.0 ms → — |

## queries

| verdict | id | calls | calls/req | p95 | total | statement |
| --- | --- | --- | --- | --- | --- | --- |
| same | e0e233d8ad75 | 3 → 3 | 0.1 → 0.3 | 269.7 ms → 247.1 ms | 782.8 ms → 698.9 ms | SELECT p.id AS product_id, p.sku AS sku, p.name AS name, SUM(l.quantity) AS quantity, SUM(l.quantity * l.unit_price) AS revenue FROM order_line l JOIN product p ON p.id = l.product_id JOIN orders o ON… |
| same | 4080653c9caa | 3 → 3 | 0.1 → 0.3 | 200.2 ms → 164.5 ms | 476.6 ms → 451.9 ms | SELECT o.status AS status, CAST(o.created_at AS DATE) AS order_day, SUM(o.total) AS revenue, COUNT(DISTINCT o.id) AS order_count, SUM(l.quantity) AS item_count FROM orders o JOIN order_line l ON l.ord… |
| same | 14eb3eff12dc | 22 → 11 | 1.0 → 0.9 | 0.2 ms → 0.1 ms | 2.0 ms → 0.5 ms | select p1_0.id,p1_0.name,p1_0.price,p1_0.sku from product p1_0 where p1_0.id=? |
| same | 62fb3169834e | 9 → 3 | 0.4 → 0.3 | 0.3 ms → 0.2 ms | 1.7 ms → 0.3 ms | select o1_0.id,o1_0.created_at,o1_0.customer_id,o1_0.status,o1_0.total from orders o1_0 where o1_0.id=? |
| same | 6ece3eeacddc | 6 → 3 | 0.3 → 0.3 | 0.4 ms → 0.2 ms | 1.0 ms → 0.4 ms | select l1_0.order_id,l1_0.id,l1_0.product_id,l1_0.quantity,l1_0.unit_price from order_line l1_0 where l1_0.order_id=? |
| same | a3b4c9a4fdef | 6 → 3 | 0.3 → 0.3 | 0.2 ms → 0.1 ms | 0.6 ms → 0.2 ms | select c1_0.id,c1_0.email,c1_0.name from customer c1_0 where c1_0.id=? |
| gone | a2ef3cae1e22 | 3 → — | 0.1 → — | 1.4 ms → — | 2.8 ms → — | select c1_0.id,c1_0.email,c1_0.name from customer c1_0 where lower(c1_0.name) like (?\|\|?\|\|?) escape ? order by c1_0.id fetch first ? rows only |

## errors

| verdict | id | type | message | target | before | after |
| --- | --- | --- | --- | --- | --- | --- |
| same | 458522ccda9f | java.lang.IllegalStateException | Payment gateway timeout | — | 1 | 1 |

Exercise the after window exactly as the before window was exercised: the same endpoints, the same number of times. A different exercise moves every number, and the verdicts then say what the traffic did rather than what the change did.

Call each endpoint once before mark before, and once again before mark after when the application was restarted in between. A cold JVM’s first call includes class loading, JIT compilation and the connection pool’s first connections, so its duration does not measure the code. A before window that holds it makes compare say better about a change that was never made, and an after window that holds it says worse. Check turns the same window into a pass or a fail.