Findings

A finding is one problem a rule detects in the data of a window: an error or an ERROR log record, a statement or an outbound call repeated within one request, a slow query, endpoint, job or outbound call, an exhausted connection pool, a JVM whose collections, heap or thread count pass a limit, or a resolved finding that came back.

GET /api/findings?since&until&service&limit=20&hideAcked=false&format=text|json answers them, and the CLI command is:

java -jar spider-sense.jar findings [--since=] [--until=] [--service=] [--limit=20] [--hide-acked] [--json] [--full]

The rules

Findings are computed over the window, not stored, so the same rules over the same window always produce the same list. Time selectors define the window, which is the last 15 minutes when nothing says otherwise.

Findings are ranked by severity (high before medium before low), then by kind, then by impact within a kind, then by id, so the list is stable between two calls over the same data. The one exception is an error that echoes another service’s error, which comes after the other errors (error). No rule produces low today. The ranking and the UI’s severity order already hold it, for a later rule that is less severe than medium. The kind order is regression, error, n-plus-one, n-plus-one-http, slow-query, slow-endpoint, slow-job, slow-external, log-error, pool-exhausted, gc-pause, heap-pressure, thread-growth. Each kind below names the impact number it is ranked by. A regression is always high, so regressions come first, ranked among themselves by the order of their original kinds before their impact.

The list is bounded. limit defaults to 20 and is at most 100. A list the limit cut says so: the JSON carries total, the number of findings before the limit, and the text rendering says how many there are and how to see the rest (The text rendering).

Two thresholds decide the slow kinds: slow.request.ms is 500 milliseconds by default and slow.query.ms is 100. status prints the thresholds in force, and Configuration says how to change them. An endpoint that spidersense.ignore.endpoints excludes (Ignored endpoints) is not an entry span and produces no finding of any kind, and neither does what its trace ran: its failure, the failures under it, its statements, its outbound calls and its ERROR log records. An entry span is what Endpoint identity defines, and it is what every count of requests below counts.

The three JVM kinds, gc-pause, heap-pressure and thread-growth, read the same metric_point series the JVM page draws. A run that is slow because the JVM is collecting or its heap is nearly full therefore has a gc-pause or a heap-pressure finding that names the cause, where its slow-endpoint findings alone would point at the endpoints' code.

What every finding carries

{
  "id": "n-plus-one:1a2b3c4d5e6f",
  "kind": "n-plus-one",
  "severity": "high",
  "state": "new",
  "service": "spring-orders",
  "title": "GET /orders/{id} runs SELECT order_line 42 times per request",
  "why": "3 of 3 requests repeated it; 42, 42 and 41 times; 38.2 ms per request in that statement",
  "subject": { "endpointId": "…", "queryId": "…", "errorId": null, "pool": null, "job": null,
               "target": null, "logger": null, "jvm": null },
  "numbers": { … },
  "statement": "SELECT … FROM order_line WHERE order_id = ?",
  "code": [ "orders.OrderService.load(OrderService.java:41)" ],
  "traces": [ "4bf92f3577b34da6a3ce929d0e0e4736", … ],
  "schema": { "tables": [ … ], "predicates": [ … ], "unindexed": [ … ] },
  "ack": null,
  "resolution": null
}
Field How to read it

id

The kind, a colon, and 12 hex characters of a SHA-256 over the kind, the service and the subject. It is stable across windows, so the same problem keeps its id between runs, and it is what ack and resolve take. A regression keeps the id of the finding it is.

kind

One of the thirteen kinds below.

severity

high, medium or low. In the text rendering it reads acked for an acknowledged finding and resolved for a resolved one that has not come back.

state

new, ongoing or regressed: what the last restart of the service changed (State).

service

The service the finding is about.

title

One line naming the subject and what is wrong with it, with a number only where it identifies the problem, such as the repeat count of an N+1 or the p95 of a slow query. The other numbers are in why.

why

The numbers that justify the claim, in a sentence. Quote this rather than restating it.

subject

The endpointId, queryId, errorId, pool, job, target, logger or jvm to pass to endpoints, queries, errors, logs or the API. Every key is present and the ones that do not apply are null. jvm is gc:<collector name>, heap or threads.

numbers

Kind-specific, listed with each kind below.

statement

The statement as the OpenTelemetry agent sanitised it, literals already ?, cut at 200 characters unless --full is given. One of 1,999 characters or more, which the store may have cut (statementCut), ends in … (cut at 2,000 characters) in the text rendering and carries statementCut: true in the JSON.

code

Application frames, most specific first, at most 5, and empty when none is known.

traces

At most 3 trace ids: the evidence, and what trace <id> opens.

schema

The tables the statement names with their indexes, the columns it filters on, and the ones no index serves. slow-query and n-plus-one only, and null when there is no catalog to read.

ack

The acknowledgement, { "at": …, "note": … }, or null when the finding is not acknowledged.

resolution

The resolution, { "at": …, "note": … }, or null when the finding is not resolved.

traces are the three slowest traces for the slow-* kinds, the three newest for error and log-error (of the records that have a trace id), the three most recent affected for n-plus-one and n-plus-one-http, and none for pool-exhausted and the three JVM kinds. A slow trace is one whose span of the finding was slow: the traces are ordered by the slowest such span each holds, the statement’s for slow-query, the call’s for slow-external, the entry span’s for slow-endpoint and the root span’s for slow-job, never by the trace’s own duration. A trace is one of the window’s when that span starts in the window, whenever the trace began, so a request under a job that was already running is evidence of the endpoint it is counted in. A trace is as often slow for its caller’s reasons as for the finding’s, and the first trace is the one hotSpan reads and trace <id> opens. A regression carries the traces of its original kind, taken after the resolution.

slow-endpoint and slow-job findings carry hotSpan among their numbers, which names where the time went in the finding’s first evidence trace:

"hotSpan": { "name": "SELECT order_line", "category": "db", "selfMs": 312.4, "share": 0.62 }

hotSpan is the span with the largest self time, that is its duration minus the time its direct children cover within it, as the Profile view computes it. It reads the spans hotSpans and breakdown read, the same way: the spans of the finding’s own service under its top spans in that trace, each counting only the part of it that lies inside its parent. A downstream service’s query belongs to that service’s endpoint, a request the service makes to itself belongs to that request, and the caller’s code around the call is not the callee’s time. An @Async span that runs on after the response counts only what it did before it. share is that self time over the duration of the top spans in that trace, which are the endpoint’s entry spans or the job’s run. category is db, http, internal or another span category. When dbShare is low, it names the span to look at first. In the text rendering it is one line, hot span: <name> · <selfMs> self · <share>, such as hot span: SELECT order_line · 312.4 ms self · 62.0%. It is null only when the finding has no trace.

Code locations

The OpenTelemetry Java agent does not record where a span was started from, so code comes from three sources. The first is the exception.stacktrace of an error. The second is the code.function and code.namespace attributes that a few instrumentations set, which is where a slow-job finding’s frames come from. The third is the code.stacktrace that Spider Sense’s own agent extension captures (The extension), on every database span that ran at least slow.query.ms, on the fifth repeat of a statement within one trace, on every non-database CLIENT span that ran at least slow.request.ms, and on the fifth repeat of an HTTP call within one trace. That stack is the one thing Spider Sense collects itself.

Those extension frames are the most accurate of the three, because they are the span’s own thread at the moment the span finished rather than a guess from an attribute. They are reduced to application frames by the same rules as exception.stacktrace, below. code is therefore reliable for error, slow-query, n-plus-one, n-plus-one-http and slow-external findings, often present for log-error, and often empty for the others. When it is empty, open a trace from traces and read the tree, which names the endpoint and the statement even when it cannot name the line.

Each kind takes its code from one place.

  • error takes it from the exception.stacktrace the error recorded.

  • log-error takes it from the exception.stacktrace attribute of the group’s newest record, when the logging bridge exported one.

  • slow-query and slow-external take it from the code.stacktrace of their slow spans.

  • n-plus-one and n-plus-one-http take it from the span of the repeated group that carries code.stacktrace, which is the fifth repeat, and fall back to the group’s newest span when none does, as in an application run without the extension.

  • slow-job takes it from the code.function and code.namespace attributes the scheduling instrumentations set on the job’s span.

The Code list of the query page reads a query group the way slow-query does, whether the group is slow or not.

A stack trace is reduced to its application frames by dropping known framework prefixes: java., javax., jdk., sun., com.sun., jakarta., org.springframework., org.hibernate., org.eclipse.jetty., org.apache., io.opentelemetry., com.zaxxer., org.h2., net.benelog.spidersilk., kotlin., scala., reactor., io.netty., ch.qos.logback., org.slf4j., org.junit., gg.jte., org.postgresql., com.mysql., org.mariadb., oracle., com.microsoft.sqlserver., org.sqlite., org.hsqldb., org.jooq., org.mybatis., com.mongodb., io.lettuce., redis.clients., com.fasterxml., com.google., kotlinx., okhttp3., feign., io.grpc. and org.jboss.. A JDBC driver, a data-access library, a JSON mapper or a client throws from frames of its own, so its prefix is on the list: otherwise its throwing frame would be the first application frame of every failure that passes through it, and all of those failures would fall into one error group. When that heuristic guesses wrong, spidersense.app.packages=com.acme,org.acme, a comma-separated list, replaces it with an allowlist. A frame of a generated class is never an application frame, whatever its package and whatever the allowlist says: one that holds $HibernateProxy, SpringCGLIB, EnhancerBySpringCGLIB, FastClassBySpringCGLIB or $ByteBuddy$, or whose position is <generated>. Such a class lives in the application’s package, but its frame is a proxy’s, so the n-plus-one of a lazily loaded association names the line that touched it rather than Product$HibernateProxy.getName(Unknown Source). A frame is read without the module or class loader the JVM writes in front of its class, such as java.base/ or app//, and a hidden class, such as a lambda’s, without the address behind its name, so org.springframework.aop.framework.CglibAopProxyLambda/0x00007f0a1c0f3b28.proceed+` is the framework's frame `+org.springframework.aop.framework.CglibAopProxyLambda.proceed. At most five frames are kept, innermost first, which means the root cause’s first. The frames of the last Caused by: section come before those of the exception that wraps it, and so on out to the outer exception, each section read top down. A Suppressed: block is left out, and a frame that repeats is counted once. The line that went wrong is usually under Caused by:, and reading the trace top to bottom would put the wrapper’s frames first and cut the root cause’s at the fifth. The same order holds for error, for log-error and for the Code list of the error page. When no spidersense.app.packages narrows it, the first frame of that order is the one an error group is keyed on (Errors).

A frame whose file is under the source roots is shown with more than its name. The UI shows its five source lines and links it into your editor, and the CLI’s findings names the change that last touched its line (Source lines and the suspect change). The text rendering itself carries neither: an agent opens the file on its own.

Source lines and the suspect change

Spider Sense runs on the machine the source is on, so it reads the line a frame names from the source file, and asks git for the change that last touched it.

A frame orders.OrderService.load(OrderService.java:41) resolves to the first <root>/orders/OrderService.java that exists under the roots of spidersense.source.dirs (Source roots). By default the roots are src/main/java and src/main/kotlin of the working directory and of each of its immediate subdirectories, which covers a single-module build and a multi-module one. A frame without a file and a line (Unknown Source, Native Method, a code.function frame) does not resolve. Neither does a frame whose path would leave its root. The path is built only from dotted Java identifiers and a plain file name with a source extension, and the file, with symbolic links followed, must lie under its root.

In the UI, the findings page and the error page show the five lines around the frame’s line under each frame that resolves, and make the frame a link that opens the file at that line in IntelliJ IDEA or VS Code (Code frames). The lines are read on request by GET /api/source?frame=… (Source) and never stored. A frame that does not resolve shows nothing more than itself.

The text rendering does not carry the lines, because an agent opens the file itself. What it gains instead, in the CLI’s findings, is the suspect change: one line under each frame that resolves to a file, naming the file relative to the working directory and saying whether that line is in the change just made. The suspect change specifies that line.

The schema block

A slow-query or n-plus-one finding of a service that ran under the agent also says which of the columns the statement filters on carry no index. The indexes are the ones the database’s catalog reports, not ones inferred from the statement’s text.

"schema": {
  "tables": [ { "table": "ITEMS", "schema": "PUBLIC",
                "indexes": [ { "name": "PRIMARY_KEY_8", "unique": true, "columns": [ "ID" ] },
                             { "name": "IDX_ITEMS_SUPPLIER", "unique": false, "columns": [ "SUPPLIER_ID", "NAME" ] } ] } ],
  "predicates": [ "items.name", "items.category" ],
  "unindexed":  [ "items.name", "items.category" ]
}

The block needs the index catalog, so it appears only for a service that ran under the agent with the extension.

tables are the tables the statement names, each with its indexes as the extension read them through DatabaseMetaData.getIndexInfo on the application’s own connection (The extension). The store keeps them per service and table (the db_table table), for as long as the service sends spans and for the retention after that (Retention), so an application that runs for longer than the retention keeps its block. The extension reads a table’s indexes on the first statement of a process that names it, however fast, and again after DDL that names it, so once the index the block asked for exists, the next run of the application, or the next statement after the create index in the same run, takes its column out of unindexed. The names use the database’s spelling: ITEMS on H2, items on PostgreSQL, and the case the table was created with on MySQL. schema is null when the database reports none, and indexes is empty for a table that has none.

predicates are read from the sanitised statement by a light parse of the common shapes: every column a where or on clause or an order by list refers to, bare (where name = ?), qualified (where i1_0.name like ?, the alias resolved to its table) or wrapped in a function (where lower(name) like ?). Each is written as table.column in lower case whatever case the statement used, in order of first appearance and once each, and a column named without a qualifier belongs to the statement’s only table. A subquery’s where counts as well, and the clause it interrupts goes on after its closing parenthesis, so where i.id in (select l.item_id from lines l) and i.name = ? names items.id and items.name. The type of a cast is not a column (where id = ?::uuid and cast(? as timestamp with time zone) name no type word), and neither is an upsert’s conflict target, the unique key the write collides on (on conflict (id) do update, on duplicate key update). Neither is the unit of an interval (interval ? day, interval ? day to second), the zone of an at time zone, or the search string and modifier of a MySQL full-text search (against (? in boolean mode)). A collation is no column (order by name collate "C"), nor is a word of a comparison (similar to, between symmetric), and the from of is [not] distinct from opens no table list, so the clause goes on after it. A locking clause names no table and no column (for update skip locked, for no key update of o1_0, for share, MySQL’s lock in share mode). Nor does what a write returns (returning id, payload), or a window: what it partitions and sorts by, inline (over (partition by … order by …)) or in a window clause, orders the rows its function reads rather than the result. An array constructor names no column either (where o.id = any(array[?, ?]) names orders.id), and brackets quote a name only around a name’s characters ([Unit Price]). Inside extract, trim, substring and overlay, from and for separate arguments rather than opening a table list, so extract(year from created_at) names created_at, and the date part that extract, dateadd or timestampdiff takes first is no column. Nor is a word of a comment, which the agent’s sanitiser keeps (-- … or MySQL’s # … to the end of the line, /* … / with the comments PostgreSQL nests inside it, an optimizer hint, an sqlcommenter suffix). A that begins one of PostgreSQL’s operators (>, >>, -, ##) starts no comment. A name the select list gives, with as or without it, is not a column when the order by sorts by it (price * qty as total … order by total, count() cnt … order by cnt), and except, intersect and minus end the clause before them as union does. A row limit or a distinct on (…) before the first item gives that item no alias, so select top 10 name … order by name and select distinct on (customer_id) customer_id … order by customer_id sort by a column. A keyword that is also a common column name (date, time, timestamp, first, last, next, row, rows, only, interval) is that column in a where or on clause when a comparison follows it (where date = ?), and the keyword anywhere else. An operator spelt as a word is no column (regexp, rlike, mod, div, xor, sounds, binary, unknown, glob, member, as in where email regexp ? or where id mod ? = ?), nor is a pseudo-column (rownum, rowid) or a function called without parentheses (sysdate, systimestamp, localtime, localtimestamp, current_user, session_user, system_user, current_role, current_schema, user); a table may still be named by one of these words (from member).

A function in an order by list is skipped whole (order by sum(l.quantity) desc names no column), because no index sorts an aggregate. unindexed is the subset of predicates that no index of their table has as its first column, the one an index can seek on. A column an index carries second is served no better by it than a column no index carries at all. A function-wrapped column is matched by its name, so an index on name takes lower(name) out of unindexed. The statement shows whether an expression index is what is really needed. The other way round, an index whose first key part is an expression serves the columns the expression names, so once create index … on customers (lower(email)) exists, customers.email leaves unindexed: PostgreSQL reports that key part as loweremail)::text). MySQL reports no column for a functional key part, so there such a key part keeps its place as an empty name (idx_customer_email_lower ()), the index serves no predicate, and EXPLAIN is the way to confirm that the statement uses it. The column after it is not the index’s first: create index … on orders ((lower(email, status) is listed as idx (, status) and leaves orders.status in unindexed.

An unqualified name is never the database’s own catalog table, such as H2’s INFORMATION_SCHEMA.USERS for a users, and the extension looks it up in the connection’s current schema first (The index catalog). The block is null whenever the parse cannot attribute every table and column of the statement with certainty: a table the catalog has no row for (the application ran in standalone mode or without the extension, or no statement on that table has run since the application started under the agent, or the statement names a schema, archive.items, in which the catalog has no such table), one name in two schemas of the same statement (s1.items a, s2.items b), whose columns cannot be told apart, an unqualified name the catalog holds in two schemas (orders with both public.orders and audit.orders read), since which one the database resolved it to cannot be told from the statement, a statement whose tables the scanner does not find, a column it cannot attribute to one table, such as an unqualified column in a join or beside a derived table (from (select …) d, items where total > ?) or the alias of a subquery, or a statement of 1,999 characters or more, which the store may have cut inside its where clause (db_statement). It says nothing about selectivity, wildcards or the plan. The spider-sense-sql-tuning skill and EXPLAIN answer those questions. The block lets the skill write the index without a round trip to the database.

In the text rendering the block sits between the statement and the code frames, one line per table, then one line for the columns:

   indexes ITEMS: PRIMARY_KEY_8 (ID) unique, IDX_ITEMS_SUPPLIER (SUPPLIER_ID, NAME)
   indexes MOVEMENTS: none
   predicates: items.name, items.category; unindexed: items.name, items.category

predicates: none stands alone when the statement has no predicate, unindexed: none when every predicate is served, and a finding without the block prints nothing for it. GET /api/queries carries the same block on every query group. Its text rendering, and so the CLI's queries table, has an unindexed column between callers and statement: the unindexed columns joined by , `, `none when every predicate is served, and — when there is no block. The catalog is also open to SQL as the db_table table, and it travels with an export. The Findings page shows the block under the statement, and the Queries page has the same unindexed column.

Where the time went

One trace may not be typical, so a slow-endpoint and a slow-job finding also say where the time went over a sample of up to 20 traces. hotSpans names the three summaries that took the most time, and breakdown says what kind of work the time was.

"hotSpans": [ { "name": "SELECT order_line", "category": "db", "selfMs": 4210.5, "share": 0.44, "count": 126 },
              { "name": "GET localhost:8081/api/books/?", "category": "http", "selfMs": 1980.0, "share": 0.21, "count": 42 } ],
"breakdown": { "db": 0.44, "http": 0.21, "internal": 0.07, "self": 0.28 }

Both are computed over the finding’s sample: the 20 slowest traces of the endpoint or the job in the window, of which the first three are the traces the finding carries. The sample is twenty traces rather than every trace because the self times are computed from each trace’s spans rather than from an aggregate, and the slowest twenty are the requests that make the endpoint slow. A share is therefore a share of those traces' time, while the finding’s own totalMs stays the window’s. Each trace is read whole or not at all: when the service’s spans in the twenty pass 20,000, the sample stops at the last whole trace under that count, slowest first, and the slowest is always read. A trace read in part would cut off whatever lies under a parent it missed and move that time into the wrong share. Only the spans of the finding’s own service under its top spans count, which is the rule dbShare already uses. The top spans are the endpoint’s own entry spans in each trace, or the job’s run, and what lies under one stops at the next entry span, which is a request of its own. The database work of a downstream service belongs to that service’s own endpoint, and the work of a request a service makes to itself, or of a message it consumes itself, belongs to the endpoint that serves it. Leaving the entry span below out makes the wait on an outbound call land in http, where the caller can see it.

Self time is a span’s duration less the time its direct children within that set cover inside it, as the Profile view computes it. A span’s children are looked up within its own trace, since a span id is unique within its trace only. Here each span counts only the part of it that lies inside its parent, down from the top span, so an @Async child that runs on after the response adds only what it did before it. A top span is never cut to its parent: a listener that runs on after its message was published is still its whole request. The shares are taken over the sum of the self times, which is the top spans' summed duration. Children that run at once each count their overlap, which makes the sum the larger, and the shares still sum to 1.

hotSpans groups every span whose self time is not self by category and name, sums each group’s self time, and lists the three largest by selfMs, largest first. The name is the span’s summary with every run of digits replaced by ?, and an outbound call is named as the repeated call names it (GET localhost:8081/api/books/?), so one call to 40 items is one row and reads the same here as it does in an n-plus-one-http. count is how many spans went into the row, and share is the row’s selfMs over the sample’s total.

breakdown is db, http, internal and self, which sum to 1. self is the top spans' own self time: the request’s own code, the framework, everything not in a child span. A top span that is itself an outbound call, a root CLIENT or PRODUCER span such as a traced load generator’s request, has no code of its own: its self time is the wait on the call, which goes to its category’s share and into hotSpans as any other call does. internal absorbs the messaging and rpc categories. The four shares answer whether the time was spent waiting on the database, waiting on the network, or working. breakdown is the number to read first when dbShare is low, because it says whether to look at the queries, at the calls or at the code without opening a trace. GET /api/endpoints/{endpointId} carries the same breakdown over the same sample, and the endpoint page draws those four shares as one bar. Both are absent, hotSpans empty and breakdown empty, when the finding has no trace or when the sample’s spans add up to nothing.

In the text rendering, hotSpans and breakdown come after the numbers and before the statement:

   hot spans: SELECT order_line · 4,210.5 ms · 44.0% · ×126
              GET localhost:8081/api/books/? · 1,980.0 ms · 21.0% · ×42
   breakdown: db 44.0% · http 21.0% · internal 7.0% · self 28.0%

The text rendering

The ranked table comes first, and the fields follow as one numbered block per row, in the same order. When the limit cut the list, a line between the heading and the table says how many findings there are and how to see the rest, as traces says it:

  • 20 of 54 findings; --limit=54 shows them all when every finding fits under the largest limit,

  • 20 of 154 findings; --limit=100 shows the first 100 when they do not,

  • 100 of 154 findings; --service= or a shorter window shows the rest when the largest limit is already in force.

The options named are the CLI’s; over HTTP and MCP the limit is the limit parameter and the service service. A list that holds every finding has no such line. The table’s state column is the finding’s state. A block is <n>. <id> — <why>, then the numbers on one line, then the hot span line when the kind has one, then the hot spans and the breakdown when it has those, then the statement when there is one, then the schema lines when the finding has that block, then the code frames one per line, then the trace ids. The numbers line is key value pairs joined by commas, and a value that is empty, such as the message of an error with none, reads —, as the tables print it.

$ java -jar spider-sense.jar findings --since=before
# findings  2026-09-17T08:19:28.122+09:00 → 08:19:34  (6s, all services, 21 requests)

| # | severity | state | kind | id | service | title |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | high | new | error | error:eecf9878a68f | spring-orders | IllegalStateException in GET /api/flaky |
| 2 | medium | new | n-plus-one | n-plus-one:4c5f46be8bc5 | spring-orders | GET /api/orders/{id}/enriched runs SELECT product 5 times per request |
| 3 | medium | new | n-plus-one | n-plus-one:e6d97708ab26 | spring-orders | GET /api/orders/{id} runs SELECT product 5 times per request |
| 4 | medium | new | slow-query | slow-query:6b3aae6f9bef | spring-orders | SELECT p.id AS product_id, p.sku AS sku, p.name AS name, SUM… is slow: p95 269.7 ms in GET /api/reports/revenue |
| 5 | medium | new | slow-query | slow-query:aae3ff87821a | spring-orders | SELECT o.status AS status, CAST(o.created_at AS DATE) AS ord… is slow: p95 200.2 ms in GET /api/reports/revenue |
| 6 | medium | new | slow-endpoint | slow-endpoint:786b594455d0 | spring-orders | GET /api/reports/revenue is slow |

1. error:eecf9878a68f — 1 occurrence in GET /api/flaky; Payment gateway timeout
   count 1, firstSeen 2026-09-17T08:19:28.962+09:00, lastSeen 2026-09-17T08:19:28.962+09:00, type java.lang.IllegalStateException, message Payment gateway timeout, endpoints [name GET /api/flaky count 1]
   orders.web.MiscController.flaky(MiscController.java:24)
   traces: 5e6769395a77c8f192949af91cbdedf2

2. n-plus-one:4c5f46be8bc5 — 1 of 3 requests repeated it; 5 times; 0.4 ms per request in that statement
   requests 3, affected 1, medianRepeats 5, maxRepeats 5, msPerRequest 0.4
   select p1_0.id,p1_0.name,p1_0.price,p1_0.sku from product p1_0 where p1_0.id=?
   traces: e7cfa77cb4c9135b623a04e470ef58d0

3. n-plus-one:e6d97708ab26 — 1 of 3 requests repeated it; 5 times; 0.8 ms per request in that statement
   requests 3, affected 1, medianRepeats 5, maxRepeats 5, msPerRequest 0.8
   select p1_0.id,p1_0.name,p1_0.price,p1_0.sku from product p1_0 where p1_0.id=?
   traces: 0936e867b3ba9250964b7efa08029ee4

4. slow-query:6b3aae6f9bef — p95 269.7 ms over 3 calls, 3 of them over 100 ms; 782.8 ms in total
   calls 3, slowCalls 3, p50Ms 266.6, p95Ms 269.7, maxMs 269.7, totalMs 782.8, callers [endpoint GET /api/reports/revenue service spring-orders calls 3]
   SELECT p.id AS product_id, p.sku AS sku, p.name AS name, SUM(l.quantity) AS quantity, SUM(l.quantity * l.unit_price) AS revenue FROM order_line l JOIN product p ON p.id = l.product_id JOIN orders o ON…
   traces: 42800a1fc7ca5ee2dec08f804cef8be6 a86465f5a9242ac8496577740771fcb7 cbc8be4f6ec5a1c033e57cbb7f269bd5

5. slow-query:aae3ff87821a — p95 200.2 ms over 3 calls, 3 of them over 100 ms; 476.6 ms in total
   calls 3, slowCalls 3, p50Ms 141.3, p95Ms 200.2, maxMs 200.2, totalMs 476.6, callers [endpoint GET /api/reports/revenue service spring-orders calls 3]
   SELECT o.status AS status, CAST(o.created_at AS DATE) AS order_day, SUM(o.total) AS revenue, COUNT(DISTINCT o.id) AS order_count, SUM(l.quantity) AS item_count FROM orders o JOIN order_line l ON l.ord…
   traces: 42800a1fc7ca5ee2dec08f804cef8be6 a86465f5a9242ac8496577740771fcb7 cbc8be4f6ec5a1c033e57cbb7f269bd5

6. slow-endpoint:786b594455d0 — p95 549.8 ms over 3 calls; 2.0 database calls and 419.8 ms per request, 93.0% of the time
   calls 3, p50Ms 411.5, p95Ms 549.8, maxMs 549.8, totalMs 1,354.9, apdex 0.833, dbCallsPerRequest 2.0, dbMsPerRequest 419.8, dbShare 93.0%
   traces: 42800a1fc7ca5ee2dec08f804cef8be6 a86465f5a9242ac8496577740771fcb7 cbc8be4f6ec5a1c033e57cbb7f269bd5

The kinds

regression

A finding of any other kind that was resolved and has come back since. Resolutions says when a finding counts as back. It is always high and always first, so a finding that came back after its fix is at the top of the list.

The id, subject, title, statement, code, traces and schema are those of the finding it is, and the impact it is ranked by is its original kind’s. numbers starts with resolvedAt, the instant of the resolution, note, the resolution’s note or null, and originalKind, followed by the original kind’s own numbers. why starts with came back after it was resolved, the note in parentheses when there is one, and continues with the original why.

What to do: read originalKind and the note, open a trace from traces, and find which change undid the fix. The fix itself is the one the original kind’s section below describes.

n-plus-one

The same query group runs 5 or more times under the same entry span in one trace, aggregated per endpoint and query group over the window. It is high when the repeats reach 20 or their summed time exceeds slow.request.ms, and medium otherwise. The impact it is ranked by is affected requests multiplied by median repeats.

The title is <endpoint> runs <statement summary> <medianRepeats> times per request.

numbers: requests (entry spans of the endpoint in the window), affected (of them, how many repeated), medianRepeats, maxRepeats, msPerRequest (the summed time of the repeated statement, per affected request).

Read it as: this endpoint ran that statement medianRepeats times in one request, and that cost msPerRequest. affected well below requests means only some inputs trigger it, which usually points at a branch or a lazily loaded collection that is only touched sometimes. code is the call site of the fifth repeat, so it names the line that issues the repeated statement. When no span of the group carries a stack, as in an application run without the extension, code comes from the group’s newest span and is usually empty. The trace tree then shows the repeats, collapsed into × n after the third, under the span that runs the loop. The schema block names the tables the repeated statement touches, the indexes they carry, and the columns it filters on that no index serves.

What to do:

  • JPA or Hibernate: load the association with its parent, with a fetch join (select distinct o from Order o join fetch o.lines) or an entity graph (@EntityGraph(attributePaths = "lines")).

  • When a fetch join would multiply rows across two collections, set batch loading instead: @BatchSize(size = 50) on the collection or the entity, or spring.jpa.properties.hibernate.default_batch_fetch_size=100 globally.

  • spring-jdbc or plain JDBC: replace the explicit loop with one query over the whole key set, where order_id in (:ids), and group the rows in memory. Chunk a very large key set rather than sending one enormous IN.

  • An IN list with a varying number of parameters produces a new statement text per size, so Spider Sense groups those calls separately.

After the fix, compare should show the query’s calls/req down to 1 and the endpoint’s verdict better, and check --max-queries-per-request= fails from then on if the count rises again.

n-plus-one-http

The same outbound HTTP call runs 5 or more times under the same entry span in one trace, aggregated per endpoint and call over the window. It is the N+1 an ORM cannot cause: a loop that fetches one remote resource per item. The trace shows it as a run of CLIENT spans of category http under one entry span. The severity, the impact, the evidence traces, the text rendering and the five numbers are the same as for n-plus-one, measured over the repeated call rather than the repeated statement. statement is null, subject.endpointId is the endpoint, and subject.target names the host that was called. When that host is another traced service, the trace names the service where the call enters it, and findings --service=<that service> lists the callee’s own findings. The title is <endpoint> calls <call> <medianRepeats> times per request:

GET /orders/{id} calls GET localhost:8081/api/books/? 6 times per request

code is the call site of the fifth repeat, so it names the line that issues the call.

The repeated call

Two CLIENT spans are the same call when they agree on their name, their target and their path once every run of digits is replaced by ?. The call is written the way it is grouped:

GET localhost:8081/api/books/?

That is the span’s name, the dependency target (localhost:8081, the host and port the call went to, as the service map names it), and the path and query of url.full with the digits replaced. A loop varies the digits, so replacing them turns three calls into one call; it is the same key the trace diff aligns on. The status is not part of the call, because the same call can answer 200 for one item and 404 for the next. The host is kept out of the digit rule, so the port stays readable. A span with no url.full is named <span name> <target> and grouped by that.

The rule is then `n-plus-one’s with the call in place of the statement: five or more of the same call under one entry span make that request affected, and the affected requests of one endpoint and call over the window make the finding.

What to do:

  • Ask for the whole set once: a batch endpoint (GET /api/books?ids=1,2,3) or a bulk query on the callee, and group the answers in memory. When the callee is yours, adding that endpoint is usually the smaller change.

  • Cache what does not change per request: a per-request cache removes the repeats inside one request, a short-lived shared cache removes them across requests.

  • Move the join to the callee: when every item needs the same enrichment, let the callee return it with the list rather than answering one item at a time.

  • When none of these is possible, make the calls concurrently, which shortens the request without removing the load.

check --max-n-plus-one=0 fails on this kind as it does on n-plus-one, because the rule counts the findings of both kinds (Check).

slow-query

A query group whose p95 exceeds slow.query.ms. Every group of the window is asked, however it ranks by total time, so a report query called twice behind a hundred frequent fast statements is still a finding. It is high when the p95 exceeds ten times the threshold, and medium otherwise. The impact it is ranked by is total time.

The group is aggregated over the calls no ignored endpoint’s trace made (Ignored endpoints), so a health check that runs the statement a hundred times fast neither dilutes the p95 of four slow requests below the threshold nor counts among calls.

The title is <statement summary> is slow: p95 <p95> in <caller>, as in SELECT items is slow: p95 812.0 ms in GET /items, where the caller is the first of callers, the endpoint that ran the group most. The summary alone would title a search and two reports over one table alike, and the p95 and the caller tell them apart.

numbers: calls, slowCalls, p50Ms, p95Ms, maxMs, totalMs, callers (the endpoint, service and call count of the nearest entry span up the parent chain, as QueryStats.callers has them, leaving out the calls of an ignored endpoint’s trace, which the queries page still lists).

A p95 far above p50 is a query that is fast for most inputs and slow for some, which usually means a plan that degrades with the data rather than a statement that is always wrong. callers lists the endpoints that ran it with their call counts, and the trace ids give the individual calls. The schema block names the tables the statement touches, the indexes they carry, and the columns it filters on that no index serves.

What to do:

  • Add the index the predicate needs, which the schema block’s unindexed lists by name. A predicate with a leading wildcard (like '%graph%') cannot use an index. Search that needs an infix match needs a search index instead, such as a PostgreSQL trigram or full-text index, H2’s FT_* functions or Lucene.

  • Rewrite the shape: a correlated subquery per row becomes a join, an unbounded select count(*) becomes a bounded count, and an order by over an unindexed column becomes an indexed one.

  • Stop selecting what is not needed: a projection instead of an entity, or named columns instead of select *.

  • Page when the endpoint does not need the whole result: Pageable in JPA, limit/offset or a keyset predicate in SQL.

Re-run the same exercise and read the query’s own p95 and total columns in compare, not only the endpoint’s.

slow-endpoint

An endpoint whose p95 exceeds slow.request.ms. It is high when the p95 exceeds four times the threshold, which is the "frustrated" bound of the Apdex, and medium otherwise. The impact it is ranked by is total time.

The title is <endpoint> is slow.

numbers: calls, p50Ms, p95Ms, maxMs, totalMs, apdex, dbCallsPerRequest, dbMsPerRequest, dbShare (from 0 to 1, the part of the endpoint’s total time spent in its own database spans, printed as a percentage), hotSpan, and hotSpans and breakdown. The endpoint’s own database spans are the ones whose nearest entry span up the parent chain is one of its requests, of their own service. A downstream service’s statements are that service’s endpoint’s, and so are the statements of a request the service makes to itself or of a message it consumes itself, which is an entry span of its own.

dbShare decides where to look next.

  • High dbShare means the endpoint is not the problem and its queries are. Look for an n-plus-one or slow-query finding with this endpoint among its callers, and fix it there.

  • Low dbShare means the time went to outbound calls or to the endpoint’s own code. breakdown says whether it is the network or the endpoint’s own code, hotSpans names the three summaries that took the most of it, and the slowest trace from traces shows the rest. hotSpan names the span with the largest self time in the first evidence trace. A CLIENT span that takes most of the duration is the cause, and a long stretch with no child spans at all is the endpoint’s own code.

What to do:

  • A CLIENT span whose child SERVER span is nearly as long means the time was spent in the other service, so run findings again with --service= set to it.

  • A CLIENT span much longer than the SERVER span it wraps is connection setup, queueing or serialisation, not the callee.

  • For the endpoint’s own code, look for work done per request that could be done once, such as compiling a pattern, building a client or reading a file. An unbounded collection built in memory and a sleep are the other common causes.

  • An endpoint that fans out to several independent calls can run them together rather than in sequence. The trace tree shows sequential CLIENT spans laid end to end when it does not.

check --endpoint="GET /orders/report" --max-p95-ms= scopes the verdict to the one endpoint being worked on.

slow-job

A job whose p95 exceeds slow.request.ms, grouped by service and span name. Every job of the window is asked, as for slow-query. A job is a root INTERNAL span: a scheduled method, an @Async call, a batch step (Endpoint identity). Severity and impact are the same as slow-endpoint. subject.job is the span name, and the title is <job> is slow.

numbers: runs, p50Ms, p95Ms, maxMs, totalMs, dbCallsPerRun, dbMsPerRun, dbShare, which are the measures of slow-endpoint over the job’s runs, then hotSpan, hotSpans and breakdown. A database span is the run’s when its parent chain reaches the run without passing an entry span: a request the job makes to its own service is that request’s work, not the job’s.

A job is started by the application itself, not by an incoming request, so it is never an entry span. It is in no request count, in no Apdex and in no check verdict, and this finding is the one place a slow scheduler tick or batch step is reported.

dbShare is read as it is for slow-endpoint. A high dbShare means the job’s queries are the problem. An N+1 in a job goes unnoticed more easily than one in a request, because no caller waits for the job to finish. The n-plus-one rule counts repeats under an entry span, and a job is none, so it does not report an N+1 inside a job, and neither does check. dbCallsPerRun and the count of a statement in hotSpans (×8 in the text) are where it shows: a statement that runs once per row of the batch. A low dbShare means the time is the job’s own code, and breakdown and hotSpans say where it went before a trace has to be opened.

What to do:

  • Run one query for the whole batch instead of one query per row.

  • Batch or move out of the tick a CLIENT call that dominates the trace.

  • A job that has grown slower than its own interval overlaps with itself, and runs with p95Ms over a few minutes say whether it has.

check says nothing about a job, so verify a fix with compare over the same exercise, or with findings --since=start after a restart.

slow-external

An outbound HTTP call whose p95 exceeds slow.request.ms: CLIENT spans of category http, grouped by service, target and span name. The target is what the service map calls the dependency, such as localhost:8081, so one row is one dependency of this service rather than every outbound call it makes. Severity and impact are the same as slow-endpoint. subject.target is the target, and the title is the span name, then the target: GET localhost:8081 is slow.

numbers: calls, errors, p50Ms, p95Ms, maxMs, totalMs, callers (as slow-query has them: the entry spans up the chain).

Open the slowest trace from traces first, to see whether the time was spent in the callee or on this side of the call. When the callee is traced too, its SERVER span is in the same tree. A SERVER span nearly as long as the CLIENT span means the time was spent in that service. A CLIENT span much longer than the SERVER span it wraps is connection setup, queueing or serialisation on this side.

What to do:

  • Stop making the call per request when the answer barely changes: cache it, or fetch it once at startup.

  • Make one call instead of N. A call inside a loop is the HTTP form of an N+1 (n-plus-one-http), and each repeat adds a network round trip to the request.

  • Run independent calls together rather than end to end.

  • Reuse the client. A RestClient or HttpClient built per request builds a connection pool per request.

  • Set timeouts. A call with no timeout waits for as long as the callee takes to answer, so a callee that stalls for a minute stalls this service’s requests for that minute.

error

An error group with at least one occurrence in the window. It is always high, and the impact it is ranked by is the occurrence count.

numbers: count, firstSeen, lastSeen, type, message (normalised, digits replaced by ?), endpoints (name and count, as ErrorGroup.endpoints has them). An error in a job’s trace, which has no entry span, is attributed to the job, so a worker’s failure reads SQLTransientConnectionException in ArchiveJob.archiveSlice (Callers and error endpoints). The title is <type short name> in <endpoint>, the endpoint being the first of endpoints.

An error group whose occurrences are mostly failed outbound HTTP calls (CLIENT spans of category http) is an error from a call. Such a span carries error.type 500 and no message, so its title names the call instead: 500 from GET localhost:8081/api/flaky, the call being the span name, the target and the path with its digits made ?, the most frequent of the group. The caller’s endpoint follows as in <endpoint> when it is another span than the call itself, which a client root span, such as a load generator’s request, is not. When the service it called also failed, the SERVER span under the call failing in every one of the finding’s traces, numbers ends with failedIn, that service’s name, and the why says so: the callee’s own error finding is the cause, and this one is its echo in the caller. Only the server span a call of the group reached counts, so a service in the middle of a chain names the service it called, never its own caller or a service further down. Such an echo is ranked after every error that is not one, whatever its count, so the finding of the service that failed comes before the findings of its callers.

The group is the service, the root cause’s type and the innermost application frame, so one row is one line throwing one exception rather than one occurrence. A wrapper such as Spring’s DataAccessException or ServletException does not split a bug by the SQL or request its message embeds, and does not merge unrelated causes under its one type. When the stack trace has no application frame, or there is none, the group falls back to the exception type and the normalised message. type and message are the outer exception’s, as the span recorded it. code is most reliable for this kind, and its top frame is the root cause’s line, so look at it first. traces are the three newest occurrences, and trace <id> shows the exception span with its message and frames and the trace’s log lines underneath. logs --trace=<traceId> gives the full log for that request.

What to do, after reading the top frame:

  • A bug: the code cannot handle an input it will receive. Fix the code, and add the test that reproduces it.

  • An expected condition answered as a failure, such as a business rule violated or an entity not found. Answer the client with a 4xx status rather than let the exception escape as a 500. The Endpoints table still counts those answers in the endpoint’s status codes (Services, Endpoints). An @ExceptionHandler that answers 409 Conflict is the usual fix.

  • A dependency failing: the trace shows the CLIENT span that failed first.

A message that carries the identifier (Order 42 is already shipped) still groups correctly, because Spider Sense normalises the digits away. check --max-errors=0 over a window that starts at the restart fails if the error occurs again.

log-error

Log records of severity ERROR or above, grouped by service, logger and normalised message, counting only the uncovered ones: records without a trace id, or with a trace id whose trace has no error span. A record an ignored endpoint’s request wrote (Ignored endpoints) is not counted: one on the ignored span itself, and one of a trace whose root is an ignored span, which is how a record with a trace id and no span id is judged. A health check that answers 200 while its indicator logs a failed dependency at ERROR is therefore no log-error and fails no check --max-log-errors. A group with at least one uncovered record in the window is a finding. It is always high, and the impact it is ranked by is the uncovered count.

The message is normalised as an error message is. subject.logger is the logger, and the title is ERROR in <logger short name>: <message cut at 80 characters>.

numbers: count (the uncovered records), firstSeen, lastSeen, logger, message (normalised), endpoints (the entry span of each record’s trace, as error counts them, the job for a record of a job’s trace, and (no endpoint) for a record without either). count covers every uncovered record of the window, never the first rows read, because check --max-log-errors sums it.

This is what catch (Exception e) { log.error(…, e); return fallback; } leaves behind: no span error, no exception event, one line in the log, and a request that answered 200 with the wrong answer. A record whose trace has an error span is already reported by an error finding and is not counted twice, so a log-error is a failure that no error finding reports. code comes from the record’s exception.stacktrace when the logging bridge exported a throwable. logs --severity=ERROR --q=<logger> lists the records themselves.

What to do, after reading the top frame:

  • A real failure being swallowed. Rethrow it, or answer the client with an error status, so the failure shows in the span and in the response rather than only in a log.

  • An expected condition logged as an error, such as a validation failure, a cache miss or a retry that then succeeded. Log it at WARN or INFO, because ERROR should mean "somebody has to look".

  • A dependency failing: the trace, when there is one, shows the CLIENT span that failed first.

  • When the fallback is the right behaviour, keep it and make it visible: record it as a counter, or set the span’s status, so the endpoint’s numbers say how often the degraded path was taken.

check --max-log-errors=0 over a window that starts at the restart fails if the record is logged again.

pool-exhausted

A JDBC pool with a point in the window where pending requests are above zero, or used equals max. It is always high, and it is ranked by pending first and used second. A pool is one process’s: when several processes of the service export a pool of one name, as every Spring Boot process names its first pool HikariPool-1, each process’s pool is judged on its own, and the worst of them is the finding. An application’s full pool beside its tests' idle one is therefore exhausted, and a second process adds no connection to the first one’s usedMax.

subject.pool is the pool.

numbers: pool, max, usedMax (the most connections in use at any point of the window), pendingMax (the most requests waiting at any point), at (the worst point: the most waiting, then the most in use). There are no traces, because the finding comes from the pool’s metrics rather than from a span.

pendingMax above zero means threads waited for a connection, and usedMax equal to max means the pool was full. A pool at its limit has two usual causes.

What to do:

  • Connections not returned. A connection taken outside a try-with-resources, a transaction left open, or a stream consumed after the transaction that produced it ended. In Spring, work that opens a transaction and then makes a slow HTTP call holds a connection for the length of that call, so move the call outside the transactional method.

  • A pool too small for the concurrency. Rule out connections not returned before raising the pool’s maximum, because a leak exhausts a pool of any size.

  • spring.datasource.hikari.leak-detection-threshold=20000 proves which of the two it is: it logs the stack that took a connection and did not give it back, and those log lines are in logs.

A pool at its limit also shows as endpoints whose p95 rises while their dbMsPerRequest does not, because the wait happens before any database span starts.

gc-pause

One collector, jvm.gc.duration per jvm.gc.name and jvm.gc.action, with a point in the window whose longest single collection is at least slow.request.ms, or whose collections summed over the export interval take at least 10% of that interval. It is high for a single collection over the threshold, and medium otherwise. The impact it is ranked by is the longest collection. Under cumulative temporality, the OpenTelemetry Java agent’s default, a point’s max is the longest collection since the JVM started, so a point stands for a collection in its own interval only when its max rose above the previous point’s; otherwise its interval’s mean collection counts, and the window’s first point is only the baseline. A collector that several processes of the service export, an application and its tests, is one finding: each process’s series is differenced on its own, and the worst interval of any of them is the collector’s.

subject.jvm is gc:<name>, and the title is <gc> paused for <worstMs>.

numbers: gc, action, worstMs (the longest single collection), shareMax (from 0 to 1, the worst export interval’s collection time over its length), collections (over the window), at. There are no traces and no code.

A collection over slow.request.ms stops every application thread for that long, so a request running at that moment is slower by the length of the pause, whatever its own code does. shareMax covers the other case: many short collections that add up to a tenth of an export interval are a throughput problem even when no single one is long. Read this finding before the slow-endpoint findings of the same window.

What to do:

  • Allocate less per request before touching any flag. The usual sources are a whole result set loaded to return a page of it, a string built by concatenation in a loop, and a response serialised into memory rather than streamed.

  • Then the heap. A young generation too small for the allocation rate collects constantly, and -Xmx and -Xms set to the same value avoid the resizing pauses of a growing heap. On a development machine the cause is often a JVM started with a small heap, such as 256 MiB, for a workload that needs more.

findings --since=start after the change says whether the pauses are gone. A heap-pressure finding beside this one says the heap is the cause rather than the allocation rate.

heap-pressure

Heap jvm.memory.used, summed over the heap pools, at 90% or more of the heap jvm.memory.limit at any point in the window. It is always high, and the impact it is ranked by is the highest ratio. A heap is one process’s: when several processes export the service, each one’s used heap is held against its own limit, and the process with the highest ratio is the finding, so an application’s full heap beside its tests' empty one is pressure.

subject.jvm is heap, and the title is heap at <ratio>% of its limit.

numbers: usedMax, limit (both in bytes), ratioMax (from 0 to 1), at (the worst point). There are no traces.

A JVM whose heap is at 90% of its limit is about to spend its time collecting, and a gc-pause finding usually appears in the same window. The fix depends on what the heap is holding.

What to do:

  • Something unbounded: a cache with no maximum, a list of everything read in a loop, a ThreadLocal never cleared, a collection on a long-lived object that only grows. Give the cache a maximum size and an expiry.

  • A whole result set in memory: streaming, paging or a projection keeps less of it in memory at once.

  • A heap genuinely too small for the workload, which is to be concluded only after the other two are ruled out.

The JVM page's memory chart over the same window says which of the three it is. A sawtooth that returns to the same floor is allocation, and a floor that climbs is something being retained.

thread-growth

jvm.thread.count at the last point of the window at least 50 above the first point, or at least twice it when the first point is 20 or more. It is always medium, and the impact it is ranked by is last minus first. The first and the last point are each process’s own: when several processes export the service, a test JVM that starts beside the application grows no thread of either, and the process that grew most is the finding.

subject.jvm is threads, and the title is threads grew from <first> to <last>.

numbers: first, last, max, at (the last point). There are no traces.

A thread count that only rises means threads are started and never stopped, and each thread reserves memory for its stack. A count settling at a new plateau after a burst is a pool that grew to its maximum, which is fine. A count that climbs for as long as the window does is a leak.

What to do:

  • Something created per request that should be created once: an executor, an HTTP client, a scheduler, a connection pool.

  • A pool with no bound, which grows until the machine refuses to create more threads. Give it a maximum and a queue.

  • Threads parked rather than finished, waiting on a call with no timeout.

The JVM page's thread chart shows whether the count plateaus or climbs, and findings --since=start after the fix says whether it still climbs.

Acknowledgements

An acknowledgement moves a known finding to the bottom of the list without removing it. A report endpoint that is slow by design, or a query that will stay slow until the schema changes, otherwise sits at the top of every answer and hides the new problem under it.

java -jar spider-sense.jar ack slow-endpoint:786b594455d0 --note="slow by design until the schema changes"
java -jar spider-sense.jar unack slow-endpoint:786b594455d0
java -jar spider-sense.jar findings --hide-acked

The id is the one findings printed in its id column. Finding ids are stable across windows, so one acknowledgement keeps holding as the window moves. unack with no such acknowledgement exits 4.

ack and resolve accept only an id a finding has:

  • An id that is not <kind>:<12 hex digits>, with a kind of The kinds other than regression, is a usage error: exit 2, and 400 over HTTP. The message says that the id column of findings is what they take, and that the ids of errors, queries and endpoints are table ids, not finding ids.

  • A well-formed id that names no finding of the data the store keeps, which is the retention window, is refused as not found: No finding <id> in the data kept (the last 24 hours), exit 4, and 404 over HTTP. An id names a finding when the store holds its subject, the thing the id is the hash of, and no rule runs: the id findings printed for a short window has to be taken whatever the rules say over the whole day, and the check costs one read of the subjects rather than a findings over the whole retention window.

    • A kind judged by a statistic of the window (slow-query, slow-endpoint, slow-job, slow-external, pool-exhausted, heap-pressure, thread-growth) needs the query group, endpoint, job, outbound target, pool or service. A p95 over the day is diluted by the day’s fast calls, so an endpoint slow for the last five minutes is a finding of those five minutes and not of the day, and its id is still one to acknowledge or resolve.

    • A kind judged by a count (error, log-error, n-plus-one, n-plus-one-http, gc-pause) needs the error group, the logger and normalised message of an ERROR record, the endpoint and the statement group or call of a trace that ran it five times or more, or the collector and its action.

Neither records anything, and MCP’s resolve tool fails with the same message.

An acknowledged finding is ranked after every other finding, in the same order among themselves. In the text rendering its severity column reads acked, and the heading counts them, as in (… 12 requests, 2 acked). hideAcked=true on /api/findings and on the MCP findings tool, and --hide-acked on the command, leave them out altogether.

An acknowledgement is one row, (finding_id, at, note) in the ack table (Storage), kept until it is withdrawn or the data is cleared. Acknowledging again replaces the row, and so does acknowledging a resolved finding. ack and unack write in the direct-file path as mark does. An acknowledgement is a decision of whoever reads the findings, not part of the collected data, so an export leaves it out.

Over HTTP:

  • POST /api/findings/{id}/ack with { "note": "…" | null } answers 201 with { "findingId": "…", "at": …, "note": … }, 400 for an id that is not a finding id, and 404 for one no finding has.

  • DELETE /api/findings/{id}/ack answers 204, or 404 when there is no such acknowledgement; a resolution is not one.

  • GET /api/acks answers { "acks": [ … ] }, newest first; resolutions are not listed.

Every finding carries "ack": { "at": …, "note": "…" | null } | null.

An acknowledged finding that recurs stays acknowledged, because a recurrence is exactly what was accepted. To be told when a finding comes back, resolve it instead.

check does not look at acknowledgements. Its rules are explicit thresholds, so an acknowledged n-plus-one is still an N+1 to maxNPlusOne. Never acknowledge a finding to make check pass.

Resolutions

A resolution records that a finding was fixed, so that findings reports it as a regression if it occurs again. It differs from an acknowledgement in one thing only: a resolved finding that recurs becomes a regression, ranked above everything else, while an acknowledged one stays acknowledged and dimmed.

java -jar spider-sense.jar resolve n-plus-one:4c5f46be8bc5 --note="fetch join"
java -jar spider-sense.jar unresolve n-plus-one:4c5f46be8bc5

Resolve a finding once check over the run after the fix passes, with what the fix was as the note. resolve takes the ids ack takes and refuses the others the same way (Acknowledgements). unresolve with no such resolution exits 4.

A resolved finding in a later window is one of two things:

  • Back, when the resolution is older than the window’s first occurrence of the finding. It is reported as a regression (regression) instead of its own kind: severity high, first in the ranking, with the same id. A resolution older than the window makes every occurrence in the window a recurrence. A resolution inside the window runs the rules again over the part of the window after it, for the finding’s service, and the finding is back when they find it there. The numbers and traces the regression carries are those of that part, so the traffic from before the fix does not count.

  • Not back, when every occurrence in the window precedes the resolution. The finding keeps its kind and carries its resolution, and it is ranked after every other finding, with the acknowledged ones. In the text rendering its severity column reads resolved, and the heading counts it, as in (… 12 requests, 2 acked, 1 resolved). --hide-acked, and hideAcked=true, leave it out as well.

A resolution is the same row in the ack table as an acknowledgement, with its resolved flag set, so a finding has one decision recorded at a time. Resolving an acknowledged finding, resolving again, or acknowledging a resolved one, replaces the row. resolve and unresolve write in the direct-file path as mark does, and MCP has the resolve tool (The tools). Resolutions are not exported, as acknowledgements are not.

Over HTTP:

  • POST /api/findings/{id}/resolve with { "note": "…" | null } answers 201 with { "findingId": "…", "at": …, "note": … }, 400 for an id that is not a finding id, and 404 for one no finding has.

  • DELETE /api/findings/{id}/resolve answers 204, or 404 when there is no such resolution; an acknowledgement is not one.

Every finding carries "resolution": { "at": …, "note": "…" | null } | null. The JSON of the list counts the resolved findings that have not come back as resolved, beside acked.

check counts regressions with --max-regressions, which is 0 in the default rule set (Check). A regressed n-plus-one still counts to --max-n-plus-one as well, and a regressed log-error to --max-log-errors.

State

Every finding carries a state, so findings --since=start says on its own which findings the last change introduced. It is the automatic counterpart of compare, which needs two marks.

State When

regressed

The finding is a regression.

new

The rules do not find it in the previous run of its service.

ongoing

The rules find it in the previous run of its service too.

The previous run of a service is the time between its two newest start marks at or before the end of the window, from the older of the two up to the newest one. Spider Sense writes a start mark each time the service starts with a new process id (Automatic start marks). With one start mark the previous run reaches back to the oldest data. With none, as for an imported session without marks or a sender that reports no process.pid, there is no previous run, and every finding of the service is new.

The rules run over the previous run once per service of the answered page, with the same thresholds as over the window. Only the rules of the kinds the service has on the page run there, and they produce ids alone: no evidence traces, no code frames, no hotSpans and no schema block, since the state asks only whether an id is there. A previous run no longer changes once a newer start mark closes it, so a running Spider Sense keeps the ids it found for each service, previous run and kind in memory, and every later answer over the same run reads them from there instead of running the rules again. The first answer after a restart therefore runs the rules over the previous run as well as over the window, and the answers after it run them over the window alone.

In the text rendering, state is a column between severity and kind. check does not read the state.