Findings

A finding is one thing worth fixing, found by rules over a window.

GET /api/findings?since&until&service&limit=20&hideAcked=false&format=text|json answers them, and the CLI command is:

java -jar spider-sense.jar findings [--since=] [--until=] [--service=] [--limit=20] [--hide-acked] [--json] [--full]

The rules

Findings are computed over the window, not stored, so the same rules over the same window always produce the same list. Time selectors define the window, which is the last 15 minutes when nothing says otherwise.

Findings are ranked by severity (high before medium before low), then by kind, then by impact within a kind, then by id, so the list is stable between two calls over the same data. No rule produces low today, and the value is reserved for gentler kinds. The kind order is error, n-plus-one, slow-query, slow-endpoint, slow-job, slow-external, log-error, pool-exhausted, gc-pause, heap-pressure, thread-growth. Each kind below names the impact number it is ranked by.

The list is bounded. limit defaults to 20 and is at most 100.

Two thresholds decide the slow kinds: slow.request.ms is 500 milliseconds by default and slow.query.ms is 100. status prints the ones in force, and Configuration says how to change them. An endpoint that spidersense.ignore.endpoints excludes is not an entry span and produces no finding of any kind.

What every finding carries

Field How to read it

id

The kind plus 12 hex characters over the kind, the service and the subject. It is stable across windows, so the same problem keeps its id between runs, and it is what ack takes.

kind

One of the eleven kinds below.

severity

high, medium or low, and acked in the text rendering for an acknowledged finding.

title

One line, the claim.

why

The numbers that justify the claim, in a sentence. Quote this rather than restating it.

subject

The endpointId, queryId, errorId, pool, job, target, logger or jvm to pass to endpoints, queries, errors, logs or the API.

numbers

Kind-specific, listed with each kind below.

statement

The statement as the OpenTelemetry agent sanitised it, literals already ?, cut at 200 characters unless --full is given.

code

Application frames, most specific first, at most 5, and empty when none is known.

traces

At most 3 trace ids: the evidence, and what trace <id> opens.

ack

The acknowledgement, { "at": …, "note": … }, or null when the finding is not acknowledged.

traces are the three slowest traces for the slow-* kinds, the three newest for error and log-error, the three most recent affected for n-plus-one, and none for pool-exhausted and the three JVM kinds.

hotSpan is where the time went in the finding’s first evidence trace, and the slow-endpoint and slow-job kinds carry it. It is the span with the largest self time, that time, and its share of the trace’s duration, and it reads as one line: hot span: SELECT order_line · 312.4 ms self · 62.0%. It is null only when the finding has no trace.

Code locations

The OpenTelemetry Java agent does not record where a span was started from, so code comes from three sources. The first is the exception.stacktrace of an error. The second is the code.function and code.namespace attributes that a few instrumentations set, which is where a slow-job finding’s frames come from. The third is the code.stacktrace that Spider Sense’s own agent extension captures, on every database span slower than slow.query.ms, on the fifth repeat of a statement within one trace, and on every non-database CLIENT span slower than slow.request.ms.

Those extension frames are the truest of the three, because they are the span’s own thread at the moment the span finished rather than a guess from an attribute. code is therefore reliable for error, slow-query, n-plus-one and slow-external findings, often present for log-error, and often empty for the others. When it is empty, open a trace from traces and read the tree, which names the endpoint and the statement even when it cannot name the line.

A stack trace is reduced to its application frames by dropping known framework prefixes: java., javax., jdk., sun., com.sun., jakarta., org.springframework., org.hibernate., org.eclipse.jetty., org.apache., io.opentelemetry., com.zaxxer., org.h2., net.benelog.spidersilk., kotlin., scala., reactor., io.netty., ch.qos.logback., org.slf4j., org.junit. and gg.jte.. When that heuristic guesses wrong, spidersense.app.packages=com.acme,org.acme replaces it with an allowlist.

The text rendering

The ranked table comes first, and the fields follow as one numbered block per row, in the same order. A block is <n>. <id> — <why>, then the numbers on one line, then the hot span line when the kind has one, then the statement when there is one, then the code frames one per line, then the trace ids.

$ java -jar spider-sense.jar findings --since=before
# findings  2026-09-17T08:19:28+09:00 → 08:19:34  (6s, all services, 21 requests)

| # | severity | kind | id | service | title |
| --- | --- | --- | --- | --- | --- |
| 1 | high | error | error:eecf9878a68f | spring-orders | IllegalStateException in GET /api/flaky |
| 2 | medium | n-plus-one | n-plus-one:4c5f46be8bc5 | spring-orders | GET /api/orders/{id}/enriched runs SELECT product 5 times per request |
| 3 | medium | n-plus-one | n-plus-one:e6d97708ab26 | spring-orders | GET /api/orders/{id} runs SELECT product 5 times per request |
| 4 | medium | slow-query | slow-query:6b3aae6f9bef | spring-orders | SELECT p.id AS product_id, p.sku AS sku, p.name AS name, SUM… is slow |
| 5 | medium | slow-query | slow-query:aae3ff87821a | spring-orders | SELECT o.status AS status, CAST(o.created_at AS DATE) AS ord… is slow |
| 6 | medium | slow-endpoint | slow-endpoint:786b594455d0 | spring-orders | GET /api/reports/revenue is slow |

1. error:eecf9878a68f — 1 occurrence in GET /api/flaky; Payment gateway timeout
   count 1, firstSeen 2026-09-17T08:19:28.962+09:00, lastSeen 2026-09-17T08:19:28.962+09:00, type java.lang.IllegalStateException, message Payment gateway timeout, endpoints [name GET /api/flaky count 1]
   orders.web.MiscController.flaky(MiscController.java:24)
   traces: 5e6769395a77c8f192949af91cbdedf2

2. n-plus-one:4c5f46be8bc5 — 1 of 3 requests repeated it; 5 times; 0.4 ms per request in that statement
   requests 3, affected 1, medianRepeats 5, maxRepeats 5, msPerRequest 0.4
   select p1_0.id,p1_0.name,p1_0.price,p1_0.sku from product p1_0 where p1_0.id=?
   traces: e7cfa77cb4c9135b623a04e470ef58d0

3. n-plus-one:e6d97708ab26 — 1 of 3 requests repeated it; 5 times; 0.8 ms per request in that statement
   requests 3, affected 1, medianRepeats 5, maxRepeats 5, msPerRequest 0.8
   select p1_0.id,p1_0.name,p1_0.price,p1_0.sku from product p1_0 where p1_0.id=?
   traces: 0936e867b3ba9250964b7efa08029ee4

4. slow-query:6b3aae6f9bef — p95 269.7 ms over 3 calls, 3 of them over 100 ms; 782.8 ms in total
   calls 3, slowCalls 3, p50Ms 266.6, p95Ms 269.7, maxMs 269.7, totalMs 782.8, callers [endpoint GET /api/reports/revenue service spring-orders calls 3]
   SELECT p.id AS product_id, p.sku AS sku, p.name AS name, SUM(l.quantity) AS quantity, SUM(l.quantity * l.unit_price) AS revenue FROM order_line l JOIN product p ON p.id = l.product_id JOIN orders o ON…
   traces: 42800a1fc7ca5ee2dec08f804cef8be6 a86465f5a9242ac8496577740771fcb7 cbc8be4f6ec5a1c033e57cbb7f269bd5

5. slow-query:aae3ff87821a — p95 200.2 ms over 3 calls, 3 of them over 100 ms; 476.6 ms in total
   calls 3, slowCalls 3, p50Ms 141.3, p95Ms 200.2, maxMs 200.2, totalMs 476.6, callers [endpoint GET /api/reports/revenue service spring-orders calls 3]
   SELECT o.status AS status, CAST(o.created_at AS DATE) AS order_day, SUM(o.total) AS revenue, COUNT(DISTINCT o.id) AS order_count, SUM(l.quantity) AS item_count FROM orders o JOIN order_line l ON l.ord…
   traces: 42800a1fc7ca5ee2dec08f804cef8be6 a86465f5a9242ac8496577740771fcb7 cbc8be4f6ec5a1c033e57cbb7f269bd5

6. slow-endpoint:786b594455d0 — p95 549.8 ms over 3 calls; 2.0 database calls and 419.8 ms per request, 93.0% of the time
   calls 3, p50Ms 411.5, p95Ms 549.8, maxMs 549.8, totalMs 1,354.9, apdex 0.833, dbCallsPerRequest 2.0, dbMsPerRequest 419.8, dbShare 93.0%
   traces: 42800a1fc7ca5ee2dec08f804cef8be6 a86465f5a9242ac8496577740771fcb7 cbc8be4f6ec5a1c033e57cbb7f269bd5

The kinds

n-plus-one

The same query group runs 5 or more times under the same entry span in one trace, aggregated per endpoint and query group over the window. It is high when the repeats reach 20 or their summed time exceeds slow.request.ms, and medium otherwise. The impact it is ranked by is affected requests multiplied by median repeats.

numbers: requests (entry spans of the endpoint in the window), affected (of them, how many repeated), medianRepeats, maxRepeats, msPerRequest (the summed time of the repeated statement, per affected request).

Read it as: this endpoint ran that statement medianRepeats times in one request, and that cost msPerRequest. affected well below requests means only some inputs trigger it, which usually points at a branch or a lazily loaded collection that is only touched sometimes. code is the call site of the fifth repeat, so it names the line that issues the repeated statement. It is empty only when the application ran without the extension, and the trace tree then names the loop.

What to do:

  • JPA or Hibernate: load the association with its parent, with a fetch join (select distinct o from Order o join fetch o.lines) or an entity graph (@EntityGraph(attributePaths = "lines")).

  • When a fetch join would multiply rows across two collections, set batch loading instead: @BatchSize(size = 50) on the collection or the entity, or spring.jpa.properties.hibernate.default_batch_fetch_size=100 globally.

  • spring-jdbc or plain JDBC: replace the explicit loop with one query over the whole key set, where order_id in (:ids), and group the rows in memory. Chunk a very large key set rather than sending one enormous IN.

  • An IN list with a varying number of parameters produces a new statement text per size, so Spider Sense groups those calls separately.

After the fix, compare should show the query’s calls/req down to 1 and the endpoint’s verdict better, and check --max-queries-per-request= locks it in.

slow-query

A query group whose p95 exceeds slow.query.ms. It is high when the p95 exceeds ten times the threshold, and medium otherwise. The impact it is ranked by is total time.

numbers: calls, slowCalls, p50Ms, p95Ms, maxMs, totalMs, callers (the endpoint, service and call count of the nearest entry span up the parent chain).

A p95 far above p50 is a query that is fast for most inputs and slow for some, which usually means a plan that degrades with the data rather than a statement that is always wrong. callers says which endpoints pay for it, and the trace ids give the individual calls.

What to do:

  • Add the index the predicate needs, and remember that a leading wildcard (like '%graph%') defeats one. Search that genuinely needs an infix match belongs in a search index.

  • Rewrite the shape: a correlated subquery per row becomes a join, an unbounded select count(*) becomes a bounded count, and an order by over an unindexed column becomes an indexed one.

  • Stop selecting what is not needed: a projection instead of an entity, or named columns instead of select *.

  • Page when the endpoint does not need the whole result: Pageable in JPA, limit/offset or a keyset predicate in SQL.

Re-run the same exercise and read the query’s own p95 and total columns in compare, not only the endpoint’s.

slow-endpoint

An endpoint whose p95 exceeds slow.request.ms. It is high when the p95 exceeds four times the threshold, which is the "frustrated" bound of the Apdex, and medium otherwise. The impact it is ranked by is total time.

numbers: calls, p50Ms, p95Ms, maxMs, totalMs, apdex, dbCallsPerRequest, dbMsPerRequest, dbShare (the part of the endpoint’s total time spent in database spans of the same trace and service, printed as a percentage), hotSpan.

dbShare is the fork in the road.

  • High dbShare means the endpoint is not the problem and its queries are. Look for an n-plus-one or slow-query finding with this endpoint among its callers, and fix it there.

  • Low dbShare means the time is elsewhere. hotSpan names the span that holds it, and the slowest trace from traces shows the rest. A CLIENT span that dominates the duration is the answer, and so is a long stretch with no child spans at all, which is the endpoint’s own code.

What to do:

  • A CLIENT span whose child SERVER span is nearly as long moves the question to the other service, and the same loop applies there with --service= set to it.

  • A CLIENT span much longer than the SERVER span it wraps is connection setup, queueing or serialisation, not the callee.

  • For the endpoint’s own code, look for work done per request that could be done once, such as compiling a pattern, building a client or reading a file. An unbounded collection built in memory and a sleep are the other usual suspects.

  • An endpoint that fans out to several independent calls can run them together rather than in sequence. The trace tree shows sequential CLIENT spans laid end to end when it does not.

check --endpoint="GET /orders/report" --max-p95-ms= scopes the verdict to the one endpoint being worked on.

slow-job

A job whose p95 exceeds slow.request.ms, grouped by service and span name. A job is a root INTERNAL span: a scheduled method, an @Async call, a batch step. Severity and impact are the same as slow-endpoint.

numbers: runs, p50Ms, p95Ms, maxMs, totalMs, dbCallsPerRun, dbMsPerRun, dbShare, hotSpan.

A job is work the application did to itself, so it is never an entry span. It is in no request count, in no Apdex and in no check verdict, and this finding is the one place a slow scheduler tick or batch step is reported.

dbShare forks the same way it does for slow-endpoint. High means the job’s queries are the problem, and a job is where an N+1 hides best, because nobody is waiting for it. Low means the time is the job’s own code, and the slowest trace shows the loop or the call that holds it.

What to do:

  • Run one query for the whole batch instead of one query per row.

  • Batch or move out of the tick a CLIENT call that dominates the trace.

  • A job that has grown slower than its own interval overlaps with itself, and runs with p95Ms over a few minutes say whether it has.

check says nothing about a job, so verify a fix with compare over the same exercise, or with findings --since=start after a restart.

slow-external

An outbound HTTP call whose p95 exceeds slow.request.ms: CLIENT spans of category http, grouped by service, target and span name. The target is what the service map calls the dependency, such as localhost:8081, so one row is one thing this service calls rather than every outbound call it makes. Severity and impact are the same as slow-endpoint.

numbers: calls, errors, p50Ms, p95Ms, maxMs, totalMs, callers.

First decide whose problem it is, by opening the slowest trace from traces. When the callee is traced too, its SERVER span is in the same tree. A SERVER span nearly as long as the CLIENT span moves the question to that service. A CLIENT span much longer than the SERVER span it wraps is connection setup, queueing or serialisation on this side.

What to do:

  • Stop making the call per request when the answer barely changes: cache it, or fetch it once at startup.

  • Make one call instead of N. A call inside a loop is the N+1 of the network, and it is worse than the database’s because the latency is a round trip.

  • Run independent calls together rather than end to end.

  • Reuse the client. A RestClient or HttpClient built per request builds a connection pool per request.

  • Set timeouts. A call with no timeout turns the callee’s bad minute into this service’s bad minute.

error

An error group with at least one occurrence in the window. It is always high, and the impact it is ranked by is the occurrence count.

numbers: count, firstSeen, lastSeen, type, message (normalised, digits replaced by ?), endpoints (name and count).

The group is the service, the exception type and the normalised message, so one row is one failure mode rather than one occurrence. code is at its best here, so the top application frame is where to look. traces are the three newest occurrences, and trace <id> shows the exception span with its message and frames and the trace’s log lines underneath. logs --trace=<traceId> gives the full log for that request.

What to do, after reading the top frame:

  • A bug: the code cannot handle an input it will receive. Fix the code, and add the test that reproduces it.

  • An expected condition answered as a failure, such as a business rule violated or an entity not found. It should still be visible, but it should answer the client properly rather than escape as a 500. An @ExceptionHandler that answers 409 Conflict is the shape.

  • A dependency failing: the trace shows the CLIENT span that failed first.

A message that carries the identifier (Order 42 is already shipped) still groups correctly, because Spider Sense normalises the digits away. check --max-errors=0 over a window that starts at the restart locks the fix in.

log-error

Log records of severity ERROR or above, grouped by service, logger and normalised message, counting only the uncovered ones: records without a trace id, or with a trace id whose trace has no error span. A group with at least one uncovered record in the window is a finding. It is always high, and the impact it is ranked by is the uncovered count.

numbers: count (the uncovered records), firstSeen, lastSeen, logger, message (normalised), endpoints (the entry span of each record’s trace, and (no endpoint) for a record without one).

This is what catch (Exception e) { log.error(…, e); return fallback; } leaves behind: no span error, no exception event, one line in the log, and a request that answered 200 with the wrong answer. A record whose trace has an error span is already reported by an error finding and is not counted twice, so a log-error is by construction the failure nothing else tells you about. code comes from the record’s exception.stacktrace when the logging bridge exported a throwable. logs --severity=ERROR --q=<logger> lists the records themselves.

What to do, after reading the top frame:

  • A real failure being swallowed. Let it fail, or answer the client properly, so the failure is visible where it is decided rather than only in a log.

  • An expected condition logged as an error, such as a validation failure, a cache miss or a retry that then succeeded. Log it at WARN or INFO, because ERROR should mean "somebody has to look".

  • A dependency failing: the trace, when there is one, shows the CLIENT span that failed first.

  • When the fallback is the right behaviour, keep it and make it visible: record it as a counter, or set the span’s status, so the endpoint’s numbers say how often the degraded path was taken.

check --max-log-errors=0 over a window that starts at the restart locks the fix in.

pool-exhausted

A JDBC pool with a point in the window where pending requests are above zero, or used equals max. It is always high, and it is ranked by pending first and used second.

numbers: pool, max, usedMax, pendingMax, at (the worst point). There are no traces, because this comes from the pool’s metrics rather than from a span.

pendingMax above zero means threads waited for a connection, and usedMax equal to max means the pool was full. A pool at its limit is a symptom, and there are two causes worth considering.

What to do:

  • Connections not returned. A connection taken outside a try-with-resources, a transaction left open, or a stream consumed after the transaction that produced it ended. In Spring, work that opens a transaction and then makes a slow HTTP call holds a connection for the length of that call, so move the call outside the transactional method.

  • A pool too small for the concurrency. This is real, but check the first cause before raising the ceiling, because a leak will exhaust any size.

  • spring.datasource.hikari.leak-detection-threshold=20000 proves which of the two it is: it logs the stack that took a connection and did not give it back, and those log lines are in logs.

A pool at its limit also shows as endpoints whose p95 rises while their dbMsPerRequest does not, because the wait happens before any database span starts.

gc-pause

One collector, jvm.gc.duration per collector name and action, with a point in the window whose longest single collection is at least slow.request.ms, or whose collections summed over the export interval take at least 10% of that interval. It is high for a single collection over the threshold, and medium otherwise. The impact it is ranked by is the longest collection.

numbers: gc, action, worstMs (the longest single collection), shareMax (the worst export interval’s collection time over its length), collections (over the window), at. There are no traces and no code.

A collection over slow.request.ms stops every application thread for that long, so a request unlucky enough to be running looks slow for no reason of its own. shareMax is the other half of the question: many short collections that add up to a tenth of the wall clock are a throughput problem even when no single one is long. Read this finding before the slow-endpoint findings of the same window.

What to do:

  • Allocate less per request before touching any flag. The usual sources are a whole result set loaded to return a page of it, a string built by concatenation in a loop, and a response serialised into memory rather than streamed.

  • Then the heap. A young generation too small for the allocation rate collects constantly, and -Xmx and -Xms set to the same value avoid the resizing pauses of a growing heap. On a development machine the honest fix is often that the JVM was given 256 MiB and the workload wants more.

findings --since=start after the change says whether the pauses are gone. A heap-pressure finding beside this one says the heap is the cause rather than the allocation rate.

heap-pressure

Heap jvm.memory.used, summed over the heap pools, at 90% or more of the heap jvm.memory.limit at any point in the window. It is always high, and the impact it is ranked by is the highest ratio.

numbers: usedMax, limit (both in bytes), ratioMax, at (the worst point). There are no traces.

A heap at 90% of its limit is a run that is about to spend its time collecting, and it usually arrives with a gc-pause finding. The question is what is being held.

What to do:

  • Something unbounded: a cache with no maximum, a list of everything read in a loop, a ThreadLocal never cleared, a collection on a long-lived object that only grows. Give the cache a maximum size and an expiry.

  • A whole result set in memory: streaming, paging or a projection turns the peak into a plateau.

  • A heap genuinely too small for the workload, which is the last thing to conclude rather than the first.

The JVM page's memory chart over the same window says which it is. A sawtooth that returns to the same floor is allocation, and a floor that climbs is something being retained.

thread-growth

jvm.thread.count at the last point of the window at least 50 above the first point, or at least twice it when the first point is 20 or more. It is always medium, and the impact it is ranked by is last minus first.

numbers: first, last, max, at (the last point). There are no traces.

Threads that only go up are threads nobody is stopping, and each one costs a stack. A count settling at a new plateau after a burst is a pool that grew to its maximum, which is fine. A count that climbs for as long as the window does is a leak.

What to do:

  • Something created per request that should be created once: an executor, an HTTP client, a scheduler, a connection pool.

  • A pool with no bound, which grows until the machine says no. Give it a maximum and a queue.

  • Threads parked rather than finished, waiting on a call with no timeout.

The JVM page's thread chart shows whether the count plateaus or climbs, and findings --since=start after the fix says whether it still climbs.

Acknowledgements

An acknowledgement takes a known finding out of the way without hiding it. A report endpoint that is slow by design, or a query that will stay slow until the schema changes, otherwise sits at the top of every answer and hides the new problem under it.

java -jar spider-sense.jar ack slow-endpoint:786b594455d0 --note="slow by design until the schema changes"
java -jar spider-sense.jar unack slow-endpoint:786b594455d0
java -jar spider-sense.jar findings --hide-acked

The id is what findings printed, and finding ids are stable across windows, so one acknowledgement keeps holding as the window moves. unack with no such acknowledgement exits 4.

An acknowledged finding is ranked after every other finding, in the same order among themselves. In the text rendering its severity column reads acked, and the heading counts them, as in (… 12 requests, 2 acked). hideAcked=true on /api/findings, and --hide-acked on the command, leave them out altogether.

An acknowledgement is one row, the finding id with an instant and a note, kept until it is withdrawn or the data is cleared. Acknowledging again replaces the row. Over HTTP it is POST /api/findings/{id}/ack with { "note": "…" }, DELETE /api/findings/{id}/ack, and GET /api/acks for the list, newest first. Acknowledgements are the reader’s rather than the session’s, so they are not exported.

check does not look at acknowledgements. Its rules are explicit thresholds, so an acknowledged n-plus-one is still an N+1 to maxNPlusOne. Never acknowledge a finding to make check pass.