The Pages
The sidebar lists twelve pages, each answering one question about the window the top bar selects. Every one of them obeys the service filter and the time range of the top bar. A service, endpoint, query or error page opened from them obeys the time range, and shows its own service in the filter, where choosing another service leads off it. The colours, the numbers and the times follow Look and feel and Behaviour and quality.
Overview
#/ answers whether anything is wrong right now.
It has five parts.
With a service chosen in the top bar, the tiles, the chart, the Response summary, the findings and the tingles are that service’s, from GET /api/overview?from&to&service (Overview); the Services panel still lists every service.
-
Stat tiles, in one row: requests, Apdex, error rate, p50, p95, p99, and rps. Each tile has the number large, the unit small, and a one-word caption. A tile takes a colour only when its value crosses a threshold. An error rate above 1% is
--err, a p95 above the slow threshold is--warn, and Apdex is coloured as Response-time buckets and Apdex says:--errbelow 0.7 and--warnbelow 0.85. The Apdex tile reads, for example,0.93, and its title names the T it was computed with. Every tile but Apdex is a link to the page behind its number, with the top bar’s filter and range: requests and rps to the Traces page, the error rate to the Errors page, and p50, p95 and p99 to the Scatter. -
Findings: the top five of
GET /api/findings?from&to&service&limit=5&hideAcked=true, the list the CLI’sfindingscommand prints (Findings). A row carries a severity dot (--errforhigh,--warnformedium,--text-mutedforlow), the kind as a chip, the service chip, the title in bold, and the finding’swhysentence dimmed on a second line. The firstcodeframe follows in monospace when the finding has one. A row click goes where the finding points:-
the endpoint page for an
endpointId; -
the query page for a
queryId; -
the error page for an
errorId; -
the JVM page for a
poolor ajvm; -
the Logs page with
severity=ERRORandqset to the logger, for alogger; -
the first evidence trace for a
jobor atarget, since neither has a page of its own; -
the first evidence trace when there is nothing else.
The panel loads on its own: the rest of the page paints as soon as
/api/overviewanswers, and the findings, which can take longer over a long window, fill the panel when they come. Until they come the panel holds its place in the page, so the chart and everything under it stay where they are. It shows a skeleton of five rows, each as tall as a row with a code frame, clipped to the height the panel’s last answer took at this width of the window, whichlocalStorageremembers, when there is one. An answer shorter than the place held shrinks the panel once and moves the page up, never down. With Live on the panel keeps the height it has across a refresh, so a tick that brings fewer rows moves nothing, for a reader at the page’s end too. The rows keep their own height, and the rest of the panel stays empty. Turning Live off, a change of the top bar’s service or range, or a reload lets the panel take the answer’s height again. A taller answer grows the panel, since a code frame is not cut to fit. For a reader scrolled past the panel the scroll position moves by the same amount, so what is being read stays put, in a browser without scroll anchoring too. A reader who can still see the panel is not moved, and the page corrects nothing after turning Live off, a change of the top bar’s service or range, or a reload. A failure of theirs puts the error box and its Try again in the panel, and leaves the rest of the page as it is. The panel head links to All findings (#/findings). Because the panel asks withhideAcked=true, acknowledged findings, and resolved ones that have not come back, are left out, so the five are always findings nobody has acknowledged or resolved, and regressions come first. With no finding in the window the panel saysNothing worth fixing in this window. -
-
Throughput and latency: a uPlot chart, 220 px high, with a Requests | Load toggle in the panel head, remembered in the hash query as
chart=requestsorchart=loadand carried by every internal link. Requests draws the requests per time bucket as bars in the silk grey, with the errors among them stacked on top in--err, so a bar is as tall as its requests. Load splits the same bars into the four response-time buckets, stacked bottom-up in the bucket colours, with the errors on top in--err. In both, p95 is a line on a second y-axis in the accent colour. Beside the chart, one third of the width, the Response summary draws five vertical bars, one per response-time bucket and one for errors, in the bucket colours, with the count above each bar and the label beneath. The bars take the height of the panel, which the chart sets. The tallest bar sets the scale, and a tooltip gives each bar’s share. With no request in the window the panel saysNo request in this window.Each bar is a link to the Traces page filtered to it:status=okwithminMsandmaxMsset to the bucket’s bounds for a response-time bucket,minMsopen as the bucket’s lower bound is, andstatus=errorfor the error bar. The bars count requests, so the link carriesby=entry, and the list holds the traces with a request in the bar (Traces). The bars are labelled from the bucket bounds,≤125 ms,≤500 ms,≤2 s,>2 s, anderror, and they keep those colours wherever a histogram or a load chart appears (Response-time buckets). Below 900 px the chart and the summary stack. -
Services: one card per service, with the name, the language chip, an
embeddedchip where it applies, rps, p95, error rate, Apdex, a sparkline (inline SVG, 120×28), and a JVM link when the service reports JVM metrics (hasJvm). A click opens the service page. The cards of the services with a request in the window come first, then those of the services seen in it without one, such as a worker. The services that sent nothing in the window are quiet, and fold under a2 quietbutton at the end of the panel, which opens to their cards and stays open across a refresh. With no service sending, nothing is folded, so the panel never holds that button alone. -
Tingles: the feed of slow requests, slow queries and errors, newest first. Repeats of one service, kind and title fold into one row, so one busy statement takes one row (Tingles). Each row has an icon for its kind (a turtle for a slow request, a database for a slow query, a bolt for an error), the service chip, a title, a detail, and a relative time. A row of several occurrences adds
× n, and its time, detail and trace are the newest occurrence’s. A click opens the trace. A tingle without a trace, as an imported one can be, is plain text: it takes no focus and no click. The panel is 420 px high at most, and the feed scrolls inside it.Chips in the panel head narrow the feed by kind, and look and act as the Findings page’s do: each carries its count of rows, which a press does not change, and several can be pressed. A value the URL names and no row holds appears as a pressed chip at
0. The chips filter on the client, and the choice lives in the hash query askind=error,slow-query, a key of the page’s own that no internal link carries. With no row of the pressed kinds the feed saysNo tingle of this kind in this window.With Live on, a new tingle folds into its row, which moves to the top with its count raised, and a tingle with no row yet is prepended with a 300 ms slide. With Live off the feed stays as the page loaded it, as the tiles and the chart do, and only the badge on the sidebar’s tingles entry counts the new ones.
Before any service has sent anything, the page shows the logo large, one sentence, and the content of the How to send data dialog inline: the -javaagent line, the OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL pair, and a curl for OTLP/JSON.
Once a service is known, the page shows its five parts even when the window holds no request, as for a worker whose jobs are not entry spans: the tiles that need a request read -, the chart stays empty, the Response summary says No request in this window., and the findings, services and tingles are listed as usual.
Findings
#/findings is the ranked list of the window’s findings: GET /api/findings?from&to&service&limit=100, in the order the API ranks it.
It shows in the browser the list the CLI’s findings command prints (Findings).
When the answer’s total is more than the rows, a line under the panel head says the limit cut the list and that the counts of the chips are over the rows shown, such as 100 of 154 findings; a service or a shorter range shows the rest. The counts are over these 100.
Chips above the table narrow the list by severity, state and kind.
Each chip carries its count over the rows the page holds, and that count does not move as other chips are pressed.
A chip is a toggle button the keyboard reaches.
A pressed chip is set in --accent-text on --accent-soft and carries a check mark, so colour is not the only sign of it.
Several chips can be pressed: a finding stays when it has one of the pressed values in every group with a chip pressed.
A value the URL names and no row holds appears as a pressed chip at 0, so it can be released.
Hide acknowledged and resolved is a chip of the same kind: it leaves out the acknowledged findings and the resolved ones that have not come back, and its count is how many of them the page holds.
The chips filter on the client, so a press asks the server for nothing.
The ranked order stays, and # keeps the finding’s rank in the full list.
The choice lives in the hash query as severity=high, kind=n-plus-one,slow-query, state=new and hideAcked=1, so a filtered list can be shared.
A group’s pressed values are separated by commas.
These keys are the page’s own: no internal link carries them, and a change of the top bar’s service, range or Live keeps them.
The table has the position (#), the severity as a dot and a word, the state chip, the kind chip, the service chip, the title, and in a right-aligned column the impact number the kind is ranked by.
That number is:
-
the original kind’s number for a
regression; -
countforerrorandlog-error; -
medianRepeats × affected, shown as42 × 3, forn-plus-oneandn-plus-one-http; -
totalMsforslow-query,slow-endpoint,slow-jobandslow-external; -
pendingMaxforpool-exhausted; -
worstMsforgc-pause; -
ratioMaxas a percentage forheap-pressure; -
lastminusfirstforthread-growth.
A number that is no duration or percentage carries its unit word, as in 12 times, 3 pending or +173 threads.
Each cell’s title says the number in words, since the column mixes units: 1,234 occurrences, 21 repeats (the median) × 228 affected requests, 72.7 s in total, 1 request waiting for a connection at the peak, +173 threads over the window.
Below 720 px the rows read as the Overview’s finding rows, and the table does not scroll sideways. A row has the severity dot with the position and the chevron under it, then the severity word and the state, kind and service chips, the title in bold, and the impact. The title wraps rather than being cut, and the column headers go out of sight, without leaving the table for a screen reader. The row is the same row at every width: it opens the same evidence in place.
A row click, or the chevron before the position, expands the row in place to show the evidence, in this order:
-
the
whysentence; -
the kind’s
numbersas a grid of keys and values, with durations, counts and rates formatted as everywhere else,callersandendpointsas a short list, andhotSpanas one line, as the CLI prints it; -
the
statement, pretty-printed, when there is one, ending in… (cut at 2,000 characters)when the store may have cut it (statementCut); -
the schema block under it;
-
the
codeframes as a monospace list; -
the
tracesas links to the trace page.
The evidence never widens the table: a value with nowhere to break, such as a class name, wraps inside its cell.
A slow-endpoint and a slow-job carry hotSpans and breakdown (Where the time went).
They are two rows of the numbers grid spanning its full width, rather than cells in it.
hot spans is a short list: each summary in monospace, with <selfMs> · <share> · ×<count> dimmed beside it.
breakdown is the same stacked bar the endpoint page draws, with the four shares written beside it.
Each code frame is a code frame: a link into your editor with its five source lines under it when it resolves, and plain monospace text when it does not.
The code frames are never folded, because a finding’s code is already its application frames only (Code locations).
The schema block is rendered as the text rendering renders it, in small monospace with the labels dimmed.
It has one line per table with the indexes it carries, such as indexes ITEMS: PRIMARY_KEY_8 (ID) unique, IDX_ITEMS_SUPPLIER (SUPPLIER_ID, NAME), with none for a table that carries none.
Then comes one line for the columns, such as predicates: items.name, items.category; unindexed: items.name.
The unindexed names are in --accent, behind the word unindexed that already says what they are.
predicates: none stands alone when the statement has no predicate, and unindexed: none appears when every predicate is served.
A finding whose schema is null shows nothing for it.
Nothing is computed in the page: the names are the ones the API carries, so they are the CLI’s.
The row also carries a link to the subject’s page, the same destination as a row click on the Overview, named by where it goes: Endpoint, Query, Error, JVM, Logs or Trace.
A finding whose subject names a statement as well as an endpoint, an n-plus-one, adds a second link, Query, to the statement’s page.
Then comes the Copy as Markdown and Copy CLI line pair over GET /api/findings/{id} and the findings command.
An expanded row stays expanded across a filter change and a Live refresh while the finding is still in the list, keyed by its id, so a row a filter hid opens again when the filter lets it back.
Its evidence follows the refresh: a part that changed, such as the numbers, is redrawn, and the other parts stay as they are.
A resolved finding that comes back as a regression therefore shows its new why at once, and its decision line drops not back since.
The state chip is the finding’s state (State), with a title saying what it means.
new is in --accent, since it is what the last restart introduced, ongoing is in the muted text colour, and regressed is in --err.
An acknowledged finding (ack not null, Acknowledgements) is listed after the others, dimmed, with acked in its severity cell and the note as its title.
A resolved finding that has not come back (resolution not null and a kind other than regression, Resolutions) is listed and dimmed the same way, with resolved in its severity cell.
A dimmed row sets its text in --text-muted and fades its dots, until the pointer is on it.
The expanded evidence of any finding ends with a Resolve button and an Acknowledge button.
Each opens a small dialog for an optional note, then sends POST /api/findings/{id}/resolve or POST /api/findings/{id}/ack.
An acknowledged finding shows the note and an Unacknowledge button (DELETE /api/findings/{id}/ack) instead.
A resolved finding, a regression included, shows the resolution’s instant, its note and a Reopen button (DELETE /api/findings/{id}/resolve), which withdraws the resolution.
The empty state says Nothing worth fixing in this window. and names the window, or shows the snippet when there was no request at all.
A filter that hides every row leaves one row saying No finding matches these filters. with a Clear the filters button, which releases every chip.
Compare
#/compare?before=<selector>&after=<selector>&until=<selector> puts two windows side by side, [before, after) and [after, until).
It shows in the browser the comparison the CLI’s compare command prints (Compare).
A bar above the content holds the Before and After selects, the Until select, and a Compare button.
Before and After list the marks of GET /api/marks newest first as name · time · service, the automatic start marks among them.
Until lists the same marks plus now, which is the default.
A recording lists only what it holds, and no Until (The recording).
The three selectors live in the hash query, so a comparison can be shared as a URL.
When both are set, the page fetches GET /api/compare?before&after&until&service.
When the URL leaves a window out, the page fills it from a default pair, the newer mark as After and the older as Before, and never replaces a window the URL names.
The default pair is the two newest marks a person or an agent made, which are all the marks but the automatic start ones.
Failing that, it is the two newest start marks of one service, its previous run and this one: the service of the top bar, or else the service of the newest mark that names one.
Two start marks of different services are never a pair, because the window between them is not one service’s previous run.
Failing that, the page says two marks are needed, adds that two can be chosen from the lists when they hold two, makes the Mark button’s dialog reachable from there, and shows the CLI’s mark before line in a copyable block, naming the jar and the --url as Copy CLI line does.
A pair of start marks compares that service’s traffic alone, because the other services did not restart at those marks.
With no service in the top bar, the page asks with the service of the two marks, keeps it in the hash beside them as scope=<service>, and leaves the top bar as it is.
A service in the top bar goes before scope, and a URL that names both windows and no scope compares what the top bar names, as does a pair chosen from the lists.
A line under the bar prints both windows and their lengths, from the before and after of the answer, as in Before 10:06:34–10:06:41 (7 s) · After 10:06:41–now (1 min 8 s).
An After window that runs to now says so rather than the time of the last answer.
A comparison narrowed by scope opens the line with whose traffic it is, as in Traffic of spring-orders · Before 10:06:34–10:06:41 (7 s) · After 10:06:41–now (1 min 8 s).
Four panels follow.
-
Totals is a row of tiles, each reading
before → afterwith the after value large: requests, requests per second, errors, p95, and Apdex. The requests per second are the API’srps, with no verdict, and set two windows of different lengths side by side as the counts cannot. The arrow and the after value are--errwhen that side got worse,--okwhen it got better, and plain otherwise. Worse means the errors grew or the p95 grew by more than 20% and 10 ms, by the API’s rule, or the Apdex fell by at least 0.01 at the two decimals the tile shows, a rule of the page’s own since the API gives the Apdex no verdict. Better is the same rule the other way, with the 20% taken of the before value, so a p95 18% faster is no change. A window with no request has no p95 and no rate, so its side reads—and the p95 tile gives no verdict. A side with fewer than 20 requests is too few to judge, a rule of the page’s own that leaves the API’s verdicts as they are. A p95 sets aside the slowest 5%, and 5% of fewer than 20 requests is less than one request, so the p95 is then the slowest request or the next one, and a single request can move the Apdex by more than 0.05. The line under the bar adds that side’s count andtoo few to judge, as in(7 s, 4 requests: too few to judge), and the p95 and Apdex tiles give no verdict and carry the same words in their captions. The errors tile keeps its verdict, because an error is a count and no estimate, and the tables keep the verdicts of the API, which agents read too. -
Endpoints is a table in the API’s order, worst first: a verdict chip (
worsein--err,betterin--ok,newandgonein--text-muted,sameplain), the name with its service chip, thencalls,errors,p50,p95,max,db calls / request, anddb ms / request. A cell readsbefore → after, with—for a side that has no value, and a pair too wide for its column puts the after value under the before one. Aneworgonerow has one side, so its cells print that side alone, without the arrow. A row click opens the endpoint page. -
Queries has the same shape, with the statement in monospace on one line and its full text in the title, marked as on the queries page when the store may have cut it, then
calls,calls / request,p95, andtotal. A row click opens the query page. -
Errors lists the verdict, the type, the message, or the
targetof a group with none, the service chip, and the countsbefore → after, or the one count of aneworgonerow. A row click opens the error page.
Live is honoured here as on every page, but only until = now moves with the clock, so the length and the numbers of an After window that runs to now grow with every tick.
The marks stay where they are.
Map
#/map is a server map over /api/map.
It draws the services, the databases and the external hosts as nodes, and the calls between them as edges.
A full-height panel holds one SVG, drawn by hand without a library.
The layout runs left to right in columns: the user node first, then the services ordered by the longest path from a source, then the external nodes (databases, hosts, destinations) in the last column.
A service that no traced call reaches is a source.
A cycle has no longest path, so the call that closes it is set aside for the layout: a walk from the sources, then from the services the user node calls, then from the rest, each by name, finds that call, and the services of a cycle start in the first service column whatever order the API lists them in.
Nodes within a column are ordered by the mean row of their callers in earlier columns, ties by name, one row each.
The rows are 96 px apart, except where a gap between two of them needs more for the edges that run through it (below).
A node with no caller in an earlier column comes after the ones that have one, by name.
So a service’s database sits level with it and most edges stay short, and the order depends on the nodes and the edges alone, not on the order the API lists them.
The layout, the lanes of the edges and the places of their labels are computed once per set of nodes and edges and kept while Live refreshes the numbers, so no node jumps.
A label whose text outgrows the width it was placed for, as a count gains a digit or an edge its first errors, would touch a neighbour where it is, so every label is placed again on the curves it has, and a refresh that widens no label moves none.
An edge that appears or leaves the window, which can move a service to another column, lays the map out again.
The set of nodes and edges does not follow the order in which the two spans of a call land.
The two sides export on batches of their own, so either span can land first.
An HTTP or RPC call counts as a call to a host nobody traces only once it ended more than 2.2 s ago, and a request whose caller’s span is not stored counts as traffic from outside only after the same wait.
By then the other span has landed, if there is one.
A host nobody traces therefore reaches the Map that much later, while a database or a queue reaches it at once.
A window that ended long enough ago holds no call still waiting, so it reads the same whenever it is asked.
The 2.2 s is two of the agent’s batch delays (otel.bsp.schedule.delay) and one flush of the writer, so a sender on a longer delay can still show such a call for a moment.
A node is a rounded rectangle of 200×64 with an icon for its kind: service, database, a trace icon for http, a log icon for messaging, and a person for user.
It carries the name, ellipsised, with the full name in a <title>.
A service’s name is ellipsised at the end.
An external name with a / in it, a database file or a URL, is ellipsised at the start instead and keeps its last path segments, so four H2 files under one directory read …/bookstore, …/orders and so on rather than four times h2:~/db/spide….
A service node adds a second line of rps, p95 and error rate, ellipsised to the space left, and a five-bar mini histogram of 40×14 in the bucket colours in the bottom-right corner, never overlapping the text.
The border is --line, and turns --err when the error rate is over 1% and --warn when the Apdex is under 0.85.
The service filter dims every node and edge that is neither the chosen service nor one of its neighbours.
An edge is a cubic curve from the right edge of the caller to the left edge of the callee, with an arrowhead.
Its stroke width is 1 + log10(calls), capped at 5, in --silk at 60% opacity, or --err when the edge carries errors.
No edge runs under a node it does not join.
An edge that skips a column bows around the nodes in between: its control points are pushed up or down by 0.6 of the ordinary row pitch, 58 px, away from the nearest node it would otherwise cross, and a further 0.6 each time that is not enough, four times at most.
When no bow clears them, the edge runs through the gaps between the rows instead, as a chain of cubics.
It crosses each column level, in a lane of its own, and changes height only in the channels between the columns.
The edges that share a gap spread evenly across its height, in the order they come from and go to, so n edges leave n + 1 equal spaces.
Each space is at least 8 px, which holds two edges apart and off the nodes alike.
A gap of the usual 32 px therefore holds three edges.
A gap with more raises the pitch of its own row, the distance to the row below it in every column, from 96 px to 64 + 8 × (n + 1) px, and the other rows keep theirs, so the map grows by what its crowded gaps need and no more.
A row is raised in the pass that finds a gap of it too low and is never lowered, so a later pass that routes an edge elsewhere can leave it a little taller than its gaps then need.
More than 20 edges through one gap are drawn as one band instead of 21 or more lanes: they share the gap’s middle, the row keeps the pitch its other columns need, and the band is one stroke in --silk, 7 to 14 px wide with the count, with round ends.
A node with more than 12 edges to the column after it, or from the column before it, is joined to them by a band as well, a bus: the node runs level to a trunk in the channel, the trunk runs up and down the channel as far as the other ends, and each of them leaves it level, with its own arrowhead; the fans of one channel share its width, in the order of their rows.
An edge is in one fan at most, the larger when it could be in two.
A band is drawn once, over the edges, whose own paths leave out the stretch they share, and a band that ends on a node carries the arrowhead for its edges.
Its tooltip says what it bundles and lists its edges with their calls and errors, twelve at most, and it takes the keyboard focus, which reads the same and highlights its edges and their nodes, as hovering it does.
The demo and any map of a handful of services stay below both thresholds, so there every edge has a lane of its own.
In the open above or below a column, an edge keeps the height of the line to its callee and moves only as far as clears the edge beside it.
A bow or a plain curve that runs level along another edge takes a lane as well.
So no two edges run level closer than 8 px for further than 60 px, the width of a channel, where lanes change, unless they are in one band.
A call that closes a cycle runs back to an earlier column, or to its own node, so it goes the other way: from the left edge of the caller to the right edge of the callee, a quarter of the node’s height below the middle, bowed below the row by 0.6 of the ordinary row pitch, or above it from a quarter above the middle, and further each time that is not enough to clear both end nodes and every node in the columns it spans, four times at most.
When no bow clears them, it runs through the gaps of the columns in between as a forward edge does, and a service calling itself in a full column takes the smallest bow, which stays in the gap beside its node.
It never shares an end or a stretch with the forward edge of the same pair.
Its label sits on the curve, with a small --bg-panel halo that breaks the line, so it reads as that edge’s and no other’s.
The label takes the place nearest 40% of the edge’s length that keeps off the nodes and the other labels, and off the other edges where it can, since a label on a crossing reads as theirs.
It moves along its own edge to find that place, and no further than 70% of the length, so it never touches the arrowhead.
An edge in a band finds that place on the part it draws itself, so its label never sits on the band and reads as the band’s.
An edge shorter than twice its label would vanish behind it, so its label sits just above the curve instead; an edge in a fan, whose own part is its stub from the trunk, has it there.
The label gives the calls, and the errors in --err when there are any.
The average and the p95 are in the hover tooltip.
Hovering a node highlights it, its neighbours and the edges among them, hovering an edge highlights it and the two nodes it joins, and hovering a band, or focusing it, highlights its edges and their nodes; everything else dims.
A band dims when every edge in it does.
A dimmed node, edge or band fades to 22%, all but its text, which turns --text-muted and stays readable.
A click on a node, or Enter or Space on a focused one, opens a drawer on the right, 420 px wide like the span drawer, with the node’s name and kind.
For a service the drawer holds the stat tiles (requests, Apdex, error rate, p95, rps), the Response summary bars, the Load chart for the window, fetched from /api/services/{name} and drawn as the Overview draws it in Load mode, and buttons for Service, Scatter and Traces, each filtered to that service.
For an external node it holds the calls, the errors, the average, the p95, and the services that call it with their counts.
For the user node, which has no numbers of its own and no caller, it holds the requests its edges carry, as its node line counts them, their errors, and the services they reach, each with its requests, errors and p95.
The SVG fits its panel by default: it scales down to the panel’s width and height, and never above its natural size.
It stops at 75%, where a node’s name reads at 9 px and its second line just under 8, so a smaller panel scrolls instead.
That makes the fitted SVG’s minimum width columns × 260 px at 75%, and it scrolls horizontally inside its panel on a narrow screen.
A window resize fits it again.
The Natural size toggle in the panel’s header draws the SVG at 100%, with a minimum width of columns × 260 px, and the panel scrolls.
The toggle is there only when the two sizes differ, so a map that fits at its natural size has none.
The panel is as high as the viewport minus the top bar, and at least 420 px.
When no call has been traced at all, the page shows Nothing has been traced yet. Attach Spider Sense to an application, or point any OTLP/HTTP sender at this collector. and the three snippets of the How to send data dialog instead.
Services
#/services lists every service: name, language, requests, rps, error rate, Apdex, p50, p95, p99, a sparkline, and when it was last seen.
A service with no request in the window, such as a worker or one that has gone quiet, reads - for rps, error rate, p50, p95 and p99, as it does for Apdex, rather than 0.0 ms and 0.0%.
The Overview’s service cards, the Overview’s tiles and the map’s service nodes follow the same rule.
Whatever the sort, a service that sent nothing in the window comes last, as the Overview’s quiet ones do.
A row click opens the service.
#/services/{name} is the page of one service, in seven parts.
-
A header with the name, the resource chips (
telemetry.sdk.language,process.runtime.nameand version,host.name,process.pid), a JVM button when the service reports JVM metrics, a Traces button that opens the Traces page filtered to the service, and on the right the compact Response summary: the five bars in 120×28, with the counts in the tooltip. -
The stat tiles of the Overview: requests, Apdex, error rate, p50, p95, p99, and rps.
-
Three charts in a row, the RED charts: requests, latency at p50, p95 and p99, and error rate as a percentage. They share a cursor, so the values of one instant can be compared across the three (Time-series charts). The first carries the Requests | Load toggle of the Overview and defaults to Load. The choice is shared with the Overview through
chart=in the hash query. -
The Endpoints table: method chip, route, calls, rps, Apdex, avg, p50, p95, p99, max, errors, a status-code mini-bar (2xx green, 4xx
--warn, 5xx--err), and total time. Clicking a header sorts by that column. The default sort is total time. A row click opens the endpoint. -
Top queries and Top errors, as two half-width panels side by side. Top queries gives the statement truncated to one line in monospace, calls, avg, p95 and total. Top errors gives the type, the message, the count and when it was last seen. Their rows link to the query page and the error page.
-
Dependencies: kind icon, target, calls, errors, avg, and p95.
-
Resource attributes, as a collapsed table of keys and values.
Endpoint
#/endpoints/{endpointId} is one route of one service.
The header gives the method, the route, the service, the Apdex and the compact Response summary.
The header also carries the time breakdown: breakdown from /api/endpoints/{endpointId} (The HTTP API), one stacked bar over the endpoint’s 20 slowest traces in the window, with the same four shares a slow-endpoint finding carries.
The bar is 120×5 px, like the status bar beside it.
The four shares come in a fixed order with a fixed colour each: db in --accent, http in --series-8 (teal), internal in --warn, and self in --silk.
Each segment is sized by its share, and the bar’s title gives every bucket with its share.
The largest bucket is named in words under the caption, so the bar can be read without telling its colours apart.
Nothing is drawn when breakdown is {}.
The same RED charts follow, with the same Requests | Load toggle.
Under them are four tabs: Slowest requests, Recent traces, Queries, and Errors.
The trace rows have the same columns as the list on the Traces page.
Slowest requests lists the traces of the endpoint’s 20 slowest requests, slowest first by the request rather than by the trace, so it adds a Request column before Duration: the endpoint’s own span in the trace, matchMs, which carries the bar, while Duration is the whole trace’s.
Scatter
#/scatter puts every request on one chart as a point, time on the x-axis and response time on the y-axis.
The chart is a full-height uPlot chart with two modes, Dots and Heatmap, chosen with a toggle above it and remembered in the hash query as mode=heatmap.
- Dots
-
The default. One 3 px dot per request in its service’s colour, errors drawn as
--errcrosses on top, and slow requests ringed in--warn. The y-axis is linear by default, with its maximum at the window’s p99 times 1.5 and a▲ n abovenote for the points that fall outside it. The Log scale toggle switches the y-axis to a logarithmic scale that runs from just under the fastest point, 0.5 ms at the most, to the slowest, so nothing falls outside it and the note goes. A request of 0 ms, which a log scale has no place for, sits on the axis’s bottom edge. Hovering a point gives the endpoint, the service, the duration and the time, and a click opens the trace. - Heatmap
-
An alternative for a dense window, on the same axes. It divides the plot area into cells 6 px wide and 24 rows tall. The rows follow the y scale, so they are log-spaced under the log scale. Each cell is filled in the accent colour with an opacity proportional to the square root of its count over the densest cell’s count. A cell holding any error gets a 1 px
--erroutline. Hovering a cell gives its time span, its response-time span, its count and its errors.
Above the chart sit the legend of services (click one to hide or show it), the Success and Failed toggles, the Log scale toggle, the mode toggle, the counts of points, errors and slow requests, and a "truncated" notice when the API cut the list. Success and Failed both start on. Turning one off hides those points, and the counts, the selection and the y-axis maximum follow.
Drag a rectangle in either mode to list the traces inside it, in a table beneath the chart.
The table asks /api/traces with from, to, minMs and maxMs from the rectangle, each widened outwards to a whole millisecond so no point drawn inside it is left out, plus the current service, plus status=error or status=ok when only one of Success and Failed is on.
A point is a request, so the table asks with by=entry and lists the traces with a request drawn inside the rectangle, a downstream one included (Traces).
The chart panel’s header then reads the selection’s time span and response-time span, beside a Clear button.
Each end of the time span reads as a tooltip’s time does (Numbers and times), so an end outside today names its day.
Esc clears the selection.
With Live on, points stream in from the right every 2 seconds, from a refetch of the last 10 seconds merged in, and the x-axis slides.
The merge adds the points the chart does not show yet, matched on start, duration, service, endpoint and trace id, so each request of a trace that crosses services keeps its own point.
The merged points are held to the 5,000 a full load asks for, the oldest going first, so a long Live session holds no more points than a full load; when that cuts any, the "truncated" notice comes on.
The y-axis maximum and the ▲ n above note follow the merged points, and the "truncated" notice stays until a full load of the window is not cut.
A change of service or range reloads the whole window and the trace list, Live or not, and nothing else does: the legend, Success and Failed, Log scale and the mode change the chart in place, and only Success and Failed ask the trace list again.
The rectangle stays on the chart through every redraw, a toggle, a Live merge or a reload, until Esc, Clear or a reload whose window no longer holds it clears the selection.
Traces
#/traces lists the requests, and #/traces/{traceId} is one of them.
The list
The query bar takes free text (q), which finds a trace id too, a duration range (minMs and maxMs, two small number fields joined by "–"), a status of all, error or ok, By trace or By request (by=trace or by=entry), and an endpoint from the current service’s endpoints when a service is selected.
By request reads the duration, the status, the service and the endpoint off one request of the trace rather than off the trace, as a Response summary bar does; a bar’s link opens the list that way.
A fraction typed in either field widens outwards to a whole millisecond before the page asks, since the API takes whole milliseconds (Traces): 2.5 in Min asks minMs=2, and 7.2 in Max asks maxMs=8.
No trace in the typed range is therefore left out.
The URL keeps the value as typed.
An endpoint belongs to one service, so a change of the top bar’s service clears the endpoint, from the URL as well.
An endpoint the window holds no call of stays chosen, as <name> (not seen), so the select names the filter the URL and the list apply.
Enter applies the query, and so does a 400 ms pause in typing.
The table gives the time, the root span’s summary in bold (rootSummary, TraceSummary), the service chips on the same line, the duration with a bar proportional to the slowest row and coloured by slow or error, the span count, the DB calls, the status code, and the error and slow markers.
A trace with no rootSummary shows its root name.
A long summary is cut to leave the chips room, down to 12 em, and under that the chips take a line of their own, so a row is one line wherever the column has the room.
The rows are newest first, with Load more under the table, which pages with the last row’s before and beforeId.
With Live on the newest page lands on top, and the rows Load more fetched stay under it until they leave the window; only a change of filter, service or range starts the list over from the newest page.
Under By request a trace may start before the window and still be listed for a request that starts in it, so a row leaves only when its whole trace ended before the window.
A trace the newest page lists again replaces its row, since a trace’s spans come in several exports and its root, start, duration and span count change as they arrive.
When no trace has been stored at all (/api/status.counts.traces is 0) and no filter is set, the page shows No trace has arrived yet. Point an application at this collector and reload. and the three snippets of the How to send data dialog instead of the table.
A filter or a window that matches nothing leaves the table, with one row that says No trace matches this window and filter.
The header
A trace opens with its root span’s summary, which is the page’s title too, the trace id in monospace (click it to copy), the start time, the total duration, the services, the span count and the error count.
The buttons are the Waterfall / Profile toggle, Export (/api/export?traceId=), and a log count that jumps to the log records.
With Live on the page refetches the trace and repaints it when spans or logs arrived since, because a trace’s spans come in several exports and from several services, so a trace opened from a fresh tingle completes on screen.
The collapsed spans, the selected span and its drawer stay.
Waterfall
The waterfall is the default view, a tree of spans indented by depth with collapsible nodes.
A span whose parent the trace does not hold is a root, and so is the first member, in the spans' order, of a parent cycle (a span that is its own parent, or spans that name each other), so the waterfall and the profile list the same spans at the same depths.
A row has a service colour bar, a category icon, the span’s summary (Span) with its name in the title, the span’s bar on the time axis the whole trace shares (offset and duration proportional), and the duration label.
The time axis runs over the trace’s own duration, however far under a millisecond.
The head above the rows reads that duration, such as 0.4 ms total.
An error has a red bar and a bolt, and a slow database span a --warn ring.
A row click opens the span drawer.
Repeated siblings fold as the CLI’s tree collapses them: more than three consecutive children of one span, none with children of its own, with the same category, service and summary, and alike in whether they failed and whether they are slow, are one row, such as SELECT product × 21.
Its bar runs from the first span’s start to the last one’s end, its duration is the total of the spans, and its title gives the count, the average and the total.
A failed or slow span among repeats that are neither keeps a row of its own, with its bolt or its ring.
A click on the row or its chevron, or Enter, expands it in place, with the spans one level deeper under it, and another folds it again.
The expanded runs stay across a Live refresh and are the same in the profile.
The span drawer
The drawer opens on the right, 420 px wide, with the span’s name, service and kind, its timing (start offset, duration, share of the trace, self time), and its status.
Its head links to the groups the span belongs to (Span), named as the Findings row names them: Query, Endpoint and Error.
A span in no group has no link.
A link carries the top bar’s query, but a span the top bar’s window no longer holds takes the narrowest range that does, since a group’s page answers over its window.
A database span’s statement comes next, under its key, db.statement or db.query.text, with a Copy button.
It is pretty-printed: SQL keywords uppercased, and a line break before each major clause and after a -- comment.
Apart from those line breaks, the printer puts a space only where the statement had one, so an operator such as >= or ::, a placeholder such as $1 or :id, and a name in any script stay whole.
Copy therefore copies a statement the database parses.
It is the attribute as the agent sent it, so it is whole where the db_statement column keeps only 2,000 characters.
Then come the other attributes as a table of keys and values, in key order, with long values in a <pre>.
A code.stacktrace the extension captured (Code locations) comes after them, under its key.
Its application frames, at most five, top first, are listed as code frames, and the stack follows folded as the error page folds one, with no toggle.
The events follow with their attributes and the stack trace in a scrollable <pre>, and the scope name comes last.
Esc closes the drawer.
Profile
The profile is the view for reading what a request did, in order, and where the time went.
It lists the same spans as a flat table: the position (#), the start offset (12.3 ms`), the gap since the previous step, the elapsed time, the *self* time, the self time as a *%* of the trace, the depth as indentation, a one-line summary, and the service.
A summary reads, for example, `SELECT orders … (h2)` or `+GET http://localhost:8081/api/books/{id} → 200.
Self time is the elapsed time minus the time the direct children cover within the span.
Two children that run at once take their overlap once, and a child that outlives its parent takes only the part inside it, so concurrent or asynchronous children never leave a parent less than its own time.
The order is chronological by default.
Clicking the Elapsed or Self header sorts by that column, descending.
The gap column then reads -.
Clicking the Start header restores the chronological order.
The three largest self times that are at least 5% of the trace are marked hot, with an --accent tint and a bold value, so the step that spent the time is visible without reading every row.
A slow step is tinted --warn and carries a turtle, and an error is tinted --err and carries a bolt.
A step is slow as the span’s slow says, as the waterfall’s ring and the CLI’s [slow] read it: a request over slow.request.ms or a statement over slow.query.ms.
A span that is neither, such as an outbound call a request makes, is not slow however long it took.
A slow or failed step keeps that tint when it is hot too, and its bold self value still marks it hot.
A run of repeated siblings is one row where its first span stands, folded as in the waterfall: its position reads 3–23, its elapsed and self times are the totals of its spans, and a row click expands it in place rather than opening a span.
Its spans, once expanded, follow its row one level deeper under every sort, and are not marked hot beside it, since its row already counts their time.
Logs
The trace’s log records come last, oldest first, each with a severity chip, its offset in the trace, the logger cut as the Logs page cuts it, and the body. A row click, or the chevron before the offset, opens it to its attributes and the stack trace of the exception it carries, as on the Logs page, and the open rows stay across a Live refresh. The panel head links to the Logs page filtered by the trace id, over all services and over the narrowest range that still holds the trace’s start a minute later, so the window the Logs page computes when it opens holds it too.
Queries
#/queries groups the SQL statements as the OpenTelemetry agent sanitised them.
Above the table are a sort selector (total, avg, p95, max, calls) and a text filter on the statement.
The table gives the statement in monospace on one line, with the full text in its tooltip, which ends in … (cut at 2,000 characters) when the store may have cut the statement (statementCut), then one Op · Table cell with the operation, then the table and the system dimmed (all three in its title), the unindexed columns, calls, avg, p95, max, total, the slow calls (in --accent when there are any), and when it was last seen.
The statement cell is the wide one: it takes the width the other columns leave and cuts its text with an ellipsis, so the table fits the page (Narrow screens).
Its line keeps the statement’s own spacing, as the pretty-printed text does.
A -- comment on it reads as /* … */, so the rest of the statement does not read as part of the comment.
Unindexed is the schema block's unindexed, the columns the statement filters on that no index leads with.
They are joined by , ` in `--accent and cut to the column width, with the full list in the title.
The cell reads none when every predicate is served, and — when the statement has no block, exactly the column the text rendering has.
A row click opens the statement.
#/queries/{queryId} is one statement: the full text pretty-printed, marked the same way when the store may have cut it, the schema block under it as the Findings page renders it, a Code list of where the statement is issued, the Copy as Markdown and Copy CLI line pair over GET /api/queries/{queryId} and the queries command, the stat tiles, a chart of calls and p95, the callers as endpoints with their counts, and Slowest calls.
Slowest calls lists the traces of the statement’s 20 slowest calls, slowest first by the call, with a Query column before Duration that gives the call’s own duration, matchMs, and carries the bar, as the endpoint page’s Slowest requests does.
The callers take a third of the width and Slowest calls two thirds, the width a trace table needs to keep its columns at 1280 px.
A caller whose endpoint has a page of its own (endpointId, Callers and error endpoints) is a link to that page; (no endpoint) stays plain text.
The Code list is the code of GET /api/queries/{queryId}, the application frames a slow-query finding on the statement would carry, as code frames; it is left out when there are none.
Errors
#/errors groups the exceptions by root cause and the application line that threw it (Errors).
A row gives the simple name of the type in monospace with the full type in the title, the message, or the target of a group with none, the service chip, the count, and the endpoints as chips, the first two and then +n.
It goes on with the occurrences over the window as a sparkline (inline SVG, 120×28, in --err), drawn from ErrorGroup.series as the services list draws its own, and when the error was first and last seen.
A row click opens the error.
#/errors/{errorId} gives the header with the Copy as Markdown and Copy CLI line pair over GET /api/errors/{errorId} and the errors command, a chart of the count, the sample’s exception chain, the endpoints, and the recent traces.
The title, and so the browser tab, reads the simple name of the type, then the group’s target when it has one, as in 500 GET localhost:8081/api/flaky.
The header reads the type and the message, or the group’s target when it has no message (ErrorGroup).
The endpoints link to their pages as the query page’s callers do.
They take a third of the width and the recent traces two thirds, as on the query page.
The chain is the chain of GET /api/errors/{errorId}, one section per cause, innermost first.
The root cause is headed Root cause, then each exception wrapping it is headed Wrapped by, out to the outer one.
There is no heading when the chain has only one exception.
Each section shows its type and its message, then its frames and its … n more line as a folded stack trace.
Above the chain, in the same panel, a Code list names the sample’s application frames as code frames: the code of GET /api/errors/{errorId}, the frames a finding would carry, the root cause’s first.
The list is left out when there are none.
With no chain the panel says This error carried no stack trace. and has no App frames | All toggle.
Code frames
A code frame on the Findings page, the error page, the query page or in the span drawer asks GET /api/source?frame=… (Source) once.
An answer is reused for ten seconds, so a Live refresh does not ask again at once.
A frame that resolves to a source file becomes a link that opens the file at that line, as the top bar’s editor setting says:
-
idea://open?file=<absolute path>&line=<n>for IntelliJ IDEA; -
vscode://file<absolute path>:<n>for VS Code.
The link’s title is the path and the line.
The five lines around the frame’s line follow under it in monospace, numbered, with the frame’s own line on --accent-soft.
A frame that does not resolve stays the plain monospace text it was, with nothing under it, and its title says that no source file was found for it under spidersense.source.dirs.
When no frame of a Code list on the Findings page, the error page, the query page or in the span drawer resolves, one muted line under the list says so and how to change it: No source found for these frames. Set spidersense.source.dirs to your source roots to open them in IntelliJ IDEA (Source roots)., with the editor the top bar names and a link to Source roots in the published manual.
The source is read when the page asks and never stored. The lines are therefore those of the file as it is now, which after an edit may no longer be the lines the frame meant. Source roots says where Spider Sense looks for the files. The Overview’s one frame per finding stays text, because the whole row is already a link.
Stack traces
The error page’s sample stack trace is a <pre> that folds runs of framework frames into one line each.
A frame is the application’s by the rules a finding’s code follows (Code locations).
A frame that holds one of generatedMarkers, a generated class’s, is never the application’s.
With appPackages set, any other frame is the application’s when it starts with one of them.
Otherwise it is the application’s unless it starts, case-insensitively, with one of frameworkPrefixes.
The page reads those rules from /api/status.codeFrames (Status) rather than keeping a list of its own, so the page and the server count the same frames as the application’s.
Application frames are in --text and the others in --text-muted.
The header is in --err, and the Caused by: lines in --warn.
The panel head has a toggle, App frames | All, which decides what is folded and is remembered in the hash query as frames=all.
App frames is the default.
Under it, every run of two or more consecutive framework frames becomes one dimmed, italic line at the run’s indentation, such as ⋯ 12 frames from org.springframework, org.apache.
The line gives the count, then the packages in order of first appearance, three at most and +n for the rest.
A package is generated for a generated class’s frame, else the framework prefix the frame matched, without its dot, or, under appPackages, the frame’s first two segments.
A single framework frame between two application frames is not folded, since its fold line would be no shorter.
A click on a fold line, or Enter, expands that run in place.
The focus the line had moves to the run’s first frame, so a keyboard user carries on from there rather than from the top of the page.
A Live refresh repaints the trace only when the trace or the mode changed, so an expanded run stays open.
Under All nothing is folded.
When /api/status carries no codeFrames, as in a recording made before it did, nothing is folded or highlighted.
The stack trace of a span event in the span drawer and that of a log record on the Logs page keep the plain highlighted <pre>.
Copy as Markdown
The expanded row of the Findings page, the error page and the query page carry two buttons, so a person can paste into an agent’s prompt the text the agent’s own findings, errors or queries command would print, or the command line that prints it.
- Copy as Markdown
-
Fetches the text rendering of that one finding or group over the page’s window when pressed, and copies it (One finding). The Findings page asks
GET /api/findings/{id}?format=textwith the list’sfrom,toandservice. The error page asksGET /api/errors/{errorId}?format=text, and the query pageGET /api/queries/{queryId}?format=text, each with the page’sfromandto. A404, a finding that has left the window since, is shown as a toast. - Copy CLI line
-
Copies
java -jar <jar> <command> --since=<from> --until=<to>, with the window’s two instants as epoch milliseconds, such asjava -jar /home/me/tools/spider-sense-0.1.0.jar findings --since=1758000000000 --until=1758000900000. The Findings page usesfindings, with--service=<name>when the top bar’s service filter is on. The error page useserrorsand the query pagequeries, each with--service=the group’s service.<jar>is/api/status.jar, the jar the server was started from, single-quoted when it holds a character a shell would read, andspider-sense.jarwhen the server does not know its jar. The line ends in--url=<endpoint>,/api/status.endpoint, when that is nothttp://127.0.0.1:4000, the CLI’s default, so the line asks the Spider Sense the page came from.
The window is the one the page last loaded, so both answer for what is on the screen rather than for a moment later.
Logs
#/logs lists the application’s log records.
The query bar takes free text (q), a minimum severity, and a traceId field, which is prefilled when you arrive from a trace.
The table is dense: the time to the millisecond, a severity chip coloured by level, the service chip, the logger dimmed, the body wrapped with its newlines kept, and a link to the trace when the record carries one.
A logger too long for its column is cut from the start at a dot, as in …service.OrderService, so the class stays and no package shows half.
The whole name is in the title.
A row click, or the chevron before the time, expands its attributes and its stack trace.
Load more pages further back with the last row’s before and beforeId.
With Live on the page tails: new lines arrive at the top, and the scroll position is kept unless you are already at the top.
The rows Load more fetched stay under them until they leave the window; only a change of filter, service or range starts the list over from the newest page.
While a row is open and the page is scrolled away from the top, Live leaves the list as it is, so the open record does not move under you.
Only a row still listed holds the list: a record that was open when a tail dropped it from the list no longer does.
A change of filter, service or range reloads the list all the same.
JVM
#/jvm shows one service, which you choose with the service filter in the top bar.
While the filter is on all services, the page offers a picker of the services that report JVM metrics (hasJvm).
A runtime header gives the JVM, the pid, the host and the CPU count.
Under it is a two-column grid of uPlot charts: heap used and committed with the limit, non-heap used and committed, the memory pools with one line per pool, GC collections per bucket as bars with each collector’s collection time as a dashed line in the colour of its bars on a right-hand ms axis, threads, CPU utilisation in percent with the system load, and loaded classes.
Every chart of the grid runs over one axis, from the first GC bucket to the end of the window, so a GC spike lines up with the heap chart above it (Time-series charts).
The Heap memory chart scales to committed with a 15% margin, since the JVM’s default limit is a quarter of the machine’s memory, and a scale that reached it would flatten the used and committed lines.
The legend gives the limit’s value, as limit 7,896 MiB.
Its dashed line is drawn only when it falls inside that scale, so a heap nearing its limit shows it.
A service that two processes export at once, an application and its tests, is charted as the sum of the two, each process’s collections counted on their own (JVM).
Every point of the CPU chart is the utilisation fraction times 100, so the sum of two busy processes reads above 100%.
One Connection pool chart follows per pool in connectionPools, with the pool’s name in the panel title.
It draws the connections in use as an accent area, the idle ones as a silk line, the maximum as a dashed --warn line, and the pending requests as --err bars on a right-hand axis.
The axes carry their units, MiB, ms, %, and connections.
Metrics
#/metrics is the explorer for every metric that arrived.
The catalog on the left is searchable, and lists each metric’s name, type chip, unit and series count.
It is a listbox with one tab stop on the selected metric: the arrow keys, Home and End move between the metrics, and Enter or Space picks one, as a click does.
It lists the metrics of the top bar’s service, and a change of service or a Live refresh reloads it with the chart, so a metric that starts arriving appears.
The chart on the right draws the selected metric over the window, one line per series, with a legend of the series' attributes.
The selected metric is metric in the URL; one the catalog does not list gives way to the catalog’s first metric, in the URL as well.
Each series is drawn at its own timestamps, so two services that export at different offsets share the time axis without one being shifted onto the other’s points.
A rate toggle applies to sums.
A histogram is drawn as its mean and p95, with the count as bars beneath.