Design

The bundled agent is OpenTelemetry Java agent 2.31.1, and the nested server jar carries Spider Silk 1.1.0.

The launcher is deliberately tiny and dependency-free because the OpenTelemetry agent appends the whole jar to the bootstrap class path (Instrumentation.appendToBootstrapClassLoaderSearch), so everything at the top level becomes bootstrap-visible. The server and its dependencies therefore live in a nested jar that the launcher extracts to ${java.io.tmpdir}/spider-sense-<version>/server.jar, skipping the extraction when the file is already present with the same size. It loads that jar through a dedicated SenseClassLoader extends URLClassLoader whose parent is the platform class loader. The server never sees the application’s classes, and the application never sees Jetty or protobuf from the server. The extension is extracted the same way, to extension.jar beside it, and is loaded by the OpenTelemetry agent’s own ExtensionClassLoader rather than by ours.

SpiderSenseAgent.premain does four things, in order.

The defaults of step 3 are:

SpiderSenseMain.main, the standalone entry point, does step 2 with --mode=standalone and then blocks on join. When its first argument does not start with - it is a CLI command instead. The launcher then loads the nested jar the same way and invokes net.benelog.spidersense.cli.Cli.run(String[]), exiting with what it returns (The CLI). It also sets the system property spidersense.jar to its own jar’s absolute path first, because init has to write that path into a project’s CLAUDE.md, and the CLI, running out of the nested jar in a temporary directory, could not find it otherwise.

Even with the class-loader exclusion in place, the collector drops any SERVER span whose server.port attribute equals its own port and whose service is the one it is embedded in. That is belt and braces, so a misconfiguration never shows the UI monitoring itself. CLIENT spans are kept: an application that calls Spider Sense’s port is doing something real, and that call belongs in its trace.

The UI server’s Jetty thread pool is marked daemon in agent mode (JettyServer.threadPool(…​) with QueuedThreadPool.setDaemon(true)) and with shutdownHook(false), so a short-lived command-line application still exits when its main returns.

A build tool that forks the application’s JVM needs the -javaagent option handed to it. The Gradle Plugin does that for bootRun and run, and Maven does it for spring-boot:run. The jar and the plugin are published together to Maven Central as net.benelog.spidersense:spider-sense and net.benelog.spidersense:spider-sense-gradle-plugin, one version for both.

The Gradle module spider-sense-server is a Spider Silk App with four concerns.

Semantic conventions: the OpenTelemetry Java agent still emits the older database attributes by default (db.system, db.statement, db.name, db.operation, db.sql.table) and the stable HTTP ones (http.request.method, http.route, url.path, http.response.status_code, server.port). With otel.semconv-stability.opt-in=database it emits db.system.name, db.query.text, db.namespace, db.operation.name and db.collection.name. The decoder normalises both generations into SpanRecord’s accessors, and also the pre-stable HTTP names (`http.method, http.target, http.status_code) for other SDKs.

Errors come from three places and are merged: the span status ERROR, the exception span event (exception.type, exception.message, exception.stacktrace), and the error.type attribute. An error group is the service, the exception type or error.type, and the message with digits and quoted strings replaced by ?.

Endpoint identity is the HTTP method plus http.route when a route exists, else the span name, and the aggregation keys on entry spans. An entry span is a span of kind SERVER or CONSUMER, or a root span (no parent) of kind CLIENT or PRODUCER that is not a database span, meaning it carries no db.system or db.system.name.

A client root span is a request someone made, which is how the load generator’s java.net.http traffic shows up. A root INTERNAL span and a root database span are work the application did to itself. A seeder’s tens of thousands of INSERT statements, or a scheduler’s tick, are not requests, and counting them would drown the endpoint list, the request totals, Apdex and check. Such spans are still stored, still have a trace row and still render in the trace tree. They are simply not endpoints, not requests and never slow request tingles.

A root INTERNAL span is a job: a scheduled method, an @Async call, a batch step. Jobs have their own finding, slow-job, so a slow one is reported without ever being counted as a request.

A health check polled every few seconds is the most frequent request of a typical Spring Boot application and the least interesting one. It is fast, it never fails, and it dilutes the request count, the Apdex, check and every slow-endpoint judgement. spidersense.ignore.endpoints takes an entry span out of all of that while still storing it, and Ignored endpoints specifies the patterns and the matching.

Query identity is the service, the database system, and the statement as the agent sanitised it. The agent replaces literals with ? by default, which is exactly the grouping wanted. A statement is shown at most 2000 characters.

The Gradle module spider-sense-extension is packaged as spider-sense/extension.jar, and is the only piece of Spider Sense that is not the stock OpenTelemetry agent. It exists for one thing the agent cannot do: say where a slow query was issued from.

It registers, through META-INF/services/io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider, a span processor that implements io.opentelemetry.sdk.trace.internal.ExtendedSpanProcessor and does its work in onEnding(ReadWriteSpan). That callback runs on the thread that is ending the span, before the span becomes immutable: the duration is already known and an attribute can still be set, which no ordinary SpanProcessor callback allows.

The first case is a slow query. When the span carries a db.system or db.system.name attribute and has taken at least spidersense.slow.query.ms, the processor writes Thread.currentThread().getStackTrace() into the span attribute code.stacktrace. The environment variable SPIDERSENSE_SLOW_QUERY_MS also works, the default is 100, which is the same threshold the server calls a tingle, and the value is read once when the processor is built.

The second case is the N+1. The individual queries of an N+1 are fast, so the threshold above would never fire on them, and an n-plus-one finding would name a statement and no line. The processor counts, per thread, how many database spans of the current trace have ended with the same statement (db.query.text or db.statement, else the span name). The count is keyed by trace id and reset when a span of another trace ends on that thread. When a statement reaches its fifth repeat, the same number that makes a query group an N+1 on the server (N+1), the stack is captured on that one span and on no later repeat. The cost is therefore one capture per repeated statement per trace, and the server’s n-plus-one rule prefers the span of the group that carries code.stacktrace for the finding’s code. A trace whose repeats end on several threads is counted per thread and may fall short of five on each. That is a known limit of keeping the counter thread-local, chosen because the alternative is a shared map with the life of every trace to manage. At most 256 distinct statements are counted per trace, and beyond that the counter stops and nothing else changes.

The third case is a slow outbound call. A CLIENT span that is not a database span (no db.system or db.system.name) and has taken at least spidersense.slow.request.ms (SPIDERSENSE_SLOW_REQUEST_MS, default 500, read once like the other threshold) gets the same code.stacktrace, so a slow-external finding names the line that made the call. The same frames are dropped, the same cap applies, and a span that already carries the attribute is left alone.

The lines are formatted as Throwable.printStackTrace writes them: \tat package.Class.method(File.java:41), one per line, no header. The server therefore reduces them to application frames with exactly the code it already uses for exception.stacktrace (Code frames). The leading frames of Thread.getStackTrace, of the processor itself and of io.opentelemetry., which is the SDK’s own end() path, are dropped, and the trace is cut at 64 frames. isStartRequired() and isEndRequired() are false, isOnEndingRequired() is true, and anything thrown inside onEnding is swallowed: a missing code location is never worth a broken span.

The module is compiled against io.opentelemetry:opentelemetry-sdk-trace and io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi at the SDK version the packaged agent bundles (otelSdkVersion in the root build.gradle), compileOnly and nothing else. The agent’s ExtensionClassLoader rewrites the unshaded io.opentelemetry references to the agent’s own shaded classes as it loads them, so the extension must not ship a copy of the SDK.

An OpenTelemetry agent extension instead of our own premain. The extension mechanism (extensions/ inside the agent jar, AgentListener) would also work, and ExtensionClassLoader is already exclusion-listed. It was rejected because the UI would then depend on the agent’s SPI and lifecycle, and the standalone mode would still need a launcher of its own. A premain that wraps the agent’s premain keeps the server a plain program that the agent happens to be pointed at over a standard protocol.

One small extension beside the premain is worth it, though. The stock agent records where an exception was thrown and nothing at all about where a query was issued from, so slow-query and n-plus-one findings named a statement and left the reader to grep for it. A stack capture on the thread that is ending the span, taken only for database spans already over slow.query.ms and for the fifth repeat of a statement within a trace, costs nothing measurable. The slow spans are by definition slow, a few microseconds against a hundred milliseconds, and the repeat capture happens once per repeated statement per trace. It turns those findings into a line to open. That is the extension: it sets one attribute and does nothing else, and the server and the standalone mode stay unaware of it.

In-process export, with a custom SpanExporter handing spans straight to the store. It is faster, but it ties the store to the agent’s shaded SDK classes and makes the standalone and embedded paths diverge. Loopback OTLP costs nothing measurable and exercises the same code path the standalone mode uses.

In-memory only storage. The first design kept everything in bounded ring buffers and aggregated on demand, which is the simplest option and free of any database inside the application’s JVM. It was rejected because the analysis screen has to survive the monitored application going down, and because two embedded instances on one machine should show one picture. H2 with AUTO_SERVER gives both, and the class-loader exclusion keeps the OpenTelemetry agent away from its JDBC.

A frontend build (React, Vue, TypeScript). SigNoz and OpenObserve are built that way. Spider Sense is a single jar whose build must stay ./gradlew build with no Node. The UI is plain ES modules, one CSS file, and uPlot for charts, served by Spider Silk’s static files. A template engine was not used either: the UI is one page whose data all comes from the JSON API, and Spider Silk’s JSON and SSE support is the part of the framework this application exercises.

A gRPC receiver. gRPC needs Netty or Armeria in the jar for a benefit no local setup has.

Profiling, Glowroot-style stack sampling. It is not part of OpenTelemetry’s stable signals in Java, and it is deferred until the profiling signal lands in the agent.