Design

Spider Sense is an observability tool for the local development loop, in the spirit of Glowroot: one jar, attached to a JVM with one option, a UI in the browser a second later.

Unlike Glowroot, Spider Sense receives everything it collects as OpenTelemetry data over OTLP. The instrumentation is the stock OpenTelemetry Java agent, not a bytecode engine of our own, and the collector accepts OTLP/HTTP from any SDK in any language.

The UI is a Spider Silk application (net.benelog.spidersilk), and the product is a sibling of Spider Silk: same author, same logo family, same "thin by design" attitude.

The Three Modes describes the three ways one jar runs, and the CLI beside them.

Influences

Product What Spider Sense takes from it

Glowroot

The deployment model: one jar, -javaagent, the UI served from the monitored JVM. The transaction/slow-trace/error/JVM page structure.

Scouter

The scatter, which Scouter calls the XLog: every request is a dot on a time × response-time scatter, errors in red, and a drag over a cluster of dots lists those traces. The profile view of one transaction as a step list with elapsed and gap times. The habit of keeping the last N minutes always visible.

Pinpoint

The server map: services, databases and external hosts as nodes, calls as edges, a node click opening that node’s numbers. The response summary (requests bucketed by response time, errors apart) and the load chart stacked by the same buckets, with Apdex as the one-number health score. The success/failed filter and the heatmap alternative on the scatter. The call tree’s self time and percentage per step. The inspector’s data-source panel (connection pool used/idle/max/pending).

SigNoz

The OpenTelemetry-native data model: services derived from service.name, endpoints from http.route, RED metrics per service, trace waterfall with a span detail drawer, logs correlated by trace id.

OpenObserve

Single binary, no external dependencies, everything on one port, a query bar above every list, a one-line "how to send data here" snippet in the UI.

How the jar is put together

spider-sense.jar
├── META-INF/MANIFEST.MF          Premain-Class/Agent-Class/Main-Class: net.benelog.spidersense.boot.VersionGate
│                                 Can-Redefine-Classes/Can-Retransform-Classes: true (copied from the OTel agent)
├── net/benelog/spidersense/boot/VersionGate.class   the entry point, which checks the Java version (Java 8)
├── net/benelog/spidersense/launcher/**   a handful of classes, no dependencies (Java 17)
├── io/opentelemetry/javaagent/**         the OpenTelemetry Java agent, verbatim (bootstrap classes)
├── inst/**                               the OpenTelemetry Java agent, verbatim (.classdata, loaded by the agent's own class loader)
├── spider-sense/server.jar               the collector + UI as a nested fat jar (Spider Silk, Jetty, protobuf, our code; Java 25)
└── spider-sense/extension.jar            the OpenTelemetry agent extension: a stack trace for a slow database span, and the index catalog of the tables it touches (below; Java 17)

The bundled agent is OpenTelemetry Java agent 2.31.1, and the nested server jar carries Spider Silk 1.2.0-SNAPSHOT.

The launcher is deliberately tiny and dependency-free because the OpenTelemetry agent appends the whole jar to the bootstrap class path (Instrumentation.appendToBootstrapClassLoaderSearch), so everything at the top level becomes bootstrap-visible. The server and its dependencies therefore live in a nested jar that the launcher extracts to ${java.io.tmpdir}/spider-sense-<user>/server-<version>-<crc>.jar, where <crc> is the CRC-32 of the nested jar’s content. A file of that name is already that content, so the launcher skips the extraction when it is present with the right size, and a rebuild at the same version gets a file of its own. The extraction writes a temporary file in the same directory and moves it into place, so two JVMs starting at once never see a half-written jar, and a file in use is never replaced. The directory belongs to one user: the launcher creates it readable and writable by its owner alone, and refuses one another user owns, because a jar in it could be replaced by that user. After a new extraction the launcher deletes the jars of other builds that no JVM has started from in the last minute. On every start, with or without an extraction, it also deletes the temporary nested-*.jar.tmp files older than a minute, which only a JVM killed during its copy leaves behind. The launcher loads that jar through a dedicated SenseClassLoader extends URLClassLoader whose parent is the platform class loader. The server never sees the application’s classes, and the application never sees Jetty or protobuf from the server. The extension is extracted the same way, to extension-<version>-<crc>.jar beside it, and is loaded by the OpenTelemetry agent’s own ExtensionClassLoader rather than by ours.

Three Java versions in one jar

Each part of the jar is compiled for the oldest Java it can run on, so one jar holds three class-file versions.

  • VersionGate, the manifest’s Premain-Class, Agent-Class and Main-Class, is compiled for Java 8, the oldest Java the OpenTelemetry agent supports.

  • The launcher and the extension are compiled for Java 17, so forwarding instruments an application on Java 17 or later.

  • The nested server is compiled for Java 25, Spider Silk’s floor, so the embedded UI, standalone and the CLI need Java 25.

The gate exists because a JVM reads the class -javaagent names before any code of the jar runs. A class file it cannot read makes it abort with processing of -javaagent failed, before main, which no try of the launcher can catch. VersionGate is therefore compiled for Java 8, so every JVM the OpenTelemetry agent supports can read it. VersionGate reads java.specification.version and hands over reflectively, to SpiderSenseAgent from Java 17 and to SpiderSenseMain from Java 25, so nothing in it names a class the JVM might not read. Below those versions it prints one line and returns, under -javaagent, or exits 2, under java -jar. Anything the hand-over throws, a class the JVM still cannot read among it, is one line on stderr too. Installation gives what each version does, as a user sees it.

What premain does

VersionGate.premain hands over to SpiderSenseAgent.premain on Java 17 or later, and that does five things, in order.

  1. Read configuration: the properties file, then the system properties spidersense.*, listed in Configuration.

  2. Unless spidersense.collector is set: extract the nested jar, create the SenseClassLoader, and invoke net.benelog.spidersense.server.SpiderSenseServer.launch(String[]) with --port=<port> --mode=agent …​, on the current thread with the context class loader set to the SenseClassLoader. launch returns once the port is bound, because Spider Silk’s start returns after binding, and it returns the port it bound, which steps 3 and 5 export to and print, so spidersense.port=0 works. launch prints nothing: the launcher says where the UI is, in step 5. A failure here is logged to stderr and swallowed: Spider Sense must never prevent the application from starting. On a JVM older than Java 25 the nested jar cannot load, so this step starts nothing and asks whether a Spider Sense answers at the port instead. When one does, the telemetry goes to it, and step 5 says so; when none does, one line on stderr says how to forward (Java versions).

  3. Set defaults for the OpenTelemetry agent, only where the user has not set the property, its environment variable or a key of the file otel.javaagent.configuration-file names already.

  4. Call io.opentelemetry.javaagent.OpenTelemetryAgent.premain(agentArgs, inst). The agent’s jar-location check only requires a Premain-Class attribute in the manifest of the jar that class came from, verified against 2.31.1’s verifyJarManifestMainClassIsThis, so our manifest satisfies it.

  5. Print one line on stdout naming the UI, the service and the CLI line that asks it (Agent). The agent has built its resource by then, so the service is the name every detector agreed on, which the extension notes and the launcher reads through the agent’s AgentInitializer.getExtensionsClassLoader(). When that cannot be read, the line names the service the launcher was told, or none.

The defaults of step 3 are:

otel.exporter.otlp.protocol=http/protobuf

The embedded receiver accepts OTLP over HTTP and has no gRPC endpoint.

otel.exporter.otlp.endpoint=http://127.0.0.1:<port>

Or spidersense.collector when it is set. The launcher uses the literal address rather than localhost, which may resolve to ::1 while the UI binds 127.0.0.1.

otel.service.name

spidersense.service if given, else unset. Unset, the agent’s detectors name the service, and the extension’s service name default names it when they do not.

otel.bsp.schedule.delay=1000, otel.blrp.schedule.delay=1000, otel.metric.export.interval=5000

Spans and log records are exported every second and metrics every five seconds, so a request is in the UI a second or two after it ends. The Map’s 2.2 s wait for the other span of a call is two of those span delays and one flush of the writer (Map).

otel.metrics.exporter=otlp, otel.logs.exporter=otlp, otel.traces.exporter=otlp

Every signal goes to the collector. When step 2 failed, or started nothing for the Java version, and no Spider Sense answers at the port, all three are none instead, because every export there would fail.

otel.javaagent.exclude-class-loaders=net.benelog.spidersense.launcher.SenseClassLoader

Appended to the user’s own list when one is set. The agent skips every class the UI server’s loader defines, so the UI’s own Jetty requests never become spans. The agent source confirms the mechanism: GlobalIgnoredTypesConfigurer already ignores ExtensionClassLoader this way, and otel.javaagent.exclude-class-loaders feeds IgnoredTypesBuilder.ignoreClassLoader.

otel.instrumentation.runtime-telemetry.enabled=true

JVM metrics. This is already the agent’s default, and the launcher sets it for clarity.

otel.javaagent.extensions=${java.io.tmpdir}/spider-sense-<user>/extension-<version>-<crc>.jar

Appended to the user’s own comma-separated list when one is set. It points at the extension. When the launcher fails to extract the extension or to point at it, it prints a warning on stderr and the application starts without the extension, so a slow query then has no code location.

VersionGate.main hands over to SpiderSenseMain.main on Java 25 or later, and prints one line and exits 2 below it. SpiderSenseMain.main, the standalone entry point, does step 2 with --mode=standalone, prints its banner with the port launch bound, and returns. Jetty’s threads are not daemons in standalone mode, so they keep the JVM serving until it is stopped. A start that fails, such as on a port another process holds, is one line on stderr and exit 2. When its first argument does not start with -, that argument is a CLI command instead. The launcher then loads the nested jar the same way and invokes net.benelog.spidersense.cli.Cli.run(String[]), exiting with what it returns (The CLI). It also sets the system property spidersense.jar to its own jar’s absolute path first, because init has to write that path into a project’s CLAUDE.md, and the CLI, running out of the nested jar in a temporary directory, could not find it otherwise.

Even with the class-loader exclusion in place, the collector drops any SERVER span whose server.port attribute equals its own port and whose service is the one it is embedded in. This second guard ensures that a misconfiguration never shows the UI monitoring itself. A standalone server runs inside no application, so the guard drops nothing there, where a span served on its port can only be another application’s. CLIENT spans are kept, because a call the application makes to Spider Sense’s port is one of its own outbound calls and is part of its trace.

The UI server’s Jetty thread pool is marked daemon in agent mode (JettyServer.threadPool(…​) with QueuedThreadPool.setDaemon(true)) and with shutdownHook(false), so a short-lived command-line application still exits when its main returns. The export the OpenTelemetry SDK sends from its own exit hook is stored before it is answered, so such an application keeps its spans (How it is written).

Configuration lists every spidersense. key, its precedence, the properties file and the environment variables. Every otel. property still works as the OpenTelemetry agent documents it, because Spider Sense only fills in defaults.

From a build tool

A build tool that forks the application’s JVM (bootRun, run, spring-boot:run) needs the -javaagent option handed to it. The Gradle Plugin net.benelog.spidersense, an included build in spider-sense-gradle-plugin, does that for bootRun and run with a jvmArgumentProvider, and its spiderSense { } block has a property for every key of the configuration table. The Spring Boot Maven plugin’s agents parameter does it for spring-boot:run, as Maven describes. The jar and the plugin are published together to Maven Central as net.benelog.spidersense:spider-sense and net.benelog.spidersense:spider-sense-gradle-plugin, one version for both.

The server

The Gradle module spider-sense-server is a Spider Silk App with four concerns.

  1. OTLP/HTTP receiver: POST /v1/traces, /v1/metrics and /v1/logs. It accepts Content-Type: application/x-protobuf, which is the agent’s format, and application/json for curl and scripts, and Content-Encoding: gzip. A web page cannot export to it, because the server answers no CORS preflight and refuses a request whose Origin is not its own (API conventions). A browser application’s telemetry goes through an OpenTelemetry Collector, whose OTLP receiver answers the browser’s CORS and exports to Spider Sense. It decodes with the io.opentelemetry.proto:opentelemetry-proto bindings, the same classes the OpenTelemetry Java SDK is generated from. The answer is an empty Export*ServiceResponse in the request’s content type. There is no gRPC receiver, because gRPC needs Netty or Armeria in the jar, and OTLP/HTTP carries the same Export*ServiceRequest messages, which every sender can export. Senders set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf.

  2. Store: an H2 file database under the user’s home, ~/db/spider-sense/sense with AUTO_SERVER=TRUE, shared by every Spider Sense process on the machine and kept after the monitored application stops, so the analysis screen is still there after a restart or a crash. Ingest goes through a write-behind queue and one writer thread, and every API answer is SQL over indexed columns. The schema, the writer, the queries and the retention sweeper are specified in Storage. The one in-memory piece is the EventBus, which hands ingest notifications to SSE subscribers. Tingles and first sightings are sent as they happen, one frame each, and the counts are coalesced into a stats frame at most once a second (Live events). Tingles are rows like everything else, and no in-memory mirror of them is kept (Tingles).

  3. JSON API and static UI: the contract in The HTTP API, and the UI in src/main/resources/public as The Pages describes it.

  4. The agent interface: findings, marks, compare, check and a Markdown rendering of every list, over the same Queries as the UI. The CLI fronts them, and an MCP adapter (POST /mcp, and mcp over stdio) serves hosts without a shell. The Loop states which of the two a host should use.

By default, the OpenTelemetry Java agent still emits the older database attributes (db.system, db.statement, db.name, db.operation, db.sql.table) and the stable HTTP ones (http.request.method, http.route, url.path, http.response.status_code, server.port). With otel.semconv-stability.opt-in=database it emits db.system.name, db.query.text, db.namespace, db.operation.name and db.collection.name. The decoder normalises both generations into SpanRecord’s accessors, and also the pre-stable HTTP names (`http.method, http.target, http.status_code) for other SDKs.

Errors come from three places and are merged: the span status ERROR, the exception span event (exception.type, exception.message, exception.stacktrace), and the error.type attribute. An error group is the service, the root cause’s type, and the innermost application frame. The root cause is the last Caused by: section of exception.stacktrace, or the outer exception when there is none. The frame is the first application frame met reading the chain from the root cause outwards, each cause’s frames top first. The same line throwing the same exception is then one error whatever wraps it. Spring’s DataAccessException subclasses and ServletException carry one outer type for every cause and a message that embeds the SQL or the request, so grouping on the outer type and message spread one bug over several groups and merged unrelated ones. The frame is keyed without its (File.java:41) and with every $<digits> of a synthetic name (lambda$load$0, Orders$1) made $?, so moving the line or adding a lambda above it keeps the group. It is judged by the default framework prefixes and the markers of a generated class (Code locations), never by spidersense.app.packages, so the id does not depend on a setting of the process that wrote the span. A trace without an application frame, and an error without a trace, fall back to the service, the exception type or error.type, and the message with digits and quoted strings replaced by ?. The normalised message stays in the group as its message either way (Errors).

Endpoint identity

Endpoint identity is the HTTP method plus http.route when a route exists, else the span name, and the aggregation keys on entry spans. An entry span is a span of kind SERVER or CONSUMER, or a root span (no parent) of kind CLIENT or PRODUCER that is not a database span, meaning it carries no db.system or db.system.name.

A client root span is a request someone made, which is how the load generator’s java.net.http traffic shows up. A root INTERNAL span and a root database span are work the application started on its own, with no request behind it. A seeder’s tens of thousands of INSERT statements, or a scheduler’s tick, are not requests, and counted as requests they would outnumber the real ones in the endpoint list, the request totals, Apdex and check. Such spans are still stored, still have a trace row and still render in the trace tree. They are not endpoints, not requests, and never slow request tingles.

A root INTERNAL span is a job: a scheduled method, an @Async call, a batch step. Jobs have their own finding, slow-job, so a slow one is reported without ever being counted as a request.

A health check polled every few seconds is the most frequent request of a typical Spring Boot application. It is fast and it never fails, so counted as a request it would raise the request count and the Apdex and lower the error rate and the Overview’s p95, and check --max-error-rate and --min-apdex would judge those numbers. spidersense.ignore.endpoints takes an entry span out of all of that while still storing it, and Ignored endpoints specifies the patterns and the matching.

Query identity is the service, the database system, and the statement as the agent sanitised it. The agent replaces literals with ? by default, so two executions that differ only in their literals are one query. At most 2,000 characters of a statement are shown. The store records nothing when it cuts, so a statement of 2,000 characters, or of 1,999 where a cut inside a surrogate pair ends, is shown as cut wherever it is shown, which may mark a whole one but never passes a cut one off as whole.

The extension

The Gradle module spider-sense-extension is packaged as spider-sense/extension.jar, and is the only piece of Spider Sense that is not the stock OpenTelemetry agent. It exists for three things the agent cannot do: say where a slow query was issued from, say which indexes the table it read carries, and say which route of a Spider Silk application a request matched. It also names a service nothing else named (below).

It registers, through META-INF/services/io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider, a span processor, CallSiteSpanProcessor, that implements io.opentelemetry.sdk.trace.internal.ExtendedSpanProcessor and applies its thresholds in onEnding(ReadWriteSpan). That callback runs on the thread that is ending the span, before the span becomes immutable: the duration is already known and an attribute can still be set, which no ordinary SpanProcessor callback allows. The repeats are counted in onStart, and an outbound call’s frames are taken there too, for the reason below.

Where a query or a call was made

The span processor writes the stack of the calling thread into code.stacktrace in four cases.

The first case is a slow query. When the span carries a db.system or db.system.name attribute and has taken at least spidersense.slow.query.ms, the processor writes Thread.currentThread().getStackTrace() into the span attribute code.stacktrace. The environment variable SPIDERSENSE_SLOW_QUERY_MS sets the threshold as well, by the rule of Configuration: the property wins, and an empty one is unset, so the variable is next. The default is 100, the same threshold at which the server calls a query a tingle. The value is read once, when the processor is built.

The second case is the N+1. The individual queries of an N+1 are fast, so the threshold above would never fire on them, and an n-plus-one finding would name a statement and no line. The processor counts, per trace, how many spans of it have started with the same statement (db.query.text or db.statement, else the span name). When a statement reaches its fifth repeat, the same number that makes a query group an N+1 on the server (N+1), the stack is captured on that one span and on no later repeat. The cost is therefore one capture per repeated statement per trace. The server’s n-plus-one rule prefers the span of the group that carries code.stacktrace for the finding’s code.

At the start, not at the end. Counting the repeats and capturing the fifth happen in onStart rather than in onEnding. The stack at the end is the stack of whichever thread ends the span, and for an asynchronous client that is a CompletableFuture completion callback on a worker, every frame of which belongs to the JDK. The finding would then carry a code location made entirely of framework frames, which the server reduces to nothing. At the start the thread is still the one that made the call, so the stack is the call site. The span already carries what the instrumentation sets on the request, the statement or the URL, so the key is the one the end would have computed. The threshold cases stay in onEnding, because the duration is known only at the end. A slow outbound call has both problems at once: it is known to be slow only at the end, and an asynchronous client ends it on the worker. sendAsync(…​).get(), which is what JdkClientHttpRequestFactory does, is such a client: the caller waits on the future while a worker ends the span. So every outbound CLIENT span that is not a database span takes a Throwable in onStart, on the thread that makes the call, and onEnding removes it, whether or not the call was slow. Its frames are turned into text only when the call has taken at least the threshold, and then they are what code.stacktrace carries, whichever thread ends the span. The cost is one Throwable per outbound call, which records the stack but formats no frame, against a call that takes milliseconds. A slow query keeps taking the ending thread’s stack, because a JDBC statement ends on the thread that issued it, and a Throwable per query would be created on far more spans than one per outbound call. At most 1,000 calls in flight keep their frames, by the same rule as the traces below; a call whose frames were dropped falls back to the ending thread’s stack.

Per trace, not per thread. The counter is one ConcurrentHashMap from trace id to that trace’s counts, read on every database and outbound span that ends. A thread-local counter would be cheaper and need no lifecycle, and the counter used to be one. A thread-local counter is wrong for anything asynchronous, because a client that completes its exchange off the calling thread ends every span of a run on a different worker and each of them counts one repeat. Most HTTP clients work that way. Spring’s JdkClientHttpRequestFactory over the JDK HttpClient calls sendAsync whether or not a timeout is set, and the reactive clients never touch the calling thread. A thread-local counter therefore gave an n-plus-one-http finding no code location at all in the common case, and the code location is the only thing the capture adds to the finding.

The map is bounded twice, because a map keyed by trace id grows for as long as the process runs unless its entries are removed. A trace is forgotten when its last open local root ends, a span with no parent or with a parent in another process, because this process’s work on the trace is then finished. The processor counts the open local roots of each trace, raising the count in onStart and lowering it in onEnding, and drops the trace’s counts when it reaches zero. A trace can enter the same process twice at once: a loop that calls an endpoint of its own application, or sends a message the same JVM consumes, opens a second local root inside the first on every turn. Dropping the counts when that inner root ends would reset them on every turn, so no statement or call of the loop would reach its fifth repeat and the finding would have no code location. A span of that trace that starts later simply starts the count again, which is what a thread-local counter did when a thread moved on. Such a span is the late work of a request that has already answered, the statements of an @Async or after-commit listener, and its counts are kept so that work’s repeats can still reach the fifth, but no local root holds them and no root’s end forgets them. The map also holds at most 1,000 traces, because a trace whose local root never ends, which is a crash or a leak in the monitored application, would otherwise keep its counts for ever, and so would every trace with late work. Inserting past the cap drops other traces until the map is back within it, first the traces no open local root holds and only then any other, so the late work of finished requests never pushes out the counts of a request still running. Among each kind, which traces are dropped is not defined, and a dropped trace can lose only the code location of its repeated statements and calls. At most 256 distinct statements and calls are counted per trace, and beyond that the counter stops and nothing else changes.

The third case is a slow outbound call. A CLIENT span that is not a database span (no db.system or db.system.name) and has taken at least spidersense.slow.request.ms (SPIDERSENSE_SLOW_REQUEST_MS, default 500, read once like the other threshold) gets the same code.stacktrace, taken at the start of the call as described above, so a slow-external finding names the line that made the call.

The fourth case is the N+1 of outbound calls, and it counts exactly as the statements do. An outbound CLIENT span that carries an HTTP method or a url.full is counted under the key <method> <url.full>, with every run of digits in the URL replaced by ?, in the same per-trace map, and the fifth repeat gets the stack, so an n-plus-one-http finding names the loop rather than only the host. The key is prefixed, so a URL is never counted as a statement of the same trace. The 256 keys a trace counts can be of either kind. The same frames are dropped, the same cap applies, and a span that already carries the attribute is left alone.

The lines are formatted as Throwable.printStackTrace writes them: \tat package.Class.method(File.java:41), one per line, no header. The server therefore reduces them to application frames with exactly the code it already uses for exception.stacktrace (Code frames). The leading frames of Thread.getStackTrace, of the processor itself and of io.opentelemetry., which is the SDK’s own start() or end() path, are dropped, and the trace is cut at 64 frames. isStartRequired() and isOnEndingRequired() are true and isEndRequired() is false, because by onEnd the span is immutable and nothing can be set on it. Anything thrown inside either callback is caught and dropped, so a failed capture leaves the span without code.stacktrace and never reaches the application.

The index catalog

The index catalog is the one part of the extension that needs a Connection, which no span processor has. The extension reads the index catalog of the tables the application’s statements touch, so a slow-query or n-plus-one finding can say which of the columns the statement filters on no index serves (The schema block).

For the catalog, the extension carries one InstrumentationModule of its own, spider-sense-schema, registered through META-INF/services/io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule. Its advice sits on the execute, executeQuery, executeUpdate and executeLargeUpdate methods of every java.sql.Statement implementation. The advice measures the call and, when it threw nothing, hands the statement to IndexCatalog. IndexCatalog reads a table’s catalog on the first statement of the process that names it, however fast, and again on the first one after DDL that names it. A catalog read only for a slow statement would outlive the index that made the statement fast: the developer adds the index the block asked for and restarts, the statement is never slow again, and the old row would keep calling the column unindexed for the whole retention. The DDL it reads is create [unique] index … on t, create table t, alter table t, drop table t and MySQL’s drop index … on t; a drop index or an alter index that names no table makes every table’s catalog be read again on the next statement that names the table, and a temporary table’s DDL is not read. A statement is remembered by its text and its connection’s JDBC URL, because one text may run on two databases, a replica or one database per tenant, and each has tables of its own. A statement it has already read on that database since the last such DDL costs two map lookups and the accessors for its connection, which ask the database nothing. The URL is asked for once per connection and kept by the driver’s connection, not the pool’s wrapper, because a pool makes a new wrapper at every borrow. It remembers 2,000 statements, a text on each database counting once; past them, a statement it has not seen is read only when it took at least spidersense.slow.query.ms (the same threshold, read the same way), so an application that writes its literals into the SQL is not tokenised on every call. DDL is the exception: a statement whose first word is create, alter or drop is read at any speed and however many texts are remembered, because a fast create index left unread would keep its table’s old catalog for the rest of the run. IndexCatalog is a helper class the agent injects beside the driver (getAdditionalHelperClassNames), because advice runs in the application’s class loader and the extension’s own classes are not visible there. Its package, schema, is the helper list: IndexCatalog, its nested classes, and Thresholds with the Settings it reads through, which the span processor also uses in the extension’s own loader, so both read the threshold one way. A unit test compares the list with the compiled package, because a class missing from it would be a NoClassDefFoundError inside the application.

IndexCatalog takes the statement’s SQL from the execute(String …) argument, or, for a PreparedStatement, from the VirtualField<PreparedStatement, String> the agent’s JDBC instrumentation fills at prepare time. A statement with neither is left alone. IndexCatalog scans that SQL for the table names after from, join, update, into and delete from, reads a comma-separated list after from, keeps a schema.table qualifier, and unquotes a quoted name.

The lookup runs once per table per process and per DDL that names it, keyed by the connection’s JDBC URL and the table name, and stops after 200 tables. It runs on the thread that ran the statement and on the connection it ran on. It reads nothing on a connection to H2’s own management_db_<port>, which H2 reports as MANAGEMENT_DB_<port>, because nobody opened that database and its SESSIONS table is no application’s (Status and ingest). It asks the standard DatabaseMetaData whether the table exists (getTables, so a name the scanner misread emits no record, and neither does a view, whose indexes are its base table’s) and for its indexes (getIndexInfo(catalog, schema, table, false, true), the statistic rows dropped, the columns in ordinal order). A name the statement does not qualify is looked up in the connection’s current schema (Connection.getSchema()), because that is the table the database resolved it to: H2 also has an INFORMATION_SCHEMA.USERS beside the application’s users, and PostgreSQL may hold an audit.orders beside public.orders. Only when the current schema has no such table, as for a later schema of PostgreSQL’s search_path, or when the driver reports no current schema, as MySQL’s does, is every schema asked, less INFORMATION_SCHEMA, information_schema and pg_catalog, and a name found in several of them is emitted once per schema. It then emits one log record through the agent’s own logger provider (GlobalOpenTelemetry.get().getLogsBridge(), scope spider-sense) with the body index catalog of <table>. The record’s attributes are spidersense.schema.table (the name as the database reports it), spidersense.schema.schema (TABLE_SCHEM, absent when the database reports none), spidersense.schema.product (DatabaseMetaData.getDatabaseProductName()) and spidersense.schema.indexes, a JSON array of { "name", "unique", "columns" } that is empty for a table without an index. The server stores that record as a catalog row and never as a log line (the db_table table). An unquoted table name is folded the way the database stores identifiers (storesUpperCaseIdentifiers, storesLowerCaseIdentifiers) before it is looked up, and a quoted one is kept as written. H2 folds to upper case and PostgreSQL to lower case. MySQL on Linux (lower_case_table_names=0) folds to neither, so there the name is looked up as the statement wrote it, because table names are case-sensitive under that setting. DatabaseCatalogIT runs the lookup against PostgreSQL and MySQL in containers.

The lookup runs on the request thread and on the borrowed connection, for two reasons. A second connection borrowed from the pool for the lookup would itself be the pool exhaustion that a pool-exhausted finding diagnoses. The connection the statement used cannot be handed to another thread, because the application returns it to the pool the moment the statement is done. The cost is the driver’s catalog query, once per table, on the first request that touches it.

On PostgreSQL that catalog query is a statement of its own, since the driver queries pg_catalog through an ordinary Statement, and it is a database span of the application’s trace when the application itself calls getIndexInfo. MySQL’s Connector/J answers the same call without an instrumented statement, so it is no span there even then. Under the packaged agent the lookup runs inside the slow statement’s own execute call, where the JDBC instrumentation’s guard against nested statements already keeps the catalog query from becoming a span. The extension does not rely on that ordering, and marks the thread with the baggage entry spidersense.schema.lookup for the length of the lookup. A sampler it wraps around the configured one (addSamplerCustomizer) drops every span started under that entry and delegates everything else. The trace therefore shows the application’s statements and none of the lookup’s. DatabaseCatalogIT checks the result on both databases: the request’s trace holds the one slow statement, and no span anywhere reads the catalog. A thread already inside a lookup does nothing, because a driver’s catalog queries pass through the same advice. Every exception is caught and dropped, so a failed lookup leaves the finding without its schema block and never reaches the application’s statement.

The instrumentation module is compiled compileOnly against io.opentelemetry.javaagent:opentelemetry-javaagent-extension-api and io.opentelemetry.instrumentation:opentelemetry-instrumentation-api at the packaged agent’s version (otelAgentVersion), beside the SDK artifacts of How the extension is built, and still ships nothing. The agent’s ByteBuddy and its shaded API are what the advice and the helper run on. Standalone mode has none of this: it has no connection to the application’s database, and the spans carry no credentials, so the catalog exists only for a service that ran under the agent.

Spider Silk routes

The third thing the extension does is a second InstrumentationModule, spider-sense-spider-silk, in the same services file. A Spider Silk application is one servlet mapped at / with Spider Silk’s own router above it. The servlet instrumentation therefore reports / as the http.route of every request, and the agent has no instrumentation for the router the way it has one for Spring MVC.

The module applies only where net.benelog.spidersilk.WebRequest is on the class path, and advises one method, WebRequest.withRoute(Route, Map). Spider Silk calls it on the request thread once the router has matched a route, before any filter or handler of the route runs. On its exit the advice calls HttpServerRoute.update(Context.current(), HttpServerRouteSource.CONTROLLER, route.path()), the call the agent’s own framework instrumentations make. The controller source outranks the servlet’s, so the span’s http.route becomes the route as registered, such as /books/{id} with group prefixes resolved, which is already OpenTelemetry’s path-template syntax. The agent then renames the span GET /books/{id}. A request no route matched, such as a static file, a 404 or a 405, never reaches the method and keeps /*.

The advice names Spider Silk’s Route, which resolves in the application’s class loader it is inlined into. The module is therefore compiled compileOnly against spider-silk-core and ships none of it. Route.path() is the only member it calls, and every Spider Silk release has it. SingleJarIT runs a Spider Silk sample under the packaged jar and checks that two ids of one route are one endpoint whose spans carry the route. -Dotel.instrumentation.spider-sense-spider-silk.enabled=false switches the module off, as it does any agent instrumentation.

The service name default

Without -Dotel.service.name, the OpenTelemetry agent names a Spring Boot application after spring.application.name, a jar after the Implementation-Title of its manifest, and any other jar after its file name with the version left in, so orders-0.1.0 is a new service on every release. A main class run from a class path, which is what an installDist start script and an IDE do, is unknown_service:java.

The extension registers ServiceNameDefault through META-INF/services/io.opentelemetry.sdk.autoconfigure.spi.ResourceProvider. It is a ConditionalResourceProvider that applies only when neither otel.service.name nor service.name in otel.resource.attributes is configured and the resource built so far is still unknown_service:java. Its order is 900: after the agent’s Spring Boot detector (100) and manifest detector (300), so either of them wins, and before the agent’s jar-name detector (1000), which then finds the service named and leaves it. It reads sun.java.command: a jar is named without .jar and without a trailing version, and a main class after the class path entry that holds it, a jar named the same way or a classes directory named after the project above its build/ or target/, and failing both after its simple name. The service name lists the whole order a user sees.

How the extension is built

The extension module is compiled against io.opentelemetry:opentelemetry-sdk-trace and io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi at the SDK version the packaged agent bundles (otelSdkVersion in the root build.gradle), compileOnly and nothing else. The agent’s ExtensionClassLoader rewrites the unshaded io.opentelemetry references to the agent’s own shaded classes as it loads them, so the extension must not ship a copy of the SDK.

The examples

Five programs under examples/ give Spider Sense something to see: three web applications on three different stacks, a worker with no web server, and a load generator. The Examples describes each of them and the two scripts that run them. scripts/demo-shared.sh builds the jar and starts one standalone Spider Sense on port 4000 that the four applications and the load generator forward to. scripts/demo.sh is the other layout, each application in agent mode hosting its own Spider Sense (bookstore 4000, orders 4001, warehouse 4002, worker 4003) with no extra process. Both take --no-build and RPS, log every process under build/demo-logs/, and print the CLI line to run next. The README and The Examples carry a prompt that hands the running demo to an AI agent: /spider-sense ($spider-sense in Codex) and a question, with no command in it. It asks for the three biggest problems, the line under examples/ behind each, and a proposed fix before any change. skills/ is linked from .claude/skills/ and .agents/skills/, where Claude Code and Codex look for a project’s skills, so an agent started in the checkout has the skill without init.

The published demo

The demo is also published as a static page, spider-sense.benelog.net/demo: the real UI over a recording of the shared demo, with no server behind it.

The recording is a DoltHub database, benelog/spider-sense-demo. It holds the tables of Storage as they are (service, span, trace, log, metric, metric_series, metric_point, tingle, mark, ack, db_table, meta) and one table more, answer. answer holds every answer the UI asks for, keyed by path and query without from and to, plus /manifest with the window, the services and the keys.

The page reads that database through DoltHub’s SQL API as it is opened (The UI). The aggregations come from answer. The lists that are the rows of one table are queried from the tables: the traces with the API’s filters, one trace with its spans and logs, the logs, the marks and the acknowledgements. Every trace of the recording therefore opens, and a filter on one of those lists runs as a query over the rows instead of returning a captured answer.

The answers exist because the aggregations are the server’s Queries and Findings, which the browser cannot run over Dolt. Dolt’s SQL has no PERCENTILE_DISC, and a second implementation of the aggregations in the page’s JavaScript would have to change with every change to Queries and Findings. The answers are computed once, when the recording is made, by the same jar that serves the UI.

scripts/demo-site.sh record runs demo-shared.sh for MINUTES (5), and marks before and after a third and two thirds of the way through, so the compare page has two windows. scripts/demo-site.mjs export then copies every row since the launch out of the H2 file into one CSV per table. The spans are copied by the trace they belong to, so a trace that straddles the end of the window is whole. The index catalog db_table is copied whole, since a table’s indexes are read once per process and usually before the window. The copy uses H2’s CSVWRITE, driven through org.h2.tools.Shell from the nested server jar, so it needs nothing but the jar. The home directory is anonymised as the agent examples are, and the window the page shows is written into meta as recording.from and recording.to.

The script then starts a fresh standalone Spider Sense on build/demo-data/sense with retention switched off. load inserts the CSVs into it with CSVREAD, after checking that the recording’s schema_version is the server’s. capture asks it every question the page does not answer from the tables, and writes the answers into answer.csv. Those are the lists unfiltered and per service, every endpoint, query, error and metric series they mention, every pair of mark names for compare that the server does not refuse, and the text rendering that Copy as Markdown asks for of every finding, query and error in the unfiltered lists. The answers say what the shared demo would: the capture server’s address becomes 127.0.0.1:4000 in every answer, and /api/status has no jar (so the UI offers spider-sense.jar), the default retention, and the default database file under ~/db/spider-sense/. push adds to DoltHub’s tables any column the schema gained since they were created, uploads each CSV through DoltHub’s import API, which lands it on a branch of its own, and merges that branch into main.

scripts/demo-site.sh recapture is the second half alone, from the rows already on DoltHub: pull, then load, capture and the push of answer. It is what to run after a change to the server’s queries, so the demo answers as the new server would without a new recording. A change to the schema needs record instead, since load refuses rows of another schema_version.

The Docs workflow redeploys the page on every push to main that touches the UI, but never the answers. So a change to the UI’s requests, the API or the queries is followed by record (or recapture), and the release procedure records the demo again too (RELEASING.md). assemble warns, and in GitHub Actions annotates the run, when the answers on DoltHub were captured before the last commit to assets/js, the api package or the query package.

scripts/demo-site.mjs assemble, which npm run docs runs, copies the UI into build/site/demo and marks index.html with data-dolthub="benelog/spider-sense-demo@main". That attribute makes app.js load assets/js/dev/replay.js instead of talking to a server. The Docs workflow needs neither Java nor the recording for it, and nothing of the recording is committed to the repository. Recording needs DOLTHUB_TOKEN, a writer’s API token of the database, kept in .envrc. The page needs nothing, since the database is public.

The agent demo

The prompt that hands the demo to an agent is published as it ran, too. Four pages replay a real session of Claude Code and of Codex CLI on it, each in English and in Korean, at agent-demo/claude-code/, …/claude-code/ko/, …/codex/ and …/codex/ko/. Nothing in them is written by hand.

scripts/agent-demo.sh record <claude|codex> <en|ko> does what a user of the prompt does. It starts scripts/demo-shared.sh --no-build and lets it take traffic for WARMUP_MINUTES (3). Only then does it run the agent headless in the checkout on that prompt, claude -p --output-format stream-json or codex exec --json; the Korean prompt is the English one’s translation. It stamps every line of the stream with the time it arrived, since neither stream carries one, and stops the demo once the agent is done.

Claude Code runs with the project’s settings only, and with none of the account’s claude.ai connectors or the user’s auto memory. A Korean session adds "language": "korean", since the user’s own setting is what makes Claude Code answer in Korean. Claude Code has no edit tool, and gets one appended line of system prompt saying that the session ends with the turn, since -p does not wait for a wait it left in the background. Codex runs in its workspace-write sandbox with the network open, since the CLI asks the running Spider Sense over HTTP, at model_reasoning_effort CODEX_EFFORT (medium). It runs with a CODEX_HOME of its own under build/agent-demo/, holding the user’s sign-in and a config.toml of model and model_reasoning_effort alone, so the user’s instructions, memories, hooks, skills and MCP servers stay out as Claude Code’s do. The script writes down the agent’s version and model for the push.

The tree is clean when a session is recorded, so what the agent reads of it is the commit. The session has a database of its own under build/agent-demo/db/, named by a file that SPIDERSENSE_CONFIG points at with spidersense.db. Its findings therefore cover that run and not what earlier ones left in ~/db/spider-sense/sense.

scripts/agent-demo.sh push (scripts/demo-site.mjs agent-push) turns the streams into two more tables of the same DoltHub database, with the home directory anonymised. agent_session has one row per session: the id, such as claude-ko, the agent, its version and model, the working directory, the prompt, when it was recorded, and how long it took. agent_event has one row per thing the terminal shows. Its kind is user, text, thinking, tool, plan or end, and it carries the tool’s name, its title and input, its output, whether it failed, and when it started and ended, in milliseconds from the start. An import replaces a table, so the push keeps the rows of every session it was not given.

The page lives in scripts/agent-demo/, which assemble copies to build/site/agent-demo with the session and the database set on <html>. It reads the two tables through the same SQL API and draws the session as a terminal styled like the agent’s own. A Claude Code session shows ⏺ Bash(…) with ⎿ under it, and a Codex session shows • Ran …, • Explored for its reads and searches, and ─ Worked for …. The page computes what is on screen from the play time alone, so the viewer can seek to any point. A pause between two events is capped at 1.6 seconds and an answer is revealed at a readable rate, while the spinner’s clock shows the recorded time. The agent’s two-minute wait therefore passes in two seconds and still says two minutes. Long outputs show their first four lines and expand on a click.

What was considered and rejected

An OpenTelemetry agent extension instead of our own premain. The extension mechanism (extensions/ inside the agent jar, AgentListener) would also work, and ExtensionClassLoader is already exclusion-listed. It was rejected because the UI would then depend on the agent’s SPI and lifecycle, and the standalone mode would still need a launcher of its own. A premain that wraps the agent’s premain keeps the server a separate program that receives the agent’s export over OTLP/HTTP, as it does in standalone mode.

One small extension beside the premain is worth it, though. The stock agent records where an exception was thrown and nothing at all about where a query was issued from, so slow-query and n-plus-one findings named a statement and left the reader to grep for it. A stack capture is taken only for database spans already over slow.query.ms, on the thread that is ending the span, and for the fifth repeat of a statement within a trace. A capture takes a few microseconds, against the hundred milliseconds or more a slow span has already taken, and the repeat capture happens once per repeated statement per trace. With it, those findings name the file and line that issued the statement. That is the extension: it sets one span attribute, code.stacktrace, which the server reads when a span carries it and does without when none does, in agent mode and standalone alike.

The index catalog is the second thing the extension does, and the first that needs bytecode advice rather than a span processor, because only the application’s own Connection can answer which indexes a table has. The advice is still confined to the extension, the catalog arrives as an OTLP log record, and the server only reads what the agent exports.

In-process export, with a custom SpanExporter handing spans straight to the store. It is faster, but it ties the store to the agent’s shaded SDK classes and makes the standalone and embedded paths diverge. Loopback OTLP is one HTTP request to 127.0.0.1 per export batch, and it goes through the same receiver the standalone mode uses.

In-memory only storage. The first design kept everything in bounded ring buffers and aggregated on demand, which is the simplest option and free of any database inside the application’s JVM. It was rejected because the analysis screen has to survive the monitored application going down, and because two embedded instances on one machine should show the same data. H2 with AUTO_SERVER gives both, and the class-loader exclusion keeps the OpenTelemetry agent from instrumenting its JDBC calls.

A frontend build (React, Vue, TypeScript). SigNoz and OpenObserve are built that way. Spider Sense is a single jar whose build must stay ./gradlew build with no Node. The UI is plain ES modules, one CSS file, and uPlot for charts, served by Spider Silk’s static files. A template engine was not used either: the UI is one page whose data all comes from the JSON API, and Spider Silk’s JSON and SSE support is the part of the framework this application exercises.

A gRPC receiver. gRPC needs Netty or Armeria in the jar, and OTLP/HTTP carries the same Export*ServiceRequest messages, as The server says.

Profiling, Glowroot-style stack sampling. It is not part of OpenTelemetry’s stable signals in Java, and it is deferred until the profiling signal lands in the agent.