The Three Modes
One jar runs three ways: as an agent that also hosts the UI, as an agent that forwards to a Spider Sense elsewhere, and as a collector and UI on their own. The jar is also a command line: with a command as its first argument it starts no server.
Spider Sense is a -javaagent, so it has to be on the JVM’s command line when the application starts.
An application that is already running cannot be brought under it, and there is no supported way to attach later.
Restart the application with -javaagent on its command line.
The examples below assume the jar’s absolute path is in $SENSE, as Installation sets it.
| Mode | Command | When |
|---|---|---|
Agent |
|
One application. The collector and the UI run inside its JVM on port 4000. |
Agent, forwarding |
|
Several applications sharing one Spider Sense, or an application that must not host a web server. |
Standalone |
|
The collector and the UI only, for anything that already speaks OTLP/HTTP, including a Java application under the stock OpenTelemetry Java agent. |
Agent
java -javaagent:"$SENSE" -Dotel.service.name=my-app -jar build/libs/my-app.jar
java -javaagent:"$SENSE" -Dotel.service.name=my-app -cp build/classes/java/main com.acme.Main
The OpenTelemetry Java agent instruments the application, and a collector and UI start inside the same JVM on port 4000. The agent exports to them over loopback. Agent mode needs one JVM option and no second process, and the UI is at http://127.0.0.1:4000.
Before the application’s own output, the launcher prints one line on stdout:
[spider-sense] UI: http://127.0.0.1:4000 service: my-app CLI: java -jar /home/me/tools/spider-sense-0.1.0.jar findings --since=start
It names the UI, the service the telemetry carries, and the command that asks this Spider Sense from a terminal, with the jar’s own absolute path.
A few seconds later the JVM may print four WARNING: lines about sun.misc.Unsafe on stderr, which come from Spider Sense’s receiver and are harmless (When it does not work).
-Dspidersense.port=4001 moves the UI when 4000 is taken, and the line’s command then carries --url=http://127.0.0.1:4001, because the CLI asks port 4000 when it is not told otherwise.
Spider Sense never prevents the application from starting: a failure in its own startup is logged to the application’s stderr and swallowed.
Agent, forwarding
java -javaagent:"$SENSE" -Dspidersense.collector=http://127.0.0.1:4000 -jar app.jar
The instrumentation is the same as in agent mode, but no UI starts in the application’s JVM. The agent exports to a Spider Sense running elsewhere, so several applications can share one Spider Sense.
Use forwarding when two applications call each other, so that a trace crossing both arrives in one place.
Forwarding is also how an application on Java 17 to 24 is measured, because the embedded UI needs Java 25 (Java versions).
Start the standalone Spider Sense first, then each application with -Dspidersense.collector= pointing at it.
When nothing accepts a connection at the collector as the application starts, the launcher says so once on stderr, with the command that starts a Spider Sense there when the collector is on this machine:
[spider-sense] nothing answers at http://127.0.0.1:4199 yet; start it with: java -jar /home/me/tools/spider-sense-0.1.0.jar --port=4199 (exports are retried)
The application starts and exports all the same, and the first export after the collector is up arrives.
Until then the OpenTelemetry agent logs each failed export with its stack trace, a few times a minute at most.
They are left on: the agent’s only switch for them, -Dotel.javaagent.logging=none, would also silence every export error that has another cause.
Forwarding is also the right mode for a test task.
A test JVM is short-lived and may be forked more than once, and only one of several embedded instances can bind the port.
Standalone
java -jar "$SENSE"
The collector and the UI run on port 4000 and nothing is instrumented.
Anything that speaks OTLP/HTTP can send to it: the two modes above, another language’s SDK, or an OpenTelemetry Collector.
The standalone jar takes its properties as --key=value arguments as well as system properties, so java -jar "$SENSE" --port=4001 is the same as -Dspidersense.port=4001.
The banner names the UI, the database, and the commands to send to it and to ask it, each with the jar’s own absolute path.
A command single-quotes a URL a shell would read as a pattern, such as 'http://[::1]:4000', so zsh takes it as it is.
It quotes the jar’s path the same way when the path holds a space, a quote, a $ or anything else a shell would split or expand, such as '/home/me/OBrien'\''s tools/spider-sense-0.1.0.jar', and so does every other line of the launcher that names the jar.
The difference is that an argument is checked: a malformed number such as --port=abc, or a key the jar does not take such as --prot=4001, is a usage error that exits 2 without starting, where -Dspidersense.port=abc is a warning and the default (Configuration lists the keys).
Every argument is a --key=value, apart from --help and --version, so a key without a value, a lone --, and a word such as a command after a flag are usage errors too.
A command goes first, as The CLI describes.
A port that is held is one line on stderr and exit 2, and the banner is printed only once the port is bound.
When a Spider Sense holds it, standalone or embedded in an application, the line names that one, which shares the default database and is most likely what you wanted:
spider-sense: a Spider Sense is already running at http://127.0.0.1:4000; open it, or ask it from a terminal (java -jar /home/me/tools/spider-sense-0.1.0.jar help lists the commands); --port= starts a second one
When anything else holds it, the line names the key that moves this one, by each of its names: spider-sense: port 4001 is in use; set spidersense.port to a free one (--port= on the command line, port in the Gradle block) (…).
A port this user may not bind, such as 80 for a user other than root, is not held, and the line says so: spider-sense: port 80 needs privileges this user does not have; set spidersense.port to a port above 1023 (…).
--port=0 binds any free port and names it in the banner.
It reads the properties file too.
Two environment variables point an OpenTelemetry SDK at the standalone Spider Sense:
java -jar "$SENSE" &
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4000
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
The protocol has to be http/protobuf, because there is no gRPC receiver.
Python, Node.js and Go
Python and Node.js have zero-code instrumentation, like the Java agent: a package installed beside the application instruments its web framework, HTTP clients and database drivers when the process starts.
Go has no such agent, so the application sets up the SDK in main and wraps its handler.
Each example below sends to a standalone Spider Sense on port 4000 under the service name my-app.
Python, with opentelemetry-instrument wrapping the usual command:
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install # the instrumentation for every library installed
export OTEL_SERVICE_NAME=my-app
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4000
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
opentelemetry-instrument --logs_exporter otlp python app.py
opentelemetry-bootstrap reads the packages already installed, so run it after the application’s own dependencies.
The last variable and --logs_exporter otlp send the logging records as well; without them only traces and metrics arrive.
Node.js, with the instrumentation loaded by --require:
npm install @opentelemetry/api @opentelemetry/auto-instrumentations-node
export OTEL_SERVICE_NAME=my-app
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4000
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_NODE_RESOURCE_DETECTORS=env,host,os,process
node --require @opentelemetry/auto-instrumentations-node/register app.js
Traces are exported by default, and the metrics and logs only when their exporter variable says otlp.
The last variable leaves out the cloud detectors, which otherwise probe a metadata server and print MetadataLookupWarning on a laptop.
Go, with the SDK set up in main:
go get go.opentelemetry.io/otel \
go.opentelemetry.io/otel/sdk \
go.opentelemetry.io/otel/sdk/metric \
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp \
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp \
go.opentelemetry.io/contrib/instrumentation/runtime
package main
import (
"context"
"log"
"net/http"
"time"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
"go.opentelemetry.io/contrib/instrumentation/runtime"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
)
func main() {
ctx := context.Background()
traceExporter, err := otlptracehttp.New(ctx) // reads OTEL_EXPORTER_OTLP_ENDPOINT
if err != nil {
log.Fatal(err)
}
tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(traceExporter)) // reads OTEL_SERVICE_NAME
defer tp.Shutdown(ctx)
otel.SetTracerProvider(tp)
metricExporter, err := otlpmetrichttp.New(ctx) // reads OTEL_EXPORTER_OTLP_ENDPOINT
if err != nil {
log.Fatal(err)
}
mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(
sdkmetric.NewPeriodicReader(metricExporter, sdkmetric.WithInterval(10*time.Second))))
defer mp.Shutdown(ctx)
otel.SetMeterProvider(mp)
if err := runtime.Start(); err != nil { // goroutines, heap, GC
log.Fatal(err)
}
mux := http.NewServeMux()
mux.HandleFunc("GET /hello", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("hello"))
})
log.Fatal(http.ListenAndServe(":8080", otelhttp.NewHandler(mux, "server")))
}
OTEL_SERVICE_NAME=my-app OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4000 go run .
The otlptracehttp and otlpmetrichttp exporters send OTLP/HTTP only, so Go needs no protocol variable.
The meter provider is what otelhttp records http.server.request.duration to, and runtime.Start adds the goroutine count, the heap and the GC to it; both appear on the Metrics page.
The reader exports every ten seconds instead of the SDK’s default of sixty, so a chart gets a new point every ten seconds.
otelhttp names each server span after the ServeMux pattern, so the Endpoints page shows GET /hello rather than one row for the whole server.
The queries appear only when the database calls make spans too, which in Go means wrapping database/sql with an instrumentation library, and in Python and Node.js happens for the drivers the packages above cover.
A start mark, the moment --since=start reads, is written when a service reports a process.pid it has not reported before (Automatic start marks).
The Node.js example reports one through its process detector.
Python reports one with OTEL_EXPERIMENTAL_RESOURCE_DETECTORS=otel,process, and Go with resource.WithProcessPID() in the resource it gives the tracer provider; without it there is no start mark, and --since=15m, or mark before and --since=before, names the window instead.
The receiver is the same for all three, so the pages, the findings and the CLI treat them as they treat a Java service.
The Java-only parts do not apply: the code.stacktrace of a slow query and the index catalog come from the Spider Sense jar’s extension.
The same two variables make a Java application send to Spider Sense from any OpenTelemetry-compatible agent, not only the Spider Sense jar: the stock opentelemetry-javaagent.jar at a version of your own choosing, a vendor’s distribution of it, or an agent already on the command line for another reason.
The stock agent is one jar, published on the project’s GitHub releases and on Maven Central as io.opentelemetry.javaagent:opentelemetry-javaagent.
The first line below fetches the latest release, and the second the version the Spider Sense jar bundles:
curl -LO https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
curl -L -o opentelemetry-javaagent.jar https://repo1.maven.org/maven2/io/opentelemetry/javaagent/opentelemetry-javaagent/2.31.1/opentelemetry-javaagent-2.31.1.jar
java -jar "$SENSE" &
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4000 OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
java -javaagent:opentelemetry-javaagent.jar -jar app.jar
This setup lacks the defaults and the extension that the Spider Sense jar adds in agent mode: the endpoint and protocol set for you, a one-second export delay, the runtime metrics on, and code.stacktrace on slow queries (What premain does and The extension).
Everything else, the pages, the findings and the CLI, is the same, because the receiver stores the stock agent’s spans, metrics and logs in the same tables.
The extension is also what reads the index catalog of the tables a slow statement touches, so a finding’s schema block exists only for a service that ran under the Spider Sense jar.
It is also what reports the route of a Spider Silk request, so under the stock agent every request of a Spider Silk application is the one endpoint GET /* (Spider Silk routes).
The receiver also accepts OTLP/JSON, so a shell can post a span with curl:
curl -X POST http://127.0.0.1:4000/v1/traces \
-H 'Content-Type: application/json' \
-d @spans.json
The HTTP API gives the three ingest paths and what they answer.
Standalone is also how you read data after the application has gone.
The H2 file under ~/db/spider-sense/ outlives every process, so java -jar "$SENSE" opens the same screens on the same rows.
The command line, the fourth way
java -jar "$SENSE" findings --since=start
An argument that does not start with - is a command, and no server starts.
The command asks the running Spider Sense over HTTP, or reads the H2 file directly when none is running, and prints text.
It is for people in a terminal and for AI agents.
The CLI documents every command, and The Loop gives the order in which an AI agent runs them.
Java versions
Standalone, the command line and the embedded UI need Java 25 or later.
An application on Java 17 to 24 is measured by forwarding: start Spider Sense standalone on Java 25 or later, and give the application -Dspidersense.collector= pointing at it.
Without a collector the agent starts no UI on such a JVM, and the telemetry goes to a Spider Sense that already answers at spidersense.port.
When none answers there, nothing is exported, and the launcher says so in one line on stderr:
[spider-sense] the embedded UI needs Java 25 or later, and this JVM is Java 21.0.8, so nothing is exported; forward to a Spider Sense on Java 25 with -Dspidersense.collector=http://127.0.0.1:4000, started by: java -jar /home/me/tools/spider-sense-0.1.0.jar
On Java 8 to 16 the launcher prints one line, this JVM is Java <version>, and Spider Sense needs 17 or later; the application runs without it, and the application starts uninstrumented.
Installation has the whole table.
Starting a JVM that is not java -jar
A build tool or a start script that forks the JVM has to pass -javaagent on to it.
For Gradle and Maven, the Gradle plugin and the Maven parameters are the shortest route, and neither needs an absolute path in the build file.
The sections below are for everything else.
A start script from installDist
The script the Gradle application plugin generates passes JAVA_OPTS and <APPNAME>_OPTS to the JVM, so nothing in the build has to change.
./gradlew installDist
JAVA_OPTS="-javaagent:$SENSE -Dotel.service.name=my-app" build/install/my-app/bin/my-app
applicationDefaultJvmArgs in build.gradle can also carry these arguments, but that edits the project.
Gradle run
The run task forks a JVM whose arguments come from the build, not from the shell, so either configure the task or avoid it.
tasks.named('run') {
jvmArgs '-javaagent:/absolute/path/to/spider-sense-0.1.0.jar', '-Dotel.service.name=my-app'
}
When the build file is not yours to edit, use ./gradlew installDist and the start script above, which change nothing in the build.
JAVA_TOOL_OPTIONS works too, but every JVM the command starts picks it up, including the Gradle daemon, which then tries to host a Spider Sense of its own on port 4000.
Use --no-daemon with it, or set spidersense.collector so that no JVM the command starts hosts a Spider Sense (Agent, forwarding).
|
Knowing it worked
java -jar "$SENSE" status
status names the mode (agent or standalone), the port, the OTLP endpoint, the database file and its size, the thresholds in force, the services seen and how many spans are stored.
The setup works when the services row names the application’s service, as in | services | 1 (my-app) |, and the span count grows after a request.
A service named unknown_service:java is an application started without -Dotel.service.name=.
The UI at http://127.0.0.1:4000 shows the same information to a person.
A restart also writes an automatic start mark for the service, so marks shows a new row with the note pid <pid> instance <service.instance.id> every time the application comes up.
--since=start therefore means the current run.
Marks describes them.
When it does not work
| Symptom | Cause |
|---|---|
|
Nothing is running on that port, or the application was started on another one. Pass |
The application starts but |
The agent did not attach. |
The service is not called what you expected |
Without |
Port 4000 is already in use |
When another Spider Sense, standalone or embedded in another application, has it, the application’s telemetry goes to that one, and the start-up line says so on stdout: |
|
The cause follows on the same line. The nested jar may not have been extracted, such as to a |
|
The application forwards to a Spider Sense that is not running. Start it with the command the line gives; the exports are retried, and the stack traces stop once it answers. |
Two applications, and each trace stops at the service boundary |
Both are embedding their own Spider Sense. Put them in forwarding mode against one standalone. |
The application crashed and the UI went with it |
The CLI reads the H2 file directly and says so on stderr. |
A finding’s |
Only errors carry stack traces, and the package heuristic can guess wrong. Set |
The window is full of another run |
Windows default to |
Four |
The protobuf library of Spider Sense’s own OTLP receiver calls |
When the application runs but Spider Sense collects nothing, run status and read the [spider-sense] lines the launcher printed.
status names the port, the mode and the span count, and the launcher prints its UI: line on stdout and a failure on stderr.
scripts/demo.sh and scripts/demo-shared.sh in The Examples run these modes with the example applications: demo.sh starts four applications each with its own embedded Spider Sense, and demo-shared.sh forwards the same four to one standalone.