Observability

Every Evolve service reports three signals. The storefront reports the same ones from your site’s own code.

Signal	Comes from	The reference sites send it to
Traces	OpenTelemetry, started by @evolve-framework/core	Honeycomb, over OTLP
Logs	logger, JSON lines on stdout	The cloud’s log service
Errors	Sentry, started with the traces	Sentry

Any backend that accepts OTLP works for traces. Nothing in Evolve is tied to Honeycomb; the reference sites just use it.

Turn it on in a service

Instrumentation patches modules as they load, so observability has to start before your service imports anything else. In a container, load it with --import:

ENV NODE_OPTIONS="--import @evolve-framework/core/observability/register"

The register module calls initObservability and then starts the event-loop watchdog, which reports a blocked event loop to Sentry with a stack.

To run a service locally without that flag, give it an entry file that starts observability first and imports the rest after:

import { initObservability } from "@evolve-framework/core/observability/configure";

initObservability();

// A static import would load the service before the instrumentation is in place.
await import("./src/cli.ts");

Use one of the two, never both: initObservability does not guard against running twice.

Point it at your backend

Observability starts before your service reads its configuration, so it takes its settings from environment variables only. Set them on the container. A value that sits only in your config store never reaches it.

Variable	What it does
OTEL_EXPORTER_OTLP_ENDPOINT	Where traces go. Without it, nothing is exported.
OTEL_EXPORTER_OTLP_HEADERS	Headers for the endpoint, such as the API key. Mount it as a secret.
OTEL_EXPORTER_OTLP_PROTOCOL	HTTP by default. Set grpc for a gRPC endpoint, such as one on port 4317.
OTEL_PROPAGATORS	How the trace travels between services. See Use one propagator everywhere.
OTEL_SERVICE_NAME	The service’s name in your traces. Defaults to SERVICE_NAME, then to the folder the service runs in.
ENVIRONMENT	The environment, on traces and Sentry events. Set it: Sentry calls an event without one production.
SENTRY_DSN	Where errors go. Without it, errors are only logged.
SENTRY_RELEASE	The release on Sentry events. Bake it into the image at build time.
LOG_LEVEL	info by default, debug when NODE_ENV is development.
OTEL_METRICS_EXPORTER	Set otlp to export metrics. They are off by default.

Honeycomb

services:
  purchase:
    env:
      ENVIRONMENT: tst
      OTEL_PROPAGATORS: b3
      OTEL_EXPORTER_OTLP_ENDPOINT: ${param:purchase-otel-exporter-otlp-endpoint}
      # x-honeycomb-team=<your key>
      OTEL_EXPORTER_OTLP_HEADERS: ${secret:purchase-otel-exporter-otlp-headers}
      SENTRY_DSN: ${param:purchase-sentry-dsn}

Any OTLP backend

services:
  purchase:
    env:
      ENVIRONMENT: tst
      OTEL_PROPAGATORS: tracecontext
      OTEL_EXPORTER_OTLP_ENDPOINT: https://otlp.example.com:4317
      OTEL_EXPORTER_OTLP_PROTOCOL: grpc
      OTEL_EXPORTER_OTLP_HEADERS: ${secret:purchase-otel-exporter-otlp-headers}
      SENTRY_DSN: ${param:purchase-sentry-dsn}

Locally, the reference sites run an OpenTelemetry collector in Docker Compose that passes traces to Jaeger. Open Jaeger to see your local requests.

Use one propagator everywhere

A request crosses four processes: the storefront, the gateway, the gateway’s proxy, and a subgraph. Each passes the trace on in a header, and each has to use the same header. A hop that uses another one starts a trace of its own, so the request shows up as separate pieces.

Choose the header for your cloud:

• b3 on Google Cloud. Google’s load balancer rewrites the W3C traceparent header between Cloud Run services to point at spans that only exist in Cloud Trace, so every service ends up as a detached subtree. It leaves b3 alone.
• tracecontext (W3C) everywhere else.

Then set it on every hop:

Hop	Where	b3	tracecontext
Storefront	frontend/site/src/instrumentation.server.ts	propagators: [new B3Propagator()]	propagators: ["tracecontext"]
Gateway	The router’s environment	OTEL_PROPAGATE_B3=true, OTEL_PROPAGATE_TRACE_CONTEXT=false	OTEL_PROPAGATE_TRACE_CONTEXT=true
Gateway proxy	The proxy’s environment	OTEL_PROPAGATORS=b3	OTEL_PROPAGATORS=tracecontext
Subgraphs	Deploy config, and the dev script locally	OTEL_PROPAGATORS=b3	OTEL_PROPAGATORS=tracecontext

The storefront only passes the trace on to the gateway. Its fetch.propagateContextUrls names SERVER_API_GATEWAY_URL, so a call to another API does not leak your trace headers.

Write logs

Import the logger and log with an object first, then a message:

import { logger } from "@evolve-framework/core/observability";

logger.info({ orderId: order.id, total: order.totalPrice.centAmount }, "Order placed");

The logger knows which request it is in. Every line carries the request’s traceId, path, method, and store context, without you passing them. Use the traceId to go from a log line to its trace.

The logger removes sensitive values before a line is written: any field, at any depth, whose name contains password, secret, token, authorization, cookie, email, or username. Name your fields so they are caught.

In production the logger writes JSON lines to stdout; your cloud’s log service picks them up. Locally it writes readable lines. A line at level error or above is also sent to Sentry, grouped by its message, so keep the message fixed and put the variable parts in the object.

Report errors

With SENTRY_DSN set, a service reports to Sentry:

• uncaught exceptions and unhandled rejections;
• every log line at level error or above;
• every unexpected error in a GraphQL request.

The GraphQL server tells expected errors from unexpected ones. A GraphQLError you throw yourself is an answer: the client gets it as it is, and nothing is reported. Give it a code in its extensions, so the storefront can act on it. Any other error is a bug: the client gets INTERNAL_SERVER_ERROR, and the details go to the log and to Sentry. So throw a GraphQLError for every outcome a shopper can cause, such as an invalid discount code.

To log every GraphQL error with its operation and variables, add useErrorLogging to your GraphQL plugins.

The storefront packages report nothing themselves. The reference sites set up Sentry in the site: on the server in instrumentation.server.ts, and in the browser through a client config that sends through the site’s own /api/capture-errors route, so ad blockers do not drop the reports.

Find the cause of a slow or failing request

Start from what you have: a traceId in a log line, or the trace id on a Sentry event. In the trace, these are the spans to read:

Span	Tells you
The site’s fetch to the gateway	How long the page waited for the API.
The gateway’s request and its subgraph fetches	Which subgraphs the query reached, and in parallel or one after another.
POST /graphql in a subgraph, and the operation under it	The operation’s name and its total time in that service.
Resolver Type.field	A field that did I/O, or failed. Fields that only read a value are left out.
Cache.wrapFn	A cached call. cache.cacheHit says whether the cache answered.
commercetools GET Product and the like	A call to commercetools. ct.duration is commercetools’ own time; the rest of the span is network.

Then look for the pattern:

1. One wide span. That call is your cost. If it is a commercetools call with a low ct.duration, the time went to the network.
2. cache.cacheHit is false where you expected a hit. Check the cache key and its TTL on the span.
3. A row of commercetools calls one after another. A resolver fetches per item. Batch the calls through a dataloader. Dataloaders have no spans of their own, so you see them only through the calls they make.
4. A gap with no spans. The service was busy with CPU work. Look in Sentry for an event-loop block from the same time; it has the stack.

An event that a service handles later, from a queue, starts a new trace. Find its handler’s trace through the log lines, by the entity’s id.

Check health

Compose HealthModule in each service and give it the checks that service depends on:

import { CompositeModule, HealthModule } from "@evolve-framework/core";
import { cache } from "@evolve-framework/core/cache";

new CompositeModule([
  // your other modules
  new HealthModule({
    name: "purchase",
    checks: {
      redis: () => cache.client?.ping(),
    },
  }),
]);

It adds:

• /healthcheck, for liveness. It runs no checks and returns the release.
• /readiness, which runs the checks and answers 503 when one fails.
• A health field in the GraphQL API, so one query through the gateway shows every service’s state and release.

Point your platform’s probes at the two routes. Requests to /healthcheck are left out of your traces.

Metrics

Metrics are off by default: the reference sites send traces through Honeycomb Refinery, which bills metrics as events. To turn them on, set OTEL_METRICS_EXPORTER=otlp and send to an endpoint that accepts OTLP metrics. The service then also reports runtime metrics. Record your own with meter:

import { meter } from "@evolve-framework/core/observability";

const ordersPlaced = meter.createCounter("orders.placed");
ordersPlaced.add(1, { store: storeKey });