Observability

#One value, both ends

next carries the request on, to the route or to the network, and it may be handed a changed one. This Around stamps a header and records where it ran. The client and the server it calls each run it once.

for
  seen <- Ref.make(Chunk.empty[(Side, Option[String])])
  stamp = new Around[Any]:
    def apply[R1, E](exchange: Exchange)(next: Request => ZIO[R1, E, Response]) =
      seen.update(_ :+ (exchange.side -> exchange.route.map(_.render))) *>
        next(exchange.request.withHeader("X-Stamped", "yes"))
  users = Routes(
    Method.GET / "users" / int("id") handle { (id, req) =>
      ZIO.succeed(Response.text(s"$id ${req.header("X-Stamped").getOrElse("no")}"))
    }
  ) @@ Middleware.around(stamp)
  body <- ZIO
    .serviceWithZIO[Client](_.batched(Request.get("http://users.internal/users/7")))
    .flatMap(_.body.utf8)
    .provideLayer(Client.inMemory(users) >>> Client.around(stamp))
  all <- seen.get
yield (body, all)

(7 yes,Chunk((Client,None),(Server,Some(GET /users/{id}))))

#Metrics

Middleware.metrics() keeps zio-http's three server metrics through zio.metrics, so a dashboard built for zio-http reads a heddle server unchanged: http_requests_total (a counter), http_concurrent_requests_total (a gauge of requests in flight), and http_request_duration_seconds (a histogram in seconds). Labels are method, path (the template), and status.

A client keeps the same three under http_client_requests_total, http_client_concurrent_requests_total, and http_client_request_duration_seconds, with host in place of path. Around.metrics() is one value for both: it picks the names by Side.

status is the response's code. An effect that fails with a Response reports that response's code, and any other failure (a ClientError, a typed domain error, an interruption) reports failed. The in-flight gauge comes back down on every exit, interruption included.

{
  val users    = Routes(Method.GET / "users" / int("id") -> { id => ZIO.succeed(Response.text(id.toString)) })
  val app      = users @@ Middleware.metrics()
  val requests = Metric
    .counterInt(server.requestsTotal)
    .tagged(MetricLabel("method", "GET"), MetricLabel("path", "/users/{id}"), MetricLabel("status", "200"))
  for
    before <- requests.value
    _      <- app(Request.get("/users/1")) *> app(Request.get("/users/2"))
    after  <- requests.value
  yield after.count - before.count
}

2.0

extraLabels rides on every metric (a service name, a region). boundaries replaces the histogram buckets. Around.MetricNames renames all three when two servers share one registry.

#Tracing

OpenTelemetry is JVM-only, so spans are their own artifact:

libraryDependencies += "rocks.earlyeffect" %% "heddle-opentelemetry" % "0.11.0"

HeddleTracing.around(tracing) is an Around over zio-opentelemetry's Tracing, and HeddleTracing.live reads it from the environment. A server opens a SERVER span named for the route (GET /users/{id}) and continues the caller's trace from its traceparent. A client opens a CLIENT span named for the method and sends its context on as traceparent. Spans carry OpenTelemetry's HTTP attribute names: http.request.method, http.route, url.path, url.full, server.address, and http.response.status_code. A server marks a 5xx as an error, a client a 4xx or a 5xx, and either one marks an effect that failed.

Here a traced client calls a traced server. The two spans are one trace, and the server's parent is the client's span. The exporter is in memory so the page can read it; in production Tracing comes from your OpenTelemetry SDK or agent.

{
  val spans    = InMemorySpanExporter.create()
  val provider = SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spans)).build()
  val tracing  = ZLayer.make[Tracing](
    ZLayer.succeed(provider.get("docs")),
    Tracing.live(),
    OpenTelemetry.contextZIO,
  )
  val users = Routes(Method.GET / "users" / int("id") -> { id => ZIO.succeed(Response.text(id.toString)) })
  val call  =
    for
      traced <- HeddleTracing.live
      _      <- ZIO
        .serviceWithZIO[Client](_.batched(Request.get("http://users.internal/users/7")))
        .provideSomeLayer[Tracing](
          Client.inMemory(users @@ Middleware.around(traced)) >>> Client.around(traced)
        )
    yield Chunk.fromIterable(spans.getFinishedSpanItems.asScala)
  call.provideLayer(tracing)
}

Chunk(SpanData{spanContext=ImmutableSpanContext{traceId=ad9cac83ed4c8948603d16c5bb1c895f, spanId=1740914478d3fe06, traceFlags=03, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=ad9cac83ed4c8948603d16c5bb1c895f, spanId=e7100117934156b7, traceFlags=03, traceState=ArrayBasedTraceState{entries=[]}, remote=true, valid=true}, resource=Resource{schemaUrl=null, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.63.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=docs, version=null, schemaUrl=null, attributes={}}, name=GET /users/{id}, kind=SERVER, startEpochNanos=1793483801987505974, endEpochNanos=1793483801989505371, attributes=AttributesMap{data={http.response.status_code=200, http.request.method=GET, http.route=/users/{id}, url.path=/users/7}, capacity=128, totalAddedValues=4}, totalAttributeCount=4, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true},SpanData{spanContext=ImmutableSpanContext{traceId=ad9cac83ed4c8948603d16c5bb1c895f, spanId=e7100117934156b7, traceFlags=03, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, resource=Resource{schemaUrl=null, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.63.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=docs, version=null, schemaUrl=null, attributes={}}, name=GET, kind=CLIENT, startEpochNanos=1793483801965053007, endEpochNanos=1793483801991126755, attributes=AttributesMap{data={url.full=http://users.internal/users/7, http.response.status_code=200, http.request.method=GET, server.address=users.internal}, capacity=128, totalAddedValues=4}, totalAttributeCount=4, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true})

#Order

Apply tracing outside metrics when both are on, so the span covers the time the metric records: routes @@ Middleware.around(HeddleTracing.around(tracing) ++ Around.metrics()). On a client the same value goes on client @@.