> ## Documentation Index
> Fetch the complete documentation index at: https://docs.anyway.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Distributed Tracing

> Automatic spans for LLM providers, custom spans with decorators, association properties, and span attributes

## Overview

Anyway provides OpenTelemetry-compatible distributed tracing for your AI applications. Traces help you:

* Debug issues by following requests through your system
* Identify performance bottlenecks
* Understand the flow of AI operations
* Correlate LLM calls with your application logic

## How Tracing Works

Every AI operation creates a **trace** composed of **spans**:

```
Trace: user-request
├── Span: validate-input (2ms)
├── Span: openai.chat.completions (1,234ms)
│   └── Attributes: model=gpt-4o, tokens=150, cost=$0.0035
├── Span: process-response (5ms)
└── Span: save-to-db (12ms)
```

## Automatic Spans

The Anyway SDK automatically creates spans for supported LLM providers:

* OpenAI API calls
* Anthropic API calls
* AWS Bedrock, Google Vertex AI, Cohere, Together AI
* Vector DB operations (Pinecone, ChromaDB, Qdrant)

## Custom Spans with Decorators

Structure your traces with workflow and task decorators:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from anyway.sdk.decorators import workflow, task
    from openai import OpenAI

    client = OpenAI()

    @task(name="validate_input")
    def validate(user_input: str) -> str:
        # validation logic
        return user_input

    @task(name="llm_call")
    def call_llm(prompt: str) -> str:
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}]
        )
        return response.choices[0].message.content

    @workflow(name="process_query")
    def process_query(user_input: str) -> str:
        validated = validate(user_input)
        result = call_llm(validated)
        return result
    ```
  </Tab>

  <Tab title="JavaScript">
    ```typescript theme={null}
    import { withWorkflow, withTask } from "@anyway-sh/node-server-sdk";
    import OpenAI from "openai";

    const client = new OpenAI();

    async function validate(userInput: string) {
      return withTask({ name: "validate_input" }, async () => {
        // validation logic
        return userInput;
      });
    }

    async function callLlm(prompt: string) {
      return withTask({ name: "llm_call" }, async () => {
        const response = await client.chat.completions.create({
          model: "gpt-4o",
          messages: [{ role: "user", content: prompt }],
        });
        return response.choices[0].message.content;
      });
    }

    async function processQuery(userInput: string) {
      return withWorkflow({ name: "process_query" }, async () => {
        const validated = await validate(userInput);
        return callLlm(validated);
      });
    }
    ```
  </Tab>
</Tabs>

## Association Properties

Add metadata to traces for filtering and grouping:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    @workflow(name="user-query", association_properties={
        "user_id": "user-123",
        "team": "growth",
        "feature": "chatbot",
    })
    def handle_query(query: str):
        return call_llm(query)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```typescript theme={null}
    await withWorkflow(
      {
        name: "user-query",
        associationProperties: {
          userId: "user-123",
          team: "growth",
          feature: "chatbot",
        },
      },
      async () => {
        return callLlm(query);
      },
    );
    ```
  </Tab>
</Tabs>

Then filter traces in the dashboard by these properties.

## Span Attributes & Semantic Conventions

These attributes are automatically set for LLM spans, following the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/):

| Attribute                        | Description                  |
| -------------------------------- | ---------------------------- |
| `gen_ai.system`                  | Provider (openai, anthropic) |
| `gen_ai.request.model`           | Requested model              |
| `gen_ai.response.model`          | Model used in response       |
| `gen_ai.usage.prompt_tokens`     | Input/prompt tokens          |
| `gen_ai.usage.completion_tokens` | Output/completion tokens     |
| `gen_ai.usage.cost`              | Estimated cost in USD        |

## Viewing Traces

In the [Anyway Dashboard](https://app.anyway.sh):

1. Navigate to **Agent Traces**
2. Use filters to find specific traces:
   * Time range
   * Model
   * Latency threshold
   * Cost threshold
   * Association properties
3. Click a trace to see the full span tree

## Context Propagation

Anyway automatically propagates trace context across:

* Async functions
* Nested workflow/task calls
* HTTP requests (with W3C trace context headers)

## Next Steps

<CardGroup cols={2}>
  <Card title="Cost Tracking" icon="dollar-sign" href="/features/cost-tracking">
    Monitor spending in real-time
  </Card>

  <Card title="Payments" icon="credit-card" href="/features/payments">
    Create payment links and accept payments
  </Card>
</CardGroup>