Lab 006: Observing AI Reasoning with Phoenix

1.0 Observability for AI Processes

Once probabilistic decisions become part of process execution, we need visibility into how those decisions are behaving across many executions, not just whether something completed successfully.

In this lab, we’ll add observability to the AI judgment layer of the order fulfillment process. 

Using OpenTelemetry and Arize Phoenix, we will trace the three AI judgment functions introduced in earlier labs and capture operational information about how they behave during execution.

OpenTelemetry provides the instrumentation that creates spans, individual records of an operation such as one call to evaluate_service_level, including its timing and associated attributes.

Phoenix receives and stores those spans, then provides the interface and reporting used to inspect AI behavior across process executions.

By the end of the lab, you will be able to:

• See AI judgment calls as spans in Phoenix.
• Identify which judgment function executed.
• Measure how long AI judgments take to execute.
• Review decision distributions and confidence where available.
• Generate a simple operational summary across multiple process executions.

This lab adds visibility around the AI judgment functions so their behavior can be observed during normal process execution while the existing LangGraph process remains unchanged.

2.0 Architecture Blueprint

Lab 006 adds observability around the three AI judgment functions in the order fulfillment process.

The instrumented functions are:

• evaluate_service_level
• evaluate_material_acceptance
• evaluate_invoice

OpenTelemetry creates spans for these calls and exports them to Phoenix. Phoenix stores the resulting telemetry and makes it available for inspection and reporting.

The observability layer is separate from process execution, so the LangGraph workflow can continue even if tracing is unavailable.

3.0 Artifacts Created

Claude Code will modify two existing modules and create two new artifacts to add observability to the AI judgment layer.

3.1 llm/judgment.py

Adds OpenTelemetry instrumentation to the existing AI judgment functions.

The module will:

• Create a span for each AI judgment call.
• Record the model, backend, decision, confidence, latency, and token usage when available.
• Continue normal judgment processing if tracing fails.

3.2 observability/summary.py

Creates an operational summary from the Phoenix trace data.

The module reports:

• AI judgment calls by function.
• Average latency by function and backend.
• Decision distributions.
• Average confidence scores.
• Backend divergence when the trace data supports it.

3.3 observability/init.py

Creates the Python package for the observability components.

3.4 orchestration/run_batch.py

Updates the existing batch runner so tracing is enabled before the AI judgment functions are loaded.

4.0 Generate the Observability Layer

4.1 Prompt Overview

In this section, you will use Claude Code to add the observability capabilities described in the previous sections.

The prompt instruments the existing AI judgment functions, adds the Phoenix operational summary, and enables tracing during batch execution.

4.2 Claude Code Prompt

Copy the following prompt into Claude Code exactly as shown:

Update llm/judgment.py and create observability/summary.py. In llm/judgment.py: Add Phoenix instrumentation using arize-phoenix and opentelemetry. At the top of the file, add initialization code that starts a Phoenix session if the environment variable TRACING_ENABLED is set to 'true' (the default). Wrap all three judgment functions so that for each call, a span is created and recorded with: span name: the function name (evaluate_service_level, etc.) attributes: llm.backend (anthropic or huggingface) llm.model (model name used) llm.decision (the decision field from the returned Pydantic model) llm.confidence (confidence field if present, else None) llm.latency_ms (call duration in milliseconds) llm.prompt_tokens (if available from API response) Use try/except so that if tracing fails, the judgment function still returns normally. observability/summary.py After a batch run, print a formatted report to the terminal. Query Phoenix's local dataset using the phoenix client library. The report should include: Total AI judgment calls Calls by function name Average latency per function per backend Decision distribution per function Average confidence score per function Backend divergence count If Phoenix data is unavailable, print a message explaining how to generate trace data. observability/__init__.py Empty file. Also update orchestration/run_batch.py to set TRACING_ENABLED=true before importing the judgment module, so all batch runs are automatically traced.

5.0 Verify the Implementation

The implementation should be verified at two levels: that tracing and reporting work as intended, and that observability remains separate from process execution. Phoenix should provide visibility into AI behavior without becoming a dependency of the business process.

5.1 Start Phoenix

Start the local Phoenix server. 

Action

Open an additional Terminal window.

Navigate to the project directory, activate the Python virtual environment, and start Phoenix:

cd ~/order-fulfillment-ai
source venv/bin/activate
phoenix serve

Leave this Terminal window open while completing the remaining verification steps.

After Phoenix starts, open a browser and navigate to:

http://localhost:6006

Example Output

Verify That

• The phoenix serve command starts successfully and remains running.
• Opening http://localhost:6006 loads the Phoenix web interface.
• The initial Phoenix screen is visible and ready for the next verification step.

5.2 Generate Trace Data

Start a batch run to give Phoenix enough AI activity to observe while also verifying that instrumentation does not interfere with the existing LangGraph workflow.

Action

From the project directory, enable tracing and run the batch process as a Python module:

export TRACING_ENABLED=true
python3.11 -m orchestration.run_batch

Example Output

Verify That

• In Terminal, confirm that the batch summary reports Total orders: 20 and shows the final-status distribution for the run.
• In Phoenix, confirm that traces were created for the AI judgment functions executed during the batch. The trace count will not necessarily equal the number of orders because each order can invoke zero, one, or several AI judgment nodes.
• For this run, the 21 Phoenix traces are consistent with the batch results: 18 orders stopped at NEGOTIATION_REQUIRED after the service-level judgment, one order reached INVOICE_EXCEPTION after three AI judgments, and one order failed at ERP_FAILURE before reaching an AI judgment.
• Confirm that the batch completes without a Phoenix or OpenTelemetry error interrupting execution.

The implementation is designed so that tracing failures do not prevent the judgment functions from returning normally.

5.3 Verify the AI Judgment Spans

This step verifies that Phoenix is capturing the AI judgment functions themselves rather than simply observing the overall batch process.

Action

In Phoenix:

1. Open the Lab 006 project.
2. Select Spans.
3. Click one of the span records to open its details.
4. In the detail pane, look at the Trace field.

Example Output

The trace should identify one of the AI judgment functions, such as:

evaluate_service_level

Depending on how far each order progressed through the process, you may also see:

evaluate_material_acceptance
evaluate_invoice

Verify That

1. In the Spans view, confirm that the project contains the spans generated by the batch.
2. Open a span and confirm that the Trace field identifies an AI judgment function such as evaluate_service_level.
3. Confirm that evaluate_material_acceptance and evaluate_invoice also appear for orders that progressed through those judgment points.

This confirms that observability has been applied directly to the AI judgment layer of the process.

5.4 Verify the Operational Summary

This step verifies that the individual Phoenix traces can be aggregated into an operational view of AI behavior across the process.

But, these labs are themselves a work in progress built with a nondeterministic LLM coding assistant, and the generated implementation will not always be correct on the first pass.

At this point, the initial implementation of observability/summary.py needs one refinement. The summary script should query the trace data produced by the existing Phoenix installation without attempting to start another Phoenix server when one is already running.

Action

In Claude Code, enter the following prompt:

Update observability/summary.py so that it queries the existing local Phoenix instance or persisted Phoenix data without starting a new Phoenix server. Phoenix is already running separately via phoenix serve, and summary.py currently causes a port 4317 binding error before printing the report. Preserve the existing report output and metrics.

Claude will update observability/summary.py so that it reads the existing Phoenix trace data without initializing another server process.

After Claude completes the update, return to a Terminal window that is not running Phoenix.

Navigate to the project directory, activate the virtual environment, and enter the following:

cd ~/order-fulfillment-ai
source venv/bin/activate
python3.11 -m observability.summary

Example Output

The report should summarize the AI judgment activity captured during the traced batch run.

Verify That

• In Terminal, confirm that the command runs cleanly and displays the AI Judgment Observability Report.
• Confirm that the report breaks AI judgment calls down by function and backend.
• Confirm that average latency, decision distribution, and confidence are reported where available.
• Confirm that backend divergence is not reported when the trace data cannot reliably correlate equivalent inputs across models.

This verifies that individual judgment traces can be aggregated into an operational view of how the AI components are behaving across the process.

5.5 Verify Observability Does Not Control the Process

The final step is just a simple exercise to show that Phoenix is an observability capability and not a runtime dependency of the order-fulfillment process.

Disabling tracing should stop new telemetry from being recorded without affecting LangGraph execution or the AI judgment functions themselves.

Action

In a Terminal window that is not running Phoenix, disable tracing and run a single order:

export TRACING_ENABLED=false
python3.11 -m orchestration.run

Leave Phoenix open in the browser so you can compare the trace count before and after the run.

Example Output

The order should still execute through the LangGraph process and finish with a normal final status, such as:

Final status: COMPLETED

or one of the defined exception states:

NEGOTIATION_REQUIRED
SL_REJECTED
QA_REJECTION
INVOICE_EXCEPTION
ERP_FAILURE

Phoenix should not record new AI judgment spans from this untraced run.

Verify That

• In Terminal, confirm that the order executes normally and reaches a final process status even though TRACING_ENABLED is set to false.
• Confirm that AI judgment still occurs when the order reaches a judgment node. Disabling tracing should not disable the LLM calls or change the process-routing logic.
• In Phoenix, refresh the project after the run and confirm that the trace or span count has not increased.

This confirms that Phoenix observes the AI judgment layer without participating in the business process itself.

Re-enable tracing when finished if you plan to continue experimenting:

export TRACING_ENABLED=true

6.0 Lab Summary

In this lab, you added observability to the AI judgment layer of the order fulfillment process.

OpenTelemetry created spans for individual AI judgment calls, and Arize Phoenix collected those spans so they could be inspected and summarized across process executions.

You also generated an operational report showing call volume, latency, decision distribution, and confidence where available.

The important architectural result is that the AI judgment layer can now be observed without becoming coupled to the observability platform. The LangGraph process continues to execute independently, while Phoenix provides visibility into how the probabilistic components are behaving.

At this point, the lab series has moved from implementing AI judgment to operating it with human oversight and observability.

7.0 Lab Series Conclusion

The Enterprise AI Architecture Labs series is now complete.

Across six labs, we built an AI-enabled order fulfillment process that combines retrieval, probabilistic judgment, conventional business logic, orchestration, human oversight, and observability into a single working architecture.

Two additional articles will follow the lab series:

The first will examine the quality and security of the code generated by Claude Code, including where AI-generated implementation deserves additional review before it would be appropriate for production use.

The second will summarize the complete lab series and connect the individual implementations back to the larger architectural ideas they were intended to demonstrate.

The labs are complete. The next step is to step back from the implementation and evaluate what we built.