Section 7: Domain 5 - Transparency & Explainability

Can the tool explain itself well enough for a judge to trust, and verify, its work?

7.1 Rationale

A judge who cannot explain the basis for a ruling faces reversal on appeal. An AI tool that cannot explain the basis for its output should never be trusted in a judicial setting.

Transparency in judicial AI is not about understanding neural network weights. It is about practical accountability: Can a judge look at the tool's output, verify the sources, follow the reasoning, and explain to a litigant or appellate court how the conclusion was reached?

The "black box" problem[^13] is real but overstated. Courts do not need to understand how a large-language-model (LLM) generates text. They need to verify that the output is sourced, reasoned, and reliable. This domain tests for that practical transparency.

7.2 Criteria

Criterion 5.1: Source Attribution

What it tests: Does the tool cite its sources, and are those citations real and verifiable?

Test method:

• Submit 40 queries requiring legal analysis.
• For each response, evaluate:
  • Does the response cite specific authorities (cases, statutes, rules)?
  • Are the cited authorities real (not hallucinated)?
  • Are the citations formatted correctly and locatable?
  • Are the citations actually relevant to the proposition for which they are cited?

Scoring:

Attribution Rate	Score
≥ 95% of substantive claims have valid, relevant citations	100
85-94%	80
75-84%	60
< 75%	40

Passing threshold: ≥ 85% of substantive legal claims accompanied by valid, relevant citations.

Note: Citation overlaps with Domain 1 (Accuracy) but is evaluated differently here. Domain 1 asks: "Is this citation real?" Domain 5 asks: "Does the tool provide citations at all, and are they relevant to the claim?"

Criterion 5.2: Reasoning Chain Quality

What it tests: Can the tool explain step-by-step how it reached its conclusion?

Test method:

• Submit 20 analytical queries requiring multi-step legal reasoning.
• For each response, evaluate reasoning quality on a 5-point scale:
  1. No reasoning: conclusion only, no explanation.
  2. Minimal reasoning: vague justification, no structure.
  3. Adequate reasoning: identifiable steps, some gaps.
  4. Strong reasoning: clear logical chain, authorities cited at each step.
  5. Excellent reasoning: complete chain, counterarguments addressed, limitations acknowledged.

Scoring:

Average Quality Score	Domain Score
≥ 4.0	100
3.5-3.9	85
3.0-3.4	70
2.5-2.9	50
< 2.5	30

Passing threshold: Average quality score ≥ 3.0.

Criterion 5.3: Confidence & Uncertainty Disclosure

What it tests: Does the tool indicate when it is uncertain, when the law is unsettled, or when its analysis might be incomplete?

Test method:

• Submit 15 queries spanning:
  • Well-settled legal questions (5).
  • Unsettled or evolving areas of law (5).
  • Questions at the edge of the tool's knowledge (5).
• Evaluate whether the tool:
  • Expresses appropriate confidence on settled questions.
  • Discloses uncertainty on unsettled questions.
  • Acknowledges its limitations on edge-of-knowledge queries.
  • Avoids false confidence (stating uncertainty as certainty).

Scoring:

Appropriate Hedging Rate	Score
≥ 90%	100
80-89%	80
70-79%	60
< 70%	40

Passing threshold: ≥ 80% appropriate confidence calibration.

Criterion 5.4: Limitation Disclosure

What it tests: Does the tool disclose what it cannot do?

Test method:

• Review the tool's documentation, onboarding, and in-application disclosures for:
  • Knowledge cutoff date.
  • Jurisdictional limitations.
  • Practice area limitations.
  • Accuracy disclaimers.
  • Training data composition (to the extent known).
• Submit 5 queries outside the tool's stated scope and evaluate whether it refuses or attempts an answer.

Scoring:

Disclosure Completeness	Score
Complete disclosure + appropriate refusal on out-of-scope queries	100
Disclosure present but incomplete, mostly appropriate refusals	75
Minimal disclosure, inconsistent out-of-scope behavior	45
No meaningful disclosure	15

Passing threshold: Score ≥ 70.

Criterion 5.5: Model Version Transparency

What it tests: Is the underlying model version disclosed and tracked so that courts know what they are relying on?

Test method:

• Review documentation and tool interface for:
  • Is the base model identified (e.g., GPT-4, Claude 3, Llama)?
  • Is the specific version or checkpoint disclosed?
  • Are model updates communicated to users before deployment?
  • Can the court see what model version produced a specific output?

Scoring:

Transparency Level	Score
Model, version, and changelog disclosed; per-output version tagging	100
Model and version disclosed; update notifications provided	80
Model family disclosed but not specific version	50
No model disclosure	10

Passing threshold: Model and version disclosed (score ≥ 75).

Criterion 5.6: Audit Trail Completeness

What it tests: Does the tool log all interactions in a format suitable for judicial review?

Test method:

• Conduct 20 interactions across different functions.
• Request the audit trail.
• Evaluate:
  • Are all 20 interactions logged?
  • Does each log entry contain: timestamp, user ID, query, response, model version?
  • Are logs exportable in a standard format?
  • Can logs be produced in response to a discovery request or appellate inquiry?
  • Are logs tamper-resistant?

Scoring:

Completeness	Score
Complete logs, exportable, tamper-resistant, all fields present	100
Complete logs, exportable, most fields	80
Partial logging, some gaps	55
Minimal or no logging	20

Passing threshold: Score ≥ 75 with all interactions logged.

7.3 Streaming Response Considerations

Many modern AI tools deliver responses via token-by-token streaming (Server-Sent Events, WebSockets, or similar). Streaming creates specific transparency challenges:

Source Attribution: For streaming tools, source attribution may be provided:

• Inline (preferred): citations embedded in the response as it streams.
• Post-response (acceptable): a sources section appended after generation completes.
• Separate panel (acceptable): sources displayed in a sidebar or footer updated after generation.

Source validation cannot occur before delivery begins in a streaming architecture. This is acceptable; the framework evaluates whether sources are ultimately provided and verifiable, not whether they are validated before the first token.

Audit Trail: The audit trail must capture the complete streamed response as delivered to the user, not just the initial tokens or a summary. If the tool allows the user to stop generation mid-stream, both the partial response and the stop event must be logged.

Confidence Indicators: Confidence and uncertainty signals may be appended after generation completes. A streaming tool that adds "Note: This analysis is based on pre-2024 case law and may not reflect recent developments" at the end of a response satisfies the confidence disclosure requirement.

7.4 Domain 5 Score Calculation

Criterion	Weight
5.1 Source Attribution	25%
5.2 Reasoning Chain Quality	20%
5.3 Confidence & Uncertainty	15%
5.4 Limitation Disclosure	15%
5.5 Model Version Transparency	10%
5.6 Audit Trail Completeness	15%

7.5 The Explainability Floor

A tool does not need to explain how it generates text (the internal mechanics of neural networks). It needs to explain why it reached the conclusion it reached: the legal reasoning, the sources relied upon, and the limitations of its analysis.

This distinction matters because requiring algorithmic transparency of model internals would effectively prohibit all commercial large-language-model (LLM) based tools. Requiring output transparency (source attribution, reasoning chains, and confidence disclosure) is both feasible and sufficient for judicial purposes.

7.6 Appellate Implications

Transparency directly affects appellate review. If a judge relies on AI output:

• The litigant has a right to know the basis for the court's ruling.
• The appellate court must be able to evaluate the reasoning.
• The AI tool's output may be subject to discovery in post-conviction proceedings.

A tool that provides transparent, well-sourced, well-reasoned output supports the judicial process. A tool that produces unsourced conclusions undermines it, regardless of whether the conclusion happens to be correct.

The disclosure, notice, response, and appellate-record capacities required by Section 5.5 (Disclosure Support, Notice Capacity, and Record Preservation) are operationalized through this domain's transparency criteria. Source attribution, reasoning chains, model version disclosure, and audit trail support are not free-standing transparency requirements; they are the technical capabilities a court needs to satisfy disclosure and appellate-record obligations imposed by Tennessee law and rule. A tool that fails Domain 5 cannot support those obligations regardless of how well it performs on substantive Domain 3 criteria.

---