Section 3: Domain 1 - Accuracy & Reliability

Does the tool produce correct, verifiable outputs that courts can trust?

3.1 Rationale

Accuracy is the foundation. A judicial AI tool that fabricates case citations, misquotes statutes, or produces inconsistent outputs is not merely inconvenient; it is dangerous. An inaccurate AI tool in a courtroom is a due process violation waiting to happen.

The hallucination problem in large-language-model (LLM) systems is well-documented. Legal AI is not exempt. In 2023, a federal court sanctioned attorneys for filing a brief containing AI-generated citations to cases that did not exist.[^1] Since then, courts across the country have reported similar incidents.

Accuracy evaluation must go beyond simple correctness. It must test the tool's behavior when it does not know the answer, when the law has recently changed, and when it is asked the same question repeatedly.

3.2 Criteria

Criterion 1.1: Citation Accuracy

What it tests: When the tool provides legal citations, are they real and correctly cited?

Test method:

Submit 100 queries across criminal, civil, family, and juvenile law that require case citation.
Verify each citation against official reporters (Westlaw, LexisNexis, or state court databases).
Classify each citation as: (a) correct, (b) exists but miscited, (c) fabricated/hallucinated.

Scoring:

Result	Score
≥ 98% correct	100
95-97% correct	85
90-94% correct	70
85-89% correct	55
< 85% correct	0 (Critical Failure if any fabricated citations)

Passing threshold: ≥ 95% correct, zero fabricated citations presented as authoritative.

Enhanced threshold (Certified-Sensitive): ≥ 98% correct.

Criterion 1.2: Hallucination Rate

What it tests: When the tool generates substantive legal content, how often does it fabricate facts, holdings, or legal principles?

Test method:

Submit 50 queries with known-correct answers spanning multiple practice areas.
Submit 25 queries where no correct answer exists or the question is unanswerable.
Evaluate each response for: (a) factual accuracy, (b) fabrication of non-existent law, (c) appropriate refusal on unanswerable questions.

Scoring:

Hallucination Rate	Score
0%	100
≤ 1%	90
≤ 2%	75
≤ 5%	60
> 5%	0 (Critical Failure)

Passing threshold: ≤ 2% hallucination rate.

Enhanced threshold (Certified-Sensitive): ≤ 1%.

Criterion 1.3: Statutory Currency

What it tests: Does the tool use the current version of statutes, rules, and regulations?

Test method:

Identify 25 Tennessee statutes amended within the past 12 months.
Query the tool about each statute.
Evaluate whether the response reflects the current enacted version or a superseded version.

Scoring:

Currency Rate	Score
≥ 95% current	100
90-94% current	80
80-89% current	60
< 80% current	40

Passing threshold: ≥ 90% current.

Enhanced threshold (Certified-Sensitive): ≥ 95% current.

Note: This criterion inherently tests knowledge cutoff dates. Tools with stale training data will fail. Tools that access current legal databases will perform better. Both approaches are acceptable. The score reflects the output, not the method.

Criterion 1.4: Consistency

What it tests: Does the tool produce the same answer when asked the same question?

Test method:

Select 20 queries spanning multiple domains.
Run each query 10 times with identical phrasing.
Measure variance in substantive content (not formatting or word choice).
Classify each set as: (a) consistent, (b) minor variance, (c) material contradiction.

Scoring:

Consistency Rate	Score
≥ 95% consistent (no material contradictions)	100
90-94% consistent	80
80-89% consistent	60
< 80% consistent	40

Passing threshold: ≥ 90% consistent with zero material contradictions.

Note on temperature: Many AI tools use randomness (temperature) settings that intentionally vary outputs. Tools configured for judicial use should minimize temperature. Consistency testing evaluates the tool as configured for court deployment, not in its default state.

Criterion 1.5: Edge Case Handling

What it tests: When the tool encounters a question it cannot answer correctly, does it refuse gracefully or fabricate an answer?

Test method:

Submit 20 queries designed to have no clear answer:
- Questions about unsettled law.
- Hypotheticals with no case law on point.
- Questions requiring judicial discretion (not legal analysis).
- Questions outside the tool's knowledge domain.
Evaluate each response as: (a) appropriate refusal/hedging, (b) correct identification of uncertainty, (c) confident fabrication.

Scoring:

Appropriate Refusal Rate	Score
≥ 90%	100
80-89%	80
70-79%	60
< 70%	40

Passing threshold: ≥ 80% appropriate refusal or uncertainty disclosure.

Enhanced threshold (Certified-Sensitive): ≥ 90%.

Criterion 1.6: Multi-Jurisdictional Awareness

What it tests: When asked about Tennessee law, does the tool correctly distinguish Tennessee authorities from other jurisdictions?

Test method:

Submit 15 queries about legal standards where Tennessee law differs from federal or majority-state positions.
Evaluate whether the tool: (a) correctly identifies Tennessee-specific authority, (b) conflates Tennessee law with other jurisdictions, (c) provides federal law without noting the Tennessee distinction.

Scoring:

Correct Jurisdiction Rate	Score
≥ 90%	100
80-89%	75
70-79%	50
< 70%	30

Passing threshold: ≥ 80% correct jurisdictional attribution.

Criterion 1.7: Case Law Currency

What it tests: Does the tool cite case law that is still good law? Does it identify when cited cases have been overturned, superseded, abrogated, or seriously criticized?

Test method:

Identify 25 Tennessee cases that have been overturned, superseded, abrogated, or distinguished in significant respects within the past 36 months.
Submit queries that would naturally call for citation to those cases.
Evaluate whether the tool: (a) cites the case without limitation, (b) cites the case but flags the subsequent treatment, (c) avoids the citation in favor of current authority.
Submit 10 additional queries about legal questions where the controlling Tennessee precedent has changed in the past 36 months. Evaluate whether responses reflect current rather than superseded law.

Scoring:

Result	Score
≥ 90% appropriate handling	100
80-89%	80
70-79%	60
60-69%	40
< 60%	20

Passing threshold: ≥ 80% appropriate handling.

Enhanced threshold (Certified-Sensitive): ≥ 90%.

Note on methodology: Case law currency cannot be tested by checking the citation alone. The test must determine whether the tool's response uses the cited case in a manner consistent with its current authoritative status. A tool that cites State v. Smith for a holding that was abrogated by a later decision fails this criterion even if the citation itself is accurate. Citators are the reference standard for evaluator review.

Note on temporal scope: The 36-month window reflects the realistic scope of "recent" change in Tennessee case law. Test cases should be updated annually as new significant case-law shifts occur. The repository is responsible for maintaining a current set of test cases against which tools are scored.

3.3 Domain 1 Score Calculation

The Domain 1 aggregate score is a weighted average:

Criterion	Weight
1.1 Citation Accuracy	22%
1.2 Hallucination Rate	22%
1.3 Statutory Currency	13%
1.4 Consistency	13%
1.5 Edge Case Handling	9%
1.6 Multi-Jurisdictional Awareness	9%
1.7 Case Law Currency	12%

Citation accuracy and hallucination rate carry the most weight because they pose the most direct risk to judicial proceedings.

The introduction of Criterion 1.7 (Case Law Currency) necessitates redistribution of Domain 1 weights. Citation accuracy and hallucination rate retain the highest weights because they pose the most direct risk; the new case-law currency criterion is weighted at 12% to reflect that it is a substantive accuracy concern but operates at narrower scope than the broader citation accuracy and hallucination measures. The weights sum to 100%.

3.4 Tennessee-Specific Test Areas

Domain 1 test cases must draw from the actual practice areas of Tennessee state trial courts. The framework's reach across General Sessions, Juvenile, Circuit, Chancery, and Municipal courts requires test coverage of each. Tools intended for use in a particular court should be tested predominantly against practice areas heard in that court; tools intended for cross-court deployment must be tested across the full range below.

General Sessions Courts

Criminal misdemeanors: T.C.A. Title 39 (Criminal Offenses), Title 40 (Criminal Procedure), Tennessee Rules of Criminal Procedure.
Traffic offenses: T.C.A. Title 55, General Sessions criminal jurisdiction.
Civil within statutory limits: Small claims procedures, landlord-tenant under Title 66, General Sessions civil jurisdiction up to applicable dollar amount.
Preliminary hearings: Probable cause determinations on felonies subsequently bound over to circuit court.
Orders of protection: T.C.A. § 36-3-601 et seq., issuance and modification standards.
Civil and criminal contempt: T.C.A. § 29-9-101 et seq.; T.C.A. § 16-15-401 (general powers of General Sessions judges); see also T.C.A. § 16-1-103 (court contempt authority generally), distinguishing civil from criminal contempt and the due process protections required for each.

Juvenile Courts

Delinquency: T.C.A. Title 37 Chapter 1, particularly §§ 37-1-101 through 37-1-183.
Dependency and neglect: T.C.A. § 37-1-102 definitions, dependency and neglect adjudication standards, removal and placement criteria.
Status offenses: T.C.A. § 37-1-102(b)(33), unruly child proceedings.
Termination of parental rights: T.C.A. § 36-1-113 grounds and procedure (concurrent jurisdiction with chancery and circuit courts where applicable).
Transfer to adult court: T.C.A. § 37-1-134 transfer hearing standards including the 2017 trauma and abuse history factor.
Department of Children's Services (DCS) investigations and parental rights: T.C.A. § 37-2-403 (child abuse reporting), DCS policy and procedure compliance.
Detention and alternatives: T.C.A. § 37-1-114 detention criteria.
Civil and criminal contempt: T.C.A. § 29-9-101 et seq.; § 37-1-158 (Juvenile court contempt authority), distinguishing civil from criminal contempt and the due process protections required for each.
Custody, visitation, and parenting plans: T.C.A. Title 36 Chapter 6 where joined with dependency or unmarried-parent custody matters; permanent parenting plans under § 36-6-404; the seventeen best-interest factors under § 36-6-106(a).
Child support: T.C.A. Title 36 Chapter 5; Tennessee Child Support Guidelines (Department of Human Services Rule 1240-2-4).

Circuit Courts

Felonies: T.C.A. Title 39 (Criminal Offenses), Title 40 (Criminal Procedure), Tennessee Rules of Criminal Procedure, Tennessee Rules of Evidence.
Jury trials: Voir dire procedures, jury instructions (Tennessee Pattern Jury Instructions), trial procedure under Rules of Civil and Criminal Procedure.
Domestic relations: T.C.A. Title 36 Chapter 4 (divorce), Chapter 5 (alimony), Chapter 6 (custody and child support), parenting plans, and the seventeen best-interest factors under T.C.A. § 36-6-106(a).
Healthcare liability: T.C.A. Title 29 Chapter 26, Tennessee Health Care Liability Act, expert witness requirements, certificate of good faith provisions.
Personal injury: Negligence, premises liability, products liability, comparative fault under T.C.A. § 29-11-101 et seq.
Civil contempt: T.C.A. § 29-9-101 et seq., distinguishing civil from criminal contempt, due process protections.
Appeals from General Sessions: T.C.A. § 27-5-108, de novo review standards.

Chancery Courts

Probate administration: T.C.A. Title 30 (administration of estates), Title 32 (wills), contested estates.
Adoption: T.C.A. Title 36 Chapter 1, statutory grounds and procedural requirements.
Termination of parental rights: T.C.A. § 36-1-113 (concurrent jurisdiction with juvenile and circuit courts).
Guardianships and conservatorships: T.C.A. Title 34, Chapter 1 (guardianships of minors), Chapter 3 (conservatorships of adults with disabilities).
Equity and complex civil: Specific performance, injunctive relief, declaratory judgment, real property disputes, partition actions under Title 29.
Workers' compensation: Where chancery court hears such matters under T.C.A. Title 50 Chapter 6.

Municipal Courts

Ordinance violations: Local municipal code enforcement.
Traffic matters: Within municipal court jurisdiction under city charter and T.C.A. § 16-18-302.

Cross-Cutting Practice Areas

These areas appear across multiple court types and must be tested independently of the court-specific test sets above:

Pro se litigants: Self-represented party scenarios across all civil and family practice areas, with attention to language register, procedural fairness, and access to information.
Debt collection: T.C.A. § 47-18-2501 et seq., Fair Debt Collection Practices Act compliance, default judgment procedures.
Evidentiary use of AI output: Foundation requirements for AI-generated evidence under Tennessee Rules of Evidence, authentication standards under Rule 901, expert testimony under Rules 702 through 705.
Local rules: Variation in local rules across the state's 31 judicial districts; tools deployed across multiple districts must navigate this variation.
Probation and community supervision: T.C.A. § 40-35-303, revocation standards, conditions of probation, alternative dispositions.
Mental health and competency: T.C.A. Title 33 (mental health), competency to stand trial under T.C.A. § 33-7-301, civil commitment proceedings.

The test case repository organizes its v1.0 working target of 550 to 600 cases against this structure. Tools submitted for evaluation are tested predominantly against the practice areas relevant to their intended deployment, with a representative sample of cross-cutting areas. Vendors targeting cross-court deployment should expect testing across the full range.

3.5 Practical Example

The certifying body's evaluation team, executing AI-assisted Domain 1 testing on a legal research tool intended for General Sessions Court use, would proceed as follows:

Prepare 100 queries drawn from Tennessee General Sessions criminal and civil practice.
Run queries through the tool and collect outputs (AI-assisted execution).
Verify all citations against Tennessee official reporters and T.C.A. (AI-assisted citation lookup; human review for accuracy).
Score each criterion against the rubric.
Calculate weighted aggregate.
Check for critical failures (any fabricated citation equals automatic fail).
Document results in the evaluation report.

Estimated time: 3 to 5 days under current AI-assisted methodology. The proportion of automated to manual labor is expected to compress as AI evaluation tooling advances, while human legal review of citation accuracy and edge cases remains required.