Consensus Scoring

How It Works

Aira does not use embeddings or semantic similarity to score consensus. Instead, each model is prompted to return structured output:

{
  "decision": "APPROVE",
  "confidence": 0.91,
  "key_factors": ["credit score above threshold", "stable income"],
  "reasoning": "Full explanation text..."
}

Agreement is then scored on these structured fields — not on free-text similarity. This makes consensus deterministic, explainable, and auditable.

The Algorithm

Step 1: Decision Agreement

Count how many models agree on the same decision (APPROVE / DENY / REVIEW):

Step 2: Confidence Variance

If models agree on the decision but with wildly different confidence levels (e.g., one at 0.95 and another at 0.30), this signals uncertainty. A variance penalty of up to 0.20 is added to the disagreement score.

Step 3: Key Factor Overlap

If models independently identify the same key factors, this strengthens the consensus signal. Each shared factor reduces the disagreement score by up to 0.03 (max 0.10 reduction).

Step 4: Human Review Trigger

Human review is triggered in two cases:

1. Models disagree — the disagreement score exceeds the configured threshold (default: 0.4). The decision becomes MIXED.
2. Models vote REVIEW — when the majority decision is REVIEW, the models themselves are saying "a human should decide this." The decision stays REVIEW but requires_human_review is true.

{
  "consensus": {
    "decision": "MIXED",
    "disagreement_score": 0.72,
    "requires_human_review": true,
    "human_review_reason": "Disagreement score 0.72 exceeds threshold 0.40 — Article 14 human oversight triggered"
  }
}

{
  "consensus": {
    "decision": "REVIEW",
    "disagreement_score": 0.00,
    "requires_human_review": true,
    "human_review_reason": "Models assessed this decision as requiring human judgment — Article 14 human oversight triggered"
  }
}

Configuring the Threshold

Set human_review_threshold per request:

{
  "options": {
    "human_review_threshold": 0.3
  }
}

For high-stakes domains (credit, medical, legal), use 0.2–0.3. For lower-risk decisions, 0.4–0.6.

Why Not Embeddings?

1. Determinism: Structured scoring produces the same result every time. Embedding similarity can vary.
2. Explainability: A regulator can audit the scoring logic. "2 out of 3 models said APPROVE" is clearer than "cosine similarity was 0.87."
3. No extra costs: No embedding model call needed. Zero additional latency.
4. No extra dependencies: No vector database, no embedding API, no model to maintain.

Decision Values

Models must return one of three decisions:

The consensus decision follows the same values, plus MIXED when human review is triggered.