ThirdLaw Evaluate

ThirdLaw Evaluate is the analytic and decision-making engine of the ThirdLaw platform. It applies programmable policies—called Laws—to analyze and assess LLM interactions in real time or batch. Evaluate serves as the policy intelligence layer between Collect (data capture) and Intervene (response), enabling organizations to identify risks, assign meaning to interactions, and decide when and how to act.

Key Capabilities

Evaluate provides the logic and judgment layer for deciding whether an LLM exchange violates acceptable use, security policy, regulatory requirements, or internal standards. Rather than relying on static filters or hand-coded logic, Evaluate supports expressive, composable Laws that can adapt to emerging threats, organizational needs, and use case complexity.

• Law Evaluation: Orchestrate custom or built-in Evaluators to process incoming LLM events and determine an adjudication.
• Evaluator Execution: Run one or more analytic engines to assess risk, compliance, or behavior in the form of an Evaluator.
• Policy Resolution: Determine whether violations have occurred and package the results for downstream action.
• Metrics & Feedback: Quantify issues and their trends to drive improvement.

Evaluation Flow

In ThirdLaw, the evaluation process follows this pattern:

1. Analytic Engines process conversation data through configured Evaluators
2. Evaluators produce structured Findings containing detection results
3. Laws interpret these Findings to determine if a policy Violation has occurred
4. Violations trigger appropriate actions or interventions

Evaluation Components

ThirdLaw Evaluate analyzes Exchange data using specialized Analytic Engines and Evaluators to identify potential issues, risks, and compliance concerns.

Analytic Engines

Analytic Engines are the core processing components that analyze Exchange data using different techniques and technologies. Each engine provides a foundation for creating specific types of Evaluators to detect specific behaviors, issues, or risks.

Evaluators

Evaluators are specialized modules that detect specific issues or risks within Exchange data. Evaluators can be customized to detect any number of issues specific to your business. Additionally, ThirdLaw provides a range of pre-built Evaluators for common use cases, including:

• Security Evaluators: (Prompt Injection - Search, Prompt Injection - Validation)
• Content safety Evaluators (Toxic Language - Validation)
• Privacy Evaluators: (PII Detection - Ensemble)
• Code detection Evaluators: (Code Detection - Search, Code Detection - Semantic)

Laws

Laws define rules and policies that process Findings from Evaluators to determine if a Violation has occurred. They determine if the Law is in Scope, combine logical operators over Findings from multiple Evaluators, and trigger Interventions when Violations are detected.

Why ThirdLaw Evaluate?

• Built for IT and Security: Designed to express policies that reflect regulatory, brand, and usage requirements.
• Composability: Mix and match evaluators for precision and coverage.
• Flexibility: Supports complex logic, thresholds, and overrides.
• Auditability: Generates traceable, versioned decisions for every interaction.
• Proactive + Reactive: Power interventions and guide continuous tuning.