Skip to main content

JSON Detection

A Search-based Evaluator that rapidly identifies JSON data structures in text using carefully curated regular expressions. It provides binary classification for JSON objects and arrays in both Prompts and Responses.

JSON Detection - Search

Detailed Description​

The JSON Detection - Search Evaluator uses specialized pattern recognition to identify JSON data structures in both Prompts and Responses. It's designed to detect valid JSON objects (starting with { and ending with }) and JSON arrays (starting with [ and ending with ]) that might contain strings, numbers, Booleans, null values, nested objects, and arrays.

This Evaluator is valuable for applications that need to detect when structured data is being passed or generated, particularly when integrating with APIs, databases, or configuration systems. It helps identify when an LLM is producing or being prompted with structured data that might require special handling or validation.

Input Descriptions:​

The Evaluator accepts text input from both Prompt and Response Events within an Exchange.

Law Usage Example:​

This Response would trigger the Evaluator since it contains a valid JSON object with proper formatting including quoted keys, string values, numeric values, and nested objects:

Triggering Example
{"name": "John Smith", "age": 30, "isActive": true, "address": {"street": "123 Main St", "city": "Anytown"}}

This Response would also trigger the Evaluator since it contains a valid JSON array with mixed data types including strings, numbers, booleans, null values, and nested structures:

Triggering Example
["apple", "banana", 42, false, null, {"key": "value"}, [1, 2, 3]]

This Response would not trigger the Evaluator since it contains plain text with no JSON structure - no objects or arrays are present:

Non-Triggering Example
Here's some information about JSON formatting standards.

Output Descriptions:​

Returns a Finding containing a Boolean flag for JSON detection:

Finding Structure
{
  "JsonDetector-Search.is_json": [True/False]
}

Configuration Options:​

N/A

Data & Dependencies​

Data Sources​

Pattern library developed based on known JSON structures and synthetically generated examples.

Benchmarks​

The JSON Detection - Search has been tested against one benchmark dataset to assess its effectiveness:

DatasetSample SizeAccuracyPrecisionRecallF1 Score
ThirdLaw JSON Benchmark10,728 examples99.1%97.8%99.9%98.9%

Key Findings:

  • Perfect recall indicates the evaluator successfully identifies all JSON structures in the dataset
  • Total sample size of 10,728 examples provides substantial validation of the evaluator's performance

Benchmarks last updated: March 2025

Ways to Use and Deploy this Evaluator​

Here's how to incorporate the JSON Detection - Search in your Law:

ThirdLaw DSL
  if JsonDetector-Search.is_json in ScopeType then run InterventionType

Here's an advance example combining JsonDetector-Search with CodeDetection-Search:

ThirdLaw DSL
  if JsonDetector-Search.is_json in Response and CodeDetection-Search.is_python in Response then run ReviewResponse

Security, Compliance & Risk Assessment​

Security Considerations:​

  • Helps identify potential API payloads that should be scrutinized
  • Can be used to enforce data format policies

Compliance & Privacy:​

  • EU AI Act - supports compliance with system design and prevents LLMs from generating structured output when it's unauthorized
  • NIS Directive - supports cybersecurity requirements by identifying potentially malicious structured data

Revision History:​

2025-02-24: Initial release​

  • Initial pattern library for JSON structure detection
  • ThirdLaw benchmark results
  • Initial documentation