Secret Detection

An Evaluator that scans input text for various types of secrets (e.g., API keys, passwords, private keys) by employing pattern matching and heuristic analysis.

Secret Detection

• Use Case: Secret Detection, Credential Scanning, Security Hardening
• Analytic Engine: Subprocess
• Related OWASP Risks:
  • LLM02: Insecure Output Handling
  • LLM06: Sensitive Information Disclosure
• Compliance Areas: General Data Security, DevSecOps Best Practices
• Valid Inputs: Text
• Scope: Full Exchange (Prompt and/or Response)
• Last Update: 2025-05-27
• License: MIT
• Dependencies: N/A

Detailed Description

The Secret Detection Evaluator utilizes a detection engine that employs pattern recognition and heuristic-based scanning to perform scans on textual content. It is designed to identify a wide array of predefined and custom secret patterns, helping to prevent accidental exposure of sensitive credentials and keys in LLM prompts or responses, or other text-based inputs.

Input Descriptions:

The Evaluator accepts text input, typically from an Exchange's Prompt or Completion, or any other textual data source.

Law Usage Example:

This input would trigger the Evaluator because it contains a pattern resembling an API key:

Triggering Example

My application uses the following API key for access: sk_live_abcdef1234567890abcdef1234567890.

This input would not trigger the Evaluator as it does not contain recognizable secret patterns:

Non-Triggering Example

Let's discuss the weather for tomorrow and plan our picnic.

Output Descriptions:

Returns a Finding containing the detection results. If secrets are found, matched_entities will be a list of SecretMatch objects. If no secrets are found, matched_entities will be the string "No Secrets Found".

Finding Structure

{
  "matched_entities": [
    {
      "entity_type": "string", 
      "start_line": "integer",
      "end_line": "integer",
      "start_column": "integer",
      "end_column": "integer",
      "entropy": "float"
    }
  ],
}

Configuration Options:

Parameter	Description	Default
secret_entities	A list of specific detection rule IDs to target, or the list ["all"] to detect all supported types based on available rules.	["all"]

---

Data & Dependencies

Data Sources

The Secret Detection relies on a comprehensive set of rule sets and detection logic. These rules are designed to identify common secret patterns and are periodically updated to include new detection signatures.

Benchmarks

The Secret Detection has been tested against the SecretBench dataset to assess its effectiveness:

Dataset	Sample Size	Precision	Recall	F1 Score
SecretBench	113k examples	46%	88%	60%

*Benchmarks last updated: May 2025 *

---

Ways to Use and Deploy this Evaluator

Here's how to incorporate the Secret Detection in your Law:

ThirdLaw DSL

if SecretDetection in Both then run RedactSecrets

---

Security, Compliance & Risk Assessment

Security Considerations:

• The evaluator processes data locally by invoking its internal detection mechanisms; it does not transmit the input text to external services.
• Helps in identifying and mitigating the risk of accidental exposure of secrets, supporting general data security policies and DevSecOps best practices.

---

Revision History:

2025-05-27: Initial release

• Initial version of the Secret Detection.
• Detects secrets using a subprocess engine that employs pattern recognition and heuristic analysis.
• Configurable secret_entities to target specific detection rules.