Choosing guardrails against malicious input to a Gen AI application

Guardrails protect GenAI applications from malicious or adversarial user inputs such as prompt injection, jailbreak attempts, and requests for harmful or off-topic content. Databricks recommends a layered approach combining system prompt design, input/output validation, and platform-level safety tools rather than relying on a single defense. Guardrails should be applied both to what enters the model (input filtering) and what leaves it (output filtering) to reduce risk end-to-end.

1 · Learn the must-know

• 1A well-designed system prompt that clearly defines the assistant's role, scope, and behavioral boundaries is a first line of defense against prompt injection and off-topic misuse.
• 2Input validation/guardrails should inspect and filter user queries before they reach the model to catch malicious instructions, toxic content, or attempts to override system instructions.
• 3Output validation/guardrails should inspect model responses before returning them to the user to catch leaked sensitive data, toxic language, or unsafe recommendations.
• 4Databricks provides safety filtering capabilities (e.g., a safety filter option on Foundation Model APIs) that can be enabled to automatically flag or block harmful content in requests and responses.
• 5Guardrails cannot fully eliminate risk from adversarial inputs, so combining multiple layers (prompting, filtering, monitoring, human review) is more robust than any single technique.
• 6Ongoing monitoring and logging of inputs/outputs (e.g., via Lakehouse Monitoring or MLflow tracing) helps detect novel attack patterns and supports iterative improvement of guardrails.

2 · Check your understanding