In this episode, we talk about practical guardrails for LLMs with data scientist Nicholas Brathwaite.
We focus on how to stop PII leaks, retrieve data, and evaluate safety with real limits. We weigh managed solutions like AWS Bedrock against open-source approaches and discuss when to skip LLMs altogether.
Most AI failures don’t come from the model being “dumb”—they come from guardrails that were never designed or tested, or they were added after the model was designed and built.
Nic talks with Sid and Andrew about what a safe, governable LLM deployment really looks like inside a Fortune 500 environment, from basic PII protections to advanced defenses against prompt injection and jailbreaks.
The discussion explores a blueprint for defense in depth:
For retrieval-augmented generation (RAG), Nic describes an intentional architecture—segmented vector databases, strict permissions, out-of-bounds detection, and query risk scoring that routes sensitive requests down a “slow path.” He also shares practical patterns for preventing credit card and unknown leakage, along with evaluation strategies that include adversarial prompts, precision/recall on PII detection, and latency/cost tracking.
All this said, the discussion leads to a healthy question about what it’s all worth vs non-LLM techniques for achieving the same results.
Beyond tooling, we talk governance and durability. Prepackaged options like AWS Bedrock Guardrails can accelerate early wins, yet most enterprises need custom rules and domain-aware models to meet industry standards in finance, healthcare, and beyond.
The outcome is a clear, actionable takeaway: build safety in, not on. Limit data exposure, validate retrieval, scrutinize outputs, and log decisions for audit. If your use case is simple, a deterministic bot might be safer and cheaper; if you need LLM reasoning, earn it with rigorous guardrails and continuous evaluation.
Connect with them to comment on your favorite topics:
