Deploying In App AI Agents in Financial Services: Security and Compliance Considerations

The business case for putting a voice agent inside your app, letting customers complete onboarding, claims, and payments by saying what they need, is compelling. The compliance and security considerations are equally important, and often underweighted in the initial enthusiasm.

This isn't a reason to avoid in app agents. It's a reason to deploy them thoughtfully, with appropriate governance frameworks in place from the start.

The regulatory landscape

Financial services is one of the most heavily regulated industries in the world. The regulations that apply to AI systems vary by jurisdiction and are evolving rapidly.

In the US, the CFPB has issued guidance on the use of AI in consumer financial services, with particular focus on explainability and fair lending. The OCC and FDIC have issued guidance on model risk management that applies to AI systems. State insurance regulators have varying requirements for AI use in underwriting and claims.

In India, the RBI has issued guidelines on digital lending and AI use in financial services. IRDAI has specific requirements for AI in insurance underwriting and claims processing.

The common thread across these regulatory frameworks is a focus on three things: explainability (can you explain why the AI made a particular decision?), fairness (does the AI treat customers consistently and without discrimination?), and oversight (is there appropriate human review of AI decisions?).

Data privacy and security

An in app agent in financial services handles sensitive personal and financial data, including, in the case of a voice agent, recordings of what customers say. The security requirements are correspondingly high.

Data minimization. The agent should collect only the data needed for the specific transaction. It should not retain data beyond the period needed for the transaction and any required audit trail.

Encryption. All data in transit and at rest should be encrypted. This is table stakes for financial services, but it's worth confirming that the agent meets the same standards as the rest of your infrastructure.

Access controls. The agent should have access only to the data and systems it needs for the specific transaction. Least privilege access is a fundamental security principle that applies equally to AI agents, especially ones authorized to call your APIs.

Audit trails. Every interaction, every decision, every action the agent takes should be logged. This is required for regulatory compliance and is essential for investigating incidents.

Model governance

An in app agent plans and executes actions that affect customers. These decisions need to be governed appropriately.

Model validation. Before deployment, the AI powering the agent should be validated against the regulatory requirements for the specific use case. This includes testing for bias, testing for edge cases, and confirming that the agent behaves as expected across the range of customer situations it will encounter.

Ongoing monitoring. After deployment, the agent's performance should be monitored continuously. This includes monitoring for drift (changes in the distribution of inputs or outputs over time), monitoring for bias, and monitoring for unexpected behaviour.

Human oversight. For decisions that materially affect customers, such as credit decisions, claims routing, and risk assessments, there should be appropriate human oversight. The agent can make recommendations, but humans should review decisions above defined thresholds.

Change management. Changes to the agent, such as model updates, new training data, and changes to the decision logic, should go through a formal change management process, with appropriate testing and validation before deployment.

The explainability requirement

Regulators increasingly require that AI decisions be explainable. A customer who is denied a loan or has a claim rejected should be able to understand why.

This requirement has implications for the design of in app agents. Systems that make decisions based on opaque neural networks are harder to explain than systems that make decisions based on explicit rules or interpretable models.

For financial services, the practical implication is that the agent's decision logic should be designed with explainability in mind from the start. This doesn't mean avoiding sophisticated AI. It means ensuring that the agent's reasoning can be surfaced and explained when required.

Vendor due diligence

For institutions that are deploying an in app agent through a vendor rather than building in house, vendor due diligence is critical.

Key questions to ask:

• Where is customer data stored, and what are the data residency requirements?
• What security certifications does the vendor hold (SOC 2, ISO 27001)?
• How is the AI model trained, and is customer data used for training?
• What is the vendor's incident response process?
• How does the vendor handle regulatory changes that affect the agent?
• What is the audit trail for the agent's decisions and actions, and how can it be accessed?

These questions should be answered before deployment, not after.

The governance framework

The institutions that deploy in app agents most successfully in financial services have a governance framework in place before they go live. This framework typically includes:

• A clear definition of which actions the agent can take autonomously and which require human review
• A model risk management process that applies to AI agents
• A data governance policy that covers what the agent collects and retains
• An incident response process for AI related issues
• A regulatory change management process for updating the agent when regulations change

Building this framework takes time. But it's significantly easier to build it before deployment than to retrofit it after.

---

Questions about deploying an in app voice agent in a regulated financial services environment? Get in touch.