Royal Bank of Canada: Bringing Together Auditors & Analysts in a Control Testing Framework

Royal Bank of Canada (RBC) is one of Canada’s biggest banks and among the largest in the world in terms of market capitalization. RBC, as the company and its subsidiaries are collectively known, is a leading diversified financial services company in North America, providing personal and commercial banking, wealth management, insurance, investor services and capital markets products and services on a global basis.

At RBC, the CAE Group has begun a transformational journey that adopts a data-driven approach to provide assurance to its clients. To address the challenges of identifying insufficient controls that could result in financial loss, the CAE Group utilizes analytics to uncover exceptions or outliers.

These in-house developed analytics can range from advanced machine learning models to simple rules-based checks. Given the complexity of RBC’s business, the design and execution of control tests also requires specialized domain knowledge.

Previously, the control testing process was manually intensive and done only periodically. The CAE Group would:

1. Select the control tests
2. Design the test procedures
3. Take samples of the resulting dataset/transactions
4. Check samples for adherence to criteria

This process would be repeated anywhere from annually to once every two years. The CAE Group, burdened by the administrative overhead, had less time to review and revise the outliers.

This process was difficult to scale, as the platforms retreated into their silos, where they built and managed their own control testing process. This duplicated effort made consolidation into CAE Group’s holistic enterprise view a cumbersome, manual process.

The Challenges to Overcome

There were two primary challenges to achieving a data-driven control testing process: a technical challenge and an organizational challenge.

The Technical

There were many technical barriers to building a unified platform:

1. Platform analysts needed the freedom to onboard and update their models in production.
2. Every solution needed to support the variability of different models and schemas of outliers.
3. Semantically, each control test required categorization to fit within the broader universe of the CAE Group analytics.
4. Lastly, managing data governance requirements across permissions, datasets and definitions was likely to be very costly for a custom application.

The Organizational

The organizational challenges were threefold:

1. For new control tests, the RBC CAE Group required a shift in mindset, such that auditors started to think of what they could test continuously, rather than periodically.
2. There were existing control tests that the RBC CAE Group could reuse from prior audits, but they needed to be updated and on-boarded.
3. The team needed to develop incentives for adopting the new platform.

At the beginning of its journey, the RBC CAE Group realized that the largest barrier to adoption was the onboarding process. The Group already had a gold mine of control tests created from past audits. In the future, each platform would be rapidly creating new control tests. They needed a self-driven way for data scientists and analysts to onboard. Additionally, they didn’t want the overhead of engineers acting as gatekeepers for each update.

Finding a Solution with Dataiku

Working with Dataiku, the RBC CAE Group became aware of three key out-of-the-box features that would aid the team:

• Tagging

• Dataiku API

• Editable Datasets

Briefly put, Dataiku provided an all-in-one platform for Control Testing at RBC.

Leveraging a self-service process, RBC data scientists and analysts are focused on building control testing in Dataiku. Once ready to onboard, they simply “tag” the final dataset which contains the final list of outliers. For rollup to organizational attributes, (e.g., Control Test Name, Description and Audit ID, etc.), an editable dataset can be entered. 

Using the Dataiku API, a background loops through each project containing the metadata and tagged dataset. The outlier dataset’s schema varies for each use case. Dataiku’s SQL API allows a schemaless approach to dump each final dataset to a centralized database. The metadata, on the other hand, has a deterministic set of columns and is imported to a summary table.

The overall benefits:

• Scalability: Data scientists and analysts are able to build control tests and onboard with minimal effort through out-of-the box Dataiku features.
• Flexibility: With this approach, each control test can be updated iteratively as auditors review and refine the logic for outliers.
• Data Governance: Through Dataiku, we have existing automated pipelines which facilitate automated capture of Data Quality and Lineage metadata. The framework fits in nicely with existing processes and automates publishing to Collibra’s RBC data governance platform.

Solving for Value

By moving from a periodic, manual process to an automated Control Test Framework process, RBC CAE Group saves 20-25% of time for a given audit. This gives the auditors insight into the strength of the control environment throughout the year, not simply at a given point of time.

The new process creates greater certainty over the status of the control, allowing auditors the freedom to explore ideas. Not only are significant time savings a benefit, but the quality of assurance is increased. It favours a more accurate view of business health at any given time, which gives decision-makers more time to react if needed.

This, in turn, has triggered a change in mindset around how analytics and data science may support the entire CAE Group, as they continue to pioneer and enable new ways to empower RBC employees to leverage data for their day-to-day activities.

The ease of onboarding, accessing data, exploring previous work, and collaborating with teammates of all profiles is paving the way for organizational transformation around what data can help us achieve.

What Dataiku Brings to the Table

In assessing the value that Dataiku helped generate with its new controls testing process, RBC noted some key advantages of Dataiku’s platform. At a high level, the platform allows for easy collaboration across RBC’s teams, and its out-of-the-box features — like tagging, which allow one team to tag a dataset and have it consumed by another team or processed downstream — make the team’s processes simpler and more efficient.

Some benefits of Dataiku’s platform in the CAE Group’s workflow:

• Tiered permissions: Access to sensitive information can be restricted on multiple levels.
• Easy API integration: The SQL API, for example, allows Data Engineers the flexibility to programmatically transfer data from one database to another for aggregation purposes.
• Speed of change: Auditors gain a near real-time view of feedback they give as projects are decoupled from one another. 
• Integration with source systems: In RBC CAE Group’s case, the amount of data scientist/analyst needs to enter is reduced because a background process in Dataiku can connect to RBC source systems.
• Governance: The platform makes it easy to identify, define, and raise Audit Issues to RBC Business Units on their risk assurance practices. The CAE Group is empowered to explain Control Test results to Regulators in a consistent and transparent way.

In short, the scalable approach powered by the Dataiku platform will help the CAE Group build a data-driven perspective of the Audit universe and answer questions about the automated coverage of assurance work with traceable, quality data.