Koodoo passes mortgage advice qualification (CeMAP)

with artificial intelligence model

Publication Date: 15th August 2023

Abstract

In this paper, Koodoo (a mortgage technology and broking firm) experimentally evaluates the
performance of various large language models (LLMs) on the entire Certificate in Mortgage
Advice and Practice (CeMAP) exam, including Module 3 which contains multiple choice case
study questions assessing a candidate’s mortgage advice knowledge. Across all three
modules of the exam, only GPT-4 and a proprietary Koodoo application demonstrated
sufficient accuracy to pass the examination. These findings not only highlight the remarkable
advance of large language model performance, but also the potential for using these models
as foundational layers to drastically increase the efficiency of mortgage advice activities in
the UK.

Introduction
Koodoo is a leading mortgage fintech company based in London. Koodoo’s digital mortgage
platform and brokerage combines technology, data and delightful user experiences to power
mortgage journeys for comparison sites, personal finance providers and marketplaces as
well as leading banks, building societies and specialist lenders.

Koodoo’s mortgage platform runs the mortgage comparison sections for major price
comparison websites such as CompareTheMarket.com, ClearScore, Go.Compare, Experian,
The Times Money Mentor, and integrates with over 85% of UK mortgage lenders.

Historically the UK mortgage industry has been heavily intermediated (business is placed
through brokers rather than direct between customer and lender), with a large emphasis
being placed on the knowledge and expertise of brokers to ensure good customer outcomes.

Since the introduction of increasingly powerful large language models and the demonstration
of such LLMs to pass complex human examinations in various areas (such as GPT-4
passing the US Bar exam), Koodoo has explored the capabilities of LLMs to support
mortgage brokers, lenders and customers with a high level of mortgage knowledge and
understanding.

Certificate in Mortgage Advice and Practice (CeMAP)

qualification
Regulated mortgage advisors in the UK are required to hold an approved mortgage
qualification to conduct business. One of the commonly recognised qualifications is the
Certificate in Mortgage Advice and Practice (CeMAP), administered by the London Institute
of Banking and Finance.

The examination consists of three modules, demonstrating a thorough understanding of the

UK Financial Services, FCA regulation and Business sourcebooks, the home buying
process, different types of mortgages, and how to assess affordability and suitability of
mortgage related products.

Applicants need a minimum of 70% in each of the three modules in order to pass. According
to the London Institute of Banking and Finance, it is estimated that applicants spend an
average of 230 hours preparing for the examinations.

Module Topics Exam format

UK Financial Introduction to Financial Services 100 multiple choice

Regulation (UKFR) Environment and Products questions

UK Financial Services and Regulation

Mortgages (MORT) Mortgage Law, Policy Practice and 100 multiple choice
Markets questions

Mortgage Applications

Mortgage Related Protection Products

Mortgage Payment Methods, Products

and Post-completion Issues

Assessment of Assessment of Mortgage Advice 6 case studies,

Mortgage Advice Knowledge 10 multiple choice
Knowledge (ASSM) questions per case study

Table 1: Breakdown of CeMap exam

Data and Methods

Data
Specimen papers for all three modules were purchased from the London Institute of Banking
and Finance. The questions, multiple choice answers, and correct answers were all
transformed into csv format.

Module Specimen papers used

UK Financial Regulation (UKFR) Module 1 - Paper A (2022_23)

Mortgages (MORT) Module 2 - Paper C (2022_23)

 Assessment of Mortgage Advice Module 3 - Paper A (2022_23)
Knowledge (ASSM)

Table 2: Specimen papers used for evaluation

Methods
Specimen papers for each module were fed into the target LLM, with simple zero-shot
prompting instructing the model that it was taking a CeMAP examination, and using its
knowledge of mortgages and applying common sense, it should select the correct answer
from the multiple choice answers provided.

The answers provided by the target LLM were then evaluated against the known correct
answers, and an accuracy score of that target LLM against a given module recorded.

As an additional challenge, Koodoo’s team developed a custom application to correctly

answer questions from the exam (Koodoo-LLM). It should be noted that at no point were the
correct answers for the specimen papers that were being evaluated provided to the target
models/applications as context.

Results
GPT-4 and Koodoo-LLM were the only models or applications to successfully pass all three
modules in CeMAP, with other models such as PaLM2, GPT-3.5-turbo and text-davinci-003
falling short of the minimum 70% pass mark.

Module 1 Results: UK Financial Regulation (UKFR)

Model Result

GPT-4 83

Koodoo-LLM 76

PaLM2-base 74

Chat GPT-3.5-turbo 77

Text-davinci-003 61

Module 2 Results: Mortgages (MORT)

Model Result

GPT-4 85

Koodoo-LLM 72

PaLM2-base 63
 Chat GPT-3.5-turbo 67

Text-davinci-003 55

Module 3 Results: Assessment of Mortgage Advice Knowledge (ASSM)

Model Result

GPT-4 85

Koodoo-LLM 72

PaLM2-base 72

Chat GPT-3.5-turbo 67

Text-davinci-003 68

Conclusion
Given the outstanding performance of GPT-4 in other similar areas such as law, it did not
come as a surprise to see GPT-4 score highest on all modules. The team’s proprietary
KoodooLLM application has only been taken through an initial training phase and the team
expects performance to increase significantly as more intensive training is conducted.

Ultimately, the team expects that both GPT-4 and Koodoo-LLM would be capable of passing
a CeMAP examination under supervised conditions, given the demonstrated capability of
both to pass specimen papers. While passing the examination does not in itself enable a
person to provide regulated mortgage advice (advisers are then expected to demonstrate
competency in the regulated firms that they operate in and then operate under limited
supervision), this presents a breakthrough moment in mortgage technology.

Koodoo considers this technology to be a foundational layer underpinning a number of tools

that will support human advisors to be more effective and efficient in providing regulated
mortgage advice, enabling human advisers to spend more time creating invaluable trust with
their clients and demonstrating their expertise in the space.