See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/378213315

Exploring Capabilities and Limitations of Generative AI Chatbots in Solving

Math Algorithm Problems

Conference Paper · July 2024

CITATIONS READS

0 82

3 authors:

Niroj Dahal Bal Chandra Luitel

Kathamandu University School of Education Kathmandu University
89 PUBLICATIONS 434 CITATIONS 131 PUBLICATIONS 957 CITATIONS

SEE PROFILE SEE PROFILE

Binod Prasad Pant

Kathmandu University
82 PUBLICATIONS 373 CITATIONS

SEE PROFILE

All content following this page was uploaded by Niroj Dahal on 08 April 2024.

The user has requested enhancement of the downloaded file.

The 15th International Congress on Mathematical Education
Sydney, 7-14 July, 2024

EXPLORING CAPABILITIES AND LIMITATIONS OF GENERATIVE AI

CHATBOTS IN SOLVING MATH ALGORITHM PROBLEMS

Niroj Dahal, Bal Chandra Luitel, and Binod Prasad Pant

Department of STEAM Education, Kathmandu University School of Education, Lalitpur, Nepal

Generative AI chatbots like Google Bard—now named a Gemini, ChatGPT—GPT-3.5 or GPT-4, Bing
Chat—now called a Copilot, and Wolfram Alpha are increasingly adept at solving math problems.
These bots can understand and generaive math expressions, providing step-by-step solutions and
explanations. This paper examines the efficacy of generative AI chatbots in solving math algorithms
for diverse users, from students to educators across all levels. This investigation is rooted in didactic
research focused on integrating generative AI chatbots in school math education. We explore how AI
recognizes math expressions and equations in various fields like arithmetic, algebra, trigonometry,
and statistics. We examine the ability of generative AI chatbots to solve math problems of varying
complexity, offer personalized solutions, and enhance students’ math learning. However, we also
highlight their limitations, such as a limited understanding of natural language models and prompt
engineering, inability to solve complex math problems, and potential bias. Future research should
focus on improving generative AI chatbots' accuracy, reliability, and problem-solving capabilities.
Despite the progress in generative AI chatbots, we emphasize the importance of human teachers in
developing students’ cognitive math skills and conceptual understanding.

INTRODUCTION
Generative AI chatbots, including ChatGPT, Google Bard, and Bing Chat, are sophisticated language
models that generate human-like responses (Ahmed et al., 2023; Thirunavukarasu et al., 2023). These
models, trained on extensive text and code datasets, can answer a variety of questions, including those
related to math algorithms. Wolfram Alpha, an AI tool with computational intelligence, can also solve
complex math problems. The proficiency of these generative AI models in solving intricate equations
has produced interest in their potential use in assisting with mathematical tasks. Likewise, math plays
a crucial role in AI development, providing data analysis and modeling tools. It’s also used to develop
new, more efficient AI algorithms and understand AI limitations. With the above, chatbots like
ChatGPT, Google Bard, Bing Chat, and Wolfram Alpha are recognized for their enhanced math skills
and ability to assist students with mathematics at all levels. They provide knowledge on various math
topics and help with geometry. Researchers have been using design-based research to develop and
evaluate these generative chatbots, analyzing training data to identify common questions and create
corresponding responses. The rapid advancement of AI technologies necessitates the design AI literacy
curricula to prepare students for a changing global market. With the above development, this paper
examines the effectiveness of generative AI chatbots in resolving mathematical algorithms for a broad
continuum of users, ranging from students to educators at all educational levels. This study is grounded
in a didactic research method that emphasizes the incorporation of generative AI chatbots into school
mathematics education. It also examines how these chatbots recognize and solve math problems

1
Dahal, Luitel & Pant

ranging from basic arithmetic to complex statistics. Finally, it critically evaluates the capabilities and
limitations of using algorithms for solving math problems.
DECIPHERING MATH EXPRESSIONS AND EQUATIONS
Math is a language of numbers, variables, and operations. Expressions such as 5 + 2, 3x - 4, and (2x +
1) (3x - 4) and equations like 2x + 3 = 7, x2 - 4x + 4 = 0, and y = mx + b are fundamental parts of this
language. Generative AI chatbots can solve these mathematical expressions and equations. For
instance, linear algebra is used in AI to represent data that AI algorithms can process. Images can be
represented as matrices and text as vectors. Further, calculus helps develop AI algorithms for learning
and prediction, like the gradient descent algorithm used in training neural networks. Statistics evaluate
the performance of these AI algorithms. Chatbots such as ChatGPT, Google Bard, Bing Chat, and
Wolfram Alpha can process math equations and expressions. They can perform basic operations like
addition, subtraction, multiplication, and division. They have the potential to revolutionize
mathematics education by providing personalized feedback, explaining mathematical concepts, and
assisting students in problem-solving for meaningful engagement. Generative AI can also enhance
teaching and learning and improve the learners' performances in school to university math (Mohamed
et al., 2022). It can evaluate individual student’s learning challenges and provide personalized support
to optimize their success in math classes. These chatbots can solve complex equations, including
systems of linear equations and quadratic equations. However, the ethical design of generative AI
systems requires a deep understanding of the relationship between mathematics and human behavior.
So, mathematicians, computer scientists, ethicists, and social scientists need to collaborate in
developing ethical guidelines for AI systems for general people and mathematics learners.

SOLVING ELEMENTARY TO ADVANCED MATH PROBLEMS

Chatbots such as ChatGPT, Google Bard, Bing Chat, and Wolfram Alpha are capable of solving
elementary math problems like addition, subtraction, multiplication, and division. For example, if
asked to calculate 35-25 or 250-105, these generative chatbots will provide the correct answers of 10
and 145, respectively. In addition to basic arithmetic, these chatbots can tackle more complex math
challenges encompassing calculus, differential equations, geometry, and trigonometry. In this regard,
Martínez-Sevilla and Alonso (2022) have suggested that AI holds the potential to revolutionize math
education. They advocate for further research into the application of AI in math education and the
development of new AI-enabled learning tools to enhance students’ math understanding. As Davies et
al. (2021) noted, “The practice of mathematics involves discovering patterns and using these to
formulate and prove conjectures, resulting in theorems.” (p. 70). These chatbots leverage deep learning
algorithms to understand problems and generate solutions. However, as they are natural language
models equipped with computational intelligence, they may not always provide the correct solution,
especially for complex problems requiring specific approaches or formulas. Nevertheless, chatbots
like ChatGPT, Google Bard, and Bing Chat can perform math operations with a certain degree of
accuracy.

MASTERING MATH WITH THE HELP OF GENERATIVE CHATBOTS

Qawaqneh et al. (2023) found that AI-based virtual laboratories (VLabs) can effectively enhance
students’ motivation toward learning mathematics. Martínez-Sevilla and Alonso (2022) further

2
 Dahal, Luitel & Pant

highlighted the potential of AI in enriching math education, suggesting that it can foster engaging and
interactive learning experiences, thereby deepening students’ understanding of math. Next, Chang et
al. (2023) explored the effects of online STEAM hands-on learning on AI learning, creativity, and
creative emotions among 25 preservice teachers. They concluded that hands-on STEAM learning can
effectively improve AI learning, creativity, and emotions. Generative chatbots like ChatGPT, Google
Bard, Bing Chat, and Wolfram Alpha can elucidate concepts and formulas and provide step-by-step
solutions to math problems. This approach helps students understand the reasoning behind the solution
and enhances their math abilities. These chatbots can also generate numerous math practice problems
for students, aiding them in finding better or unique solutions. Students can assume a more active role
in their learning process by leveraging generative chatbots like ChatGPT, Google Bard, Bing Chat,
and Wolfram Alpha. On the whole, using customized instructions can maximize the potential benefits
these platforms offer.

PROFICIENCIES AND CONSTRAINTS OF CHATBOTS IN RESOLVING MATH

PROBLEMS
Monte-Serrat and Cattani (2023) have emphasized the crucial role of mathematics in AI development,
which has implications for the ethical design of AI systems. Generative AI chatbots, capable of solving
a wide range of math problems from basic arithmetic to complex calculus, can generate proofs for
mathematical theorems and aid students in understanding intricate concepts. Wang and Lester (2023)
have argued that AI literacy is essential for preparing students for future jobs and ensuring their
participation in a democratic society where AI plays an increasingly important role. AI chatbots can
comprehend natural language questions about math, even if they are not phrased formally. This makes
them accessible to a wider range of students, including those not fluent in mathematical notation. These
chatbots can generate step-by-step solutions to math problems, explain math concepts concisely, and
generate proofs for mathematical theorems. However, these powerful tools for math problem-solving
have limitations. They may struggle with challenging math equations and large statistical analyses that
require specific approaches or formulas. They may not always provide the best solution to a problem
or solve issues requiring real-world contextual knowledge. Despite these limitations, chatbots like
Google Bard, ChatGPT, Bing Chat, and Wolfram Alpha are evolving. Their capabilities will continue
to grow and advance. They are moving towards a multimodal model that outperforms current chatbots
in many ways, including their capacity to solve mathematical problems. Chatbots such as Google Bard,
ChatGPT, and Bing Chat lack a comprehensive understanding of geometry and are unable to correct
misunderstandings effectively. The accuracy and effectiveness of their solutions may depend on the
complexity of the equation, the input data, and the instructions provided. These chatbots are anticipated
to become more effective at solving increasingly complex mathematical problems.
Overall, generative AI chatbots, such as Google Bard, ChatGPT, Bing Chat, and Wolfram
Alpha, can potentially transform math education. They can provide personalized instruction, facilitate
self-paced learning, and make mathematics more accessible. These chatbots can solve complex
equations quickly and accurately, providing step-by-step explanations to aid understanding. However,
their use should complement rather than replace learning.

3
 Dahal, Luitel & Pant

CONCLUSION
Generative AI chatbots like Google Bard, ChatGPT, Bing Chat, and Wolfram Alpha are powerful tools
for solving math problems. They can explain concepts and formulas, provide step-by-step solutions,
and swiftly tackle simple and complex problems. These chatbots can be used to create practice
problems, explain underlying ideas and formulas, and offer step-by-step solutions. This can be
beneficial for anyone needing math help, including students, teachers, and teacher educators. However,
these chatbots have limitations. They may struggle with understanding the subtleties of human
language, leading to misinterpretations and mistakes in their solutions. They are only as good as their
training data and might struggle with problems outside of their training set.
Despite these challenges, generative AI chatbots have shown significant potential in solving
math problems. They can understand and generate math expressions, provide step-by-step solutions,
and enhance students’ learning experiences. However, there are areas where human intelligence
continues to outperform AI, necessitating a nuanced approach to education in the age of AI. In
conclusion, while generative AI chatbots have made significant strides in math education and problem-
solving, students must continue interacting with human teachers to develop their cognitive and
evaluative math skills and conceptual understanding. Hence, by blending the strengths of humans and
machines, we can forge a more effective educational approach.
References
Ahmed, I., Kajol, M., Hasan, U., Datta, P. P., Roy, A., & Reza, M. R. (2023). ChatGPT vs. Bard: A comparative
study. UMBC Student Collection. https://dx.doi.org/10.36227/techrxiv.23536290.v2
Chang, Y. S., Wang, Y. Y., & Ku, Y. T. (2023). Influence of online STEAM hands-on learning on AI learning,
creativity, and creative emotions. Interactive Learning Environments, 1-20.
https://doi.org/10.1080/10494820.2023.2205898
Davies, A., Veličković, P., Buesing, L., Blackwell, S., Zheng, D., Tomašev, N., Tanburn, R., Battaglia, P.,
Blundell, C., Juhász, A., Lackenby, M., Williamson, G., Hassabis, D., & Kohli, P. (2021). Advancing
mathematics by guiding human intuition with AI. Nature, 600(7887), 70–74.
https://doi.org/10.1038/s41586-021-04086-x
Martínez-Sevilla, Á., & Alonso, S. (2022). AI and mathematics interaction for a new learning paradigm on
monumental heritage. In Mathematics Education in the Age of Artificial Intelligence: How Artificial
Intelligence Can Serve Mathematical Human Learning (pp. 107-136). Springer International Publishing.
Mohamed, M. Z. b., Hidayat, R., Suhaizi, N. N. b., Sabri, N. b. M., Mahmud, M. K. H. b., & Baharuddin, S. N.
b. (2022). Artificial intelligence in mathematics education: A systematic literature review. International
Electronic Journal of Mathematics Education, 17(3), em0694. https://doi.org/10.29333/iejme/12132
Monte-Serrat, D. M., & Cattani, C. (2023). Towards ethical AI: Mathematics influences human
behavior. Journal of Humanistic Mathematics, 13(2), 469-493.
Qawaqneh, H., Ahmad, F. B., & Alawamreh, A. R. (2023). The impact of artificial intelligence-based virtual
laboratories on developing students’ motivation towards learning mathematics. International Journal of
Emerging Technologies in Learning, 18(14), 105–121. https://doi.org/10.3991/ijet.v18i14.39873
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023). Large
language models in medicine. Nature Medicine, 1-11.
Wang, N., & Lester, J. (2023). K-12 education in the age of AI: A call to action for K-12 AI
literacy. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-
00358-x

View publication stats