Conversational Interface for Process Mining Tools Using Large Language Models (LLMs)
Name
David Damola Obembe
Abstract
Business process mining techniques allow business analysts to answer questions about the performance of business processes using data extracted from information systems. In exist-ing process mining tools, business analysts often need to navigate across many diagrams and charts to find answers to their questions. Given the advancements in Large Language Models (LLM) for question answering, there is an opportunity to extend process mining tools to provide a conversational style to answer process mining questions. This study eval-uates the ability of LLMs to answer process mining questions. The study evaluates two prompting approaches to answer process mining questions using LLMs. In the first ap-proach (the direct approach), relevant business process performance metrics are included in the prompt. The LLM is asked to answer the question directly based on these metrics. In the second approach (the SQL approach), the LLM is told that there are tables available in a da-tabase to answer the questions. The LLM is asked to return an SQL query to answer each question. The study evaluates these two approaches using two LLMs, namely GPT-4 and Claude V3. The results show that these LLMs can answer process mining questions with precision and a recall ranging from 67% to 87%. Claude V3 slightly outperformed GPT-4 when using the direct approach, whereas GPT-4 performed better when using the SQL ap-proach. The study also examines the effect of using XML tags to separate different sections of the prompt. This approach did not show any perceivable benefits.
Graduation Thesis language
English
Graduation Thesis type
Master - Innovation and Technology Management
Supervisor(s)
Shefali Sharma, Marlon Dumas
Defence year
2024