Yohan Park

Course: CS453 Automated Software Testing (Period: Feb 2025 - Jun 2025)

Full project title: Knowledge graph-guided autoregressive test generation for diversity-based LLM prompt testing

TL;DR:

• I proposed the project topic as a team leader and independently designed the system architecture.
• Our team implemented autoregressive, knowledge graph-guided, diversity-based prompt testing for Large Language Models (LLMs).
• Delivered the system collaboratively with team members.

Abstract. With the emergence of LLM-based applications, techniques to efficiently test LLM-applied software is becoming more and more critical. The core difficulties in LLM prompt testing includes the nondeterministic and uncontrollable outputs, time and cost expensiveness in iterative refinement, and the fact that problem-answer pair datasets are expensive and labor intensive to create.

To be specific, generating effective test inputs for prompt testing presents several challenges. Firstly, generated test inputs must be assigned a ground truth. Secondly, these inputs should exhibit diversity, as this diversity can potentially lead to the successful discovery of failures within the system under test. Finally, the test generation process itself needs to be generalizable and controllable, which ultimately assists developers in creating efficient prompt development workflows. The project aims to address “How to Generate Test Inputs of LLM Prompts?” effectively for LLM-based applications.

Read the full report.

Contents to be updated.