Questions Before Answers? A Comparison of Codebase Chatbots and Tutorials for Codebase Learning

<p dir="ltr">LLM-based chatbots are commonly used to assist and guide programmers while coding. However, it is unclear how effective they are compared to human-written tutorials, especially when dealing with a medium-sized unfamiliar codebase. We conducted a between-subject experiment with 15 experienced programmers to compare the effectiveness of LLM chatbots with codebase support and human-written tutorials in helping programmers learn and get started with a new codebase.We found that programmers assisted by LLM chatbots were more confused and less confident, while those using tutorials had a better understanding of the code structure but struggled with low-level coding challenges. We also found that, in determining productivity, a programmer’s personal experience with the programming language is more important than any other factor, such as experience with LLM systems and the framework.</p>