Prompt engineering is a key consideration for accounting professionals to get the most from their use of AI. Crafting the right prompts for each task can seem daunting especially when having experienced less than ideal responses from AI Large Language Models in the past. However, there are considerations both within and beyond prompts themselves that can assist auditors and accountants in accomplishing more with ChatGPT and other generative AI tools.
Before the Prompt - Custom Instructions in ChatGPT
Before diving into prompt engineering more thoroughly, it is worth exploring a lesser known feature of OpenAI’s platform that isn’t often considered in discussion around prompt engineering or leveraging ChatGPT more broadly. Within the user’s profile at the bottom of the ChatGPT homepage is an option to customize ChatGPT to the user. This feature can be incredibly valuable for users before ever considering a formal prompt. Users will see the suggestions that OpenAI has included to consider for each of the inputs both about you as a user and how you prefer it to respond. As expected with ChatGPT, these fields are open ended and can be experimented with for optimal results. In general, for our use cases we’ll consider letting ChatGPT know that we’re an auditor/accountant and are looking for concise and objective answers. With those considerations in mind, let’s look at the different types of responses received with the exact same prompt, both with and without these custom instructions.
To begin, we’ve entered the prompt below which provides a bit of context for ChatGPT4 to consider in its response. As expected, it responds with a summary of the information requested, explaining a bit of the background of lease accounting convergence, the standard itself and the steps to actually calculate and apply lease accounting (likely what the client is most interested in). However, without any customization to ChatGPT, the response is over 550 words, likely more reading than a client bargained for when reaching out to their accounting advisor for a bit of guidance.
Let’s try again and enter some very basic background information on the user, their business and what is desired from the AI’s responses.
This time the exact same prompt received a response of just over 400 words, even including the section below on advising the client, which we’d exclude from our email response. This may not seem like much but when compared to the alternatives of attaching lengthy accounting standard guidance or paring down the original answer from ChatGPT, which included irrelevant considerations like IFRS, quickly setting up custom instructions can be incredibly beneficial when using ChatGPT to handle a large volume of queries. Of course, another alternative would be responding within the original conversation asking ChatGPT to condense and focus on the basics of the standard and next steps for the client but this back and forth would need to be repeated across every similar conversation.
If you were in a similar position to this hypothetical professional, you might also consider creating a personal GPT for use with your clients with further background information on the type of responses you aim to receive as currently these custom instructions apply only to ChatGPT 3.5 and ChatGPT 4.
Rethinking Prompt Specificity
Turning to prompt engineering itself, several focus areas come to mind for accountants and auditors when using AI language models. First, the issue of prompt specificity is critical to obtaining optimal results. However, often on the topic of specificity users encounter the seemingly conflicting issues of time invested into the prompt versus the time savings the AI is generating. Something to the effect of “If I spend lots of time writing out a very specific prompt, I might as well have looked up and written out the answer myself” is not an uncommon sentiment from users. Instead of thinking about specificity in terms of the length of the prompt and the details included, instead think of the context that is most crucial to the task at hand. Revisiting the example from the custom instructions case, an accounting advisor may know from a first glance that the lease in question should be accounted for as a finance rather than operating lease, as most all leases are under ASC 842. Instead of asking, "Provide my client with an overview of lease accounting", a more specific prompt would be, "Provide my client steps to account for a finance lease of an office building be accounted for under ASC 842". Without adding much additional effort to writing the prompt, the focus has shifted to information that is most relevant to the client. With the additional guidance of the custom instructions users can expect a prompt that focuses on the needs of the client without providing more than the necessary background information. Specificity doesn’t have to mean a lengthy prompt, but rather one that requires thoughtful consideration of the most relevant information.
Clarity and Context
Going hand in hand with specificity is context. While custom instructions, creation of personal GPTs and prompts can all refine the focus of the LLM, accountants must consider the context of their inputs. For example, in the custom instructions, mentioning the user's location as the US would steer ChatGPT toward answers within the context of US GAAP. Further context that can be included in prompts can include the financial, operational, or regulatory background relevant to the query. For example, as US GAAP has many accounting standards that apply to particular industries, it is usually worthwhile to include the client’s industry in prompts related to financial accounting guidance. Accountants should be cognizant of these considerations when creating their custom instructions and prompts.
Use of Accounting Terminology
Using the correct accounting terminology and citing specific standards or sections ensures the AI can accurately interpret the request and provide information that is more likely to be relevant to the query and at the level of expertise that is useful to the user. For example: "Can you provide a journal entry example for recording a revenue transaction that meets the criteria of ASC 606, focusing on the allocation of transaction price to performance obligations for a software service contract?”. Revisiting our lease example once more, with a bit of guidance and iteration (covered in further detail below), ChatGPT can extract the relevant paragraphs of the Accounting Standards Codification that pertain to a specific scenario, in this case a sublease.
Accountants can explicitly request these citations in their prompts and it's worth considering doing so when users of the information require further verification.
Focus on Objectives and Outcomes
When stating the objectives clearly and eliciting objective answers from AI the outputs will often be more actionable in an accounting context. This is an area where Microsoft Copilot’s interface is ideally suited for providing concise and actionable results with many common considerations listed as toggleable inputs. These inputs are important to consider for all LLMs and should be built into standard practice for prompt engineering when they are not clearly delineated by the interface itself.
These considerations can be built into ChatGPT’s prompt as well and are all worth including when thinking of the objectives. The above prompt and response might be used for a presentation planning the audit engagement and built into a slide deck (something Copilot is also adept at drafting on behalf of users) along with the planned audit responses which is also suggested as a follow up prompt.
Iteration and Retooling
While many accounting professionals aim to use LLMs in a method that provides results quickly and with minimal need to iterate, in practice iterating and training of an AI model is comparable to training new team members. While LLMs have vast knowledge bases to pull from, they also need guidance to arrive at a desired outcome. As seen with Copilot, developers are making user interfaces more customizable and predefined which can have both positive and negative results for users. On the positive side, the predefining of many of the considerations discussed earlier will assist users in prompting but may serve to detriment the patience for iteration. Using the iterative process to hone in on the most relevant and accurate information is one that can be frustrating but necessary to arrive at the right results. Accounting professionals should keep in mind that due to LLMs like ChatGPT's ability to remember instructions from previous prompts makes iteration a useful tool to build LLMs that provide the desired results. Custom GPTs are another option to retain the style of prompts that produce the desired results once refined.
Closing Thoughts
Generative AI’s ability to transform the way auditors, accountants and professionals more broadly approach their work is only just beginning to be explored with the widespread adoption of LLMs. As AI moves beyond the input and output of text prompts and responses and into more substantial and complex tasks involving document inputs and fully fledged audit procedures delivered by bespoke AI agents. ChatGPT, Copilot and others are already capable of analyzing documents uploaded into the models, and it is worth exploring the capabilities of these functions. As prompt inputs evolve with the capabilities of generative AI, it will be important to keep these and other prompt engineering considerations in mind to continue to generate optimal results from accounting professionals' use of AI.
.svg)
.svg)
