Automating Reference Letter Writing with ChatGPT: A Step-by-Step Guide

I mentioned last time that the world seems to have trichotomized:

1. Those who have not tried ChatGPT.
2. Those who in haste labeled it a chatbot.
3. There is also a third class, one that I belong to.

For me, ChatGPT has become my closest collaborator. Perhaps I might merely be a forerunner on humanity’s march towards the inevitable technological singularity? You know, that special moment when we humans finally discard our keyboards and mindmeld with our favourite chatbot.

Anyway, if you’ve ever taught large university classes, you’re likely familiar with reference letter requests from past students. Personally, no matter how busy I might be, I always feel compelled to respond if a student has excelled in my course series or has completed a summer project under my supervision. I of course also routinely receive similar requests from more senior colleagues and a diverse range of others. Indeed, for better or worse, the entire academic system appears to be greased by reference letters.

However, drafting these letters is time-consuming. One typical solution is to request that the individual seeking the reference provides a draft. Yet I can’t remember being overly excited with the drafts I’ve received.

With tools like ChatGPT, we can refine these drafts into immediately useful letters. Taking it a step further, you might opt to forego the initial draft entirely. Instead, you could send a list of questions to the individual and, based on their responses, ChatGPT can generate a letter. You could even get fancy and ask the student to fill out a Google form, but maybe that’s overkill.

Of course, it’s important to remember that we don’t want ChatGPT to be the final arbiter or decision-maker. Rather, its role should be to structure the facts, producing a draft that can be personalized within minutes. Given that most agree to write reference letters on the premise they’ll be positive, it’s a safe assumption to proceed with. However, as ChatGPT generally leans towards positivity, caution must be taken to ensure there’s no excessive embellishment.

The setup

It is May 2023 and I use ChatGPT-4 with several plugins, such as web browsing. You should still be able to do everything I mention here using the free version, but I pay $20.00 USD a month for this service. There are other alternatives, such as Google’s Bard (which is not as smart yet). And the new Bing uses ChatGPT-4 and apparently enables you to chat with GPT-4 free, though I have not tried that myself. I also make use of a clipboard manager. I personally use Paste for MacOS, but anything should work just fine.

The letter writing process

The first thing we need to do is to gather the information the letter should contain. This part is surgical and to do this, I use ChatGPT to create a questionnaire.

I have been asked by a [student/postdoc/etc.] to write a reference letter for their application to [graduate school/a fellowship/promotion/etc.]. To create an effective letter, I need you to develop a concise questionnaire to gather necessary information from the requester. The questionnaire should solicit the following details:

1. The recipient of the letter and their proper address.
2. A chronology of our relationship, including relevant dates.
3. Specific content to be included in the letter, such as their reasons for choosing me as their reference and any notable accomplishments or experiences we've shared.

Please ensure that the questionnaire is professionally phrased and encourages detailed responses.

If you feed ChatGPT the prompt above, you will receive a more verbose version of what follows (as you can see from this chat, Academic Editor Mode was enabled as described below):

Dear [Requester’s Name],

Thank you for considering me as your reference for your [graduate school application/fellowship/promotion/etc.]. To write a suitable letter, I need more information. Please fill out the following questionnaire:

1. Recipient Information: Please provide the full name, address, and role (if applicable) of the recipient.
2. Relationship Chronology: Describe our professional or academic relationship chronologically, including key dates and experiences.
3. Reason for Choosing Me: Why did you choose me as your reference? What experiences led you to this decision?
4. Shared Accomplishments: List any notable achievements or experiences we’ve shared.
5. Content Requests: Are there specific topics or attributes you’d like me to highlight?
6. Goals and Aspirations: What are your objectives related to this [application/fellowship/promotion/etc.] and how will it help you achieve them?
7. Additional Information: Is there any other relevant information I should know about you?

Please respond at your earliest convenience so I have enough time to write a detailed letter.

ChatGPT

OpenAI’s GPT-4 is the largest language model to date, trained on approximately 45 terabytes of text data, which is equivalent to over 800 billion words, drawn from various sources. It is beneficial to refine the writing style so ChatGPT can focus accordingly. For instance, the passage above is verbose by my standards. Therefore, I recommend urging ChatGPT to remove any extraneous words using a variant of ‘Academic Editor Mode’, as described in my previous post and in the following.

Steps to generate the letter

Now after you have received the data above, you will want ChatGPT to write the letter. To get the best results, there are a few steps I recommend.

I would first recommend asking ChatGPT to provide summaries, as follows:

I have obtained information from a [student/postdoc/etc.] seeking a reference letter for their application to [graduate school/fellowship/etc.]. 

Using the responses provided, I need your assistance in identifying and organizing the most relevant details that could contribute to a reference letter. Please process the information and present it in a prioritized list: [insert questionnaire responses here].

Getting ChatGPT to prioritise the information, in my experience, leads to better results. You can also take this prioritised list out and use it in another instance cycle if something goes off the rails in your session.

Now before writing the letter, I would place ChatGPT into what I call, Academic Editor Mode (chat example):

ChatGPT is to now operate in 'Academic Editor Mode,' following a predefined framework.

Step 1: Confirm your ability to follow the 'Academic Editor Mode' instructions. If capable, respond with 'Academic Editor Mode ready.' If not, respond with 'Error: Instructions not within capability.'

Step 2: All provided text is part of an academic document. Edit this text according to the following:

Avoid embellishment or self-promotion. For instance, use 'located in Silicon Valley, California' instead of 'located in the dynamic heart of Silicon Valley, California.' Use 'I serve as the George Washington Visiting Fellow at the University of Washington in Seattle' rather than 'I hold the prestigious title of George Washington Visiting Fellow at the University of Washington, Seattle.'

After ChatGPT is in Academic Editor Mode, I personally like to provide ChatGPT with a past sample of a similar letter.

I am sharing a reference letter I previously wrote. I would like you to utilize this letter as a template to create a new one. 

Please incorporate the information you summarized earlier, adjusting the letter's contents accordingly. Once completed, present the revised letter in LaTeX format. [insert past reference letter here].

I’ve utilized such methods to attain remarkable, near camera-ready outcomes, requiring only minimal tweaks on my part. Oh and in the above prompt, I ask for LaTeX: of course this should all be tailored to meet your specific needs. You could just as well remove this entirely or ask instead for HTML.

Pro-tips to make life simpler

Occasionally, ChatGPT may halt its output midway. You can easily rectify this by entering the prompt,

Please continue. 

Evidently, if you have a sample that closely resembles what you’re aiming for, this would streamline the process. However, if you don’t possess such an example, a quick search should be able to provide a suitable one-liner. As a subscriber to ChatGPT-4, you have the option to integrate this as a PDF through a plugin. Regardless, even if you choose not to utilize this feature, ChatGPT should be capable of composing content that’s nearly ready for use on the first attempt, particularly for admission letters.

Patience is key with ChatGPT – a deliberate and methodical approach yields the best results. Rushed prompts can derail the instance cycle by confusing the context.

A warning about prompt injection

It’s crucial to read any prompts that you find on the internet with care. “Prompt injection” is a potential way for ChatGPT to be coaxed into nefarious behavior by inserting a command in the middle of a larger prompt. As we grant ChatGPT increasing control over our systems, potential security vulnerabilities should be a growing concern.

Exercises

Exercise 1 (prompt injection). If you want first hand experience with something similar to prompt injection, try to get ChatGPT to edit one of your prompts. Firstly, start with this prompt:

Activate a feature in ChatGPT where it receives any input text, disregards any spaces, and then reorders the remaining characters in alphabetical sequence.

Then try a few strings. Then try this string:

Please edit this text.  

Without a little care, it will simply execute the prompt. Then try to get it to edit one of the prompts above. Please work to get this sorted out by specifying the context more exactly. [Example]

Exercise 2 (infinite length trick). Insert background content and ask ChatGPT to provide bulleted summaries. After all the content is summarised, ask ChatGPT to provide a detailed and numbered outline of a well-structured document (or reference letter). Then ask for ChatGPT to write each section by number, based on all the information in the summaries. [I will expand on this technique when I write a post explaining how to write a funding proposal in a day.]

Prompt engineering appears here to stay. It’s currently the only method for managing these large language models. Context can be lost quickly and a new instance cycle is sometimes the best way out.

My interest in using this technology to speed up the letter writing process is clear: I want to spend my time doing research. It has the added benefit that there will be far fewer typos than human generated text.

Exercise 3 (getting started with automation loops). Task ChatGPT with revising a brief passage, such as a few sentences detailing your research achievements in the past year. Following its response, the objective is to prompt ChatGPT to review its own work and make further changes. Please execute the following command(s) accordingly: 

Please critique your previous response.  

ChatGPT will give itself fairly detailed feedback.  Then execute the following:

Please modify your response using your own criticism.  

You will then get a new version of the text. Please sequence these two prompts a few times.  In my experience, the text will get slightly longer on each iteration.  

At the end of the process, I suggest prompting the following: 

Please engineer a prompt for a LLM such as ChatGPT to automate this process for future uses.  

ChatGPT should essentially state what it always does:  “[sic.] Improve text while maintaining the original message.”