The Dr. Fox Effect & ChatGPT: Charisma, Cognitive Biases, and Digital Communication

What is the Dr. Fox effect, and what does it say about digital charisma?

The Dr. Fox effect is what happened when students rated an actor on par with a real scientist because of his humour, charisma and warmth, although his talk was gibberish. ChatGPT's clear, polite and convincing answers can work on readers in the same way, which is the digital charisma this post is about.

In the heart of 1970, whispers of an unusual lecture buzzed through the corridors of the University of Southern California School of Medicine. Two groups of students were curious and looking forward to hearing a new class on "Mathematical Game Theory as Applied to Physician Education." Little did they know, the day's lesson held a twist.

Among the presenters, one was a genuine scholar, while the other was actor Michael Fox, cleverly posing as "Dr. Myron L. Fox."

As the lecture unfolded, the students who joined the Dr. Fox lecture were drawn into the charismatic web spun by the excellent performer. Though his speech was peppered with nonsensical jargon, contradictions, and gibberish, they rated him on par with the real scientist.

His secret weapon? An attractive mix of humor, charisma, and warmth.

This experiment, known as the Dr. Fox effect, parallels ChatGPT's abilities. Just as Dr. Fox, ChatGPT can craft convincing yet potentially misleading responses, taking advantage of people's cognitive biases.

Even Stack Overflow fell victim to the ChatGPT spreading and now has experienced a decline in traffic, despite researchers finding that 52% of ChatGPT's responses to questions may need to be corrected.

The following sections will analyze how ChatGPT affects community engagement and communication.

How does ChatGPT affect community engagement?

One of the most appreciated aspects of Stack Overflow is the availability of multiple answers to a single question, along with varied perspectives from developers, aiding in selecting the best solution.

ChatGPT can provide multiple options if explicitly instructed in the prompt. However, its answers lack human-based reasoning. This can be problematic for junior developers. While they can ask questions without fear of negative feedback, they might struggle to identify flawed architectural approaches behind seemingly correct answers.

Therefore our focus should be on promoting best practices and fostering a collaborative environment. This way, developers will feel comfortable communicating within the company and seeking trusted answers offline and online rather than solely depending on "smart" assistants.

Ask for evidence, not confidence

An answer from an AI tool, or a block of code it wrote, should be judged by the evidence that comes with it and not by how sure it sounds. Dr. Fox shows that a smooth, confident delivery can hide empty content, so the check has to work whatever the tone.

Two studies, one on answers and one on code, point to the risk. In a study of ChatGPT answers to 517 programming questions from Stack Overflow, Kabir and colleagues (arXiv, 2023, conditionally accepted for CHI 2024) found that 52% of the answers contained incorrect information. Twelve programmers in the study still preferred ChatGPT's answer over the Stack Overflow one 35% of the time, because of its comprehensive, well-articulated style, and overlooked the misinformation in the incorrect answers 39% of the time.

In a user study of security tasks, Perry and colleagues (arXiv, 2022, ACM CCS 2023) found that participants with an AI code assistant built on OpenAI's codex-davinci-002 model wrote significantly less secure code than those without one, and were more likely to believe their code was secure. Participants who trusted the AI less and engaged more with the wording and format of their prompts wrote code with fewer security vulnerabilities.

So ask the tool to separate what it checked from what it guessed. The prompt below does that. Treat its answer as a list of things to verify, not as proof, because a claim that something was checked is still only a claim. Run the command or open the link yourself before you rely on it.

Split your last answer into CHECKED and ASSUMED. For each CHECKED item, give the command, test or documentation link that shows it, and say how I can run or open it myself. Put everything you recalled or guessed under ASSUMED. Do not change any files and stop after the list.

The two studies do not test this prompt; it is our suggestion.

How can digital charisma help communication practice?

Moreover, we can take advantage of the Dr. Fox effect. Now, everyone has the potential to become an excellent presenter, sharing brilliant ideas with others. ChatGPT offers a unique opportunity, especially for non-English speaking developers, to practice communication with stakeholders and colleagues. It has become the best soft skill simulator ever created.

Since ChatGPT shares all the data with its analytics and others, users can't put it under an NDA. However, by crafting our prompts to be artificial and using fictional scenarios without real names, numbers, and proprietary ideas, we can quickly learn from ChatGPT's clear and polite way of responding.

For instance, here are five examples showcasing ChatGPT's ability to transform a potentially misleading or toxic communication into a more professional tone:

Prompt:

I need to convey feedback to my colleague. Can you help me rephrase the following statement to make it more professional and constructive? Some incorrect sentences.

Case 1

• Original: "Your idea doesn't work. Try something else."
• Reworked: Thank you for the suggestion. I believe there might be alternative approaches to consider. Do you have other suggestions we could explore?

Case 2

• Original: "I don't understand why you did this."
• Reworked: Can you help me understand the reasoning behind this approach?

Case 3

• Original: "You should've told me earlier."
• Reworked: Could we set up a regular check-in or communication method in the future? This will ensure we're both informed and address issues promptly.

Case 4

• Original: "Add more content to the report."
• Reworked with ChatGPT's help: Expand the report by including detailed analyses of our Q2 sales figures, a comparison with Q1, and insights on market trends. Also, incorporate relevant charts and infographics for better visualization.

Case 5

• Original: "The app crashes sometimes."
• Reworked with ChatGPT's help: We've observed intermittent app crashes, particularly when accessing the 'Profile' section on Android devices running version 10.0. Can you investigate and rectify the root cause?

Such interactions refine communication, fostering collaboration, client satisfaction, faster deliveries, and outstanding outcomes.

Conclusion

The wave ChatGPT has created in the online Q&A sphere is undeniable. Technologies are reshaping the digital dialogue. As we stand at this intersection of AI and human intellect, we must recognize and use the potential of such platforms to our advantage.

Next step