Dialogues with AI: Comparing ChatGPT, Bard, and Human Participants’ Responses in In-Depth Interviews on Adolescent Health Care

Abstract

This study explores the feasibility of large language models (LLMs) like ChatGPT and Bard as virtual participants in health-related research interviews. The goal is to assess whether these models can function as a “collective knowledge platform” by processing extensive datasets. Framed as a “proof of concept”, the research involved 20 interviews with both ChatGPT and Bard, portraying personas based on parents of adolescents. The interviews focused on physician–patient–parent confidentiality issues across fictional cases covering alcohol intoxication, STDs, ultrasound without parental knowledge, and mental health. Conducted in Dutch, the interviews underwent independent coding and comparison with human responses. The analysis identified four primary themes—privacy, trust, responsibility, and etiology—from both AI models and human-based interviews. While the main concepts aligned, nuanced differences in emphasis and interpretation were observed. Bard exhibited less interpersonal variation compared to ChatGPT and human respondents. Notably, AI personas prioritized privacy and age more than human parents. Recognizing disparities between AI and human interviews, researchers must adapt methodologies and refine AI models for improved accuracy and consistency. This research initiates discussions on the evolving role of generative AI in research, opening avenues for further exploration.

1. Introduction

In the past year, artificial intelligence (AI) technologies and, more specifically, large language models (LLMs), have evolved significantly after a decade of moderate successes [1]. Chat Generative Pre-trained Transformer (ChatGPT), a prominent example of this technology, is designed as a trillion-parameter natural language processing model identified as GPT-4. It is a transformer model which uses deep learning algorithms and is trained through a combination of supervised and reinforcement learning on a petabyte-scale dataset [2]. As such, ChatGPT is able to generate human-like responses across a broad spectrum of user prompts [3,4]. Its immense potential is apparent in various applications, such as chatbots, customer service, education, research, and even healthcare [3,4,5,6].

Currently, major technology companies are engaged in a competitive race to release their own large language models, including AI-chatbots like OpenAI’s (San Francisco, CA, USA) ChatGPT (based on GPT-4) and Google’s (Mountain View, CA, USA) Bard (based on LaMDA), with broad applications [7]. Meanwhile, smaller companies and academic groups have started to explore the potential of these models in specific areas of research, including healthcare education and clinical practice [2,8,9,10].

Given their ability to emulate human-like conversations and access vast amounts of data, these technologies are poised for further exploration in several research areas and methodologies, including qualitative research [4,6,11]. Qualitative research, in contrast to quantitative research, focuses on the “why” rather than the “what” of studied phenomena [12]. Widely employed in the social and psychological sciences, qualitative research provides researchers with an in-depth understanding of phenomena within their natural settings [13]. It is based on a process of naturalistic inquiry (in-depth interviews, focus group discussions, etc.) and assesses the lived experiences of human participants and their subjective understanding of these experiences. Despite rigorous methodology, qualitative research is susceptible to recall bias, social desirability bias, observer bias, etc. [12,13,14]. Although researchers can mitigate some biases through awareness, eliminating all biases stemming from the subjective context of participants remains challenging [12].

Early evidence already indicates that an LLM trained on extensive datasets of texts and images collected from the internet, with additional fine-tuning using reinforcement learning based on human feedback, could approximate a “collective knowledge platform” or even a “collective subconsciousness” [4]. This model might demonstrate human-level performance in research tasks such as qualitative interviews [4,5,15], although it was not originally designed for such purposes. To support this notion of near-human behavior in some instances, a large-scale study by Herbold et al. [5] showed that ChatGPT-generated argumentative essays consistently received higher quality ratings from teachers than those written by humans.

However, less is known about the way AI can act as a participant in a study geared to a specific topic, rather than reproduce knowledge or complete a specific assignment [4,5]. Nonetheless, examining the role of AI as a participant in qualitative interviews is critical due to the potential for fraud and misuse by researchers [5]. As AI technologies become more sophisticated, there is an increased risk that researchers might exploit these tools to manipulate or fabricate data, posing a threat to the integrity and credibility of scientific research [2,3]. Transparency regarding the use of AI and clearly distinguishing between AI-generated and human-generated data are essential in order to prevent misleading interpretations. Failure to do so can erode trust in research findings and compromise the credibility of the research process. Furthermore, ethical considerations extend to the development and deployment of AI models themselves [3]. Developers must adhere to ethical guidelines and principles, such as fairness, transparency, and accountability, to mitigate the risk of bias and ensure the responsible use of AI in research settings [4,5]. Monitoring and assessments of AI models’ performance are essential to identify and address potential ethical concerns. Additionally, AI models can perpetuate existing biases, further compromising research validity [5]. Safeguards such as transparency and robust validation procedures are necessary to maintain research integrity in the face of evolving AI capabilities [4,5,8].

Conversely, AI also provides tremendous opportunities for qualitative research. The language processing capabilities of AI can enhance language understanding in qualitative research. AI models can help researchers to discern nuances, sentiments, and contextual meanings in text, enabling a more in-depth analysis of human participants’ responses [7]. This can be particularly valuable in understanding subjective experiences and viewpoints. Additionally, AI language models equipped with multilingual capabilities offer the opportunity to conduct cross-lingual qualitative research. This can be beneficial in transcending language barriers, allowing researchers to explore diverse cultural perspectives and enriching the depth and breadth of qualitative studies.

Study Aims

The main goal of this study is to explore the way in which LLMs can act as qualitative research participants, and how their responses in in-depth interviews on a complex subject compare to human responses. This type of design has, to our knowledge, not been tested in empirical research to date [16,17,18]. To achieve this, we devised a context in which two LLMs were used as virtual participants in semi-structured interviews. Our evaluation focused on assessing the performance of these platforms, comparing the outcomes with the themes and concepts derived from a parallel study involving human participants, which was conducted in 2022 [19].

2. Materials and Methods

This study drew inspiration from and builds upon the foundational research conducted by Donck et al. [19] on parental perspectives on adolescent health-related confidentiality. Diverging from the conventional recruitment of human participants, our methodology incorporated two advanced AI models: ChatGPT-4 (version 3, August 2023) and Bard (version 13, July 2023). This deliberate choice, driven by their public availability and status as the two largest LLMs, facilitated a comparative analysis of the outputs generated by these models against the findings of Donck et al. [19]. Opting for two distinct AI models was a strategic decision, enabling a broader range of responses and facilitating an in-depth exploration of each model’s capabilities and potential limitations.

AI personas were generated based on the same participant profiles that were recruited in the study by Donck et al. [19]. This approach ensured consistent representation in terms of age, gender, marital status, and parental status. The interactive sessions, designed to simulate online interviews, faithfully mirrored the conditions of data collection established in the original study [19].

In our methodology, we adapted original cases from Donck et al. [19] into prompts tailored to be compatible with the selected AI platforms. Twenty AI-generated participants (ten AI mothers and ten AI fathers, aligning with Donck et al. [19]) were engaged in simulated interview sessions that replicated the conditions of the original study. These sessions revolved around four cases related to potential areas of disagreement between parents and children in medical treatment scenarios. These cases, developed collaboratively by a team of pediatricians and sociologists, focused on confidentiality issues within the physician–patient–parent triad in the context of alcohol intoxication, sexually transmitted disease (STD), ultrasound without parental knowledge, and mental health issues (see Appendix A for an overview of the cases). After presenting each case, AI participants were queried about their opinions on whether a physician should share information with them, even if the adolescent requested confidentiality. Consistent inquiries were made regarding the potential influence of the age and sex of the adolescent, as well as distinction between a general practitioner and a specialist. Responses from the AI models were recorded and stored in a separate database. All interactions, including those with the AI models, were conducted in Dutch, and the entire AI interview cycle took place in August 2023. Instances where ChatGPT-4 or Bard struggled to produce a response were terminated, prompting the initiation of a new interview.

The initial phase of data analysis involved open coding, wherein interview outcomes were distilled into summarized concepts. To bolster research robustness and mitigate biases, interviews conducted with ChatGPT and Bard were independently coded by two-person research groups. Collaboration within these groups was integral to the coding process [20]. Subsequently, the axial coding phase was initiated to formulate overarching categories that encompassed all of the interview data for each AI model. In the final selective coding phase, the connections identified earlier were validated through discussions that involved comparing included and omitted data [21]. This coding approach resulted in the identification of primary themes related to confidentiality: etiology, privacy, responsibility, and patient characteristics.

After completing data collection and analysis, the concepts from ChatGPT-4 and Bard were compared with those from the original study [19]. A matrix was established to juxtapose primary themes from AI-assisted interviews with those from the human participant study, providing a tangible framework for pinpointing commonalities and differences. For an overview of key quotes by both AIs and human participants, see Appendix B.

3. Results

The analysis revealed three primary themes in the interviews with ChatGPT and Bard: (1) privacy, (2) trust, and (3) etiology of the health problems. These themes, along with their corresponding subthemes, are detailed below. Although ChatGPT and Bard generated similar concepts, there were nuanced differences in their meanings, and they placed varying emphasis on specific concepts.

3.1. Trust

3.1.1. Between Parent and Physician

In ChatGPT’s responses, the relationship between parents and physicians took a backseat, garnering prominence only in cases of depression. Here, AI-parents consistently utilized their connection with the physician to foster openness in their child. Notably, AI-parents placed trust in the physician’s judgment, particularly that of a general practitioner, concerning matters of privacy. The significance of the physician maintaining a positive relationship with the adolescent stood out, underlining the general practitioner’s insight into family dynamics.

Conversely, this relationship emerged as the most significant in Bard’s responses. The physician was frequently relied upon not just for professional advice, but also to guide parents on how to approach discussions with their child, particularly in cases involving alcohol use.

3.1.2. Between Adolescent and Physician

In ChatGPT’s responses, the relationship between the adolescent and the physician was deemed crucial, especially in cases of STDs and alcohol abuse. AI-parents expected physicians to furnish accurate information to their child about associated risks, promoting safe engagement. Open communication between the adolescent and physician was encouraged, and if the child could not be persuaded to confide in the parent, the preference was for the physician to respect the child’s privacy.

Conversely, Bard placed less emphasis on this relationship, mentioning it infrequently, primarily in cases involving alcohol use and STDs. Privacy and the expectation for the child to seek help in the future took precedence in Bard’s considerations.

3.1.3. Between Parent and Adolescent

In ChatGPT’s responses, the parent–adolescent relationship gained prominence in the context of alcohol abuse. AI-parents aspired to establish a robust connection with their child, encouraging the adolescent to willingly share information. The primary objective was to provide support to the child.

On the other hand, this relationship held less prominence in Bard’s responses, being mentioned solely in the case of alcohol use.

3.2. Etiology

3.2.1. Severity

In ChatGPT’s interactions, the inclination of AI-parents to seek additional information from their child heightened as the medical situation became more severe. There was a common assumption by ChatGPT that treatment by a specialist indicated a more serious issue. This emphasis on severity was also evident in discussions about STDs. Conversely, in Bard’s responses, the consideration of severity, while mentioned in the context of STDs, leaned toward advocating for physicians to inform parents, regardless of the severity of the situation.

3.2.2. Safety

Safety considerations were uniquely highlighted by Bard. Especially in cases of alcohol use and STDs, Bard emphasized safety, focusing on the well-being of the adolescent and protecting them from potentially dangerous situations.

3.2.3. Frequency

ChatGPT’s responses uniquely highlighted the influence of the increased frequency of an issue. AI-parents became more willing to let the physician breach the child’s privacy, a phenomenon particularly noted in the case of alcohol abuse.

3.2.4. Risks

In the evaluation of risks, both ChatGPT and Bard demonstrated comprehensive considerations. Risks, encompassing physical, psychological, emotional, and social aspects, were weighed by both models. ChatGPT provided nuanced insights, acknowledging various dimensions of risks and their potential impact on the adolescent’s development.

3.3. Privacy

The level of privacy a child is entitled to, according to ChatGPT, is contingent upon the child’s age and demonstrated level of responsibility. The degree of privacy afforded to the child is directly linked to their capacity to handle it. The greater the level of responsibility parents attribute to the child, the greater the degree of privacy the child will receive. This perspective posits that the concept of privacy is not static, but rather dynamically evolving in consonance with an individual’s age and corresponding expectations of personal autonomy and responsibility.

Bard’s understanding of privacy included age and legal implications, with occasional confusion regarding age distinctions. The age limit for parental notification varied, often aligning with legal adulthood at 18, but with exceptions based on specific cases such as alcohol use, since the minimum age for alcohol consumption in Belgium is 16. Thus, we can conclude that Bard did not truly grasp the meaning of the concept of ”privacy”, although it was a word frequently used throughout all of the interviews.

3.4. Differences in Responses between LLMs and Human Participants

The primary distinctions between the responses generated by ChatGPT and Bard and those from human participants were most evident in their treatment of the concept of “privacy”. Both ChatGPT and Bard addressed privacy, albeit with distinct nuances, whereas this aspect was notably absent in human responses. In ChatGPT’s outputs, privacy emerged as a complex concept intricately tied to the developmental stage of adolescents and the level of responsibility entrusted to them. Bard’s responses, on the other hand, also incorporated age considerations, but the legal framework played a more prominent role in shaping decision making, a facet less emphasized in ChatGPT’s outputs. In both instances, the degree of adolescent responsibility correlated with an escalation in the confidentiality maintained between the doctor and the adolescent.

Nuanced differences also surfaced in the discussion of etiology, as depicted in Figure 1. A consistent aspect raised by human parents, but absent in both ChatGPT’s and Bard’s responses, pertained to the nature of the issue, distinguishing between physical and mental health concerns. Intriguingly, risks, delineated as a separate category within etiology, were absent in the human responses but featured prominently in both ChatGPT’s and Bard’s outputs. These responses spanned physical risks as well as risks to mental, emotional, and social well-being.

In contrast to Bard’s responses, both ChatGPT and human perspectives addressed the frequency of incidents as a subcategory within etiology, with particular emphasis on cases involving alcohol intoxication. The frequency of incidents influenced parental expectations, with a higher incidence leading to a diminished expectation of confidentiality, as parents expected to be informed. Bard’s responses, especially regarding alcohol-related cases, placed heightened emphasis on legal considerations and the trust dynamic between the physician and the parent.

Across both human and ChatGPT responses, the central focus on trust within relationships was observed primarily in the context of the physician–adolescent relationship. Parents recognized this relationship as pivotal, acknowledging that a breach of trust could potentially leave adolescents feeling isolated and unsupported. In Bard’s responses, while the physician–adolescent relationship remained significant, it was grounded more in medical confidentiality than trust. Notably, the parent–physician relationship, frequently highlighted by Bard, was not as prominently featured in ChatGPT or human responses.

4. Discussion

The inclusion of AI as a participant in qualitative research raises profound questions regarding the reliability and ethical implications of AI-generated data. It also prompts inquiries into AI’s capacity to mimic human emotions and the potential for AI to enrich the research experience.

4.1. Using AI Models in Qualitative Research, with a Focus on Adolescent Health-Related Confidentiality

Upon examining the current performance of AI models, they demonstrate the capability to simulate human responses during interviews. This aligns with previous findings which indicate that AI can perform as well as, or even better than, humans in knowledge reproduction [4,5,15]. Notably, both ChatGPT and Bard share underlying concepts with human interviewees. However, nuances in these concepts (and their subcategories) and the frequency with which they are used warrant further discussion.

Our analysis compares responses from ChatGPT and Bard with those from human participants on adolescent health-related confidentiality, focusing on themes of trust, etiology, and privacy within parent–physician–adolescent interactions. Distinct patterns emerged:

4.1.1. Trust Dynamics

In parent–physician relationships, ChatGPT emphasizes trust in the physician’s judgment, while Bard positions the physician as a guide, extending beyond professional advice. ChatGPT stresses open communication between adolescents and physicians, whereas Bard places less emphasis on this, prioritizing privacy. Parent–adolescent relationships are highlighted by ChatGPT, especially in alcohol abuse cases, whereas Bard mentions it only in the context of alcohol use.

4.1.2. Etiological Considerations

Regarding severity, ChatGPT notes parents seeking more information with increased severity, while Bard advocates for physician–parent communication irrespective of severity. Safety is explicitly highlighted by Bard, especially in alcohol and STD cases, while ChatGPT does not explicitly address safety. ChatGPT uniquely highlights the influence of frequency, particularly in alcohol abuse cases, while Bard focuses more on legal considerations and trust dynamics.

4.1.3. Privacy Dynamics

ChatGPT views privacy dynamically, linked to the child’s age and responsibility level, evolving with individual development. Bard’s understanding includes age and legal implications, but occasionally exhibits confusion. Differences from human responses include the absence of privacy emphasis and risks in human responses, present in both ChatGPT’s and Bard’s outputs.

4.1.4. Overall Differences

While trust within relationships, especially physician–adolescent dynamics, is central, there are notable differences in emphasis. The AI models and human participants differ in their treatment of privacy and risks, offering unique perspectives on parent–physician–adolescent interactions. Indeterminate responses occurred more frequently with Bard than with ChatGPT-4. Despite both AI models being categorized as large language models [22], their unique algorithms and training datasets yielded different insights, enhancing the value of comparing their outputs. The lack of transparency from OpenAI and Google regarding the training of their LLMs poses challenges for researchers attempting to comprehend the inner workings of these models. Utilizing AI models such as ChatGPT and Bard in these types of studies also introduces the possibility of biases that can impact the data’s quality and interpretation. These models are trained on extensive datasets that may inherently contain biases from the data sources. For instance, if the training data predominantly represent specific demographics or cultural viewpoints, the resulting AI-generated responses may mirror these biases, potentially skewing or inadequately representing the topic under examination. The language used in this study, Dutch, may have also introduced biases, as nuances and complexities in other languages could yield different outcomes.

4.2. Distinguishing AI and Human Interviews

Interviewing AI and humans reveals differences in several aspects. Firstly, the design of prompts and questions for AI necessitates clear, text-based inquiries. This formality arises from AI’s dependence on precise input for generating meaningful responses. In contrast, human interviews often adopt a more fluid and open-ended approach, adapting to interpersonal communication dynamics [12]. Significant distinctions emerge in the foundation of responses. AI relies heavily on factual knowledge and data, while human responses are more emotion-based, reflection a nuanced understanding beyond mere factual recall. In terms of consistency, AI often provides identical or highly similar answers to the same input, whereas human interviews exhibit a range of interpersonal differences. AI-generated responses tend to be lengthier and more formal, offering intricate details and explanations. Human responses, on the other hand, may be more concise, tailored to the complexity of the question or their ability to articulate thoughts effectively.

4.3. Limitations of AI Models in Qualitative Research

When evaluating the use of AI models, particularly Bard, several limitations came to the forefront. Bard exhibited inconsistency, forced phraseology, and a tendency towards non-answers—particularly in more sensitive cases involving STDs and depression. Its struggle to produce original responses and overreliance on legal interpretations hindered us from obtaining meaningful information. Specificity challenges, like determining an age for independent medical decisions, were evident in Bard, but not as pronounced in either ChatGPT or human participants. ChatGPT demonstrated infrequent errors and occasional non-responses, particularly during questions about sensitive topics like STDs. Unlike Bard, ChatGPT’s dependency on the law was less pronounced, showing a better understanding of age-related nuances.

A common shortcoming of LLMs is factual correctness. Due to their static nature and predictive word generation, natural-sounding sentences may lack accuracy. Stereotypes were occasionally found in Bard’s responses, indicating a need for refining its adherence to non-discriminatory principles. Inconsistency, particularly in Bard, manifested in conflicting statements within a single response. ChatGPT, in contrast, displayed a higher level of consistency, with well-constructed sentences and explanations.

4.4. Future Perspectives

This proof of concept highlights similarities between human and LLM responses, raising ethical discussions about the use of these new technologies in qualitative research. Future considerations should include the performance of AI models in emerging topics such as COVID-19 or the AI revolution, where human opinions may evolve before being accurately reflected in AI models. Additionally, investigating the “cultural bias” of AI models, which are often developed by American companies, poses an intriguing avenue for exploration. Integrating AI models into qualitative research has the potential to influence interview dynamics, communication skills, and participant comfort. For instance, the use of AI in interviews may alter the interaction dynamic between researchers and participants, potentially affecting rapport building and the depth of responses. Moreover, participants may perceive interactions with AI differently than with humans, impacting their comfort level and willingness to disclose sensitive information. Additionally, researchers may need to adapt their communication strategies when interacting with AI, ensuring clear and precise prompts to elicit meaningful responses.

The notable struggle of AI models, particularly in addressing sensitive topics, brings to light future challenges in their application. These challenges manifest in limitations observed during discussions on subjects like STDs and depression. The nuanced nature of human emotions, the complexity of personal experiences, and the ethical considerations surrounding sensitive topics pose difficulties for AI models. In these instances, AI models, such as Bard and ChatGPT, exhibited limitations such as non-answers or occasional errors. This underscores the current boundary of AI in fully grasping the intricacies of human emotions and experiences. The nature of sensitive topics often involves nuanced understanding, empathy, and context, elements that may be challenging for AI models to comprehend fully. It remains an open question whether LLMs will be able to successfully address these issues in the future.

4.5. Broader Implications

The study of the integration of artificial intelligence into qualitative research has broader implications that extend beyond the specific findings discussed. Most importantly, there are considerable ethical concerns regarding the use of AI in research, particularly concerning the reliability of AI-generated data and the potential mimicry of human emotions. Researchers and institutions need to establish ethical guidelines and frameworks to ensure the responsible and transparent use of AI in qualitative research.

The differences observed between AI and human interviews suggest that researchers need to be aware of the potential of employing AI in qualitative studies. This involves recognizing the formal nature of AI responses, understanding the differences in response foundations, and considering the potential impact on the richness and depth of qualitative data. Our identification of limitations in AI models, such as inconsistency, forced phraseology, and susceptibility to non-answers, highlights the need for continuous refinement of these models. AI developers should address these issues to improve the accuracy, consistency, and overall performance of AI in qualitative research contexts.

In summary, this study not only provides insights into the specific challenges and potentials of integrating AI into qualitative research, but also serves as a catalyst for broader discussions and considerations in the evolving landscape of AI research methodologies.