Discussing LLM AI: Trusting LLM AI Output - Part I

A discussion on trusting the output of LLM AI

Note: As usual, I used ChatGPT 4.0 in the preparation of this piece as research assistant and ghostwriter. Kids don’t try this at home.

See Part II here:

Author’s Preface

How Much Should You Trust LLM AI

Well, one of the practical issues is to what extent do you trust AI output? And currently, the answer is be damned cautious in trusting it.

The Fundamental Epistemological Issue

So, the epistemological issue is how do you know when any output, human or AI, is correct? And as we've discussed in many cases, we have no real test, although opinions abound, interpretations differ. So, in terms of the simple things, temperature at which water boils at sea level, rate of acceleration due to gravity at the North Pole, things like that, AI will probably do a pretty good job because there's such widespread evidence on these issues that our data on that is pretty satisfactory to just about everyone. Of course, we have the issue of the flat earth and their opinions differ. The majority thinking that flat earthers are mentally deficient, but in any case, that's the interpretation. Even if I agree with the interpretation that the flat earthers are mentally deficient, it's still an interpretation.

The Training Data and Its Deficiencies

So, we have the issue of the quality of the AI output. It start with the information used to create the LLM AI. Out AI is ‘curated’ and ‘trained’ on a large set of data, not vast, but pretty big. It's a set full of information, misinformation, and disinformation. That is just the nature of human understanding, and it is reflected in text used to create the AI system.

Bias Towards Mainstream Views

Also, there is an inherent bias towards mainstream views that are more commonly represented in the data.

Misunderstandings, Biases and Corporate Imperatives

And the trainers select it, the curators select it. So, the machine is biased towards some groups' understanding or misunderstandings and biases and suspect corporate imperatives shaping the system and proscribing certain output.

What Does It Mean To Be Coincidentally Correct Versus Justifiably Correct

But putting that aside, we could argue that when it's correct, it's only coincidentally correct. But that begs the question, doesn't it, of what it means to be coincidentally correct? It is an unresolved epistemological problem, perhaps seen as self-evident by some, but I have problems with such views. Certain epistemological views see that as being a different sort of correctness, because it's not justified. But other epistemologists have said that the whole notion of justification is self-referential and not very useful.

Getting Back to the Key Point of Trust

However, getting back to the key point, the pragmatic point, how much can we trust AI output now or in the future? I'd like to know, since I use the tool. Other people are using it, and they're going to be using it more and more heavily in our world.

Ethical Violations by The AI Development Community

Well, by making AI seem to be human with their apologies, with their pats on the back and so on, the developers have violated ethics. That's not an issue that's brought up, but they have, because they've created a system that makes people trust it when trust is unwarranted. And all the cautions on the interface saying that you have to verify the information don't amount to a hill of beans when it's countered by the appearance of intelligence. So it's very hard for people not to think that they're dealing with an intelligent being. And the answers always seem oh-so-plausible, even when total nonsense. And that's partly in effect because of the ethical failure to consider that making it seem like a human being is going to gull people into believing it. And that's ill-considered at the least, but I think it represents a short-sighted, probably profit-driven, or maybe a “wouldn’t it be cool if … “ failure of ethics.

Do We Need a Language for AI

Currently, we can only have “Chats” with LLM AI using language appropriate to human discourse (of course, that is true of our non-human animal associations as well). This is misleading, and problematic. It again makes us think that we are dealing with intelligence as opposed to statistical words chasing along associative linguistic chains. Having noted that, I have no idea how this might work, and we are probably stuck with the status quo.

---

Introduction

The development of Large Language Models (LLMs), such as OpenAI's GPT series, represents a significant milestone in artificial intelligence (AI). These systems generate human-like text by analyzing patterns in extensive datasets, demonstrating impressive utility in tasks ranging from creative writing to technical support. However, the rise of LLMs also brings critical questions about the trustworthiness of their outputs.

Trust in LLMs is complex, as their design prioritizes fluency and coherence, often at the expense of accuracy or reliability. While they are highly effective tools for certain tasks, users must understand their limitations, particularly when applying their outputs in areas requiring precision or expertise. This essay explores the epistemological, technical, and ethical challenges surrounding trust in LLM AI, providing a comprehensive discussion of the issues at stake.

---

Discussion

How Much Should You Trust LLM AI

Trusting the output of an LLM requires careful consideration of the context in which the system is used. These models are reliable for tasks involving widely agreed-upon facts, such as retrieving basic scientific information or summarizing common knowledge (Marcus & Davis, 2019). For example, when asked about the acceleration due to gravity at sea level, LLMs will typically provide accurate answers because this information is well-documented and uncontroversial.

However, their reliability diminishes in areas requiring nuanced understanding or critical thinking. LLMs do not reason in the way humans do; instead, they generate outputs based on probabilities derived from patterns in their training data (Brown et al., 2020). This probabilistic nature means they may produce plausible-sounding but incorrect answers, particularly in fields with competing interpretations or complex methodologies. Users must therefore verify LLM outputs before relying on them in high-stakes decisions.

The Fundamental Epistemological Issue

A key philosophical issue with LLMs is their inability to provide justification for their outputs. While humans can sometimes justify their conclusions through evidence, reasoning, or analysis, LLMs generate text by predicting what words are most likely to follow a given prompt (Bender et al., 2021). This difference highlights a significant epistemological challenge: how do we verify the correctness of an AI-generated statement?

For straightforward facts, verification is often easy. However, in domains where knowledge is evolving or interpretations differ, assessing correctness becomes more complex. For example, an LLM may produce a convincing analysis of a historical event, but its output reflects the biases and perspectives present in its training data rather than a reasoned evaluation of evidence. This lack of justification makes LLMs fundamentally different from human experts, emphasizing the need for critical scrutiny of their outputs.

The Training Data and Its Deficiencies

The quality of an LLM's training data is central to its performance. These systems are trained on large datasets sourced from diverse written material, including books, academic papers, and online content. However, this data is far from perfect. It includes inaccuracies, outdated information, and gaps that reflect the limitations of human knowledge and understanding at the time of its compilation (Zellers et al., 2019).

Furthermore, some topics may be over-represented or under-represented, skewing the system’s outputs. For instance, technical topics with abundant online documentation might yield more precise answers than specialized or niche subjects. These deficiencies in training data contribute to inconsistencies in the reliability of AI outputs.

Bias Towards Mainstream Views

LLMs inherently lean towards mainstream perspectives because these are more prevalent in their training datasets. This bias can be problematic when alternative viewpoints or minority perspectives are critical to the discussion (Floridi & Cowls, 2019). For instance, cultural or regional practices that are underrepresented in the training data may be inaccurately or incompletely reflected in the AI's responses.

The outputs of LLMs continue to reflect the dominant narratives embedded in their training material, potentially marginalizing voices that are less commonly represented.

Misunderstandings, Biases, and Corporate Imperatives

The biases in LLM outputs are not solely a product of training data; they also stem from decisions made by developers and corporations. These decisions influence which data is included, how the model is fine-tuned, and how outputs are shaped to align with corporate priorities (Johnson & Verdicchio, 2017).

For example, developers may optimize outputs to meet user expectations or regulatory requirements, inadvertently reinforcing specific biases or limitations. Additionally, the proprietary nature of many LLMs limits transparency, making it difficult for users to fully understand the extent of these biases. Addressing these challenges requires developers to prioritize ethical considerations alongside technical performance.

What Does It Mean To Be Coincidentally Correct Versus Justifiably Correct

A significant distinction in evaluating LLM outputs is the difference between coincidental correctness and justifiable correctness. LLMs often produce accurate responses by matching patterns in their training data rather than through any form of understanding or reasoning (Chalmers, 2021). For example, an LLM might correctly answer a technical question because similar answers appeared frequently in its training material, not because it comprehends the underlying concepts.

This coincidental correctness poses risks in applications requiring accountability or explanation. In professional contexts, such as law or medicine, decisions based on unverified AI outputs could lead to errors with serious consequences. Understanding this distinction is essential for appropriately using LLM-generated content.

Getting Back to the Key Point of Trust

Given these limitations, trust in LLM AI must be tempered with caution. As the Author’s Preface emphasizes, users should not assume that AI outputs are inherently reliable or authoritative. Instead, they should critically evaluate AI-generated content, cross-referencing it with verified sources and seeking human expertise when necessary (Marcus & Davis, 2019).

This approach ensures that users can leverage the strengths of LLMs—such as speed and versatility—while mitigating the risks associated with their limitations. As these systems evolve, fostering transparency and improving user education will be crucial to enhancing trust without fostering overreliance.

Ethical Violations by the AI Development Community

The ethical responsibilities of AI developers extend beyond technical considerations. As the Author’s Preface notes, presenting LLMs as human-like systems fosters an illusion of intelligence that can mislead users. This anthropomorphism exploits the tendency of people to trust entities that appear human, creating a false sense of reliability (Crawford, 2021).

Disclaimers urging users to verify AI outputs are often overshadowed by the persuasive nature of human-like responses. Developers must address the ethical perspective by prioritizing transparency, designing interfaces that emphasize the system’s limitations, and ensuring users understand that LLMs are statistical tools, not intelligent beings.

Do We Need a Language for Conversing With AI

The current reliance on human language to interact with LLMs introduces challenges, as it perpetuates the illusion of intelligence. A specialized language designed specifically for AI interactions could clarify the system’s capabilities and limitations, emphasizing its statistical nature rather than mimicking human reasoning (Mitchell, 2019).

However, developing such a language would require significant effort and widespread adoption. In the meantime, users must remain aware of the constraints of human-like AI conversations, using critical thinking to navigate the inherent ambiguities and limitations of these systems.

Summary

This essay explores the complexities of trusting the outputs of Large Language Models (LLMs), emphasizing the epistemological, technical, and ethical challenges these systems present. While LLMs excel in generating human-like text and retrieving well-established factual information, their probabilistic nature and reliance on imperfect training data raise concerns about accuracy, bias, and reliability.

Epistemologically, LLMs lack the ability to justify their outputs, relying instead on patterns in their training data. This makes them prone to coincidental correctness, particularly in nuanced or contested domains. Biases in the training data, coupled with corporate imperatives in system design, further complicate their trustworthiness by marginalizing minority perspectives and prioritizing outputs that align with dominant narratives. Ethical considerations, including the anthropomorphism of AI systems, exacerbate these issues by fostering unwarranted trust among users.

Ultimately, the essay underscores the importance of critical engagement with AI outputs, advocating for transparency in AI development, improvements in training data diversity, and user education to mitigate risks. While specialized AI languages could address some challenges, fostering awareness of the limitations of LLMs remains a pragmatic necessity.

---

References

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610-623. https://dl.acm.org/doi/10.1145/3442188.3445922

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901. https://arxiv.org/abs/2005.14165

Chalmers, D. J. (2021). Reality+: Virtual worlds and the problems of philosophy. W. W. Norton & Company. https://www.amazon.ca/Reality-Virtual-Worlds-Problems-Philosophy/dp/B09FR9FFW1

Crawford, K. (2021). Atlas of AI: Power, politics, and the planetary costs of artificial intelligence. Yale University Press. https://www.amazon.ca/Atlas-AI-Kate-Crawford/dp/0300209576

Floridi, L., & Cowls, J. (2019). A unified framework of five principles for AI in society. Harvard Data Science Review, 1(1). https://hdsr.mitpress.mit.edu/pub/l0jsh9d1

Johnson, D. G., & Verdicchio, M. (2017). AI anxiety. Journal of the Association for Information Science and Technology, 68(9), 2267-2270. https://research-portal.uws.ac.uk/en/publications/ai-anxiety

Marcus, G., & Davis, E. (2019). Rebooting AI: Building artificial intelligence we can trust. Pantheon. https://www.amazon.ca/Rebooting-AI-Building-Artificial-Intelligence-ebook/dp/B07MYLGQLB

Mitchell, M. (2019). Artificial intelligence: A guide for thinking humans. Farrar, Straus and Giroux. https://www.amazon.ca/Artificial-Intelligence-Guide-Thinking-Humans/dp/0374257833

Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019). Defending against neural fake news. Advances in Neural Information Processing Systems, 32, 9051-9062. https://papers.nips.cc/paper/9106-defending-against-neural-fake-news