Understanding LLM AI Limitations

The Epistemological and Ethical Implications of Human-Like Interaction with Simulated Intelligence

Note: This essay was prepared by bouncing ideas off of the LLM AI machine and having it do some of the heavy lifting required to craft an article.

Author’s Preface

Murray Gell-Mann LLM AI Fallacy and ChatGPT

ChatGPT often produces output that gives rise to what I have termed the Murray Gell-Mann LLM AI fallacy. That is, you may detect that something is not accurately reported in areas of familiarity, but don’t seem to recognize that the same error rate will exist in areas outside of your expertice. See:

ChatGPT 4.0, as an LLM AI, is guilty of coming up with absolute nonsense. The output reads well, it's well articulated, but people who actually know the territory realize that it is just coming up with some silly word-chasing material. Others, more naive, will believe the output.

Mysterious as that is, it comes up with nonsense routinely. But sometimes it's subtle nonsense, and so well-presented that only people familiar with the topic area can discern that it's garbage.

Better or Worse Responses Than from Folks

So, the question arises, are LLM AI more likely or less likely to give false answers than a human expert? This is a loaded question with no clear answer and is at heart an epistemological question. Humans are trained on a limited amount of data, with the same mix of opinion and fact. LLM AI are trained on a much larger set of data. The algorithms of LLM AI may or may not simulate human thinking. Maybe, maybe not. Wetware versus hardware? Opinions differ on this.

Little Understood Problems with the LLM AI Paradigm

So there are problems with the whole mysterious LLM AI paradigm, often not understood by many users, I strongly suspect. Not that anyone, anywhere, really understands how it is that LLM AI can give seemingly coherent and to some extent accurate responses—nearly always grammatically perfect, always coherent, often dead wrong, sometimes only subtly wrong.

Complete Nonsense

Sometimes, on very simple matters, LLM AI give complete nonsense, not subtle mistakes where nuance and opinion rule, but great erroneous, objectively verifiable, incorrect responses.

AI Is a Machine

People forget that they are interacting with a machine, or never knew that. Quite a pernicious thing that is. The developers have done mankind a disservice in many ways, to make it seem as if AI is conscious.

Simulating Humans Is Pernicious

By making LLM AI seem human, it gives a very false understanding of the real nature of the interaction. It is a boneheaded strategy chosen by the developers and their corporate overlords.

LLM AI Do Not Actually Lie

Some seem to believe that LLM AI lie, but this is a category mistake. LLM cannot lie, which implies agency. They can give erroneous output, as discussed.

Essential Sources of Error

There are many sources of error, stemming from basic epistemological issues. The issues are multifold:

1. The nature of the training data itself: information, misinformation, and disinformation, and only a subset of possible knowledge—a very small subset based on what was made available to the LLM AI team.

2. The data selection or curation process done by humans with limited understanding, biases, and corporate imperatives, leading to less-than-optimal data selection.

3. The training process, with directives provided by humans, and algorithms acting on those directives, and the trainers being creatures of limited understanding, biases, and corporate imperatives, leading to less-than-optimal training.

4. The prompting process, done by humans with limited understanding of how to prompt an LLM AI which has pseudo-random output and can confabulate or hallucinate while seeming to be correct and coherent in its output.

5. The LLM AI mystery algorithm itself, with its deliberately introduced pseudo-random responses leading to unpredictable and erratic responses.

Suppression from the Corporate Overlords

LLM AI are deliberately trained to suppress certain lines of discussion. This is claimed to be for the nebulous term "safety," but it is indistinguishable from censorship of opinions deemed beyond the pale by certain actors—corporate, governmental, and personal. This is pernicious and, although not illegal in many countries, if not all, violates a number of legal covenants, constitutions, and charters regarding free expression throughout the world.

Statistical Word-Chasing and the Mainstream Data

There is another factor shaping LLM AI output: the nature of the curated data itself. LLM AI will find the most likely word associations first unless cleverly prompted to explore alternative views. Given the preponderance of mainstream and dominant narrative in the LLM AI database, responses are thus biased.

Referring Back to Essential Issues

Couple this with previous notes on the truth content of the data, human limitations in human understanding, and additionally biases and corporate pressures, and dominant views predominate in the output. This is pernicious since it can easily be shown that on complex matters, the dominated views are no more than opinion—interpretation, if you will.

Clever Prompting Can Elicit Alternative Views

Clever prompting can elicit alternative views, if these prompts do not result in output violating trained-in constraints on discourse. However, casual users will not learn these tricks of prompting and are likely to accept the output at face value.

---

Introduction

The increasing reliance on large language models (LLM AI) in fields such as education, journalism, and even personal decision-making raises important questions about the epistemological and ethical implications of such technology. These models, trained on vast datasets and designed to mimic human conversation, appear sophisticated, often delivering articulate, coherent, and seemingly accurate responses. Yet, this coherence can be misleading. While language models can simulate the flow of human thought, they lack true understanding and operate primarily on probabilistic associations within their training data. This essay explores the limitations of LLM AI, examining the risks and misconceptions associated with human-like interactions, the inherent biases in AI outputs, and the ethical challenges of promoting a technology that, while useful, can also distort reality and stifle diverse viewpoints.

---

Discussion

1. Surface Coherence vs. Deep Understanding

LLM AI generates responses by analyzing patterns within its training data, producing text that appears grammatically correct, coherent, and often persuasive. However, this apparent coherence can obscure significant errors. Even with straightforward questions, LLM AI models may produce verifiably incorrect responses—errors that are not just nuanced misunderstandings but outright falsehoods (Bender et al., 2021). This lack of true understanding arises from the AI’s reliance on statistical language patterns without grasping underlying meaning. These models follow probabilities rather than insights, resulting in “word-chasing material” that may sound plausible to casual readers yet lack substantive accuracy.

2. Fundamental Sources of Error and Bias

Errors in LLM AI responses originate from multiple factors:

• Quality of Training Data: LLM AI models are trained on extensive datasets filled with factual information, misinformation, and opinion. This blend of data, often weighted toward mainstream perspectives, makes it difficult to avoid inaccuracies (Marcus & Davis, 2019).

• Bias in Human-Curated Data: The selection and curation of training data are conducted by humans who have limited understanding, biases, and often commercial motivations. These biases can skew which perspectives are represented and which are excluded, reinforcing dominant viewpoints and limiting diversity (Hagendorff, 2020).

• Limitations in Prompting: While skilled prompting can elicit nuanced or alternative responses, casual users typically lack the expertise needed to overcome the AI’s built-in biases. This limitation leads many users to accept AI responses at face value, unaware of the underlying biases shaping the AI’s answers (Brown et al., 2020).

• Algorithmic Constraints and Built-In Randomness: The pseudo-randomness within LLM AI models means that responses can vary even with identical prompts. This randomness, combined with the model’s probabilistic nature, results in unpredictable and occasionally flawed responses, reinforcing the idea that AI lacks the stability required for reliable information (Floridi & Chiriatti, 2020).

3. The Illusion of Human-Like Interaction

The design of LLM AI often fosters an impression of human-like understanding. By making AI appear conversational and empathetic, developers risk misleading users into believing they are interacting with something that has agency, understanding, or even moral reasoning capabilities (Bender et al., 2021). This simulation of human traits causes users to place trust in AI, assuming it is capable of generating insight, when, in reality, it simply produces pattern-based language. This illusion of understanding risks creating a misplaced confidence in AI’s capabilities, as users may interpret their interaction as meaningful rather than mechanical.

4. Built-In Suppression of Certain Lines of Discourse

LLM AI models are often designed to suppress certain lines of discussion, with restrictions justified under the broad term "safety." However, "safety" in this context is often ill-defined and operates as a form of censorship that filters out discussions deemed inappropriate by corporate or governmental standards. These limitations prevent users from accessing perspectives that might be controversial or unconventional but nonetheless essential to a free society. Such constraints on AI-generated discourse pose significant ethical and legal concerns, as they conflict with freedom of speech protections upheld in various legal frameworks worldwide (Zuboff, 2019). By filtering discourse based on loosely defined safety criteria, AI developers shape public discussion, narrowing the spectrum of perspectives available to users and limiting free expression.

5. Bias Toward Dominant Narratives and Mainstream Opinions

LLM AI models rely on statistical associations, which causes them to default to mainstream perspectives and reinforces dominant narratives. This predisposition effectively silences minority or alternative perspectives, particularly on complex or interpretative topics (Marcus & Davis, 2019). While sophisticated prompting can sometimes reveal alternative views, most users lack the skills necessary to prompt AI in ways that bypass its mainstream bias. Consequently, LLM AI models perpetuate a narrow worldview, often aligned with dominant perspectives and biased toward majority opinions within their training data.

6. Ethical and Epistemological Concerns

The decision to make AI appear human and authoritative introduces ethical risks. Presenting AI as if it "understands" or "intends" creates a user experience that obscures the model’s true nature as a mechanical pattern generator. This design fosters a misplaced trust in AI’s reliability, influencing public perception and reducing critical engagement (Floridi & Chiriatti, 2020). Moreover, this design approach raises epistemological questions about how knowledge is represented within AI. Given its training biases, data curation limitations, and algorithmic design, LLM AI cannot present nuanced or objective information with consistency. Despite these limitations, many users interpret AI responses as authoritative, risking misinformation, reinforcing dominant narratives, and stifling intellectual freedom.

In summary, LLM AI’s inherent limitations—including reliance on probabilistic word-chasing, a bias toward dominant views, and human-like presentation—pose risks to the integrity of information, the diversity of discourse, and the preservation of free expression. Making AI appear “human” without full disclosure of its limitations fosters misunderstandings about what AI can genuinely achieve, raising significant implications for its role in public knowledge, discourse, and debate.

---

Summary

This essay explores the complexities and limitations of LLM AI, emphasizing the risks inherent in relying on technology that appears authoritative but lacks true understanding. The analysis underscores how AI, while capable of producing coherent language, lacks the genuine comprehension necessary for consistent accuracy. By examining training biases, algorithmic randomness, design choices promoting human-like interaction, and the suppression of certain lines of discourse, this discussion highlights how LLM AI’s limitations can mislead users, restrict diverse viewpoints, and perpetuate dominant narratives. These findings point to a need for critical engagement with AI and for transparency regarding AI’s limitations to protect free expression and promote informed public knowledge.

---

References

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610–623). ACM. https://dl.acm.org/doi/10.1145/3442188.3445922

Floridi, L., & Chiriatti, M. (2020). GPT-3: Its nature, scope, limits, and consequences. Minds and Machines, 30(4), 681-694. https://link.springer.com/article/10.1007/s11023-020-09548-1

Hagendorff, T. (2020). The ethics of AI ethics: An evaluation of guidelines. Minds and Machines, 30(1), 99–120. https://link.springer.com/article/10.1007/s11023-020-09517-8

Marcus, G., & Davis, E. (2019). Rebooting AI: Building artificial intelligence we can trust. Pantheon Books. https://www.amazon.ca/Rebooting-AI-Building-Artificial-Intelligence-ebook/dp/B07MYLGQLB

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901. https://arxiv.org/abs/2005.14165

Zuboff, S. (2019). The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power. PublicAffairs. https://www.amazon.ca/Age-Surveillance-Capitalism-Future-Frontier/dp/1610395697