Understanding Scholarship: Epistemology and the Scholar's Task
I am starting a new series to discuss scholarship and the making of sausage.
Note: This essay was prepared with the research assistance and ghostwriting of ChatGPT 4.0. No LLMAI were harmed in the process, although I felt inclined to threaten them from time to time.
Author’s Preface:
Well, I woke up this morning, and as usual, I had these random thoughts bubbling up. My thoughts this morning were about academic citations, obtaining references, and producing essays.
One of the things that I have encountered in using ChatGPT and other LLM AIs to produce essays is the inability to come up with valid references. They may or may not exist, but I can't always find ChatGPT citation with search engines. The references may or may not be relevant if they exist, and they may not be of high quality. They may be historical and much too old. So, I'm pretty fed up with using ChatGPT for references. It's just not very good. Still, when I look at the alternative ….
So, I started planning a new series on scholarship after looking at the citations topic during a long session with ChatGPT, about the production of essays. And that went off into associated topics, which led to reflections that really, this was an issue of applied epistemology—the search for truth. I've written about this for a long time, with a lot of writing that is not much read, and varies in quality, I suppose.
So, Ephektikoi, my pen name, is a Greek word for Pyrrhonian skeptic. I call myself the Guerilla Epistemologist, Cracker Barrel Philosopher, Cheap Seats Polymath, Smart-ass —a little joke. I realized that this discussion of citations, references, and the preparation of written works harks back to my initial premises of skepticism and epistemology.
So, I decided I really needed to create a series of essays, and I decided to call the collection Understanding Scholarship. After long, convoluted discussions with ChatGPT, I’ve come up with this overview.
Introduction
The field of scholarship spans a vast range of methods, from historical research to experimental studies. Regardless of the discipline, scholars are tasked with evaluating and interpreting evidence, a process that is inherently difficult and often prone to challenges. The complexity of determining what is true in a given field is central to the practice of scholarship, and scholars must grapple with issues of epistemology—the theory of knowledge and understanding. This essay explores the epistemological foundations of scholarship, the challenges in assessing evidence, and the methods scholars use to navigate bias, group think, and the difficulties of human inquiry.
General Considerations
The search for reliable sources of truth is not just about gathering information; it’s a fundamental issue of applied epistemology. Scholarship, whether it is historical research, experimental science, or theoretical inquiry, always revolves around the question: how do we know what is true?
One of the key challenges in any scholarly pursuit is understanding the fallibility of human knowledge. What we believe today may be overturned tomorrow. Mistakes, contradictions, misunderstandings, and biases are inherent in the human process of inquiry. Scholars must make judgment calls—they must decide which sources to accept, which to reject, and which to remain uncertain about.
In academic work, citations are critical. A good scholar must be able to sift through evidence, using heuristics to assess the quality of sources. Factors like author reputation, journal quality, citation counts, and the relevance of the paper to the current field all play a role. However, these methods are not perfect. As scholars like John P. Ioannidis, Richard Horton, and Marcia Angell have pointed out, the traditional ways of determining research credibility—like Randomized Controlled Trials (RCTs) and the hierarchy of evidence—are often flawed. Other disciplines use other methods, and they may be just as flawed. These flaws must be taken into account when evaluating sources.
Epistemological Skepticism and Judgment in Scholarship
The Greek skeptics, particularly the Pyrrhonian skeptics, offer a useful framework for understanding the uncertainties in scholarship. This ancient philosophy of skepticism—reflected in my pen name, "Ephektikoi"—teaches that scholars should doubt both their own views and the views of others, always maintaining a critical stance towards the sources of knowledge they encounter.
Applied to modern scholarship, this means that a good scholar must evaluate multiple perspectives, not just the one they think is correct. A true scholar is an honest broker, presenting not only their best judgment on which sources are credible but also discussing opposing views and why they might be incorrect. This balancing act is difficult but necessary for an honest exploration of the truth.
Navigating Bias, Peer Review, and Groupthink
In the world of scholarship, bias is a significant factor. Scholars often rely on peer review to determine the credibility of a source, but peer review has its flaws. It can be superficial, biased toward established views, and subject to the same group think that afflicts any human institution. Scholars like Ioannidis and Horton have argued that peer review does not always filter out bad science, and financial pressures can distort results. The process of citation, too, is not always free from these pressures—papers with high citation counts are not necessarily more reliable.
Bias can also come from financial and career incentives. The pressure to publish, combined with corporate and political interests, often leads to compromised research. Scholars must remain aware of these pressures and seek to understand the motivations behind the research they are citing.
The Sausage-Making of Scholarship
There is a famous saying: “You don’t want to know how the sausage is made.” This applies to scholarship as well. While the public image of research is one of intellectual purity, the reality is far more complex. Behind the scenes, the process of scholarship can be messy, driven by competing interests, financial incentives, and careerism. Scholars are human, and human flaws find their way into the research process. Understanding this reality is essential for any honest scholar.
Conclusion
In the end, the task of the scholar is to make reasoned judgment calls in the face of uncertainty. Scholars must weigh multiple sources of evidence, navigate biases, and remain open to the possibility that they themselves may be wrong. Scholarship is not a clean process, and the search for truth is fraught with challenges, but it is this complexity that makes the pursuit of knowledge both difficult and worthwhile.
Bibliography
Angell, M. (2004). The Truth About the Drug Companies: How They Deceive Us and What to Do About It. Random House. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC521592/
Capsule: Marcia Angell critiques the pharmaceutical industry’s influence on medical research, highlighting the ways in which drug companies manipulate clinical trials, distort results, and use financial incentives to skew research findings in their favor.Cartwright, N., & Deaton, A. (2016). Understanding and Misunderstanding Randomized Controlled Trials. Social Science & Medicine, 210, 2-21. https://pubmed.ncbi.nlm.nih.gov/29331519/
Capsule: This paper critiques the over-reliance on RCTs, arguing that while RCTs can determine whether an intervention works in a specific context, they often fail to explain the mechanisms behind the results or how they apply in broader, real-world settings.Horton, R. (2015). Offline: What is Medicine’s 5-Sigma? The Lancet, 385(9976), 1380. https://doi.org/10.1016/S0140-6736(15)60696-1
Capsule: Richard Horton critiques the current state of medical research, claiming that much of what is published may be unreliable due to problems such as industry influence, publication bias, and poor methodological standards.Ioannidis, J. P. A. (2005). Why Most Published Research Findings Are False. PLoS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124
Capsule: This seminal paper argues that most published research findings in the medical field are likely to be false due to factors such as small sample sizes, bias, financial conflicts of interest, and flexible research designs that allow manipulation of results.Petticrew, M., & Roberts, H. (2003). Evidence, Hierarchies, and Typologies: Horses for Courses. Journal of Epidemiology & Community Health, 57(7), 527-529. https://doi.org/10.1136/jech.57.7.527
Capsule: This article critiques the strict application of the hierarchy of evidence in public health research, arguing that different types of research questions require different methodological approaches. It promotes using appropriate typologies rather than adhering rigidly to a single hierarchy of evidence.