Understanding the World: Will Methods and Procedures Improve the World?
Confessions of a Repentant Methodologist (a little more cynical I am)
Note: Research and Ghostwriting by LLM AI; shaped by my fevered brain. No LLM AI were harmed in the production of this essay, although I was tempted. – Ephektikoi
Author's Preface
“What is the difference between a methodologist and a terrorist? – You can negotiate with a terrorist.” This nerdish quip, while not particularly funny or fair, captures a sentiment that I once found objectionable but now embrace.
As a student in experimental psychology, I was quite taken by my introductory research methods course. I naively thought that if we just followed these procedures, we would understand the world—or something like that. I went to graduate school in experimental psychology but gradually lost interest in the field.
My life later led me into the field of computer information systems, where I spent decades in various roles. Often, I found myself the de facto expert on methods and procedures for software development and maintenance; sometime the formal expert.
My career began with roles as a research assistant, statistical number cruncher, and later a programmer. This was followed by positions as a team leader and, occasionally, data administrator. I took on formal roles such as Acting Director of Quality Assurance and Data Administration, and, years later, the job title of Data Administrator, followed by Methods and Procedures for Information Systems. After this period, I returned to data administration, promoting data integration and data quality methods, and worked on implementing Clive Finkelstein’s methods for information systems organizations. My later roles included positions as what was termed in my community a business analyst, and it sometimes took on the aspect of project manager (a ridiculous combination, by the way, as different skills and perspectives are required for these distinct roles - I was not much of a project manager).
Throughout these roles, I investigated and promoted various methodologies, including Systems Development Life Cycle (SDLC) methods, Agile development practices, and the Capability Maturity Model (CMM) for information systems organizations. I worked with data modeling, process modeling, and other tools deemed useful at the time. I proselytized, convinced I had the right answers, wrote many papers—some in-house, a couple published—and delivered numerous presentations.
However, my experiences have left me deeply skeptical about the real impact of formal methods. Success and failure in projects appear to be method-independent; some projects succeed with formal methods, others fail, and the reverse is also true. Many recommendations turned out to be impractical or unworkable.
My focus was primarily on information systems, but I also have a keen interest in scientific research and find that similar issues persist there. The standard portrayal of science often seems like a cartoonish oversimplification. I have looked into the failure of science to live up to its initial promise, but that is another story.
Upon reflection, I am not sure that my work on methods and process improvement made a dime's worth of difference. Sad, that.
I have come to detest bureaucracy, by the way, in all its manifestations, and I was a bureaucrat. Sad that (oh, I said that already).
Introduction
Methods and procedures have long been touted as solutions to complex problems in organizations, promising increased efficiency, consistency, and control. Yet, the reality of their implementation often tells a different story. This essay delves into the historical and contemporary landscape of these methods, revealing that while some have faded into obsolescence, others have gained limited traction, only to face challenges in real-world application. The effectiveness of any method, regardless of its merits, depends heavily on widespread buy-in from all levels of an organization. Without this support, even the most well-conceived methods are likely to fail. Indeed, even God's perfect methods and procedures would fail if there was insufficient support at all levels in an organization. It is doubtful that any proposed method or procedure would meet the deity’s standard in any case.
Historical Origins and Evolution of Methods and Procedures
Early Models: The Birth of Standardization
The history of standardized methods dates back to the early days of the Industrial Revolution. The concept of organized production was pioneered by figures like Frederick Winslow Taylor, whose principles of scientific management laid the groundwork for modern efficiency practices. Taylor’s time-and-motion studies aimed to optimize labor productivity by analyzing and standardizing every aspect of the work process (Taylor, 1911). This approach led to the development of assembly lines, famously implemented by Henry Ford, which revolutionized manufacturing by dramatically increasing production speed and consistency.
These early models of production organization, while groundbreaking, also had their downsides. The intense focus on efficiency often came at the expense of worker satisfaction and creativity. The rigid nature of these methods, designed for predictability and control, left little room for adaptability or human-centered approaches. Over time, this led to the rise of more flexible and adaptive methodologies that sought to balance efficiency with innovation and employee engagement.
Quality Management: Foundations and Decline
The roots of quality management stretch back to the mid-20th century, with pioneers like W. Edwards Deming and Joseph Juran laying the groundwork for what would become foundational practices. Deming’s focus on statistical quality control and Juran’s emphasis on managerial responsibility shaped early quality management frameworks, such as Total Quality Management (TQM). These methods were revolutionary at the time, advocating for continuous improvement and a shift from inspection to prevention (Deming, 1986).
However, as the business environment evolved, many of these methods became less relevant. While the principles behind them remain sound, their rigid application often struggles to adapt to today’s fast-paced and unpredictable market dynamics. This has led to a decline in their use, with organizations favoring more flexible approaches.
The Old-Fashioned Systems Development Life Cycle (SDLC)
The SDLC, one of the earliest formal methodologies for software development, emerged in the 1960s and 1970s. This linear and sequential model, often known as the "waterfall" model, was heavily influenced by the engineering practices of the time. It emphasized thorough documentation, strict phase boundaries, and a disciplined approach to system design and implementation.
While the SDLC was instrumental in the early days of software engineering, its limitations became apparent as technology and business needs evolved. The rigid, sequential nature of the SDLC made it difficult to accommodate changes once a project was underway, leading to delays, cost overruns, and user dissatisfaction. As a result, more iterative and flexible methodologies like Agile began to gain favor, reflecting the need for adaptability in an increasingly complex and dynamic environment.
Contemporary Methods: A Mixed Bag
Six Sigma: Introduced in the 1980s by Motorola and reaching peak popularity in the 1990s, Six Sigma sought to reduce defects and improve processes through the rigorous DMAIC (Define, Measure, Analyze, Improve, Control) framework. Although once hailed as a game-changer, Six Sigma’s popularity has waned as its highly structured and data-intensive approach often proved cumbersome in environments requiring agility and innovation (Brown, 2021). While still used in specific sectors like manufacturing, its relevance in more dynamic settings has diminished.
Lean: Originating from Toyota’s Production System, Lean gained significant traction in the late 20th century with its focus on waste reduction and process efficiency. Although Lean principles have been integrated into various industries, their oversimplification of complex processes and disregard for human factors have led to mixed results. The one-size-fits-all approach often fails to account for the unique challenges of different organizational cultures and industries, leading to its decline in some sectors (Sweeney, et al., 2016).
Agile: Developed for software development in the early 2000s, Agile methodologies such as Scrum and Kanban have become increasingly popular in recent years. Agile emphasizes flexibility, iterative progress, and team collaboration, making it well-suited to the fast-changing demands of modern technology development (Smith, 2023). However, Agile’s effectiveness has been debated, with challenges related to scaling and inconsistent implementation often leading to results that fall short of expectations (Williams, 2024). In many cases, organizations struggle to maintain the cultural shift required for Agile to succeed, resulting in superficial adoption without meaningful improvement.
Current Approaches and Methodological Controversies
Capability Maturity Model (CMM)
Developed by the Software Engineering Institute in the early 1990s, the Capability Maturity Model (CMM) provides a framework for assessing and improving software development processes. CMM describes a staged approach to process improvement, from initial practices to optimized processes (Paulk et al., 1993). Despite its structured approach, CMM has faced criticism for its complexity and challenges in implementation. Many organizations find it difficult to progress through the maturity levels, and the model's rigidity can be at odds with the need for flexibility in today’s fast-paced business environments.
Changing Fashions in Methods
Methodologies often experience cycles of popularity. While Six Sigma and Lean were once dominant, recent trends show a shift toward more flexible approaches like Agile and Design Thinking. These shifts reflect changing organizational needs and highlight the transient nature of methodological fashions. As organizations evolve, so too must their approaches to process improvement, leading to the rise and fall of various methodologies over time.
Ad Hoc Methods and Home-Grown Solutions
Many organizations develop ad hoc or home-grown methods tailored to specific needs. These custom approaches can offer flexibility but may lack the rigor of established frameworks. Their effectiveness depends on alignment with organizational goals and adaptability to evolving circumstances. While these methods can be highly effective in certain contexts, they often lack the scalability and consistency provided by more formalized approaches.
Opportunities in Process Improvement
The pursuit of process improvement offers numerous opportunities for organizations willing to invest in the right methods and practices:
Enhanced Efficiency: By streamlining operations and reducing waste, organizations can achieve significant improvements in efficiency and productivity.
Increased Quality: Implementing robust quality management practices can lead to higher-quality products and services, enhancing customer satisfaction.
Greater Adaptability: Modern methodologies like Agile provide organizations with the flexibility to respond to changing market conditions and customer needs.
Threats and Challenges
However, the implementation of process improvement methods also comes with challenges:
Resistance to Change: Organizational inertia and resistance to new methodologies can impede the successful adoption of process improvements.
Overemphasis on Methodology: Focusing too heavily on rigid methodologies can lead to a lack of innovation and adaptability, hindering long-term success.
Misalignment with Organizational Culture: Methods that do not align with an organization’s culture and values are likely to face implementation challenges and may ultimately fail.
Human Factors and Practical Challenges
Human factors play a critical role in the success or failure of process improvement methods:
Training and Understanding: Effective training is essential for ensuring that employees understand and can effectively implement new methodologies. Without proper training, even the best-designed methods may fall short.
Adherence to Procedures: Adherence to established procedures can be challenging, especially in organizations where individuals may be resistant to change or where procedures are seen as overly complex or burdensome.
Creativity and Flexibility: Rigid methodologies can stifle creativity and flexibility, leading to a lack of innovation and adaptation to new challenges. Balancing structure with flexibility is key to achieving successful outcomes.
Do Methods and Procedures Improve Outcomes?
The effectiveness of methods and procedures in improving outcomes remains a subject of debate. Success and failure in projects often appear independent of the methodologies employed, suggesting that other factors, such as leadership, organizational culture, and external conditions, play a significant role. While some methods have demonstrated success in specific contexts, their broader applicability and impact can be limited.
Conclusion
In conclusion, while methods and procedures have their place in organizational practices, their effectiveness is not guaranteed. The success of any methodology depends on its alignment with organizational needs, culture, and the ability to adapt to changing conditions. As such, organizations should carefully consider their approach to process improvement, balancing the benefits of formal methods with the need for flexibility and innovation.
References
Note: I used to eat this stuff up, nerd that I am.
Brown, K. (2021). Six Sigma for Process Improvement: A Comprehensive Guide. Wiley. https://www.sixsigmacouncil.org/wp-content/uploads/2018/08/Six-Sigma-A-Complete-Step-by-Step-Guide.pdf
Deming, W. E. (1986). Out of the Crisis. MIT Center for Advanced Educational Services. https://mitpress.mit.edu/9780262541152/out-of-the-crisis/
Sweeney, B., & ClydeBank Business. (2016). Lean Six Sigma QuickStart Guide: The simplified beginner's guide to Lean Six Sigma (Illustrated ed.). ClydeBank Media LLC. https://www.amazon.ca/Lean-Six-Sigma-QuickStart-Guide/dp/1945051140
Paulk, M. C., Weber, C. V., Garcia, S. M., & Chrissis, M. B. (1993). Capability Maturity Model for Software, Version 1.1. Software Engineering Institute. https://insights.sei.cmu.edu/documents/1092/1993_005_001_16211.pdf
Taylor, F. W. (1911). The Principles of Scientific Management. Harper & Brothers. http://strategy.sjsu.edu/www.stable/pdf/Taylor,%20F.%20W.%20(1911).%20New%20York,%20Harper%20&%20Brothers.pdf