I'm taking a closer look at ordinal scaling and its use in various contexts, which seems endemic but misapplied. I believe I can find scholars to support me in this view by appropriate cherry picking.
You are perfectly correct, people (scientists) build complex models/systems often (making assumptions = a leap of faith) incorrectly applying mathematical tools in situations they are not designed for .. and when models break (start spewing obvious garbage) nobody understands why, because assumption-error was introduced 100s of steps back (and you are lucky if there was only one).
The sad thing is, there is a marked inability to replicate studies in various fields. Medicine, nutrition (I think), Psychology, just about any "soft science."
I use that term "soft science" more and more advisedly, pessimistically. Maybe it is just technological shamanism, but when it comes to medicine and nutrition and so on, it has implications for practice. Big implications.
There are lots of reasons for this failure to replicate but I suspect that the instability and inappropriateness of the measures used is part of it. Moreover, I think it is not just the misapplication of statistics, but the fundamental unsuitability, fitness for purpose, of statistical models for such complicated and complex fields. I suspect this is leading to multiple wild goose chases.
I would like to be wrong, but I can reach no other conclusion. There are a few other dissenting voices on these things, working at a higher pay grade than me of course.
We have to distinguish social factors vs mathematical factors.
We also have to distinguish between industry and academia.
Social factors: Everybody lives under pressure "publish or die", things are rushed to review at the last moment, the experimental data has often been cherry picked to fit the journal and showcase authors' "stunning" foresight what and how to investigate all in hope of getting a tenure (things do ease a bit once tenure has been obtained). Nobody has time to evaluate/understand "failed" experiments .. Nobody has time to repeat experiment (possibly several times) to ensure it was not a fluke.. Nobody has time to try to replicate other's experiments because they have not been described in details (intentionally) with many unspoken assumptions and details left out. It is actually more time consuming trying to replicate/understand somebody elses junk science than making your own junk science. Notable exceptions are comparative studies, but they are only done for very selective, stending out papers/results.
Mathematical factors: Many experiments start from very different initial conditions, you have to have a strong convergence to arrive to reasonably similar results. The more complex experiment more likely things will diverge (chaos theory). Rarely people take time to repeat experiments even less to understand the dynamics and mathematical reasons for divergence.
Models are simply overly complex to understand now days.
Academia suffers from "non-cosequences" of their BS, they can publish anything, everything works on paper. There are no real-life consequences if things do not work like described, or go wrong.
Industry does not have that luxury, and often industry is not motivated to publish their own research/results because when properly done they are time consuming and expensive, and they offer competetive advantage you do not want to throw away.
So yeah, you want to find real science, you have to go to industrial/private research.
I liked your comment on chaos theory. That seems to be a good way to conceptualize the more complex and complicated areas where we look vainly to find scientific answers.
You may have noticed that I question the fitness for purpose of any statistical methods for inference in those heavily confounded, non-linear, feedback filled, and just so damn difficult areas of research where we seem to make only illusory progress - medicine being one with the greatest practical import.
I have asserted that fitness of purpose, as sometimes defined in quality assurance has two perspectives (I worked in that field at one time); roughly
1- Do the right job (use the right tool)
2 - Do the job right (use the tool correctly).
I, and others far more trained than I, argue that statistics fails in chaotic domains in both regards. John. P. Iaonnidis seems to focus on #2. I focus now on #1, having grave doubts about the statistical methods I learned as an applied discipline. I worked my ass off on that also, only to think now that I studied a system that was inapplicable to chaotic domains. Interpretations will of course differ.
You are perfectly correct, people (scientists) build complex models/systems often (making assumptions = a leap of faith) incorrectly applying mathematical tools in situations they are not designed for .. and when models break (start spewing obvious garbage) nobody understands why, because assumption-error was introduced 100s of steps back (and you are lucky if there was only one).
The sad thing is, there is a marked inability to replicate studies in various fields. Medicine, nutrition (I think), Psychology, just about any "soft science."
I use that term "soft science" more and more advisedly, pessimistically. Maybe it is just technological shamanism, but when it comes to medicine and nutrition and so on, it has implications for practice. Big implications.
There are lots of reasons for this failure to replicate but I suspect that the instability and inappropriateness of the measures used is part of it. Moreover, I think it is not just the misapplication of statistics, but the fundamental unsuitability, fitness for purpose, of statistical models for such complicated and complex fields. I suspect this is leading to multiple wild goose chases.
I would like to be wrong, but I can reach no other conclusion. There are a few other dissenting voices on these things, working at a higher pay grade than me of course.
Repeatability of experiments is a big problem.
We have to distinguish social factors vs mathematical factors.
We also have to distinguish between industry and academia.
Social factors: Everybody lives under pressure "publish or die", things are rushed to review at the last moment, the experimental data has often been cherry picked to fit the journal and showcase authors' "stunning" foresight what and how to investigate all in hope of getting a tenure (things do ease a bit once tenure has been obtained). Nobody has time to evaluate/understand "failed" experiments .. Nobody has time to repeat experiment (possibly several times) to ensure it was not a fluke.. Nobody has time to try to replicate other's experiments because they have not been described in details (intentionally) with many unspoken assumptions and details left out. It is actually more time consuming trying to replicate/understand somebody elses junk science than making your own junk science. Notable exceptions are comparative studies, but they are only done for very selective, stending out papers/results.
Mathematical factors: Many experiments start from very different initial conditions, you have to have a strong convergence to arrive to reasonably similar results. The more complex experiment more likely things will diverge (chaos theory). Rarely people take time to repeat experiments even less to understand the dynamics and mathematical reasons for divergence.
Models are simply overly complex to understand now days.
Academia suffers from "non-cosequences" of their BS, they can publish anything, everything works on paper. There are no real-life consequences if things do not work like described, or go wrong.
Industry does not have that luxury, and often industry is not motivated to publish their own research/results because when properly done they are time consuming and expensive, and they offer competetive advantage you do not want to throw away.
So yeah, you want to find real science, you have to go to industrial/private research.
I liked your comment on chaos theory. That seems to be a good way to conceptualize the more complex and complicated areas where we look vainly to find scientific answers.
You may have noticed that I question the fitness for purpose of any statistical methods for inference in those heavily confounded, non-linear, feedback filled, and just so damn difficult areas of research where we seem to make only illusory progress - medicine being one with the greatest practical import.
I have asserted that fitness of purpose, as sometimes defined in quality assurance has two perspectives (I worked in that field at one time); roughly
1- Do the right job (use the right tool)
2 - Do the job right (use the tool correctly).
I, and others far more trained than I, argue that statistics fails in chaotic domains in both regards. John. P. Iaonnidis seems to focus on #2. I focus now on #1, having grave doubts about the statistical methods I learned as an applied discipline. I worked my ass off on that also, only to think now that I studied a system that was inapplicable to chaotic domains. Interpretations will of course differ.