The DK theory (for short since I’ll be referencing it a lot..) is based in an observable “fact” that to most seems intuitive, and a moral lesson to avoid overconfidence. The DK study took students and had them do some tests. Students were asked to predict their results and those results were broken into 4 groups based on how they scored- lowest, highest, and two groups in the “middle” on scores. The actual score was compared to the predicted score and the differences between the two were graphed out to get the results and conclusion. These are critical points to both a common misunderstanding of the DK effect and some things we will discuss in a bit. So to start, a common misconception is the simplification that the theory implies people who know less overestimate. Really the data generally shows most people will overestimate but the variance will tend to be less among those who score high.
So as an example- a person who scores a 90% on a creative writing test may estimate they would get 92% or 95% whereas a person who scores a 30% may have estimated they’d get a 50% or 60%. So the person with the higher score might be off a few or ten percent whereas the person with the lower score is more likely to be off by a wide margin. What is important about this is cognition. If someone asks you a question like: “what is the Capitol of Australia?” Many might answer Sydney or Melbourne or some well known large city they’ve heard a lot- the same as many assume NewYork City is the Capitol of New York State. If you are SURE you don’t know- like I ask questions such as exact atomic weights of atoms or something you have NO idea- you’re more likely to predict you’ll do poorly.
So it might be oversimplification as commonly presented that the DK effect is “not knowing enough to know what you don’t know,” more accurately it predicts that the more you know about something, the more accurate your predictions on knowing it will be. Now is where some serious doubts on the study creep in. Your feelings on how well you will do at something can easily change. If you are in a poor mood and I ask how you think you’ll do on a test you might say “I’m going to screw it up like I screw up everything else…” if I ask you when you’re having a great day or feeling motivated you might answer very differently. So there is this problem in general when dealing with self estimates. Our moods and confidence can be dramatically impacted by circumstances of the time the question is asked.
The next major hurdle we hit is that the results of the study can be, and have been, replicated fairly faithfully using random generated data. So if you have a computer create random numbers for what a score will be, then have a computer assign random scores, you can replicate the same pattern on a graph. This could suggest that the results themselves are more of a coincidence in numbers or an effect of how the data is measured. What do I mean? Well- bias implies that we think in a way that regardless of reality will lead us to weight one thing more heavily. If I have a red card and a black card and ask 100 people to chose a card, the results probably won’t be 50/50 but they shouldn’t be too far off if there is no significant bias. If I have a machine randomly select one of the cards 100 times, my results should fairly consistently compare to my human tests. If they do not, or is likely the humans had some sort of bias in their selection-
Assuming the machine is designed without bias.
So the fact that we can reproduce very similar results using random data with reasonable consistency does imply that the results could be no better than random data or require a larger pool and more data to make a conclusion. In dealing with how humans think we need a lot of data because in addition to mood and such, bias can be culture based and may not exist or manifest differently across groups.
So how do I explain that simply? Say that you take your temperature. It is 102f degrees. Do you then assume that your body and everyone else has a temperature of 102 degrees? What if you take it the next day or week and it is 97f degrees… that’s a critical component missing from the DK data- a proper study would require observation of a group over time and comparative data. Do the same people who scored low and vastly overestimated their scores also overestimate when they score high?
The DK effect was first reported as such in the 1990’s, and since then various studies have found conflicting data using computer models and human test subjects. The D of DK themselves has said the popular interpretation of the study is incorrect and that the intended conclusion was more about your own personal bias than a tool to measure others. In the words of some of those who have consisted conflicting studies- when the original DK findings were revealed they just didn’t question it. It was intuitive and seemed the logical conclusion. The data was there. It was only over time and as the effect rose to prominence that more people decided to investigate and reproduce it. More modern studies with additional controls have found something closer to under 10% of people can be said to grossly overestimate things they lack a basic knowledge of while most people will overestimate their performance regardless. This is similar but distinct from the common interpretation.
Do also keep in mind that the subject matter may, and very likely does, play an effect too. For example- the tests in the DK study were on grammar, logical reasoning, humor. Most people believe their logic is correct- that’s a given- otherwise we would all mostly defer to whatever we were told even when it contradicted what we believed. Grammar is a more technical subject but can be a bit subjective, and humor is highly subjective. Would you get similar results if the tests were on hard sciences or technical fields? Would an average person be as prone or prone at all to overestimate their ability to do trigonometry or understand particle physics or a knowledge of COBOL programming language?
Ironically, many of us are bias to believe the DK effect. It supports what we think we see around us- “idiots who don’t know anything acting like experts on a subject.” It is more often applied to others than to the self, so it can support the ego or a belief that one has the superior position in a matter. It has the much loved appeal of “hidden truths and lies you are told..” where we mostly all tend to want to have some “secret knowledge” even if it isn’t so secret or so correct- so long as it satisfies our need to feel “elite” or “informed” or like we know the truth even if others don’t. One can hopefully see the irony in supporting a position that knowledge of the DK effect elevates them over those who don’t have that knowledge. So overall the idea that idiots are too dumb to know they’re idiots is an appealing one most of us would want to believe- again- irony- how many people genuinely consider themselves to be idiots outside those with self esteem issues?
So of course inherently the gospel of idiots being too dumb to know is going to apply only to idiots, whom most of us will believe are people who are not us, and thusly will support our idea that we are correct when we encounter such an idiot who doesn’t know how dumb they are.
Of course- the DK effect hasn’t been formally “disproven” or anything- we can say it just hasn’t been completely “proven” and is often misunderstood or itself a victim of bias in the popular sentiment. We can also say that even conflicting studies have largely shown that there are SOME percentage of people who really do fit the overall conclusion of the DK study- so to be clear I am. It saying that there is no validity to the DK effect conclusion- just outlining where there is room for debate and that the common interpretation is usually slightly off base.
Assuming the machine is designed without bias.
So how do I explain that simply? Say that you take your temperature. It is 102f degrees. Do you then assume that your body and everyone else has a temperature of 102 degrees? What if you take it the next day or week and it is 97f degrees… that’s a critical component missing from the DK data- a proper study would require observation of a group over time and comparative data. Do the same people who scored low and vastly overestimated their scores also overestimate when they score high?
Of course- the DK effect hasn’t been formally “disproven” or anything- we can say it just hasn’t been completely “proven” and is often misunderstood or itself a victim of bias in the popular sentiment. We can also say that even conflicting studies have largely shown that there are SOME percentage of people who really do fit the overall conclusion of the DK study- so to be clear I am. It saying that there is no validity to the DK effect conclusion- just outlining where there is room for debate and that the common interpretation is usually slightly off base.