Statistical and media truths
- Science is suspicious of unexpected findings
- News seek attention
- News consumer has hunter's fears
- Interpretation of public data involves risks
Jussi Melkas. The author is Senior Adviser at Statistics Finland. He is a regular contributor to Tieto&trendit magazine. This article was published in Statistics Finland's Tieto&trendit magazine, Vol. 7/2011.
Last spring a group of experts at the International Agency for Research on Cancer, which is part of the WHO, decided to classify the mobile phone among products that may possibly cause brain cancer. The decision aroused considerable attention, and many people interpreted the news as meaning that the radiation caused by mobile phones is dangerous.
This is not quite the truth of the matter. In the classification system used by the WHO, possibly carcinogenic to humans represents a smaller risk than probably or certainly carcinogenic. The Working Group had come to the conclusion that there is some evidence of a link between the mobile phone and brain cancer, but that there is insufficient knowledge of what causes the link. There may, for example, be a third factor involved. At least it cannot be established with certainty that there is no connection between the mobile phone and cancer.
The Working Group did not say anything about how strong the link between the mobile phone and brain cancer is. In all probability the link can be fairly weak, in other words there may be a very slight risk of cancer.
The conclusion gave rise to a proliferation of different headlines around the world. The tabloid-type media made the most of the risk aspect. "Mobile phones can cause cancer!" As a result of the decision people were demanding restrictions to children's use of mobile phones. The media with close connections to the mobile phone industry, on the other hand, questioned the expert group's work: "The industry rejects the killer phone report: no problems have been observed".
Depending on the approach, the risks of the mobile phone were presented in different ways, by referring to other substances on the above mentioned list of possibly carcinogenic agents. If the media wanted to stress the risks of using mobile phones, they mentioned that DDT and exhaust gases belong to the same group. Both of these are hazardous substances but not as carcinogens. If, on the other hand, the intention was to play down the risk, it was pointed out that coffee and talcum powder are also in the same group as mobile phone radiation.
The different ways in which the Working Group's report was handled shows the conflict that exists between the scientific approach to information and its use by the media for news value. Often it is the scientific, moderate, cautious point of view that loses the battle.
Science is suspicious of unexpected findings
In scientific thinking a great deal of attention is paid to the reliability of the methods by which information is produced. New information is always compared to earlier findings. Surprising results are examined critically and always analysed to see whether the changes could be attributed to some other reason than a real change. Among other things, the possible effects of measurement methods and of chance are assessed.
From the point of view of the subject discussed here, scientific and statistical thinking more or less converge. For example, the quality criteria for Finland's official statistics are to a great extent the same as those set for science, emphasising, for example, the reliability and comparability of the data.
Emphasis is also placed on the significance of randomness, both in science and in statistics. Even the best set up research or statistics can produce misleading results due to the effect of chance.
The WHO Working Group's solution is a very apt example of scientific thinking and interpretation of results. The possibility of cancer was not excluded, but neither was it confirmed.
News seek attention
Galtung and Ruge's classic list of the criteria of newsworthiness includes such things as negativity, prominence, unambiguity and unexpectednesss, which are clearly in conflict with the criteria set for science and statistics.
The differences in approach are further highlighted by the logic behind the presentation of news in the electronic media and the tabloids – a logic that stresses striking, simple, eye-catching presentation. One seldom sees a news story about a "small, poorly planned study which has produced evidence that x may cause cancer". The media tend to cut corners: "According to research, x causes cancer."
Editors compete for the public's attention and they are usually reluctant to leave the results of research unpublished simply because an expert has pronounced the research methodology unsatisfactory.
For some time I assisted a certain news team by assessing the reliability of statistical surveys presented to them. My own attitude was in itself a problem: it seemed audacious to say that a study published with great amount of fuss and noise was really not worth the publicity. It was, of course, even more difficult for the news team itself to accept this. "Why are we not publishing a story about this interesting and entertaining research, even though the neighbouring tabloid has it on their front page?" My assistantship only lasted a few months.
News consumer has hunter's fears
The media are always in a hurry, and this haste is not really reconcilable with the schedule of scientific knowledge, which is slow to gather and takes time to confirm. First results are generally based on research that is methodologically inferior, i.e. carried out quickly and cheaply.
Some time ago, preliminary findings suggested that antioxidants may prevent cancer. When the matter was investigated by more thorough (time-consuming, expensive) methods, it turned out that the findings of the earlier studies could no longer be confirmed. In the meantime, the preliminary results had been very visibly publicised, and the food industry, which had noticed a marketing opportunity, had begun to add antioxidants to their products. In the light of new research findings the benefits of the investments seem to have been at the very least questionable. Who is to blame?
The continuously growing offer of factual information often leads to simplifications, which in turn create unfounded fears. For example, the search for causes of rare but dangerous diseases may tempt journalists to incite unnecessary panic. Reading the health news in the tabloids, it is easy to understand what the age-old story of the tree of knowledge of good and evil is about. Stirring up fear and threats spells billions of euros and dollars in the media society.
The basic problem lies in us ourselves. The media consumer's interest in information seems to stem more or less from the same motive in the information society as among the hunter-gatherers roaming the African savannah: to draw his or her attention to threats and to seek quick solutions to them.
Interpretation of public data involves risks
The conflict between the cultures of science and the media is very topical now that the exploitation of statistics and other material has developed into a new sub-genre of journalism. This new genre, called data journalism, has been made possible by the fact that public material is now not only available to officials and civil servants. Data journalism is winning ground from researchers and statisticians by analysing and reporting unrefined and uninterpreted material. In itself this development is positive: at last the information society is beginning to produce information for all the members of society.
Simon Rogers described data journalism on the website of the British daily, The Guardian, in August, as follows: "data journalism is 80 per cent perspiration, 10 per cent great idea and 10 per cent output". The percentages would be fairly apt if applied to the production of statistics. I suspect that in the work culture of journalism, the "perspiration" part of the work in reality plays a lesser role, while the "great idea" part is gaining more importance.
Rogers also says that data journalism is telling a story as well as possible. "... if data journalism is about anything, it's the flexibility to search for new ways of storytelling". Indirectly Rogers is saying that the statisticians and the scientific world have not been very good at finding the latent stories in their material.
Knowing the enthusiasm of journalists, the danger of the future is that more and more often material will be used to produce stories that are not there in the first place. Good stories, whether they are true or not, tend to stay alive, especially on the web, and for this reason data journalism should not be left exclusively to journalists. Researchers and experts should become more active as story tellers, as critics of the stories told and as partners in collaboration with the journalists.
An important source and inspiration for the column has been the Understanding uncertainty blog at http://understandinguncertainty.org/
Päivitetty 2.11.2011