Science: definition – The intellectual and practical activity encompassing the systematic study of the structure and behaviour of the physical and natural world through observation and experiment.
Cod: definition – Not authentic; fake.
Science is underpinned by two fundamental rules: If you can’t measure it, you can’t do science on it and garbage in = garbage out.
The first should be a bit obvious really, science is all about quantifying stuff, finding correlations, measuring change and so on, which obviously requires some sort of measurement, the data collected must be objective and not biased and so on.
The second rule is the killer though, because it means this measurement has to be meaningful, samples have to be representative and so on because it doesn’t matter how complex your analysis or how much data you pour into it if that data is badly collected, the result you get from all your work will be meaningless.
It’s also important to remember that even if you do actually measure things and go to great lengths to ensure the data you have is properly collected, you do have to ensure there isn’t some other factor which is skewing the effect you’re trying to observe. Any decent study will try hard to at least acknowledge such difficulties and any that don’t should be treated with utmost caution.
When it comes to studying prohibited drugs like cannabis and understanding their effect on people we have a blindingly obvious problem in that because it’s all illegal, you can’t employ any form of direct measurement or use any of the normal, accepted sampling methods for collecting your data. As a result, a lot of the supposedly scientific studies we see are frankly little more than cod science.
Data collection is something of a fine art and isn’t something I could give a presentation about, but take a look at this page for a gentle introduction to why it has to be done right
Consequences from improperly collected data include
- inability to answer research questions accurately
- inability to repeat and validate the study
- distorted findings resulting in wasted resources
- misleading other researchers to pursue fruitless avenues of investigation
- compromising decisions for public policy
- causing harm to human participants and animal subjects
So, how do we go about doing this with illegal drugs, when we can’t use any of the standard sampling techniques which we would employ for studying any legal commodity? The simple answer is we can’t and other ways have to be found to arrive at data and more often than not this stretches credibility beyond breaking point. This is especially true when it comes to the sometimes lurid claims about cannabis and mental health, particularly those emanating from Kings College in London.
Now let’s be clear, claims made by respected scientists are very alarming and need to be taken seriously. The essential claim is that cannabis containing high levels of THC and low levels of CBD (the two major active constituents of cannabis) may detrimentally affect the mental health of the consumer.
Studies released from these scientists have received a lot of publicity in the media and have been used by the government to justify its repressive campaign of prohibition. Over recent months and years the debate has shifted somewhat towards using these studies to support the call for law reform, because if they are true than it becomes important to regulated the strains of cannabis on sale to ensure healthy levels of CBD in the product sold to consumers. Indeed, this is probably the most powerful argument the law reform campaign has ever had.
No-one, however, seems to have taken a step back and cast a critical eye over the standard of some of this research and the truth many of the claims are are built on truly shoddy data. By way of illustration we’ll look at a couple of studies which are central to the cannabis and mental health debate, yet are arguably glaring examples of cod science.
The claim that most cannabis being sold on the streets today is high potency (high THC/low CBD) varieties is central to the mental health claims. Given cannabis is termed a “controlled drug” you might be forgiven for assuming we know all about this, but we don’t. To date, the only national study into what is actually being sold was carried out in 2008 for the Home Office in which samples of cannabis were collected and sent for analysis. It’s almost a master class in how not to collect data. The actual “sampling method” was carried out by asking police around the country to provide samples from cannabis they had seized while enforcing the law. The report describes the “methodology”:
For operational reasons some forces chose to send in material from only one Borough Command Unit or from one of several forces collection points. Some forces experienced internal logistics problems; others were very enthusiastic and sent in everything received during the trial period.
This quote is all there is in the study by way of highlighting possible problems with the data, there is no discussion or acknowledgement as to the potential complications it could have introduced and no discussion as to the limitations of the study.
On the basis of that “sampling technique” some complex analysis came to the conclusion we apparently now take as fact, that high potency herbal cannabis dominates the trade in the UK. Yes, seriously, that is the standard of data collection being presented as authoritative research. It’s pathetic.
Her’s another one:
Proportion of patients in south London with first-episode psychosis attributable to use of high potency cannabis: a case-control study by Marta Di Forti et al of Kings College.
We analysed detailed data for history of cannabis use, aiming to: compare the patterns and types of cannabis used between patients with first-episode psychosis and a population control sample; use the data for pattern of cannabis use to develop a cannabis exposure measure that accurately estimates the risk of psychotic disorders; and calculate the proportion of cases of psychosis in our study area attributable to use of cannabis, particularly high-potency cannabis, if we assumed causality.
The conclusion it drew was
The ready availability of high potency cannabis in south London might have resulted in a greater proportion of first onset psychosis cases being attributed to cannabis use than in previous studies.
The study collected the following data
type of cannabis used—ie, the type most used by the subject (none scores 0, low potency [hash-type] scores 1, high potency [skunk-type] scores 2
They collected their data by asking a series of questions to people presenting with first-episode psychosis at the Maudsley hospital in South London and a control sample of people recruited from the street. They asked about past cannabis use including
we used a seven-item composite cannabis exposure measure derived from the lifetime frequency of use and the most used type (none scores 0, hash less than once per week every week scores 1, hash at weekends scores 2, hash every day scores 3, skunk less than once per week scores 4, skunk at weekends scores 5, skunk every day scores 6) to investigate which patterns of use conferred the greatest risk.
The study actually calls this high THC cannabis “skunk”, The use of this word – which was employed in the collection of the data – is extremely suspect. The term “skunk” has been used extensively in the media in connection with the “reefer madness V2.0” claims promoted by tabloids like The Daily Mail, it can easily be argued that people asked at random will be less likely to admit to using something ‘known to make you mad’, whilst those suffering psychosis may be eager to put the blame on the dreaded “skunk”.
This paper is all on free access so please do read it all. It makes some powerful recommendations which law reform campaigns are keen to use to justify the need to regulate the cannabis trade in terms of the THC/CBD content, although the authors simply warn of the dangers. It presents the study in a typically well referenced way, featuring the usual complex analysis and looks very authoritative. All built on the data collected as outlined above. Seriously, you couldn’t make this up.
Now we should be willing to accept this conclusion could be correct, and it seems self evidently true that we need to get to the bottom of all this, as Dr Di Forti states in a Kings College news release about the study:
When a GP or psychiatrist asks if a patient uses cannabis it’s not helpful; it’s like asking whether someone drinks. As with alcohol, the relevant questions are how often and what type of cannabis.
But can a study based on such simplistic data as this really provide the sort of evidence we need, let alone the firm conclusions they seem to come to? If you believe in a theory and go out to find evidence to support that theory, you’ll probably be successful and frankly that’s what seems to have happened here. Even the use of the word “Skunk”, given the extensive media stories in such gutter press publications as the Daily Mail is likely to skew the answers. It’s so full of holes as to be a joke.
Cannabis, or cannabis under prohibition?
The reality is we have very little solid science to base any of the claims of harm from cannabis on because the data used is often of such low quality. But even if we did have properly structured surveys, we would still only be looking at the effects of cannabis under a prohibition regime and that adds another layer of uncertainty. Under a regime of prohibition the police try as hard as they can to make what is sold as uncertain as possible, it’s all pure harm maximisation. There could be something else going caused by prohibition on which hasn’t been considered, a “confounding factor” as scientist like to call it.
One example: Plants grown in high intensity plantations, with hundreds of plants in a confined space as much of the so-called “skunk” is grown, requires heavy use of organo phosphate (OP) pesticides. Now we have absolutely zero understanding of how much OP pesticides residue is in cannabis sold here. It could be that OP pesticides are actually a real cause of psychosis and it’s that, not the high THC/low CBD factor causing the claimed correlation between the use of high potency cannabis and psychosis. It could be, no-one knows and no-one is looking.
As I wrote above, it doesn’t matter how many studies you do, how much dodgy data you collect or how much complex statistical analysis you put it all through, if the data is garbage, your result will be garbage. We need something better, we need studies carried out where cannabis is legal, where we can be certain people are using specific strains of known potency and where we can go out and collect firm data.
This used to be impossible, now it’s simple. Just go to Colorado or Washington state and do it there where it’s been legal for some time. It’s strange, indeed almost suspicious, that no-one over here seems to be suggesting this be done. Of course in Colorado they are monitoring the situation very carefully and so far they don’t seem to be seeing anything close to a public health emergency (Denver post):
Poison center calls about marijuana exposure have been on the decline since 2015 and marijuana-related emergency room visits dropped between 2014 and 2015, according to the report
So enough of this stupid situation, if it really is the case that we need more studies into cannabis and mental illness, don’t do it here, do it in Colorado with real data. Enough of this cod science, or would that spoil a good Daily Mail story and undermine the drugs policy?