Dr. Stephen Bustin: I was very grateful to hear the comment this morning--or, I was interested to hear the comment, Cindy Brenner's [sp] comment about technology. And I think the problem with QPCR is that everyone thinks it's all sussed. And in reality, it's nothing like that at all.

I would like to take you through an example of the problems [unintelligible] associated with QPCR. I'm sure you all, the same as me, you read all these exciting biomarkers that tend to be found for all kinds of different conditions. And usually, what you first read is some kind of hyped article in something like the Scientist, where you have a breakthrough.

And you get all excited. Particularly, it's in your--vaguely in your area of expertise. You then go and look at the paper and try and see what the people have done and whether the hype that has been printed in the Scientist, for example, in this case, matches reality.

And so, in this case, what the authors were claiming was that there was a truncation associated--protein associated with metastasis and then used QPCR to show that this truncated RNA and protein for that matter is expressed and differentially between the cells that are metastatic and those that are not metastatic.

In terms of the QPCR results, these are the data. And they were considering a two- to threefold difference in mRNA levels as significant.

So, obviously, when you see this kind of result, you immediately wonder how relevant this really is. And so, we want to be certain that they have done the experiments in an appropriate way.

So, the first thing you obviously do is you look at the materials and methods to see what they have done. And it's RNA based. So, there's four essential things that you look for. You look for any evidence that they have--the authors have analyzed the integrity of their RNA.

You want to be certain that the assay is detecting what it's supposed to be detecting. You want to have some information about PCR efficiency. And you want to look at how they normalized their expression data.

So, if you look at the RNA quality, then you may find the same thing. When you look at papers, typically, there's no information at all provided about RNA quality. This doesn't mean they haven't done the RNA quality analyses. But, they certainly haven't reported anything on this.

Primer specificity, they have two prime--a set of primers that binds upstream and downstream from this deletive region here. So, the upstream primer is specific for the deletion variant because half of it binds here, and half of it binds there and excludes this deleted bit here.

I was just looking at this. The first thing you notice is that it is an amplification that amplifies within a single axon. So, the first thing you'd want to know is make sure the DNA's treated there, the samples.

So, you go and look at the materials and methods to see what they've done in terms of specificity. And the first problem arises when you actually scan the primers, put them into something called primer blast. It's immediately clear that these primers are not terribly specific because you can see that there's at least three products here that are very small and with a significant sequence similarity with the target gene. So, this is a problem with the primer specificity here.

The problem is that these--there's a nice review published a few years ago. If you have primers that bind in close proximity on genomic DNA or RNA for that matter, we get all kinds of off-target side effects that are not limited to the complementarity of the three-prime ends of the primers but that result in primer, dimer, and other types of nonspecific amplification that you will detect if using [unintelligible] green and that will interfere with your assay if you're using [unintelligible] based assays.

So, this is the primer sequence that they published. And when I did my primer blast search, there was a conundrum here because, as you can see, there seems to be a mismatch between the primer blast and the actual sequence that amplify. And that is here in this region here. Okay? But, this is easily resolved because it turns out that these primers actually also amplify and target this region here. So, far from being specific for the splice variant, these primers actually will also amplify the full-length RNA, DNA I suppose, and it's converted into cDNA. So, these are not specific primers.

Now, how you would miss this when you do a proper quality analysis of your assay is not clear to me. But, clearly, they can't have done this kind of analysis.

So, how efficient is the PCR? Well, there's no information at all provided about this. However, you can come to some conclusions by analyzing their primers and probe and amplicon [sp]. These are structures of the forward primer, of the reverse primer, and the joint primer.

And you can see very clearly, there's significant potential here for cross-dimerization between the forward primer. There's some for the reverse primer. And there's possibly some interaction of reverse and forward primer, suggesting that this assay is not going to be the most efficient of PCR assays.

If you look at the amplicon, there are real problems here, too. You've got a secondary structure at the forward and at the reverse primer-binding site, which is always a sign of a poor assay.

Here's an example of my own. On the left, we have an amplicon, where upstream and downstream primer-binding sites are clear. Here, we have a similar situation as we had with this previous assay, where both are--have secondary structure.

If you look at the relative efficiency of these two assays, you can see that one is in the 100 percent region, 96 percent, and one--it's come off the screen--is about 82 percent. So, there's a clear difference based on the amplicon structure.

The third thing they did was to quantify their target. They didn't actually use the truncated version. They used the full-length version of the RNA. And that's what the standards look like. And again, you can see very clearly that there is significant secondary structure associated with this. So, this would not be a very good assay.

And again, these are the primers they used for the actual standard curves. And again, you can see very clearly, it's not very good.

So, are they normalized, the next question? Well, there are two things they use, a delta-delta-CQ and use 18S RNA. Now, I think most of you know that you can't do a delta-delta-CQ if you don't know the relative efficiencies of amplification of your target gene and your reference gene.

They have no idea of their relative efficiencies, or at least they don't tell us about it. So, we can't decide whether they of equal--of similar or if they are of equal efficiency.

18S RNA, again, something--this is the target they use for their 18S amplification. Firstly, you can see it's much bigger. And secondly, you see there's no secondary structure here. So, I would guess that the efficiency of this template [sp] is going to be quite different to their target.

And of course, as far as 18S is concerned, it's now seven years ago that this paper was published, and it shows very clearly--and if you want to read about it, read the paper why 18S is not a good normalizer for gene expression.

So, does it matter if someone's assay isn't very good? Well, here's an example of a good assay characterized by the parallel slopes of the amplification plots. And as you can see, it doesn't matter where you draw your threshold line. You get more or less the same result within 20 or 30 percent. It doesn't matter [unintelligible] over wide dynamic range.

So, a good assay is characterized by a very robust assay that gives you reproducible data. A poor assay would be something like this. And you can see there's not a very large dynamic range. And there's a massive difference in the calculated copy numbers of your targets.

And this would typically be an assay where the primers interact or where there's secondary structure within the amplicon.

Now--and does it matter? Well, it does matter because this is a famous--infamous paper now, which used real-time PCR to suggest that there might be a link between gut pathology and children with developmental disorder and measles virus. And these are the sort of results they produced in this paper.

In fact, they give you no results at all. They give you a summary table. But, these are the data they used. For example, these are children with developmental disorders. But, they amplify for measles virus gene. And so are these.

But, clearly, those of you that are familiar with QPCR will understand that this is not really amplification. This is simply an artifact and nonspecific drifting of the fluorescence, clearly not amplification. Yet, they scored both of these as positives.

Just as another example of the sort of things that people do, these are the result they got for their various samples. If they had degraded RNA, they got roughly a CQ of 30. For their positives that were fresh or frozen, they got the same CQ. And [unintelligible] also got the same CQ. It didn't matter what condition the RNA was in. They always got the same result.

And so, ultimately, this meant they were looking at contamination. They were looking at DNA contamination.

So, if you look at the literature, there's a real problem. There's in fact two problems. The first problem is that people don't report what they're doing. And the second problem is that, very often, they report what they're doing, and then you find there's a real problem with what they've reported.

For example, the first paper I showed you, where it's clear from their primers that they cannot be observing specific amplification of a truncated primer--of a truncated amplicon.

So, what we decided to do was to actually look at the literature and see what--and to what--what the problem was and to what extent there was a problem. And together with my colleague Joe Vanderzample [sp] from University of Ghent, we looked at 82 publications with impact facts from 1.8 to 32, from 2009 to 2011. And we looked at 20 papers per publication. Okay? So, we looked at a total of roughly 1,600 papers published in 2011, some going back to 2010 or '09.

And we asked a series of questions and then farmed this out, obviously, to lots of people to score these parameters that we were interested in and that are of interest to anyone who is interested in RTQPCR.

And so, here's an example of a single paper scored. As you can see that the majority of criteria we're interested in, this paper actually scores. So, it fulfils 80 percent of the criteria that we considered to be essential in order to be able to assess the quality of a real-time PCR paper. Okay? We then took this and looked at the 20 papers, in this case Developmental Biology. And this is the one paper I just showed you. We can see the majority of papers are down here with a median of three.

So, this means that we consider 16 parameters to be essential for anyone to be able to judge the quality of a paper. But, the median information you get is three of those. Okay? So, we can't really make up your mind how good the data people are producing are.

If you look at journals in terms of the impact factor, it is clear that the higher the impact factor, the less information you get, until you end up with something like Nature or Cell, where very often there's no information at all. They use QPCR data and didn't even refer to QPCR in the materials and methods. This is despite the fact that there's plenty of room to do this in the online supplements.

Now, I briefly talked about RNA quality. And I want--I don't want to talk about that today because--well, I could talk to you about it all day. But, you could argue that they have done the RNA quality assessments. They just haven't reported them.

Normalization's completely different. You either get it right, or you don't get it right. And so, I would like to concentrate on normalization.

Now, we showed a long time ago that you can really show whatever you'd like, depending on which reference gene you use. And this was--oh, it's 10 years ago now.

So, we asked again the over 20 papers from Developmental Biology. And the first thing we wanted to know was how many reference genes do people use because the recommendation since 2002 has been to use at least two or three. And secondly, have these reference genes been validated? And as you can see, one look tells you that it's a pretty dark state of affairs here, that most people use one reference gene. And most people don't validate that reference gene.

Now, if you look at the situation of the--in publications and impact factors of less than five, then you can see that 18 percent of these journals don't have a single paper in them that look at--that use more than one reference gene. And only 6 percent of journals have more than six--in fact, it's seven papers. So, the majority of papers in the published literature do not normalize their QPCR data appropriately.

And as you go up the scale, it gets significantly worse until you end up with something like Science, which in 2010, not a single paper used more than one reference gene using QPCR. Okay? So, roughly two thirds of journals and with an impact factor of greater than 10 don't have a single paper within the 20 papers we analyzed that used more than one reference gene. The situation is very similar for the validated reference genes.

Here, at the lower end of the scale, 30 percent of journals don't have a single paper that has a validated reference gene. By that, I mean they've actually checked to see whether their reference gene, the single reference gene they're usually using, is in any way altered by the experimental conditions that they're subjecting their cells to, or in case they're using biopsies, that there isn't any difference in the expression of that target reference gene amongst their various samples.

And you won't be surprised to see that, again, as you go up towards the high-impact-factor journals, it gets worse. So, essentially, you can end up with a paper--or, you do end up with a paper published in somewhere like Nature or Cell that uses QPCR either to try and corroborate [unintelligible] data or in fact come up with a statement based on the QPCR data that either doesn't mention QPCR at all in its materials and methods. We have absolutely no idea how they did the experiment.

Or, if they do mention QPCR, they actually do the experiment inappropriately. So, the data really are not genuine, or they're not believable.

Now, it doesn't matter if you're looking at 100-fold or 500-fold or even 50-fold differences in mRNA levels. But, most people are look--trying to convince you, as the first paper I showed you does, that it's twofold, threefold levels of mRNA differences that are biologically significant.

The point I'm making is that, for most papers that you look at, you have no idea whether the statements people make are valid. And that's for the vast majority. And for--very often, when they do make a statement that's based on some materials and method you read, it is inappropriate.

And there's this very nice negative correlation between impact factor and what I would call quality of the technical information.

Now, we've known about this for a very long time. And so, in 2009, we published a paper that attempts to list a series of important steps that we think are important for assay design, assay reporting, 80-odd individual parameters. And here's an example of the parameters that we consider essential for--to provide information on nucleogast [sp] extraction, listed as either essential or desirable.

So, you can see it's fairly comprehensive. So, in addition to being something that is useful for a reviewer to note when he reviews the paper and for the reader to be reassured when he reads the results and tries and applies them to his own research, these are, of course, also very useful if you're trying to design your own assay because, as you can see from the very first example I showed you at the beginning of my talk, a lot of the assays that people use are poorly designed and do not actually amplify what people think they amplify.

The paper has really--it's now gone up to number nine of the most cited articles in clinical chemistry. So, people have begun to take notice of the problem. And in terms of citations, we had 530. So, it is clearly--it has clearly hit a spot.

And BMC journals are implementing the guidelines or a pracy [sp] version. And they are recommended. And we're trying to get people that are not normally interested in QPCR also to understand that there could be a problem with the data people are reporting.

In particular, the companies such as Agilent have been very supportive about this. And in fact, all the major suppliers of reagents and instruments have in one way or another, Mikey [sp] compliance and Mikey interests that seem to--they try and educate their customers in best practice. Okay? The ultimate goal is a Mikey app. And there are two Mikey apps. They're freely available, either for the iPhone or for the Android system. And really, what this allows you to do is to either--when you write your own paper or you're reviewing your paper, just go to every step of the guidelines and see how the paper you're either writing or reviewing scores. Okay? Now, because there's such a problem with QPCR assay design and data analysis and understanding how assays should be optimized, I have started a new venture, which is going to be a serious of iBooks published on iTunes that are going to be available as of this month.

And these really take you through in this case assay design step by step and because, unlike--people think it's very straightforward. But, as you can see from the very first example I gave you, even a high-impact-factor journal publishing high-impact research can produce data that a very simple analysis would tell the reviewer or the reader that there's a real flaw with those data.

And as I say, unfortunately, the higher the impact factor, the more problematic the data are.

So, thank you very much. I'd like to thank the people that helped with the survey. And I hope I haven't depressed you too much being between these fantastic technical talks. But, I think listening to Cindy Brenner's quote this morning was rather interesting because it does tell you that there's no point having all these great ideas if we're still failing to get the very, very basics right and a technique that is as simple as QPCR. Thank you very much.