Researchers led by the University of Cambridge have employed Eve, a “Robot Scientist,” to help address the reproducibility - or replicability - crisis which has long been an issue scientists and researchers have hoped to overcome.
Image Credit: Shutterstock.com/ Andrey Suslov
Since the scientific revolution in the 17th century, reproducible results have been a scientific cornerstone that has ushered in wave after wave of progress.
However, in spite of the necessity and value of reproducible results, this is not tested on a large scale, and when reproducibility is tested, it can be extremely difficult to observe reproducibility.
This is what is known as the reproducibility crisis, which impedes the capability to reproduce experiments, yet this failure is a familiar aspect of research. Still, the ongoing issue of reproducibility is one of the largest crises modern science is facing.
The researchers developed a system that combines automated text analysis with Eve to aid the process of finding reproducible research results by analyzing more than 12,000 research papers that cover the biology of breast cancer cells. After managing to reduce the data set down to just 74 papers of significant scientific interest, just 22 of the initial 12,000 were reproducible.
One of the big advantages of using machines to do science is they’re more precise and record details more exactly than a human can.
Professor Ross King, Department of Chemical Engineering and Biotechnology, University of Cambridge
Of course, the seemingly obvious solution to combat the reproducibility crisis would be to recruit more scientists to try to reproduce the work of other scientists. However, there are a series of sociological and career concerns that inhibit a clear path to doing so: it is difficult to find funding, and many authors do not appreciate having their results doubted or challenged.
Attempts have been made to determine exactly what factors are crucial to the reproducibility of scientific results. However, due to the costs, the current funding model, and difficulties that arise when trying to confirm experimental results, it is improbable that human researchers will be able to conduct experiments that prove just a small portion of published results.
King and his team set about addressing these challenges by finding a feasible way to boost the amount of reproduced scientific results; one way is to bring automation to the table.
Thus, to ensure effective automation, the integration of text mining (extracting results from the literature) in combination with AI-based lab automation (to test the reproducibility of the results) could be a viable method.
Having received funding from DARPA, the research team developed an experiment with the application of AI and robotics in order to tackle the reproducibility crisis; by having AI systems read scientific papers and extract knowledge, the team used Eve to try to reproduce the results.
The cancer literature is enormous, but no one ever does the same thing twice, making reproducibility a huge issue.
Professor Ross King, Department of Chemical Engineering and Biotechnology, University of Cambridge
“Given the vast sums of money spent on cancer research, and the sheer number of people affected by cancer worldwide, it’s an area where we urgently need to improve reproducibility,” King continued.
From an initial data set of over 12,000 peer-reviewed papers, the researchers applied automated text mining to identify and pull statements where changes in gene expression as a response to drug treatment in breast cancer were clearly observed.
Subsequently, two human-led teams working with Eve and two breast cancer cell lines made an attempt to reproduce the results of the 74 papers the text mining was able to extract.
Evidence of repeatability with statistical significance was discovered in 43 of the papers, which indicates results are replicable under identical conditions, and there was strong evidence for reproducibility found in 22 papers, meaning the results could be replicated under similar lab conditions by different scientists.
While just under 1/3 of the papers demonstrated robust reproducibility in the automation-assisted experiment, the researchers made clear that this does not undermine the reproducibility or robustness of the remaining papers.
There are lots of reasons why a particular result may not be reproducible in another lab … Cell lines can sometimes change their behaviour in different labs under different conditions, for instance. The most important difference we found was that it matters who does the experiment, because every person is different.
Professor Ross King, Department of Chemical Engineering and Biotechnology, University of Cambridge
This work demonstrates that semi- and fully-automated laboratory techniques could be crucial to overcoming some of the challenges that the reproducibility crisis presents. Furthermore, the team believes that reproducibility should be a standard part of the whole scientific process.
It’s quite shocking how big of an issue reproducibility is in science, and it’s going to need a complete overhaul in the way that a lot of science is done…We think that machines have a key role to play in helping to fix it.
Professor Ross King, Department of Chemical Engineering and Biotechnology, University of Cambridge
References and Further Reading
Collins, S., (2022) ‘Robot scientist’ Eve finds that less than one third of scientific results are reproducible. [online] University of Cambridge. Available at: https://www.cam.ac.uk/research/news/robot-scientist-eve-finds-that-less-than-one-third-of-scientific-results-are-reproducible
Roper, K., et al., (2022) Testing the reproducibility and robustness of the cancer biology literature by robot. Journal of The Royal Society Interface, [online] 19(189). Available at: https://royalsocietypublishing.org/doi/10.1098/rsif.2021.0821
Disclaimer: The views expressed here are those of the author expressed in their private capacity and do not necessarily represent the views of AZoM.com Limited T/A AZoNetwork the owner and operator of this website. This disclaimer forms part of the Terms and conditions of use of this website.