The researchers used Bridges-2 to train AI tools to analyze research reports and identify missing steps in the design and conduct of clinical trials. The goal is to create an open-source AI tool that can help authors and journals improve the quality of reporting in clinical trials.
Randomized, controlled trials are essential for evaluating the safety and effectiveness of new treatments. By randomly assigning patients to different groups, researchers can ensure that the results are not biased. It is also important for scientists to clearly define their goals and criteria for success before conducting the trial, to avoid bias in the interpretation of results.
However, many research reports do not provide enough detail about the design and conduct of clinical trials, making it difficult for other researchers to evaluate the quality of the study. The AI tool developed by the University of Illinois Urbana-Champaign team aims to address this issue by automatically identifying missing steps in clinical trial reports.
According to Halil Kilicoglu, an associate professor of information sciences at the University of Illinois Urbana-Champaign, improving the reporting of clinical trials is crucial for ensuring the validity of medical research. “Clinical trials are considered the best type of evidence for clinical care. If a drug is going to be used for a disease, it needs to be shown that it’s safe and effective,” says Kilicoglu.
The AI tool developed by the researchers has the potential to revolutionize the way clinical trials are conducted and reported. By using artificial intelligence to identify missing steps in research reports, authors and journals can improve the quality and transparency of clinical trial reporting, ultimately leading to better patient care and more reliable medical research.
Overall, the use of AI tools in analyzing and improving the reporting of clinical trials has the potential to enhance the credibility and validity of medical research. By ensuring that research reports provide accurate and comprehensive information about the design and conduct of clinical trials, AI can help to advance the field of medicine and improve patient outcomes.
The National Science Foundation’s Advanced Cyberinfrastructure for Computational Science and Engineering (ACCESS) program provides researchers with access to high-performance computing resources to advance their studies. Through this program, the Illinois team of scientists obtained valuable time on Bridges-2, a supercomputer system managed by the Pittsburgh Supercomputing Center (PSC). PSC is a key member of the ACCESS program, providing researchers with the computational power and support they need to conduct cutting-edge research.
Utilizing AI for Clinical Trial Analysis
The Illinois team focused their research on analyzing 200 articles describing clinical trials from the medical literature between 2011 and 2022. To ensure proper evaluation of these articles, the team turned to the CONSORT 2010 Statement and the SPIRIT 2013 Statement, which provide guidelines for reporting clinical trial results. The researchers aimed to develop artificial intelligence (AI) models that could assess the adherence of scientific papers to these guidelines.
By leveraging Bridges-2’s high-performance computing capabilities, including powerful graphics processing units (GPUs) and specialized AI training tools, the Illinois team explored the use of natural language processing (NLP) algorithms to evaluate the articles. The team trained the AI models using a portion of the articles as labeled training data, allowing the models to learn and improve their performance over time.
Lead researcher Dr. Halil Kilicoglu highlighted the importance of Bridges-2 in supporting the development of deep learning models for the project. The system’s GPU resources and pre-installed software streamlined the AI training process, making it easier for Kilicoglu’s team to conduct their research effectively.
Evaluating AI Performance
The researchers evaluated the AI models’ performance using the F₁ score, a metric that assesses the models’ ability to identify missing checklist items in the articles while minimizing false positives. Initial results showed promising F₁ scores of 0.742 at the sentence level and 0.865 at the article level, indicating the AI’s effectiveness in evaluating SPIRIT/CONSORT adherence.
Despite these positive results, the research team aims to further enhance the AI models by incorporating more data and refining the training process. One approach involves using distillation, where a larger AI model trained on a supercomputer teaches a smaller AI model to identify adherence to reporting guidelines. This strategy aims to improve the AI’s accuracy and efficiency in evaluating scientific papers.
Future Implications and Goals
Looking ahead, Kilicoglu and his team plan to make these AI tools accessible to journals and scientists at no cost. By providing researchers and publishers with automated tools for assessing reporting guidelines adherence, the team hopes to streamline the article review process and enhance the quality of scientific publications. These AI tools can serve as valuable resources for researchers, enabling them to identify and correct checklist omissions in their draft papers.
By leveraging the computational resources and support provided by PSC through the ACCESS program, the Illinois team’s research showcases the potential of AI-driven analysis in improving the quality and transparency of clinical trial reporting. Through continued advancements in AI technology and collaborative efforts with leading research institutions, the future of scientific publication evaluation looks promising.
