The Phases of Living Evidence Synthesis Using AI AI: Living Evidence Synthesis (Version 1)

Background: Living evidence (LE) synthesis refers to the method of continuously updating systematic evidence reviews to incorporate new evidence. It has emerged to address the limitations of the traditional systematic review process, particularly the absence or delays in update publication. The emergence of COVID-19 accelerated the progress in the field of LE synthesis, and currently, the applications of artificial intelligence (AI) in LE synthesis are expanding rapidly. However, in which phases of LE synthesis should AI be used remains an unanswered question. Objective: (ⅰ) To document which phases of LE synthesis where AI is used. (ⅱ) To investigate whether AI improves the efficiency, accuracy, or utility of LE synthesis. Methods: We searched Web of Science, PubMed, the Cochrane Library, Epistemonikos, the Campbell Library, IEEE Xplore, medRxiv, COVID-19 Evidence Network to support Decision-making (COVID-END), and McMaster Health Forum. We used Covidence to facilitate the monthly screening and extraction processes to maintain the LE synthesis process. Studies that used or developed AI/semi-automated tools in the phases of LE synthesis were included. Results: A total of 24 studies were included, including 17 on LE syntheses, with four involving tool development, and seven on living meta-analyses, with three involving tool development. (ⅰ) A total of 34 AI/semi-automated tools were involved, comprising 12 AI tools and 22 semi-automated tools. The most frequently used AI/semi-automated tools were machine learning classifiers (n=5) and the Living Interactive Evidence synthesis platform (n=3). (ⅱ) 20 AI/semi-automated tools were used for the data extraction/collection and risk of bias assessment phase, and only one AI was used for the update publication phase. (ⅲ) Three studies demonstrated the improvement in efficiency achieved based on time, workload, and conflict rate metrics. Nine studies applied AI/semi-automated tools in LE synthesis, obtaining a mean recall rate of 96.24%, and six studies achieved a mean F1 score of 92.17%. Additionally, eight studies reported precision values ranging from 0.2% to 100%. Conclusions: Artificial intelligence and semi-automated tools primarily facilitate data extraction/collection and risk of bias assessment. The use of AI/semi-automated tools in LE synthesis improves efficiency, leading to high accuracy, recall, and F1 scores, while precision varies across tools. Clinical Trial: https://doi.org/10.17605/OSF.IO/4FVDQ