Methods

How much time can Collie save?

Last updated: October 24, 2026

Our figures are estimates, not guarantees. They depend on your topic, your team and how you use Collie. We keep two kinds of numbers apart: measured timings of the steps Collie automates, and estimated human hours built from published studies.

1. What we measured

We ran Collie locally on one workstation (Intel Core Ultra 9, 31 GB RAM, Windows 11, R 4.5.3), 3 runs per step, each on a fresh review. Data: the public SYNERGY dataset of a PTSD-trajectory review (4,469 records retrieved from OpenAlex), plus a second file with 294 reformatted copies of records the original search listed twice, so we knew every true duplicate. The meta-analysis used the 13 BCG vaccine trials distributed with the R package metadat. Times are for the automated part only and include the request to the local server.

Measured on a local install (median, range of 3 runs)
StepTime
Import 4,469 RIS records and detect duplicates0.93 s (0.81–0.98)
Import a second RIS file (294 records) and detect duplicates across 4,7630.43 s (0.43–0.46)
PubMed: live hit count of a two-concept search1.2 s (1.2–1.5)
PubMed: live import of 500 records with duplicate check7.0 s (6.1–7.6)
Relevance ranking of about 5,260 records (offline)0.44 s (0.39–0.50)
Random-effects meta-analysis in R (k = 13) with forest, funnel and leave-one-out plots5.2 s (5.0–5.3); 6.4 s with SVG, PNG, PDF and TIFF at 600 dpi
Risk-of-bias traffic-light and summary figures (SVG, PNG, PDF)4.2 s (4.1–4.7)
PRISMA 2020 flow counts; PRISMA 2020 checklistunder 0.1 s each

Duplicate check: all 294 planted duplicate pairs were flagged in every run. Collie also flagged 7 lower-similarity groups for a person to judge, such as a preprint and its published version. Collie never merges records on its own. This is one dataset with artificial formatting differences; real exports from several databases are messier, so we do not claim a general accuracy figure. The hosted service runs over the internet and queues R jobs, so absolute times there are longer.

2. What manual work costs, according to published studies

Published baselines (per reviewer unless stated)
TaskPublished figureSource
Whole meta-analysismedian 1,110 person-hours (range 216–2,518)Allen & Olkin 1999
Registration to publicationmedian 65.8 weeks (range 6–186)Borah et al. 2017
Title/abstract screeningabout 21 s to 7.7 min per record; 1 min used as a planning valueNussbaumer-Streit et al. 2021; Shemilt et al. 2016
Full-text screening4.3–5 min per article, plus about 4 min to retrieve itNussbaumer-Streit et al. 2021; Shemilt et al. 2016
Data extraction41–65 min per study (one person); 107 min with verification; 172 min dualNussbaumer-Streit et al. 2021
Risk of bias8.8 min (RoB 1) to 28 min per study outcome (RoB 2)Hartling et al. 2009; Minozzi et al. 2020
Single vs dual screeninga single reviewer missed 13% of relevant studiesGartlehner et al. 2020
Text-mining and prioritised screening30–70% less screening workload; 64–92% in active-learning simulations at 95% recallO'Mara-Eves et al. 2015; van de Schoot et al. 2021; Ferdinands et al. 2023
AI (large language model) screeningpooled sensitivity 0.92 (0.81–0.96), specificity 0.94 (0.90–0.97); single studies vary widelyXie et al. 2026; Dennstädt et al. 2024
Deduplicationabout 70 min by hand per dataset; automated methods 2–109 sBorissov et al. 2022

Caveats: Allen & Olkin is 1999 data from one company; Borah measures calendar time, not effort; per-task figures mix measurements and planning assumptions; prioritisation results are simulations that accept missing up to 5% of relevant studies; AI screening results depend on the prompt, the criteria, the dataset and the model version.

3. The model behind the calculator

For each stage we compare manual hours with hours using Collie, in three scenarios: conservative (lowest times and savings), central and optimistic (highest). Time people spend reading, judging and deciding stays in the "with Collie" column.

Typical review: 3,000 records, 2 reviewers, 30 included studies (estimated hours)
ScenarioManualWith CollieSaved
Conservative117 h99 h18 h (16%)
Central225 h158 h67 h (30%)
Optimistic475 h258 h217 h (46%)

Most of the estimated saving comes from title/abstract screening, so it grows with the number of records. The totals cover only the modelled stages, not a whole review.

4. Limits

5. Sources

  1. Allen IE, Olkin I. Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA. 1999;282(7):634–5. doi:10.1001/jama.282.7.634
  2. Borah R, et al. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7:e012545. doi:10.1136/bmjopen-2016-012545
  3. Nussbaumer-Streit B, et al. Resource use during systematic review production varies widely: a scoping review. J Clin Epidemiol. 2021;139:287–96. doi:10.1016/j.jclinepi.2021.05.019
  4. Shemilt I, et al. Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Syst Rev. 2016;5:140. doi:10.1186/s13643-016-0315-4
  5. Hartling L, et al. Risk of bias versus quality assessment of randomised controlled trials. BMJ. 2009;339:b4012. doi:10.1136/bmj.b4012
  6. Minozzi S, et al. The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability. J Clin Epidemiol. 2020;126:37–44. doi:10.1016/j.jclinepi.2020.06.015
  7. Gartlehner G, et al. Single-reviewer abstract screening missed 13 percent of relevant studies. J Clin Epidemiol. 2020;121:20–8. doi:10.1016/j.jclinepi.2020.01.005
  8. O'Mara-Eves A, et al. Using text mining for study identification in systematic reviews. Syst Rev. 2015;4:5. doi:10.1186/2046-4053-4-5
  9. van de Schoot R, et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat Mach Intell. 2021;3:125–33. doi:10.1038/s42256-020-00287-7
  10. Ferdinands G, et al. Performance of active learning models for screening prioritization in systematic reviews. Syst Rev. 2023;12:100. doi:10.1186/s13643-023-02257-7
  11. Xie et al. Performance of large language models in automated medical literature screening: a systematic review and meta-analysis. J Evid Based Med. 2026;19(3):e70166. doi:10.1111/jebm.70166
  12. Dennstädt F, et al. Title and abstract screening for literature reviews using large language models. Syst Rev. 2024;13:158. doi:10.1186/s13643-024-02575-4
  13. Borissov N, et al. Syst Rev. 2022;11:172. doi:10.1186/s13643-022-02045-9
  14. De Bruin J, et al. SYNERGY – Open machine learning dataset on study selection in systematic reviews. DataverseNL; 2023. doi:10.34894/HE6NAQ (CC0 1.0)