Review Papers Claim by Claim

 Quick Explanation Copied

 Long Explanation

What the Study Reports

Khattabi, El Ayachi, and Biniz assembled a three-phase referencing pipeline — a nine-step NLP preprocessing stage (lowercasing, spelling correction, negation handling, tokenization, stop-word removal, lemmatization, POS tagging), WordNet-based semantic similarity scoring, and paragraph-level reference insertion — and compared six measures on a 1000-article Scopus-derived Kaggle dataset Khattabi et al. 2025, International Journal of Computing. The paper ranks Resnik first, Mihalcea second, Wu-Palmer third (42% accuracy), and Lin, Leacock-Chodorow, and Jiang-Conrath far behind (14–28% accuracy).

Internal Arithmetic Fails

Applying F1 = 2PR/(P+R) to Table 4's own precision and sensitivity columns yields Resnik ≈ Error: 'NoneType' object is not subscriptable% against the reported 64%, and Mihalcea ≈ Error: 'NoneType' object is not subscriptable% against 59%. Either sensitivity is not the recall entering the F1 (a unit mismatch between paragraph- and reference-level scoring) or at least one column is wrong; the manuscript offers no reconciliation. Accuracy also conflicts within the paper (Wu-Palmer: 42% in Table 4 vs 38% in the results text). The full audit is below.

Evaluation Design Cannot Carry the Conclusions

• Circular test set: the test database "contains some paragraphs collected from the reference database documents", so the system is scored on recovering near-duplicates from its own corpus rather than generalizing to new papers Test-set circularity and self-annotation details.
• Self-generated ground truth: labels were produced semi-automatically "based on semantic similarity metrics (Wu-Palmer, Resnik, Mihalcea, etc.)" with manual correction on a subset — the annotations partially derive from the very measures being ranked, a criterion-contamination risk.
• Test-set-tuned thresholds: the 90th-percentile rule fits each cutoff to the observed test-score distribution; no held-out validation, no significance tests, no confidence intervals, and the number of evaluation paragraphs is never stated. Jiang-Conrath outputs a distance (lower = more similar), yet the pipeline applies a uniform above-threshold insertion rule with a 1.5 cutoff, so the rule is either undocumentedly inverted or directionally wrong for that measure.
• Editorial integrity: Beel et al. appears three times ([1], [8], [13]) under one DOI, reference [34] is a co-author's biography parsed as a citation, the Letizia/PHOAKS paragraph repeats verbatim, the heading "SYSTEM ARCHETECURE PROPOSED" is misspelled, and the introduction's 20%/30%/40% impact figures are attributed to survey papers that do not report them.

Context and Falsification

A 57% accuracy ceiling also sits below contemporary scholarly-IR practice: a BioBERT-augmented BM25 hybrid reported a ~3-point P@10 gain on TREC 2019 precision-medicine data with a public GitHub repository Zhang et al. 2023, BMC Bioinformatics (public-code retrieval baseline), and a deep-learning scholarly-document extractor reached 93% F1 with released data Deep-learning scholarly metadata extraction (public code/data). The authors fairly acknowledge WordNet's coverage limits, weak paraphrase handling, threshold sensitivity, and English-only scope Authors, but the remedy is not incremental: contextual embeddings plus an independent, human-verified gold standard are prerequisites. The central ranking would be falsified — and could reverse — by re-running the identical pipeline on a held-out corpus with human-annotated insertions and fixed pre-registered thresholds; materially lower scores than 57/58/64 would refute the efficacy claim.

 BGPT Paper Review

Study Novelty

Combines six pre-existing WordNet similarity measures with standard NLP preprocessing for citation insertion; no new measure, architecture, or dataset contribution — comparable multi-measure comparisons date to the 2000s Mihalcea-era literature.

Scientific Quality

Internally inconsistent statistics (F1 values incompatible with the same table's precision/sensitivity; accuracy conflicts between text and table), a circular test set drawn from the reference corpus, labels semi-generated by the evaluated measures, test-set-tuned thresholds, no significance testing, duplicated references, a co-author biography parsed as reference [34], and verbatim-repeated paragraphs.

Study Generality

The pipeline concept transfers to any English scholarly corpus, but validation covers one small in-house test set, and WordNet reliance plus English-only processing restrict domain and language generality.

Study Usefulness

The negative results (Leacock-Chodorow and Jiang-Conrath unsuitable), the nine-step preprocessing recipe, and the threshold methodology are practically informative, but 57% accuracy and no released code limit direct adoption.

Study Reproducibility

No code or data links, unstated test-set size, thresholds fitted to the test distribution, and a partially manual annotation protocol — independent replication is effectively impossible from the manuscript alone.

Explanatory Depth

Formulas and pipeline steps are described, but the paper never analyzes why Resnik outperforms the other information-content measures, why JCN behaves anomalously, or why F1 exceeds the precision/recall-implied values.