Evidence Paper Review

Brief critique of ARPD dataset paper

The authors present ARPD a 2011‑paper Arabic academic corpus across seven domains and evaluate clustering and classification methods; they report SVM yields ~98% classification accuracy while clustering scores (Silhouette) remain low because of high dimensionality — full dataset and files released on Zenodo ARPD dataset summary and availability

Comprehensive review of Open source Arabic research paper dataset for natural language processing (ARPD)

What the paper does (facts taken verbatim)

• Introduces ARPD: an open academic Arabic corpus with 2011 documents spanning seven classes: Art, Law, Business, Religion, Agriculture, Arabic, Education; dataset released on Zenodo in PDF, text, CSV formats ARPD dataset release and classes
• Data collection and cleaning pipeline: journals selection, PDF to text (PyPDF2), manual QC and Unicode fixes, multiple preprocessing versions (Preprocessing1..3), TF-IDF vectorization Methods preprocessing and TF-IDF
• Experiments: clustering (Kmeans, mini-batch Kmeans, DBSCAN, PSO, GWO) and classification (SVM, RF, DT, NB) with evaluations Silhouette, ARI, Davies-Bouldin, accuracy/precision/recall/F1; train/test split 70/30 Experiments and evaluation metrics

Key quantitative results (reported by authors)

• Classification: SVM accuracy ≈98% across preprocessing scenarios; RF similarly high (97–98%); per-class precision often >83% and many F1 scores ~97–99% for several classes Classification performance summary
• Clustering: Silhouette coefficients are low (maximum ~0.1 or less), ARI up to ~0.49 for some Kmeans settings, Davies-Bouldin best values around 1.5–2 for some methods; bio-inspired GWO gave best Davies-Bouldin in some runs while PSO often worse on DB index Clustering performance summary

Strengths

• Clear gap addressed: academic Arabic paper corpora are scarce; ARPD adds multi-domain academic documents (2011 papers) and multiple convenient file formats which helps reproducible benchmarking Motivation and novelty
• Comprehensive evaluation across preprocessing pipelines and algorithms gives immediate baseline numbers for future work Comprehensive evaluation
• Public Zenodo deposit: eases reuse and verification by other researchers Data availability Zenodo

Limitations and blindspots (critical)

• Selection bias from open access journals and regional focus (major sources affiliated with particular universities) — limits representativeness across Arabic scholarly publishing; authors acknowledge journal selection and open access constraints Source selection and bias
• Manual cleaning and ad hoc fixes: necessary but creates unknown annotation bias and reduces scalability; authors report deleting many files and manual corrections for Unicode and layout errors which may affect reproducibility Manual cleaning and file rejection
• High dimensional TF-IDF feature sizes (Preprocessing1 features >1.5M) make clustering challenging and increase chance of overfitting for classifiers unless feature reduction or embeddings used — authors note curse of dimensionality drives low Silhouette scores High dimensionality and clustering issues
• Code release unclear: dataset on Zenodo but full experiment scripts, seeds, and hyperparameters are not explicitly linked (authors reference Google Colab and libraries used) — this reduces reproducibility despite dataset availability Software and reproducibility notes

Technical critique and suggestions

1. Feature representation — TF-IDF alone is brittle at millions of sparse features. Recommend providing precomputed dense embeddings (AraBERT or multilingual transformers) and/or dimensionality reductions (PCA/TruncatedSVD, UMAP) to improve clustering and to enable fairer comparisons; the paper does not evaluate transformer embeddings TF-IDF use and missing embeddings
2. Clustering evaluation — Silhouette and DBI are sensitive to high dimensionality; recommend reporting cluster stability (resampling), and external topic-coherence metrics if labels available; authors report ARI which is useful but should be paired with stability tests Clustering metrics used
3. Train/test splits and variance — provide cross validation or multiple random seeds and confidence intervals around classifier scores; paper reports single-split numbers which can overstate performance variance Train test split described
4. Label provenance and inter-annotator checks — authors selected classes by journal domain; if any manual labeling occurred, report inter-annotator agreement and rules for ambiguous papers (not provided) Labeling and selection process

Reproducibility and recommended minimal artifacts

To make ARPD maximally useful, authors or contributors should publish alongside the dataset: (1) full ingestion and cleaning scripts with deterministic seeds, (2) preprocessing pipelines that generate Preprocessing1..3 variants, (3) sample model configs for SVM/RF/PSO/GWO with hyperparameters, and (4) example notebook reproducing main tables and figures — the paper currently lists tools used (PyPDF2, PyArabic, Arlstem, scikit-learn, Pyriad) but does not attach full pipelines Software stack used

Quick actionable advice for researchers who want to use ARPD

• Start by using the Preprocessing3 CSV to avoid raw OCR errors; then compute transformer embeddings (AraBERT family) and run UMAP + HDBSCAN for clustering before relying on Kmeans Preprocessing recommendations and content
• Report cross validated classifier metrics (kfold or repeated splits) and per-class confusion matrices to reveal class imbalance effects — authors provide per-class tables but not cross-validated uncertainty Per-class results available

Figure reproductions

Overall evaluation scores (objective)

Short conclusions with confidence

ARPD is a valuable, novel resource for Arabic NLP in academic domains and provides strong baseline classification numbers (SVM ~98%) while exposing clustering weaknesses driven by TF-IDF high dimensionality; confidence in these statements is moderate because results are reported but experiment scripts and multiple-seed variance are not fully published Paper final claims about utility

What evidence would overturn the main claims

Reproducing experiments with independent code and showing large performance variance or lower SVM accuracy across randomized CV folds, or finding that ARPD selection bias makes it nonrepresentative of Arabic academic writing would overturn claims of broad utility — the paper's own limitations section points to selection and preprocessing effects Paper limitations and blindspots