BGPT: Paper Review: SoMe: A Realistic Benchmark for LLM-based Social Media Agents

Explore by Goal

Quick Explanation Copied

Core takeaway

SoMe is a tool-enabled, large-scale benchmark for 8 social-media agent tasks (RED/SES/MID/UBP/UEA/UCS/MCR/SMQ) built on millions of real posts/users/reports, and the paper reports that mainstream agentic LLMs struggle—especially on content-open tasks (scoring often <50) and with error modes like tool-chain planning mistakes and tool-use hallucination.

(Evidence: )

Long Explanation

Paper Review (Critical + Evidence-based): SoMe: A Realistic Benchmark for LLM-based Social Media Agents

Target: AAAI 2026 paper (DOI: 10.1609/aaai.v40i2.37113)
What it claims: A realistic, tool-enabled evaluation benchmark for agentic LLMs on 8 social-media tasks using millions of real-world posts/users/reports and annotated queries.

Dataset & evaluation scope (from the paper)

Scale: 9,164,284 posts; 6,591 user profiles; 25,686 reports; 17,869 annotated task queries.
Platforms/modalities: data from 32 social media platforms (English + Chinese emphasis), with non-text post content converted into captions/OCR/ASR as described.
Agent tooling: 8 agent tools for acquiring/managing/analyzing data, designed to follow Model Context Protocol (MCP).
Models evaluated: 13 agentic LLMs (API-based and open-source), run with tool access via MCP.

Visual 1: Multi-task performance (avg score) across models

Interpretation (from reported tables): the paper reports that “current closed-source and open-source LLMs cannot handle social media agent tasks satisfactorily,” and shows many average scores in the ~30–55 range, even when task completion can be high for some models.

Visual 2: Task Completion Rate (TCR) vs average score

Why this matters: SoMe distinguishes “task completion” from “evaluation scores,” and the paper emphasizes that passing tool sequences (high TCR for some models) does not automatically yield high semantic-quality scores, especially for open-ended tasks scored by LLM-based metrics.

Visual 3: Example error modes (tool-chain planning & tool-use hallucination)

The paper provides (i) One-attempt Success Rates (OSR) for correct tool-chain planning in 100 samples of RED, and (ii) hallucination in tool-use via an experiment where tool responses are intentionally altered; it reports tool-response hallucination rates such as 14%, 8%, 8%, 6%, 7%, 13% for several open-source models.

(Critical) Methodological audit: what is known vs uncertain

Known from the paper text

Benchmark construction: SoMe claims to be “the first versatile and realistic platform” for tool-enabled evaluation of social-media agents, with 8 tasks and millions of real posts/users/reports plus 17,869 annotated queries.
Evaluation metrics: SoMe uses LLM-based semantic scoring for RED/SES/SMQ and computes ACC + TCR for several other tasks by having LLMs filter redundant info and extract answers before comparing to ground truth.
Reported weaknesses: The paper reports most models do not exceed 70 in most tasks; RED/SES/MID are highlighted as particularly low for many models (many scores below 50).

Uncertain / potentially biasing factors (flagged)

LLM-based metrics: Using an LLM scorer for semantic comparison can introduce grader-model dependence and calibration issues (e.g., a scorer that is itself weaker in certain linguistic phenomena could systematically under/over-score). This risk follows directly from the paper’s described scoring approach (no independent human correlation reported in the provided text).
Ground truth generation pipeline: Some tasks’ ground truth is described as automatically generated from templates with no LLM/human edition, while other tasks use multi-round human–LLM interactive pipelines based on a model (Qwen3-32B), and final filtering is done by 10 annotators. Uncertainty remains about residual systematic template bias and about whether the human–LLM pipeline achieves consistent “answer norms” across tasks.
Language/platform composition: The paper states a predominant focus on English and Chinese and data from 32 platforms. External validity to other languages or platforms is therefore uncertain.
Tool-use “hallucination” operationalization: The hallucination rates come from an experiment where tool responses are intentionally altered to test whether the agent feigns completion. This is informative for robustness to malformed/incorrect tool outputs, but it depends on the specific altered-response design described in the paper.

(Critical) What the paper’s results do and do not prove

Supports: The paper provides evidence that tool-chain planning and tool-use robustness remain bottlenecks for many models (via OSR and hallucination-rate experiments).
Does not directly prove: That architectural changes alone will resolve semantic-scoring failures (since the paper’s metrics incorporate LLM-based scoring/extraction). The results justify system-level debugging, but the causal mechanism “why the score is low” may be confounded by the scoring pipeline itself.
Does not directly establish: Longitudinal real-world performance of deployed agents, because the benchmark evaluates against static ground truth annotations and specific tool configurations.

Suggested directions to strengthen the benchmark (within the paper’s own stated setup)

Measure grader agreement: Add calibration/consistency checks between LLM-based semantic scoring and human judgments for a subset of RED/SES/SMQ, to quantify whether model improvement is real or scorer-dependent. (This follows because the paper explicitly uses LLM-based scoring.)
Tool-misuse robustness stress-tests: Expand the altered-tool-response experiments (beyond one induced modification scheme) to cover multiple failure modes (wrong schema, partial outputs, stale timestamps). The paper already demonstrates hallucination sensitivity via an altered-response setup.
Generalization checks: Report performance stratified by platform and language (English vs Chinese) to bound external validity claims more tightly, since the paper specifies predominant language focus.

Author reviews (bespoke BGPT links)

Feedback:

Updated: April 25, 2026