User study with AI-powered fact-checking system for COVID-19 related topics

HIGHLIGHTS

How can transparency and interactivity in AI systems help users make better decisions when fact-checking information? I led the UX research and experimental design for an AI-powered fact-checking platform that visualizes the reasoning process behind automated claim verification. Through eye-tracking and behavioral analytics, I investigated how interface interactivity and evidence presentation shape user engagement, trust, and cognitive effort during misinformation detection.

Role: Lead Researcher (Experimental Design, UX Evaluation, Data Analysis)

Timeline: May 2021 - December 2023

Collaborators: Dr. Nilavra Bhattacharya, Dr. Anubrata Das, Dr. Jacek Gwizdka

RESEARCH GOALS

Investigate how interactive transparency in AI fact-checking tools influences user behavior and decision-making.
Examine whether visualizing model reasoning (e.g., article stance, source credibility) leads to better understanding and more critical engagement.
Quantify how transparency impacts attention, trust, and cognitive workload in AI-assisted verification tasks.

RESEARCH APPROACH

To address these goals, I adopted a quantitative approach that integrates:

Experimental Control: A within-subjects eye-tracking study comparing an interactive vs. non-interactive fact-checking interface, isolating the impact of transparency.
Behavioral Analytics: Continuous logging of dwell time and interaction behavior to quantify engagement.
Physiological Measurement: Eye-tracking of fixation patterns and pupil dilation to capture attention and cognitive effort.
Self-Report Validation: Post-task NASA-TLX surveys to measure perceived workload and trust confidence.

This combination of quantitative precision and user-centered interpretation allowed us to connect interface design features with measurable cognitive outcomes. This is a methodology directly applicable to UX evaluation in AI-driven products.

SYSTEM DESIGN

The system retrieves articles supporting or refuting a claim, displays each article’s stance and source reputation, and predicts overall claim correctness.
Two versions were developed:

Interactive: Users can adjust credibility sliders and stance weights to explore how AI predictions change.
Non-Interactive: Displays static model outputs only.

This contrast allowed isolation of interactivity’s effect on user cognition and engagement.

*Fact-checking system (interactive interface- left, non-interactive - right). In the interactive interface, users can manipulate evidence parameters to see how model predictions update in real time.*

EVALUATION

Study Design

Type: Controlled, within-subjects laboratory experiment.
Participants: 40 adults (balanced gender, native-level English, non-experts in fact-checking).
Setting: UT Austin iSchool usability lab using Tobii TX-300 eye-tracker.
Duration: ~90 minutes per participant (training + 2 interface blocks × 12 trials + break between blocks).
Stimuli: 24 COVID-19-related claims (8 TRUE / 8 FALSE / 8 UNSURE) each paired with five supporting or refuting news articles.

Tasks

Review each claim presented by the system.
Examine evidence (source names, reputation, stance, and predicted correctness).
Optionally open linked news articles to read the full text.
Rate perceived claim correctness before and after system use.
Complete NASA-TLX workload survey after each interface block.

Measures Collected

Behavioral: Interface dwell time, number of article clicks, interaction frequency.
Eye-tracking: Fixation count, fixation duration, and pupil dilation across six AOIs (claim, source, reputation, stance, correctness, headline).
Self-Report: Perceived workload and task effort (NASA-TLX).

Analysis Approach

Statistical tests: Wilcoxon signed-rank and two-way ANOVA for interface × AOI and interface × claim-type effects.
Effect sizes: η² and r for engagement and attention.
Visualization: Heatmaps, fixation plots, and dwell-time distributions to interpret focus shifts across UI elements.

This rigorous design enabled clear attribution of behavioral and cognitive differences to the presence of interactivity rather than content or topic bias.

KEY FINDINGS

1. Interactivity Enhances Engagement

Users spent ≈ 33 % more time in the interactive interface (p < .05).
External news dwell time also increased significantly (effect size = 0.33). Users tend to seek additional evidence rather than relying solely on AI predictions.

2. Higher Visual Attention on Transparent Elements

Fixation counts (+40 %) and durations (+35 %) were higher for interactive cues (p < .05).
Headlines, stance bars, and reputation sliders drew the most attention, confirming users engaged with explainability features.

3. Ambiguous Claims Drove Deeper Reasoning

Claims without clear right or wrong answers elicited the highest fixation metrics across both interfaces, suggesting that users devoted more cognitive resources to ambiguous or uncertain content.

4. Cognitive Load Remained Stable

NASA-TLX scores showed no significant workload difference between interfaces (p > .05).
Interactivity therefore improved understanding and trust calibration without adding cognitive strain.

5. Design Implications

Transparent affordances can shift users from passive acceptance to active evaluation.
Future designs should pair interactive explanations with clear uncertainty cues to support informed reasoning at scale.

OUTCOME

This research provided empirical evidence that interactive transparency enables more reflective, trust-calibrated fact-checking behavior.
Its design and methodological framework have informed later Good Systems projects on responsible AI and continue to guide best practices for evaluating human-AI collaboration in UX research.

Transparent Fact-checking System