Skip to main content

Bias Classification Methodology

How we analyze and label editorial bias in news coverage of the Iran-Israel conflict.

Important: Bias labels reflect editorial tendency based on publicly available data and algorithmic analysis. They do not represent judgments on accuracy, credibility, or journalistic quality. All classifications should be considered as analytical opinions, not statements of fact.

Overview

Our bias classification system uses a hybrid approach combining two independent signals:

  1. Source-level bias — Static ratings based on media bias databases
  2. Article-level bias — Per-article LLM analysis of content and framing

These two signals are combined using a weighted formula to produce a final bias label for each article.

Source-Level Bias

Each news source in our registry is assigned a static bias score from -1.0 (strongly pro-Iran) to +1.0 (strongly pro-Israel) based on data from:

  • Media Bias/Fact Check (MBFC) — Independent media bias rating organization
  • AllSides — Media bias ratings using multi-method analysis
  • Academic research on media coverage of Middle East conflicts

Source-level scores provide a baseline that reflects the editorial tendency of the publication on this specific topic (Iran-Israel relations), not their overall political lean.

Article-Level Bias (LLM Analysis)

Each article is individually analyzed using a large language model (Llama 3.1 8B via Groq) to assess:

  • Language and framing choices
  • Selection and emphasis of facts
  • Attribution and sourcing patterns
  • Emotional tone and connotation

The LLM returns a bias score (-1.0 to +1.0) and a confidence value (0.0 to 1.0) indicating how certain the analysis is. Low-confidence results have less influence on the final score.

Combined Scoring

The final bias score combines both signals using a weighted formula:

combined = (35% × source_score) + (65% × article_score × confidence)

The article-level analysis receives higher weight (65%) because it assesses the specific content, while the source-level provides a stabilizing baseline (35%). When only one signal is available, it's used directly without weighting.

Bias Thresholds

The combined score is mapped to one of three labels:

Pro-Iran Score < -0.25
Neutral -0.25 ≤ Score ≤ 0.25
Pro-Israel Score > 0.25
Pro-Iran (3)Neutral (5)Pro-Israel (4)

Graceful Degradation

If the LLM classification is unavailable (e.g., API downtime, rate limits), articles fall back to source-level bias only. Articles from unknown sources remain unclassified until manually reviewed. The system prioritizes partial classification over no classification.