SHAP — Shapley Additive Explanations

Takeaway

SHAP attributes a model’s prediction to features using Shapley values from cooperative game theory, satisfying axioms like efficiency, symmetry, and additivity.

The problem (before → after)

Before: Feature importance methods are inconsistent across models and lack principled guarantees.
After: A unified framework yields consistent attributions under clear axioms and links to many existing methods.

Mental model first

Imagine splitting a group prize fairly based on each person’s marginal contribution across all possible team orders. Shapley values average those contributions to give each feature credit.

Just-in-time concepts

Shapley value φ_i: Average marginal contribution over all subsets S not containing i.
Additive explanations: f(x) ≈ φ_0 + ∑ φ_i.
Model-specific approximations: TreeSHAP computes exact values for tree ensembles efficiently.

First-pass solution

Define a background distribution; estimate contributions via sampling or exact algorithms for special model classes; visualize with beeswarm and force plots.

Interaction values measure pairwise effects.
Causality-aware SHAP adjusts attributions under interventions.
Runtime: Use model-specific algorithms when possible.

Code as a byproduct (marginal contribution sketch)

from itertools import permutations

def shapley_value(player, players, value):
    contrib = 0.0
    for perm in permutations(players):
        S = set()
        for p in perm:
            if p == player:
                before = value(S)
                after = value(S | {p})
                contrib += (after - before)
                break
            S.add(p)
    return contrib / len(list(permutations(players)))

Principles, not prescriptions

Choose background data carefully; attributions depend on it.
Use consistent axioms to compare methods.

Common pitfalls

Explanations can be brittle under distribution shift.
Misinterpreting correlation as causation.

Connections and contrasts

See also: [/blog/adversarial-examples] (robustness vs explanations), [/blog/variational-inference] (uncertainty-aware explanations).

Quick checks

What guarantees does SHAP satisfy? — Efficiency, symmetry, dummy, additivity.
Why background data? — Defines the baseline prediction.
When exact? — Tree ensembles via TreeSHAP.