Heavy-tailed distributions

evergreen#foundations#distributions

Bar: explain why clickstream popularity is Zipf/Pareto, recognize a power law on a log-log plot, and know when the mean/variance simply don't exist.

What "heavy-tailed" means

A distribution whose tail decays slower than exponential — extreme values are rare but not negligibly rare. Consequence: a handful of observations dominate the totals, and classic intuitions built on the Normal (where everything clusters near the mean) break.

Common heavy tails in this domain:

Log-normal — $\log X$ is Normal. Dwell times, session durations, time-between-visits.
Pareto / power law — $P(X>x)=(x/x_m)^{-\alpha}$ for $x\ge x_m$ . The "80/20" rule.
Zipf — discrete power law over ranks: frequency $\propto 1/\text{rank}^s$ . Page popularity, search terms, event-type counts.

The dangerous part: moments may not exist

For Pareto with exponent $\alpha$ :

\mathbb{E}[X]=\frac{\alpha\,x_m}{\alpha-1}\ (\alpha>1);\qquad \text{Var}[X]<\infty\ \text{only if } \alpha>2.

$\alpha\le 2$ : infinite variance — the sample mean is unstable, CLT-based confidence intervals lie.
$\alpha\le 1$ : infinite mean — the sample average keeps growing as you collect more data.

So "the average session has X pageviews" can be a meaningless statistic. Use medians and quantiles, and report the tail explicitly.

Diagnostics (how to spot it)

Log-log CCDF plot: a power law is a straight line on $\log P(X>x)$ vs $\log x$ ; the slope is $-\alpha$ . A log-normal curves; an exponential drops off a cliff.
Mean-vs-sample-size plot: if the running mean never settles, suspect infinite/near-infinite moments.
Zipf check: plot frequency vs rank on log-log; slope near $-1$ ⇒ Zipfian.

Worked (Zipf/80-20 intuition): with frequency $\propto 1/\text{rank}$ , the top page gets twice the 2nd, three times the 3rd… A few pages soak up most traffic; the long tail is enormous but individually tiny. For Pareto $\alpha=1.16$ , the top 20% of items hold ~80% of the mass (that's where "80/20" comes from).

Why this matters here

Sampling & evaluation: uniform sampling under-represents the tail; rare-but-important events (high-value pages, rare conversions) get lost. Class imbalance (1–3% conversions) is the same phenomenon (Evaluation theory, Difficulty knobs).
Synthetic realism: to fake a believable clickstream you must inject Zipf page popularity and heavy-tailed dwell times, or your background is too tidy to be a fair test (Generating background traffic).
Don't trust the mean in dashboards or features; heavy tails quietly wreck naïvely-Normal statistics.

By-hand exercise (meets the bar)

Pareto with $x_m=1$ . For which $\alpha$ is the mean finite but the variance infinite? (Answer: $1<\alpha\le2$ .)
Sketch how an exponential vs a power-law tail look on a log-log CCDF, and explain why only one is straight.