Cost functional method

The original paper propose is to minimize the Kolmogorov-Smirnov standard statstics, given by:

but we can give a weight $w(x)$ to this measure, defining the weighted-KS statistics for a theoretical distribution $F$:

In fact, one can show that to choose $w(x)=\sqrt{F(x;\theta)(1-F(x;\theta))}$ is to make the measure variations uniformly over the interval $x_j\geq x_{\min}$.

With these two stats, the standard one and the uniformly one, the long tail has the same weight as the data in the $x_{\min}$ neighborhood. So, if we want to maximize the long tail, we can build a weight that is low near $x_{\min}$ and grows in the tail. One solution is to consider $w(x)=F(x;\theta)$ as $F(x_{\min})=0$ and $F(\infty)\to1$. That is:

This weight is, however, instable and does not give a good statistical measure.

In other way, we can define a cost functional:

where $N$ is the original data size. The interpratation is imediate: $\lambda\geq0$ is the cost of the longtail length and minimize $\mathcal{K}_{w}\big[F\big]$ while considering the importance of the long tail is to minimize the functional $\mathcal{L}[F,x_{\min}]$. This revealed to be much more stable and rigorous.

Fitting with cost functional¶

The quantity $\mathcal{L}[F,x_{\min}]$ does not represent any formal statistical measure; instead, we need information theory to give a formal sense to it. The cost $\lambda$ is, somehow, like a lagrangian multiplier so that

represents the lost information if we drop $N-n_{\text{tail}}+1$ points from the original data. Minimizing $\mathcal{L}[F,x_{\min}]$, then, is minimize the KS measure minimizing the lost of information in the tail.

$\lambda$ is a trade off parameters which tells us, for a given KS value, how plausible a given $x_{\min}$ can be for the cost functional. So we have to apply the p-value bootstrap test to verify what are the plausible values for $\lambda$; in other words, $\lambda$ cannot be a free parameters for the model. The values that we can accept for $\lambda$ are that ones which can give rise to best values for $x_{\min}$ while keeping the traditional null hypothesis test, $p\geq 0.1$.