Estimators

Description

Given a population, from which random variables are assumed to follow a distribution $F$ with parameter $\theta$, we seek to take random sample $\overrightarrow{Y} := \left(Y_1,... , Y_n \right)$ from this population and use them to estimate the true value of $\theta$.

Estimator $\hat{\theta}_n$: a statistic being used to estimate $\theta$.
(A rule, often expressed as a formula, that tells how to calculate the value of an estimate based on the measurements contained in a sample)

We can make different estimators. Some may be meaningless, some bad, and some good
How to asses our estimators?

Experiment

Let's consider an experiment

Biased coin: It does not have a 50/50 chance
Parameter $\theta$: The probability of getting heads in one attempt
Fixed sample size: $n=10$

Here is the estimators

$\hat{\theta}_{10,1}$
$$\hat{\theta}_{10,1} = \left\{\begin{matrix} 1, & Y_1 = Y_{10}\\ 0, & Y_1 \neq Y_{10} \end{matrix}\right.$$ Returns 1 if the first element in the sample equals the last element, otherwise 0. (1 if Y[0] == Y[-1] else 0)
$\hat{\theta}_{10,2}$

$$\hat{\theta}_{10,2} = \left\{\begin{matrix} 1, & Y_1 = \text{H}\\ 0, & Y_1 = \text{C} \end{matrix}\right. ~~~=~~~Y_1,~~\text{where heads = 1 and tails = 0}$$
Returns 1 if the first element in the sample is heads, otherwise 0. (1 if Y[0] == "H" else 0)
$\hat{\theta}_{10,3}$ $$\hat{\theta}_{10,3} = \frac{1}{10}\sum_{i=1}^{10}Y_i,$$ where heads = 1 and tails = 0
This is the ratio of the number of heads in the sample to the total sample size $$\hat{\theta}_{10,3} = \frac{\text{number of heads}}{10}$$

Bias

The bias of an estimator $\hat{\theta}_n$ that is being used to estimate $\theta$ is defined to be $$\text{Bias}(\hat{\theta}_n , \theta) = E[\hat{\theta}_n] − \theta$$

If $\text{Bias}(\hat{\theta}_n , \theta) = 0$, then $\hat{\theta}_n$ is unbiased
Otherwise $\hat{\theta}_n$ is biased

To calculate bias we take 1000 samples of size $n=10$ and compute the estimates of $\theta$.
The expectation $E[\hat{\theta}_{10}]$ is the mean value of these estimates over all 1000 samples.
Variance is calculated similarly.

Our Data

True Probability of Getting Heads:

θ =

Results Scroll carefully! 1000 rows dynamically loaded

$\overrightarrow{Y}$

$E\left[\hat{\theta}_{10}\right]$

$\text{Bias}\left(\hat{\theta}_{10}, \theta \right)$

$Var\left[\hat{\theta}_{10}\right]$

Conclusions

For $\theta = 0.5$ all estimators are seem to be unbiased, the $\text{Bias}\left(\hat{\theta}_{10}, \theta \right) \approx 0$
With $\theta \neq 0.5$ the 1^st estimator is clearly biased, while 2^nd and 3^rd estimators are unbiased
We can proof that 2^nd and 3^rd estimators are biased.
$$E[\hat{\theta}_{10, 2}] = E[Y_1] = \theta$$ $$E[\hat{\theta}_{10, 3}] = E[(Y_1+Y_2+...+Y_{10})/10] = \frac{E[Y_1]+E[Y_2]+...+E[Y_{10}]}{10} = \frac{\theta+\theta+...+\theta}{10} = \theta$$
Regardless of $\theta$ the variance of the 3^rd estimator is significantly lower than that of the 2^nd.
Thus, if we take only one sample of 10 attempts, the 3^rd estimator is more likely to yield a result closer to the true parameter