19  Homework: Comparing Estimators

$$ \newcommand{X}{} \newcommand{Y}{} % A hex digit after # inside a macro body is eaten by build-time expansion, so digit-leading hexes spell # via .

$$

\DeclareMathOperator{\RMSE}{RMSE} \DeclareMathOperator{\mathop{\mathrm{E}}}{E} \DeclareMathOperator{\mathop{\mathrm{\mathop{\mathrm{V}}}}}{V} \DeclareMathOperator{\mathop{\mathrm{sd}}}{sd} \DeclareMathOperator{\mathop{\mathrm{\widehat{V}}}}{\widehat{V}} \DeclareMathOperator{\mathop{\mathrm{bias}}}{bias} \newcommand{\thetaprior}{\theta^{\text{prior}}} \newcommand{\nprior}{n^{\text{prior}}} \newcommand{\yprior}{y^{\text{prior}}}

Introduction

Summary

This homework continues where the last one left off. You’ve now seen that biased estimators can mess up coverage, and you’ve learned exactly how: coverage depends on the ratio bias/se. Now we’ll dig into the tradeoffs involved in choosing an estimator.

We’ll start with the bias/variance tradeoff. An estimator can be bad in multiple ways: too much bias, too much variance, or both. We’ll use root-mean-squared error to summarize overall accuracy.¹

Then we’ll talk about consistency—the one property that almost everyone agrees they want an estimator to have. It’s really the bare minimum: if you’re willing to collect enough data, you can get as close to the estimation target as you want. We’ll see which of our prior-based estimators are consistent and which aren’t.

And we’ll see what consistency doesn’t promise. An estimator can be consistent—its typical error vanishing as you collect more data—and still give you interval estimates that get worse as n grows. We’ll also work out how big a sample has to be for a chosen precision.

Finally, you’ll use the expectation and variance calculations from your previous homework to study a different sampling design.

The Point

Knowing this stuff will help you choose between estimators yourself, think about the choices others make, and communicate about these decisions and their implications. It’ll help us think about calibration later on, too, because when we know our point estimators are close to the estimation target, we can get away with using approximations like Taylor series to understand their sampling distributions.

Bias, Variance, and Tradeoffs

Review: Prior-Based Estimators

In the last homework, we worked with estimators that incorporate prior information. Recall the general form: if we have \nprior prior observations with mean \thetaprior, we can combine them with our sample to get \tilde{Y}_{\nprior} = \frac{1}{\nprior + n} \cb*{ \nprior\thetaprior + n\bar Y }. You calculated the bias and standard deviation of this estimator: \begin{aligned} \text{bias} &= \frac{\nprior( \thetaprior - \theta)}{\nprior + n} \\ \mathop{\mathrm{sd}}(\tilde Y_{\nprior}) &= \sqrt{\frac{n\theta(1-\theta)}{(\nprior + n)^2}} \end{aligned} Throughout, assume the prior mean is not exactly the population proportion, \thetaprior \neq \theta, and that the population is not degenerate, 0 < \theta < 1.

And you saw, both in the sampling distribution plots and in the coverage simulation, that bias causes problems for interval estimation. The lecture gave us the precise relationship: coverage depends on the ratio bias/se.

Now we’ll dig deeper into the tradeoffs involved in choosing an estimator.

The Bias/Variance Tradeoff

Here’s the figure from last homework showing three estimators at three sample sizes. Recall that estimator ‘a’ is the sample mean (unbiased), estimator ‘b’ uses a fixed number of prior observations (bias shrinks with n), and estimator ‘c’ scales the prior observations with sample size (constant bias).

Figure 21.1: Sampling distributions of three estimators at three sample sizes.

As the figure shows, there are multiple ways to be a not-so-great estimator. You can have no bias—or small bias—but so much variance that in many surveys (i.e. many draws from the sampling distribution) you’re way off. That’s what we see happening with ‘estimator a’ at sample size n=10. You can have low variance but comparatively high bias, like we see with ‘estimator c’ at sample sizes n=40 and n=160. We often use root-mean-squared-error to summarize the typical magnitude of an estimator’s error.

\RMSE(\hat\theta) = \sqrt{ \mathop{\mathrm{E}}\sb*{ \cb*{\hat \theta - \theta}^2 } }

Famously, we can decompose its square into a sum of squared bias and variance.

\begin{aligned} \mathop{\mathrm{E}}\sb*{ \cb*{\hat\theta - \theta}^2 } &= \mathop{\mathrm{E}}\sb*{ \cb*{ \p*{\hat\theta - \mathop{\mathrm{E}}[\hat\theta]} + \p*{\mathop{\mathrm{E}}[\hat\theta] - \theta} }^2 } \\ &= \mathop{\mathrm{E}}\sb*{ \cb*{ \hat\theta - \mathop{\mathrm{E}}[\hat\theta]}^2 } + 2\mathop{\mathrm{E}}\sb*{ \cb*{ \hat\theta - \mathop{\mathrm{E}}[\hat\theta]} \cdot \cb*{\mathop{\mathrm{E}}[\hat\theta] - \theta} } + \mathop{\mathrm{E}}\sb*{ \cb*{ \mathop{\mathrm{E}}[\hat\theta] - \theta }^2 } \\ &= \mathop{\mathrm{\mathop{\mathrm{V}}}}[\hat\theta] + 2 \times 0 + \underset{\text{bias}(\hat\theta)^2}{\cb*{ \mathop{\mathrm{E}}[\hat\theta] - \theta }^2} \\ \end{aligned}

Just to check that you’re following the math, do this quick exercise.

Exercise 21.1  

Exercise

Explain why \mathop{\mathrm{E}}\sb*{ \cb*{ \hat\theta - \mathop{\mathrm{E}}[\hat\theta]} \cdot \cb*{\mathop{\mathrm{E}}[\hat\theta] - \theta} } = 0.

SolutionSolution

It’s the expected value of a mean-zero random variable \cb*{ \hat\theta - \mathop{\mathrm{E}}[\hat\theta]} times a constant \cb*{\mathop{\mathrm{E}}[\hat\theta] - \theta}. Linearity of expectations justifies rewriting this as that constant times the expectation of the mean-zero random variable, i.e., times 0.

Now here’s a real one. We’re talking about different estimators for the same estimation target. This exercise is an opportunity to think about how you might choose between them. I’ve asked you to answer a few questions and sketch a few things to help you think through the issues, but I’ve tried to leave this pretty open-ended because this really is a question without a single right answer. To some extent, it’s about what you think is important. You don’t have to stand by this choice for the rest of your life, so it’s okay if you miss something important and wind up changing your mind later, e.g. when you talk with your classmates or read the solution. It’s really just to get you started thinking about these kinds of choices and what you might want to consider when making them.

Exercise 21.2  

Exercise

Suppose we increase our number of prior observations, \nprior, without changing their mean \thetaprior. What happens to the bias of the estimator \tilde{Y}_{\nprior}? E.g. does it stay the same, increase, decrease, increase then decrease, decrease then increase, etc.? What happens to its standard deviation? And what happens to its root-mean-squared-error, \RMSE(\tilde{Y}_{\nprior})? Is there a bias/variance tradeoff going on?

Thinking of the magnitude of the bias, the standard deviation, and the root-mean-squared error as functions of \nprior, sketch all three on the same axes. This sketch doesn’t have to be super precise, but try to convey the general shape of each function and do identify precisely the location (value of \nprior) and value (height of the function at the location) of any important features, e.g. minima or maxima of one curve, points where two curves cross, etc.

Having done all this, propose a choice for \nprior and explain why you’d make it. Your choice can depend on n and \thetaprior, because you know all that stuff. But it can’t depend on \theta. That’s something you don’t know.

SolutionSolution

Let’s recall our formulas for the bias and standard deviation of \tilde{Y}_{\nprior} and sum their squares to get its mean squared error.

\begin{aligned} \text{bias} &= \frac{\nprior(\thetaprior-\theta)}{\nprior+n} \\ \text{sd} &=\frac{\sqrt{n\theta(1-\theta)}}{\nprior+n} \\ \text{rmse} &= \sqrt{\frac{ (\nprior)^2 (\thetaprior-\theta)^2 + n\theta(1-\theta)}{(\nprior+n)^2}} \end{aligned}

The bias of this estimator increases in magnitude as \nprior grows. In particular, it grows more or less linearly when \nprior is small relative to n and level off then slows, taking the value (\thetaprior-\theta)/2 for \nprior=n and converging to \thetaprior-\theta as \nprior goes to infinity. The standard deviation of this estimator decreases as \nprior grows, taking on half the value it has for \nprior=0 when \nprior=n and converging to zero as \nprior goes to infinity. The root-mean-squared error of this estimator, decreases then increases as \nprior grows.

I’m going to give you a long, detailed answer for the rest of this question. I wasn’t expecting anything like it from you, but I thought it might interest you and maybe clarify a few things you might’ve thought about when you approached the question yourself. I’ll start with precise plots of the bias, sd, and rmse curves for \thetaprior=.5 and three values of \theta: .6, .55, and .525. So that we can compare across different values of \theta, I’ll plot each twice: once on a common set of axes (left) and once with the minimums of \text{rmse} aligned (right).

bias (red), sd (blue), and rmse (green) as a function of nprior when thetaprior=.5, theta=.6, and n=100

bias (red), sd (blue), and rmse (green) as a function of nprior when thetaprior=.5, theta=.6, and n=100

bias (red), sd (blue), and rmse (green) as a function of nprior when thetaprior=.5, theta=.55, and n=100

bias (red), sd (blue), and rmse (green) as a function of nprior when thetaprior=.5, theta=.55, and n=100

bias (red), sd (blue), and rmse (green) as a function of nprior when thetaprior=.5, theta=.525, and n=100

bias (red), sd (blue), and rmse (green) as a function of nprior when thetaprior=.5, theta=.525, and n=100

One salient feature is the minimum of the rmse curve, which I’ll indicate with a dotted line. We can find it by setting the derivative of the mean squared error with respect to \nprior to zero and solving for \nprior. We use the quotient rule [f(x)/g(x)]' = [f'(x)g(x) - f(x)g'(x)]/g(x)^2. Since we’re looking for a minimum, we can ignore the denominator. We want x=\nprior where f'(x)g(x) - f(x)g'(x) = 0 for f(x) = x^2 (\thetaprior-\theta)^2 + n\theta(1-\theta) and g(x) = (x + n)^2. \begin{aligned} 0 &= f'(x)g(x) - f(x)g'(x) \\ &= 2x(\thetaprior-\theta)^2(x+n)^2 - 2x^2(\thetaprior-\theta)^2(x+n) - 2n\theta(1-\theta)(x+n) \\ &= 2x(\thetaprior-\theta)^2 (x+n)\cb*{(x+n) - x} - 2n\theta(1-\theta)(x+n) \\ \end{aligned} which happens, cancelling the common factor of 2n(x+n), for x=\theta(1-\theta)/(\thetaprior-\theta)^2.

Another salient feature is the point where the magnitude of the bias and the standard deviation are equal, which is indicated by a crossing of the red and blue lines in the plots. We can solve for this as follows. \frac{\nprior\lvert\thetaprior-\theta\rvert}{\nprior+n} = \frac{\sqrt{n\theta(1-\theta)}}{\nprior+n} \qqtext { where } \nprior = \sqrt{n\theta(1-\theta)} / \lvert\thetaprior-\theta\rvert

One interesting thing to notice is that, while both of these points shift to the right as \thetaprior gets closer to \theta, the point where rmse is minimized shifts to the right faster. An implication of this that we can see in the plots is that choosing \nprior to minimize rmse will result in an estimator with a larger \lvert\text{bias}\rvert/\text{sd} ratio when \thetaprior is close to \theta than when it is far away. Why does this matter? Because the bigger the ratio of \lvert\text{bias}\rvert/\text{sd}, the worse our interval estimators’ coverage probability will be. When this ratio is roughly 1.96, we’ll get about 50% coverage. We see that in the slides when discussing implications of bias. So it tells us that choosing \nprior to minimize \RMSE is not necessarily a good idea from an inferential perspective. When we do this, poor coverage isn’t a price we pay for choosing \thetaprior badly—it’s our ‘reward’ for choosing it well. In light of this, I think choosing \nprior to minimize \RMSE is a bad idea. We could try to do something a bit more complicated like choosing \nprior to minimize \RMSE subject to a constraint on the magnitude of the bias relative to the standard deviation, e.g., \lvert\text{bias}\rvert/\text{sd} \le 1/2, to ensure close-to-nominal coverage. But what I’d suggest is simpler: just set \nprior=0 and use the sample mean.

All of this assumes we can actually choose \nprior to minimize \RMSE. It isn’t obvious that we can given that our \RMSE formula involves the unknown population proportion \theta, but you can do something a lot like that by minimizing an estimate of \text{rmse}. People do this often, although typically in more complex estimation problems. The most popular technique for this is probably Cross-Validation, which we’ll talk about later in the semester. Stein’s Unbiased Risk Estimate is an interesting alternative. We won’t get there this semester, but it doesn’t require all that much background to understand, so take a look at the wikipedia page if you’re curious.

Consistency

We say an estimator is consistent if it converges to the estimation target as sample size increases to infinity. That’s being vague. There are a couple ways this is imprecise.

First, what does it mean to be the same estimator at different sample sizes? E.g., the definition of estimator ‘c’ in Figure fig-3-sampling-distributions depends on n, so is that ‘an estimator’? The resolution for this one is easy—if we really want to be precise, we explicitly specify an estimator for each sample size: we say ‘an estimator sequence’ is consistent instead of ‘an estimator’. When we say ‘an estimator’ is consistent, we expect the person we’re talking about understands what sequence of estimators we actually mean. This is just a language thing.

The second thing that’s imprecise is what it actually means for a random variable, like an estimator, to converge to something. It turns out that there are a lot of different, useful ways to think about this happening. One of the simpler versions is called convergence in mean square. A sequence of random variables Z_1,Z_2,Z_3,\ldots converges in mean square to a a random variable Z, which is often but not necessarily a constant, if the root mean square difference between them, \sqrt{E[(Z_n - Z)^2]}, gets arbitrarily small (i.e., converges to zero) as n \to \infty. And we say an estimator \hat\theta is consistent in mean-square if it converges in mean square to the estimation target \theta as sample size n goes to infinity.

Here are a few exercises to get you thinking about what consistency can and can’t look like.

Exercise 21.3  

Exercise

If we know that an estimator \hat\theta is mean-square consistent, do we know that its root-mean-squared error goes to zero? What about its bias and standard deviation?

If we know that an estimator \hat\theta is not consistent in mean square, what does that tell us about its root-mean-squared error, bias, and standard deviation?

SolutionSolution

As a matter of definition, an estimator is consistent if (and only if) its root-mean-squared error goes to zero. And because mean squared error is the sum of its squared bias and squared standard deviation, this implies that both its bias and standard deviation go to zero.

If an estimator is not consistent in mean square, then its root-mean-squared error does not go to zero, so at least one of its bias or its standard deviation does not go to zero.

Exercise 21.4  

Exercise

Of the three estimators depicted in Figure fig-3-sampling-distributions — estimators ‘a’, ‘b’, and ‘c’ — which are mean-square consistent? Using the figure, explain your reasoning.

Because consistency is about what happens as n\to\infty, it’s not really possible to know whether an estimator is consistent by looking at what happens at 3 sample sizes, like we have in the figure. You’d have to guess what the next row would look like, and the next, etc. But you don’t need to guess. You have the actual estimator definitions in the review section at the start of this homework. That said, rather than phrasing your explanation in terms of any formulas, talk about what you see in the figure and what the formulas tell you that you would see if we were to add rows for n increasing to \infty.

SolutionSolution

Estimators ‘a’ and ‘b’ are mean-square consistent. The bias of ‘a’ is zero at all sample sizes and the bias of ‘b’ is decreasing to zero as sample size increases. And both have standard deviations that go to zero as sample size increases. Estimator ‘c’ is not mean-square consistent because its bias isn’t improving with sample size.

Convergence in Probability

Now let’s think about another notion of convergence. If an estimator’s sampling distribution ends up in the right location (\text{bias} \to 0) with arbitrarily little spread (\text{standard deviation} \to 0), then it makes sense that every draw from that sampling distribution will be close to the estimation target. Or almost every draw, anyway.

We tend to visualize our estimate as a single dot, e.g. the black dot below. That’s what it is. One number. But when we want to think about what estimates we could have plausibly gotten, we tend to think about the dots we’d get if we were to repeat our survey a hundred times or a billion, each time calculating an estimate the same way. The sampling distribution of our estimator. We can plot actual dots, like the purple ones below, or histogram them, to get a sense of what this looks like. Hopefully all this is familiar-verging-on-boring to you by now.

Whereas convergence in mean square is about the typical distance between these dots and our estimation target, convergence in probability is about the fraction of these dots that falls within a small distance \epsilon. We say a random variable \hat\theta converges in probability to \theta if, for anyone’s idea of ‘sufficiently close’, the probability that \hat\theta is sufficiently close to \theta goes to one. P(\lvert\hat\theta - \theta\rvert \le \epsilon) \to 1 \qqtext{ as } n \to \infty \qqtext{ for any } \epsilon > 0 Or equivalently, the probability that it isn’t sufficiently close goes to zero. That is, if P(\lvert\hat\theta - \theta\rvert > \epsilon) \to 0 \qqtext{ as } n \to \infty \qqtext{ for any } \epsilon > 0

If \hat \theta is an estimator and \theta is our estimation target, we say \hat\theta is consistent in probability if it converges in probability to \theta.²

NoteConvergence in Probability and Interval Estimation

You can rephrase consistency in probability as a question about what happens to the coverage probability of a sequence of interval estimates \hat\theta \pm \epsilon as sample size n goes to infinity. \hat\theta is consistent in probability if, for every \epsilon > 0, the coverage probability of the interval estimate \hat\theta \pm \epsilon goes to one as n goes to infinity. Why? Because \theta is in the interval \hat\theta \pm \epsilon if and only if \lvert\hat\theta - \theta\rvert \le \epsilon, so the probability that interval covers is same as the probability that \lvert\hat\theta - \theta\rvert \le \epsilon.

What Consistency Implies

We now have two notions of consistency on the table and nothing connecting them. Chebyshev’s inequality connects them.

Centred on the estimation target rather than on the estimator’s own mean, it says P(\lvert\hat\theta - \theta\rvert > \epsilon) \le \frac{\mathop{\mathrm{E}}\sb*{(\hat\theta-\theta)^2}}{\epsilon^2} = \frac{\RMSE(\hat\theta)^2}{\epsilon^2}. Fix \epsilon and let n grow. If the root-mean-squared error goes to zero, the bound on the right goes to zero, so the probability on the left does too. Which implies that mean-square consistency implies consistency in probability: if your typical error vanishes, then you land close to the target with probability approaching one.

Negligible and Non-Negligible Bias

Mean-square consistency is a promise about typical error, and even that promise buys less than it sounds like. Coverage runs on the ratio \text{bias}/\text{sd}, and for our prior-based estimator that ratio is simple, because the \nprior + n in the bias and in the standard deviation cancels. \frac{\mathop{\mathrm{bias}}(\tilde Y_{\nprior})}{\mathop{\mathrm{sd}}(\tilde Y_{\nprior})} = \frac{\nprior(\thetaprior - \theta)}{\sqrt{n\theta(1-\theta)}}. So the ratio is governed by \nprior against \sqrt n. And \nprior is something we choose.

Exercise 22.1  

Exercise

Take three ways of growing the number of prior observations as n \to \infty: \nprior = 10 (which is what estimator ‘b’ in Figure fig-3-sampling-distributions does), \nprior = \sqrt n, and \nprior = n^{3/4}.

1. For each, say what happens to the bias, the standard deviation, the root-mean-squared error, and the ratio \text{bias}/\text{sd}. Which of the three are mean-square consistent?

2. By hand, on paper, sketch the sampling distribution of \tilde Y_{\nprior} at a large sample size for two of the three: one where the bias is negligible compared to the standard deviation, and one where it isn’t. On each sketch, mark the estimation target \theta, the mean of the sampling distribution, and the mean \pm two standard deviations, the way we’ve been marking them in green and blue. Photograph your sketches and include them below.

3. In words: for the case where the interval estimate gets worse as n grows, explain how that can happen even though the estimator is consistent.

TipHints

• The three choices make the number of prior observations grow at different speeds—or not at all. To see how fast the bias goes to zero, write it as \frac{\nprior(\thetaprior-\theta)}{\nprior+n} = \frac{\thetaprior-\theta}{1 + n/\nprior}, and compare n/\nprior across the three choices.
• The standard deviation is \sqrt{n\theta(1-\theta)}/(\nprior+n), which behaves like \sqrt{\theta(1-\theta)/n} whenever \nprior is small compared to n.
• For the sketch you don’t need precision. The mean sits at \theta + \mathop{\mathrm{bias}}, the distribution has width about 2\mathop{\mathrm{sd}}, and ‘negligible’ means \theta sits comfortably inside that width.

SolutionSolution

1. The bias is (\thetaprior-\theta)/(1 + n/\nprior), so it’s of order \nprior/n: order 1/n for \nprior=10, order n^{-1/2} for \nprior=\sqrt n, and order n^{-1/4} for \nprior = n^{3/4}. It goes to zero in all three cases. The standard deviation is of order n^{-1/2} in all three. So the root-mean-squared error goes to zero in all three, and all three are consistent in mean square. The ratio is a constant times \nprior/\sqrt n: it goes to zero for \nprior=10, stays constant for \nprior=\sqrt n, and grows in magnitude like n^{1/4} for \nprior = n^{3/4}.

2. All three sampling distributions narrow as n grows. What differs is where the target sits relative to the width. For \nprior=10 the bias shrinks faster than the width, so eventually \theta sits comfortably inside the mean \pm two standard deviations band. For \nprior = n^{3/4} the width shrinks faster than the bias, so eventually \theta sits outside the band.

3. Consistency asks whether the typical error goes to zero—both the bias and the standard deviation. Interval coverage asks whether the bias is small compared to the width of the interval—and the width is headed to zero too. Both shrink; which one wins the race decides whether the intervals get better or worse. For \nprior = n^{3/4} the width wins, so the target drifts further and further outside while the estimator gets steadily closer to it. Being close isn’t the same as being close relative to how precise you’re claiming to be.

How Big a Sample?

Exercise 22.2  

Exercise

A new, wildly controversial candidate is on the ballot, and nobody has any idea what turnout will look like. You’re polling to estimate the proportion \theta of eligible voters who will turn out, drawing people independently and uniformly, and you want a 95% interval estimate accurate to within 1 percentage point—an interval of the form \hat\theta \pm .01, using the normal approximation.

1. Roughly how many voters should you poll? Choose the closest from this list: 1{,}000, 10{,}000, 100{,}000.

2. Suppose you only needed to be accurate to within 2 percentage points, i.e. \hat\theta \pm .02. Would you need half as many voters, or fewer than half? Say why, in a sentence.

SolutionSolution

1. Setting the half-width to .01 and using the worst case \theta(1-\theta) = 1/4, n = \frac{1.96^2 \times 1/4}{.01^2} \approx \frac{3.84 \times .25}{.0001} = 9{,}600, so the closest choice is 10{,}000 voters.

2. Fewer than half. Doubling the width means halving the quantity being squared, so the sample size falls by a factor of four: about 2{,}400 voters for \pm .02.

The Rotational Design, Revisited

A population of m=2n people consists of n people whose response is 0 followed by n people whose response is 1, arranged in a circle. Choose a starting person uniformly at random and sample that person and the next n-1 people around the circle. Let \hat\theta be the proportion of 1s in the sample.

Exercise 23.1  

Exercise

1. Start with n=2: the circle is 0,0,1,1, and you sample two consecutive people. List the possible samples and work out the sampling distribution of \hat\theta.

2. Now let n be any positive integer. Work out the sampling distribution of \hat\theta. Count the starting positions that produce each possible number of 1s in the sample.

3. Calculate the expectation and variance of \hat\theta.

4. What happens to the variance as n grows? Compare it with the variance when you draw n people independently from the same population. What does this comparison tell you about increasing the sample size?

TipHelp with the sums

To calculate an expectation from a sampling distribution, multiply each possible value by its probability and add. If your calculation involves sums of consecutive integers or their squares, these identities may help: \sum_{k=1}^{n-1} k = \frac{(n-1)n}{2} \qquad\text{and}\qquad \sum_{k=1}^{n-1} k^2 = \frac{(n-1)n(2n-1)}{6}. For example, when n=4, the first sum is 1+2+3=6, and its formula is 3\cdot4/2=6. The second is 1^2+2^2+3^2=14, and its formula is 3\cdot4\cdot7/6=14.

SolutionSolution

1. The four samples, by starting person: (0,0), (0,1), (1,1), (1,0). So \hat\theta is 0 with probability 1/4, 1/2 with probability 1/2, and 1 with probability 1/4. Its expectation is 0 \cdot \frac14 + \frac12 \cdot \frac12 + 1 \cdot \frac14 = \frac12, which is \theta: the estimator is unbiased. Its variance is \mathop{\mathrm{E}}[\hat\theta^2] - \mathop{\mathrm{E}}[\hat\theta]^2 = \p*{0 \cdot \tfrac14 + \tfrac14 \cdot \tfrac12 + 1 \cdot \tfrac14} - \tfrac14 = \tfrac38 - \tfrac14 = \tfrac18. That’s the general answer below, at n=2.

2. Write K for the number of 1s in the sample, so \hat\theta = K/n and K runs from 0 to n. One starting position gives K=0, the run sitting exactly on the block of zeros, and one gives K=n, the run sitting exactly on the block of ones. Each count K=k with 1 \le k \le n-1 comes up twice: once as the run slides across one boundary between the blocks and once as it slides across the other. So out of the 2n equally likely starting positions, K=0 and K=n each have probability 1/(2n), and each interior value has probability 1/n.

3. From those probabilities, \mathop{\mathrm{E}}[K] = \frac{1}{2n}\cdot 0 + \frac{1}{n}\sum_{k=1}^{n-1} k + \frac{1}{2n}\cdot n = \frac{(n-1)n/2}{n} + \frac12 = \frac n2, which gives \mathop{\mathrm{E}}[\hat\theta] = 1/2 = \theta: unbiased at every sample size. And \mathop{\mathrm{E}}[K^2] = \frac{1}{n}\sum_{k=1}^{n-1} k^2 + \frac{1}{2n}\cdot n^2 = \frac{(n-1)(2n-1)}{6} + \frac n2 = \frac{2n^2+1}{6}, so \mathop{\mathrm{\mathop{\mathrm{V}}}}(K) = \mathop{\mathrm{E}}[K^2] - \mathop{\mathrm{E}}[K]^2 = \frac{2n^2+1}{6} - \frac{n^2}{4} = \frac{n^2+2}{12}, and therefore \mathop{\mathrm{\mathop{\mathrm{V}}}}(\hat\theta) = \frac{\mathop{\mathrm{\mathop{\mathrm{V}}}}(K)}{n^2} = \frac{1}{12} + \frac{1}{6n^2}.

4. As n grows, \mathop{\mathrm{\mathop{\mathrm{V}}}}(\hat\theta) settles down to 1/12 and stays there—it does not go to zero. The sample mean of n independent draws would have variance 1/(4n), which vanishes. So here you can call half the population, with n growing without bound, and the standard deviation of your estimate still settles at \sqrt{1/12} \approx .29 instead of shrinking to nothing. A big sample, and it says almost nothing.

Which is the lesson of this design. You can’t just act like everything works the same all the time.

Extra Credit: An Arrangement You Don’t Know

Keep the circle of 2n people and the same sampling rule, but now suppose you don’t know how many responses are 1 or how they are arranged. You want a variance bound that works for every such population.

Exercise 23.2  

Exercise

1. Consider the alternating arrangement 0,1,0,1,\ldots. Why is the sample proportion essentially the same at every starting point?

2. Give an informal argument that the half-and-half block arrangement in the previous exercise has the largest variance. It may help to consider how the number of 1s in the sample changes when you move the starting point one place, and when you move it n places.

3. Use this argument and your previous calculation to state a bound on the variance that holds no matter how many responses are 1 or how they are arranged.

SolutionSolution

1. In the alternating circle every run of n consecutive people reads the same alternation, 1,0,1,0,\ldots, so its count of 1s is fixed by the pattern alone: exactly n/2 when n is even (at every start, which makes \hat\theta = 1/2 and the variance zero) and either (n-1)/2 or (n+1)/2 when n is odd, according to which response the run starts on (a variance of just 1/(4n^2), nothing next to what the block arrangement gave us).

2. The variance of the count of 1s is its average squared distance from its mean over the 2n starting positions; the variance of \hat\theta is that divided by n^2, so it is large only if the count sits far from its mean at many starts. Two facts limit how far and how often. Sliding the start one person along changes the count by at most one, so any distance has to be climbed a step at a time; and starting n people along calls the complementary half of the circle, whose count is the total minus the count you had, so however far above its mean the count gets at one start, it sits exactly as far below it at the complementary start. Between a high start and its complement the walk has to come all the way back down, one level per step at best, so the tallest it can be is n/2 above the mean, and the arrangement that spends the most starts at large distances is the one that climbs one level per step to that height and back down through every level—which is exactly what the half-and-half block does, its counts running 0,1,\ldots,n,\ldots,1 as the start slides. Any other arrangement of the ones, whatever their number, gives a shorter or flatter walk, so its starts sit closer to the mean on average. (On a small circle you can check this by listing every arrangement and computing the variance of each—a good way to test the reasoning.)

3. So the bound is the block arrangement’s variance: \mathop{\mathrm{\mathop{\mathrm{V}}}}(\hat\theta) \le \frac{1}{12} + \frac{1}{6n^2} \approx \frac{1}{12}, no matter how the responses are arranged. And the block attains it, so the bound is tight rather than just safe. It’s the same strategy as using \theta(1-\theta) \le 1/4 to plan a sample: find the worst case first, then bound by it.

---

1. I like to say ‘typical distance’ and ‘typical error’ instead of ‘root-mean-squared distance’ and ‘root-mean-squared error’ because it’s shorter, gets the essential idea across, and doesn’t have a technical meaning that might conflict with my use of it this way like ‘average distance’ would.

2. This is nonstandard terminology. What I’m calling ‘consistency in probability’ is usually called weak consistency for reasons that are a little hard to explain without a fair amount of background in topology and measure-theoretic probability. If you say ‘consistent in probability’, people who usually say weak consistency will know what you mean and maybe even follow your lead, but it may take them a second. ‘Convergence in probability’ is standard terminology, for what it’s worth.