Convergence rate and consistency at zero for decreasing densities on [0, 1]

J-B. Salomond

CREST, Université Paris Dauphine

Abstract

In this paper, we consider the well known problem of estimating a density function defined on a compact set under qualitative assumption. More precisely, we study monotone non increasing densities in a Bayesian setting, and derive convergence rate for the posterior distribution for two types of priors, namely, a Dirichlet process and finite mixture prior. We prove that the posterior distribution based on both prior concentrates at the rate (n/ log(n))^{−1/3}, which is the minimax rate of estimation up to a log(n) factor. We also prove that the posterior probability of the continuous limit of the density at 0 is consistent, which is of particular interest as the classical non parametric maximum likelihood estimator is known to be inconsistent at this point.

1 Introduction

The non parametric problem of estimating monotone non increasing curve, and monotone densities in particular, has been well studied in the literature, both form a theoretical and applied perspective. For instance, Groeneboom (1985) and more recently, Balabdaoui and Wellner (2007) studied very precisely the asymptotic properties of the non parametric maximum likelihood estimator (NPMLE) , namely, the Grenander estimator, which is proved to be consistent and to converge at the minimax rate n^{−1/3} when the support of the distribution is compact. Monotone non increasing densities arise naturally when considering survival analysis, where it is natural to assume that the uncensored survival time have a monotone non increasing density.

In this paper, we study the asymptotic properties of a Bayesian estimator of a monotone non increasing density function with a compact support. Following Khazaei et al. (2010) we study two families of non parametric priors on F, the class of monotone non increasing densities with support on [0, 1]. We obtained the two families of prior a concentration rate of order (n/ log(n))^{−1/3}. Interestingly, the NPMLE is not consistent at 0, (see Sun and Woodroofe (1996) and Balabdaoui and Wellner (2007) for instance). However, we prove that the posterior distribution of f(0) is still concentrate around the true value a the rate (n/ log(n))^{−1/3}. The non parametric prior are constructed from the mixture representation of monotone non increasing density. It is known since Williamson (1956) that any monotone non increasing density on ℝ^{+} has a mixture representation

f(x) = ∫_{0}^{∞} \frac{𝕀_{[0,θ]}(x)}{θ}dP (θ)  (1)

where P is a probability distribution on ℝ^{+}, namely a mixing distribution. In order to indicate the dependence on P, we shall note f_{P} the density function admiting representation (1). Let F be the set

F = {f s.t. f > 0, f ↘ ∫_{0}^{L} f = 1}

Mixtures models have been well studied in a Bayesian framework and various types of prior have been considered for the mixing distribution. The most popular non parametric mixture prior model is the Dirichlet Process prior (DP) introduced by Ferguson (1983) and in their article, Wu and Ghosal (2008) studied properties of general mixtures models. Using the mixture representation od monotone non increasing densities (1) we construct non parametric priors on the set F by considering a prior on the mixing distribution P. We thus fall in the set up of non parametric mixture prior models. We consider two types of prior

Type 1 : Dirichlet Process prior P ∼ DP (A, α) where A is a positive constant and α a probability distribution on [0, L]

Type 2 : Finite mixture P = ∑_{j=1}^{K} p_{j}δ_{x_{j}} with K a non zero integer and δ_{x} the dirac distribution on x. We define a prior on P by a distribution Q on K and given K, distributions π_{x,K} on (x_{1}, …, x_{K}) and π_{p,K} on (p_{1}, …, p_{K})

Each of these are prior on the set of probability distributions. For X^{n} = (X_{1}, …, X_{n}), a sample of n independent and identically distributes random variables with a common probability distribution function f_{0} in F with respect to the Lebesgue measure, we denote Π(·|X^{n}) the posterior probability measure associated with the prior Π. Most of the results on the convergence rates of posterior distribution requires that the prior puts enough mass on Kullback Leiber neighborhood of the true density

Π({P, KL(f_{P} , f_{0}) ≤ ϵ}) > 0  (2)

where KL(f_{1}, f2) is the Kullback Leiber divergence between f_{1} and f_{2} defined by KL(f_{1}, f_{2}) = ∫ f_{1} log(f_{1}/f_{2}). In particular, Wu and Ghosal (2008) proved that this condition is satisfied by the Type 1 prior in the case of monotone non increasing densities, under mild conditions on f_{0}. The study of the Type 2 prior is more delicate as it does not satisfy (2) in general. Moreover, it has been proved (see Khazaei et al. (2010) ) that for this choice of prior, Π({P, KL(f_{P} , f_{0}) = ∞}) = 1. However, we achieve to prove that the posterior based on a Type 2 prior concentrate at the same rate without the Kullback Leiber property.

The paper is organised as follow. the main results are given in section 2, where conditions on the priors are discussed. The proof are given afterward.

\underline{3}

2 Main results

Convergence rate of the posterior distribution have been well studied in the literature and some general results links the rate to the prior (see Ghosal et al. (2000)). We deduce from these results the optimality of the Bayesian estimator (up to a log(n) factor).The following theorem gives general conditions on the prior to achieve a posterior convergence rate of (n/ log(n))^{−1/3}.

Theorem 1. Let X^{n} = (X_{1}, …, X_{n}) be a iid sample with a common probability distribution function f_{0} ∈ F; and let α be a positive probability density on [0, 1] with respect to the Lebesgue measure that satisfy for θ close to 0, and t > 1

α(θ) ≲ θ^{t}  (3a)

Define also Q a probability distribution on ℕ and π_{p,K} a probability distribution on the simplex of ℝ^{K} satisfying for some positive constants C_{1}, C_{2}, a_{1}, …, a_{K}, c

e^{−C_{1}K log(K)} ≥ Q(K) ≥ e^{−C_{2}K log(K)}  (3b)
π_{p,k}(p_{1}, …, p_{K}) ≥ K^{−K}c^{K}p_{1}^{a_{1}} … p_{K}^{a_{K}}  (3c)

and finally, let (x_{i})_{i} be the order statistics of K iid random variables from α. If d is either the L^{1} or Hellinger distance, then for Π a Type 1 or Type 2 prior, there exists a positive constant C such that

Π (f, d(f, f_{0}) ≥ C (\frac{n}{log(n)})^{−1/3} |X^{n}) → 0,  P_{0} a.s.  (4)

when n goes to infinity.

The proof of this theorem is given in section 3. Condition (3) are roughly the same than in Khazaei et al. (2010). This Theorem is thus an extension of their result when the density’s support is compact. Given that f_{0} is in F, we can uniquely define f_{0}(0) by considering the left limit of f_{0}(x) when x ↘ 0. Let f_{0}(0) = lim_{x↘0} f_{0}(x), under some mild conditions on the first derivative of f, we get the consistency and the convergence rate of the posterior distribution and of the Bayesian estimator associated with the absolute loss, namely, the posterior median.

Theorem 2. Let f_{0} ∈ F such that, f′(0^{+}) < 0. Let X_{i} ∼^{iid} f_{0} for i = 1 … n, and let Π be a prior satisfying (3), then, if ϵ_{n} = C(n/ log(n))^{−1/3}

P^{π}(|f_{P} (0) − f_{0}(0)| > ϵ_{n}|X^{n}) → 0  (5)

Consider the posterior median f̂_{n}(0) = inf{x, P^{π}[f_{p}(0) ≤ x] > 1/2} thus

P_{0}(|f̂_{n}(0) − f_{0}(0)| > ϵ_{n}|X^{n}) → 0  (6)

Thus the Bayesian approach is yield a consistent estimator of f_{0}(0), which is not the case of the maximum likelihood approach. It is known that integrating the parameter as done in Bayesian approaches induces a penalization. This is particularly useful in testing or model choice problems but can also be effective in estimation problems, see for instance Rousseau and Mengersen (2011). The

\underline{4 3 Proofs}

problem of estimating f_{0}(0) under monotonicity constraints is another example of the effectiveness of penalization induced by integration on the parameters. However, contrariwise to Rousseau and Mengersen (2011), we have not clearly identified how the penalisation act, and only observe that it leads to a consistent posterior distribution and a consistent estimator. Furthermore, we only studied here the consistency in probability and it is not clear whether or not stronger consistency results, such as almost sure consistency, should be expected.

3 Proofs

We present here the proof of the two main theorem. The proof of Theorem 1 adapted from Ghosal et al. (2000), and is based on a piecewise constant approximation of f_{0} in the sense of Kullback Leiber divergence. We therefore detail the construction of this approximation in the following section. Note that this is not the construction proposed in Khazaei et al. (2010) which was adapted to the L^{1} distance but not to the Kullback Leiber divergence. This construction is adapted from the one proposed by van der Vaart and Wellner (1996) in section ??.

3.1 Proof of Theorem 1

The piecewise constant approximation of f_{0} is base on a sequential subdivision of the interval [0, L] with more refined subdivision where f_{0} is less regular. We then identify the piecewise constant density by a mixture of uniformes. The following Lemma gives the form of the probability distribution P such that f_{P} is in the Kullback Leiber neighbourhood of f_{0}.

Lemma 3. Let f ∈ F be such that f(0) ≤ M < +∞. For all 0 < ϵ < 1 there exists m ≲ M^{2/3}ϵ^{−1}, p = (p_{1}, …, p_{m}) and x = (x_{1}, …, x_{m}) such that P = ∑_{i=1}^{m} δ_{x_{i}}p_{i} satisfy

KL(f, f_{P}) ≲ ϵ^{2}, ∫ f log (\frac{f}{f_{P}} )^{2} ≲ ϵ^{2}  (7)

where f_{P} is defined as in (1) and KL denote the Kullback Leiber divergence.

Proof. For a fixed ϵ, let f be in F. Consider P_{0} the coarsest partition :

0 = x_{0}^{0} < x_{1}^{0} = 1

at the i^{th} step, let P_{i} be the partition

0 = x_{0}^{i} < x_{1}^{i} < ⋯ < x_{n_{i}}^{i} = 1

and define

ε_{i} = max_{j} {(f(x_{j−1}^{i}) − f(x_{j}^{i}))(x_{j}^{i} − x_{j−1}^{i})^{1/2}}

For each j ≤ 1, if (f(x_{j−1}^{i}) − f(x_{j}^{i}))(x_{j}^{i} − x_{j−1}^{i})^{1/2} ≥ \frac{ε_{i}}{\sqrt{2}} we split the interval [x_{j−1}, x_{j}] into two subsets of equal length. We then get a new partition P_{i+1}. We continue the partitioning until the first k such that ε_{k}^{2} ≤ ϵ^{3}. At each step i, let n_{i} be the number of intervals in P_{i}, s_{i} the number of interval in P_{i} that have been divided to obtain P_{i+1}, and c = 1/\sqrt{2}.

s_{i}(cε_{i})^{2/3} ≤ ∑_{j}(f(x_{j−1}^{i}) − f(x_{j}^{i}))^{2/3}(x_{j}^{i} − x_{j−1}^{i})^{1/3}

≤ (∑_{j} f(x_{j−1}^{i}) − f(x_{j}^{i}))^{2/3} (∑_{j} x_{j}^{i} − x_{j−1}^{i})^{1/3} ≤ M^{2/3}L^{1/3}

The proof of Theorem 1 is based on an adaptation by Rivoirard and Rousseau (2009) of theorem 2.1 of Ghosal et al. (2000). It consists in obtaining a lower bound on the prior mass of Kullback Leiber neighbourhoods of any density in F. An interesting feature of mixtures distributions having a parameter dependent support is that in many cases, the prior mass of sets the sets {f, KL(f_{0}, f) = +∞} is 1 for most f_{0} ∈ F.

References

Balabdaoui, F. and Wellner, J. A. (2007). Estimation of a k-monotone density: limit distribution theory and the spline connection. 35:2536–2564.

Ferguson, T. S. (1983). Bayesian density estimation by mixtures of normal distributions. pages 287–302.

Ghosal, S., Ghosh, J. K., and van der Vaart, A. W. (2000). Convergence rates of posterior distributions. Ann. Statist., 28(2):500–531.

Groeneboom, P. (1985). Estimating a monotone density. pages 539–555.

Khazaei, S., Rousseau, J., and Balabdaoui, F. (2010). Bayesian nonparametric inference of decreasing densities. In 42èmes Journées de Statistique, Marseille, France France.

Rivoirard, V. and Rousseau, J. (2009). On the bernstein von mises theorem for linear functionals of the density. Technical report, Université Paris Dauphine.

Rousseau, J. and Mengersen, K. (2011). Asymptotic behaviour of the posterior distribution in overfitted mixture models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(5):689–710.

Sun, J. and Woodroofe, M. (1996). Adaptive smoothing for a penalized NPMLE of a non-increasing density. J. Statist. Plann. Inference, 52(2):143–159.

van der Vaart, A. W. and Wellner, J. A. (1996). Weak convergence and empirical processes. Springer Series in Statistics. Springer-Verlag, New York. With applications to statistics.

Williamson, R. E. (1956). Multiply monotone functions and their laplace transforms. Duke Mathematical Journal, pages 189 – 207.

Wu, Y. and Ghosal, S. (2008). Kullback leibler property of kernel mixture priors in bayesian density estimation. Electron. J. Stat., 2:298–331.