NUS ST2334 Lecture Notes

ST2334 Probability and Statistics

A/P Ajay Jasra Office 06-18, S16 National University of Singapore E-Mail:[email protected]

Department of Statistics and Applied Probability National University of Singapore 6 Science Drive 2, Singapore, 117546

Contents 1 Introduction to Probability 1.1 Introduction . . . . . . . . . . . . . . 1.2 Probability Triples . . . . . . . . . . 1.2.1 Sample Space . . . . . . . . . 1.2.2 σ−Fields . . . . . . . . . . . 1.2.3 Probability . . . . . . . . . . 1.3 Conditional Probability . . . . . . . 1.3.1 Bayes Theorem . . . . . . . . 1.3.2 Theorem of Total Probability 1.4 Independence . . . . . . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

2 2 2 2 3 4 5 6 6 7

2 Random Variables and their Distributions 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . 2.2 Random Variables and Distribution Functions . . . 2.2.1 Random Variables . . . . . . . . . . . . . . 2.2.2 Distribution Functions . . . . . . . . . . . . 2.3 Discrete Random Variables . . . . . . . . . . . . . 2.3.1 Probability Mass Functions . . . . . . . . . 2.3.2 Independence . . . . . . . . . . . . . . . . . 2.3.3 Expectation . . . . . . . . . . . . . . . . . . 2.3.4 Dependence . . . . . . . . . . . . . . . . . . 2.3.5 Conditional Distributions and Expectations 2.4 Continuous Random Variables . . . . . . . . . . . . 2.4.1 Probability Density Functions . . . . . . . . 2.4.2 Independence . . . . . . . . . . . . . . . . . 2.4.3 Expectation . . . . . . . . . . . . . . . . . . 2.4.4 Dependence . . . . . . . . . . . . . . . . . . 2.4.5 Conditional Distributions and Expectations 2.4.6 Functions of Random Variables . . . . . . . 2.5 Convergence of Random Variables . . . . . . . . . 2.5.1 Convergence in Distribution . . . . . . . . . 2.5.2 Convergence in Probability . . . . . . . . . 2.5.3 Central Limit Theorem . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

8 8 8 8 8 9 9 10 11 13 15 17 18 18 19 21 24 26 31 31 32 33

3 Introduction to Statistics 3.1 Introduction . . . . . . . . . . . . . 3.2 Maximum Likelihood Estimation . 3.2.1 Introduction . . . . . . . . 3.2.2 The Method . . . . . . . . 3.2.3 Examples of Computing the 3.3 Hypothesis Testing . . . . . . . . . 3.3.1 Introduction . . . . . . . . 3.3.2 Constructing Test Statistics 3.4 Bayesian Inference . . . . . . . . . 3.4.1 Introduction . . . . . . . . 3.4.2 Bayesian Estimation . . . . 3.4.3 Examples . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

34 34 34 34 34 35 37 37 38 39 39 39 40

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . . . . . . . . . MLE . . . . . . . . . . . . . . . . . . . . . . . . . . . .

i

. . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

4 Miscellaneous Results 4.1 Set Theory . . . . . . . . 4.2 Derivatives and Functions 4.3 Summation . . . . . . . . 4.4 Exponential Function . . 4.5 Taylor Series . . . . . . . 4.6 Integration Methods . . . 4.7 Distributions . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

ii

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

43 43 43 44 45 45 45 46

1

Course Information Lecture times: This course is held from January 2015-April 2015. Lecture times are at 08001000 Monday and Wednesday at LT25. The notes are available on my website: http://www.stat. nus.edu.sg/~staja/ (also IVLE). You should register as usual for tutorials and for manual registration please email me with your name and matriculation number and I will attempt to put you into a convenient tutorial group. Office Hour: My office hour is at 1500 on Wednesday in 04-02, Department of Statistics and Applied Probability. I am not available at other times, unless there is a time-table clash. If you would like to see me or have a question, please email me or your tutors. I will try to respond as quickly as I can. Assessed Coursework: During the course there will be one assignment which makes up 40% of the final grade. The dates of when this assessment will be handed out will be provided at least two weeks beforehand and you are given two weeks to complete the assessment. The assessment is to be handed to me in person or, at the statistics office, S16, level 7. Due to the number of students on this course I do not accept assessments via e-mail (unless there are extreme circumstances, which would need to be verified by the department before-hand) or by student submission on IVLE (again there are too many students for this). There is NO mid-term examination. Exam:. There is a 2 hour final exam of 4 questions. No calculators are allowed and the examination is closed-book (with the exception of the below information). You will be given a formula sheet to assist you in the examination (Tables 4.1 and 4.2) and you may bring a single side of hand written A4 to the examination with any information you deem useful for the examination; you may write on both sides of the A4. Problem Sheets: There are also 10 non-assessed problem sheets available on my website (also IVLE). Typed solutions will be available on my website. Course Details: These notes are not sufficient to replace lectures. In particular, many examples and clarifications are given during the class. This course investigates the following concepts: Basic concepts of probability, conditional probability, independence, random variables, joint and marginal distributions, mean and variance, some common probability distributions, sampling distributions, estimation and hypothesis testing based on a normal population. There are three chapters of course content, the first covering the foundations of probability. We then move on to random variables and their distributions which features the main content of the course and is a necessary prerequisite for further work in statistical modeling and randomized algorithms. The third chapter is a basic introduction to statistics and gives some basic notions which would be used in statistical modeling. The final chapter of the notes provides some mathematical background for the course, which you are strongly advised to read in the first week. You are expected to know everything that is in this chapter. In particular the notion of double summation and integration is used routinely in this course and you should spend some time recalling these ideas. Some additional help on integration will be given later on in the course. References: The recommended reference for this course is [1] (Chapters 1-5). For a non-technical introduction the book [2] provides an entertaining and intuitive look at probability.

Chapter 1

Introduction to Probability 1.1

Introduction

This Chapter provides a basic introduction to the notions that underlie the basics of probability. The level of notes is below that required for complete mathematical rigor, but does provide a technical development of these ideas. Essentially, the basic ideas of probability theory start with a probability triple of an ‘event-space’ (e.g. in flipping a coin a head or tail), sets of events (e.g.‘tail’ or ‘tail and head’) and a way to compare the ‘likelihood’ of events by a probability distribution (the chance of obtaining a tail). Moving on from these notions, we consider the probability of events, given other events are known to have occurred (conditional probability) (the probability a coin lands tails, given we know it has two ‘heads’). Some events have probabilities which decouple in a special way and are called independent (for example, flipping a fair coin twice, the outcome of the first flip, may not influence that of the second). The structure of this Chapter is as follows: In Section 1.2, we discuss the idea of a probability triple; in Section 1.3 conditional probability is introduced and in Section 1.4 we discuss independence.

1.2

Probability Triples

As mentioned in the introduction, probability begins with a triple: • A sample space (possible outcomes) • A collection of sets on outcomes (σ−field) • A way to compare the likelihood of events (probability measure) We will slowly introduce and analyze these concepts.

1.2.1

Sample Space

The basic notion we begin with is that of an experiment, with a collection of possible outcomes to which we cannot (usually) determine exactly what will happen. For example, if we flip a coin, or if we watch a football game and so on; in general we do not know for certain what will happen. The idea of a sample space is as follows. Definition 1.2.1. The set of all possible outcomes of an experiment is called the sample space and is denoted by Ω. Example 1.2.2. Consider rolling a six sided fair die, once. There are 6 possible outcomes, and thus: Ω = {1, 2, . . . , 6}. We may be interested (for example, for betting purposes) in the following events: 1. we roll a 1 2. we roll an even number 3. we roll an even number, which is less than 3

2

3 Notation Ω ω A Ac A∩B A∪B A\B A∆B A⊆B ∅

Set Terminology Collection of objects Member of Ω Subset of Ω Complement of A Intersection Union Difference Symmetric difference Inclusion Empty set

Probability Terminology Sample Space Elementary event Event that an outcome in A occurs Event that no outcome in A occurs Both A and B A or B A but not B A or B but not both If A then B Impossible event

Table 1.1: Terminology of Set and Probability Theory 4. we roll an odd number One thing that we immediately realize is that each of the events in the above example are subsets of Ω, that is each event (1)-(4) correspond to: 1. {1} 2. {2,4,6} 3. {2} 4. {1,3,5} As a result, we think of events as subsets of the sample space Ω. Such events can be constructed by unions, intersections and complements of other events (we will formally explain why, below). For example, letting A = {2, 4, 6} in the case (2) above, we immediately yield that the event (4) is Ac . Similarly letting A be the event in (2) and B be the event of rolling a number less than 4 we obtain that event (3) is A ∩ B. For those of you that have forgotten basic set notations and terminologies, see Table1 1.1. In particular, we will think of Ω as the certain event (we will roll a 1, 2, . . . , 6) and its complement Ωc = ∅ as the impossible event (we must roll something).

1.2.2

σ−Fields

Now we have a notion of an event, in particular, that they are subsets of of Ω. A particular question of interest is then: are all subsets of Ω events? The answer actually turns out to be no, but the technical reasons for this lie far out of the scope of this course. We will content ourselves to use a particular collection of sets F (a ‘set of sets’) of Ω which contains all the events that we can make probability statements about. This collection of sets is a σ−field. Definition 1.2.3. A collection of sets F of Ω is called a σ-field if it satisfies the following conditions: 1. ∅ ∈ F 2. If A1 , . . . , ∈ F then

S∞

i=1

Ai ∈ F

3. If A ∈ F then Ac ∈ F T∞It can be shown that σ−fields are closed under countable intersections (i.e. if A1 , . . . , ∈ F then i=1 Ai ∈ F ). Whilst it may seem quite abstract, it will turn out that, for this course, all the sets we consider will lie in the σ−field F . Example 1.2.4. Consider flipping a single coin, letting H denote a head and T denoting a tail then: Ω = {H, T } and F = {Ω, ∅, {H}, {T }}. Thus, to summarize, so far (Ω, F ) is sample space and a σ−field. The former is the collection of all possible outcomes and the latter is collection of sets on Ω (the events) that follow Definition 1.2.3. Our next objective is define a way of assigning a likelihood to each event. 1 For

those of you that have forgotten set theory, Section 4.1 has a refresher

4

Exercise 1.2.5. You are given De-Morgan’s Laws: [ c \ \ c [ Ai = Aci Ai = Aci . i

i

i

Show that if A, B ∈ F , A ∩ B and A \ B are in F (just use the rules of σ-fields).

1.2.3

Probability

We now introduce a way to assign a likelihood to an event, via probability. One possible interpretation is the following: suppose my experiment is repeated many times, then the probability of any event is the limit of the ratio of times the event occurs over the number of events. We note that this is not the only interpretation of probability, but we do not diverge into a discussion of the philosophy of probability. Formally, we introduce the notion of a probability measure: Definition 1.2.6. A probability measure P on (Ω, F ) is a function P : F → [0, 1] which satisfies: 1. P(Ω) = 1 and P(∅) = 0 2. For A1 , A2 , . . . disjoint (∀i 6= j, Ai ∩ Aj = ∅) members of F then P

∞ [

∞ X Ai = P(Ai ).

i=1

i=1

The triple (Ω, F , P), comprising a set Ω, a σ−field F of subsets of Ω and a probability measure P, is called a probability space (or probability triple). The idea is to associate the probability space with an experiment: Example 1.2.7. Consider flipping a coin as in Example 1.2.4, with Ω = {H, T } and F = {Ω, ∅, {H}, {T }}. Then we can set: P({H}) = p P({T }) = 1 − p p ∈ [0, 1]. If p = 1/2 we say that coin is fair. Example 1.2.8. Consider rolling a 6-sided die, then Ω = {1, . . . , 6} and F = {0, 1}Ω (the power set of Ω the set of all subsets of Ω). Define a probability measure on (Ω, F ) as: X P(A) = pi ∀A ∈ F i∈A

where ∀i ∈ {1, . . . , 6}, 0 ≤ pi ≤ 1 and

P6

i=1

pi = 1. If pi = 1/6, ∀i ∈ {1, . . . , 6} then:

P(A) =

Card(A) 6

where Card(A) is the cardinality of A (the number of elements in the set A). We now consider a sequence of results on probability spaces. Lemma 1.2.9. We have the following properties on probability space (Ω, F , P): 1. For any A ∈ F , P(Ac ) = 1 − P(A). 2. For any A, B ∈ F , if A ⊆ B then, P(B) = P(A) + P(B \ A) ≥ P(A). 3. For any A, B ∈ F , P(A ∪ B) = P(A) + P(B) − P(A ∩ B). 4. (inclusion-exclusion formula) For any A1 , . . . , An ∈ F then: P

n [ i=1

n X \ X X Ai = P(Ai ) − P(Ai ∩ Aj ) + P(Ai ∩ Aj ∩ Ak ) − · · · + (−1)n+1 P Ai . i

where, for example,

i
P

i
=

i
Pn

j=1

Pj−1 i=1

etc.

i=1

5 Proof. For (1), A ∪ Ac = Ω, so 1 = P(Ω) = P(A ∪ Ac ) = P(A) + P(Ac ) (A ∩ Ac = ∅). Thus, one concludes by rearranging. For (2), as B = A ∪ (B \ A) and A ∩ (B \ A) = ∅, (recall Exercise 1.2.5) P(B) = P(A) + P(B \ A). The inequality is clear from the definition of a probability. For (3), A ∪ B = A ∪ (B \ A), which is a disjoint union, and B \ A = (B \ (A ∩ B)) hence P(A ∪ B) = P(A) + P(B \ A) = P(A) + P(B \ (A ∩ B)). Now (A ∩ B) ⊆ B so by (2) P(B \ (A ∩ B)) = P(B) − P(A ∩ B) and thus P(A ∪ B) = P(A) + P(B) − P(A ∩ B). For (4), this can be proved by induction and is an exercise. To conclude the Section, we introduce two terms: • An event A ∈ F is null if P(A) = 0. • An event A ∈ F occurs almost surely if P(A) = 1. Null events are not impossible. We will see this when considering random variables which take values in the real line.

1.3

Conditional Probability

We now move onto the notion of conditional probability. It takes the concept of a probability of an event, given another event is known to have occurred. For example, what is the probability that it rains today, given that it rained yesterday? We formalize the notion of conditional probability: Definition 1.3.1. Consider probability space (Ω, F , P) and let A, B ∈ F with P(B) > 0. Then the conditional probability that A occurs given B occurs is defined to be: P(A|B) :=

P(A ∩ B) . P(B)

Example 1.3.2. A family has two children of different ages. What is the probability that both children are boys, given that at least one is a boy? The older and younger child may be either boys or girl so the sample space is: Ω = {GG, BB, GB, BG} and we assume that all outcomes are equally likely P(GG) = P(BB) = P(GB) = P(BG) = 1/4 (the uniform distribution). We are interested in: P(BB|GB ∪ BB ∪ BG)

= = =

P(BB ∩ (GB ∪ BB ∪ BG)) P(GB ∪ BB ∪ BG) P(BB) P(GB ∪ BB ∪ BG) 1 1/4 = . 3/4 3

One can also ask the question: what is the probability that in a family of two children, where the younger of two is a boy, that both are boys? We want: P(BB|BG ∪ BB)

= = =

P(BB ∩ (BG ∪ BB)) P(BG ∪ BB) P(BB) P(BG ∪ BB) 1/4 1 = . 1/2 2

6

1.3.1

Bayes Theorem

An important result in conditional probability is Bayes theorem. Theorem 1.3.3. Consider probability space (Ω, F , P) and let A, B ∈ F with P(A), P(B) > 0. Then we have: P(A|B)P(B) P(B|A) = . P(A) Proof. By definition P(A ∩ B) . P(A)

P(B|A) =

As P(B) > 0, P(A ∩ B) = P(A|B)P(B) and hence: P(B|A) =

1.3.2

P(A|B)P(B) . P(A)

Theorem of Total Probability

We will use Bayes Theorem after introducing the following result (the Theorem of Total Probability. We begin with a preliminary definition: Definition 1.3.4. A family of sets B1 , . . . , Bn is called a partition of Ω if: ∀i 6= j Bi ∩ Bj = ∅

and

n [

Bi = Ω.

i=1

Lemma 1.3.5. Consider probability space (Ω, F , P) and let for any fixed n ≥ 2, B1 , . . . , Bn ∈ F be a partition of Ω, with P(Bi ) > 0, ∀i ∈ {1, . . . , n}. Then, for any A ∈ F : P(A) =

n X

P(A|Bi )P(Bi ).

i=1

Proof. We give the proof for n = 2, the other cases being very similar. We have A = (A∩B1 )∪(A∩B2 ) (recall B1 ∪ B2 = Ω, and B2 = B1c ), so P(A)

=

P(A ∩ B1 ) + P(A ∩ B2 )

=

P(A|B1 )P(B1 ) + P(A|B2 )P(B2 ).

Example 1.3.6. Consider two factories which manufacture a product. If the product comes from factory I, it is defective with probability 1/5 and if it is from factory II, it is defective with probability 1/20. It is twice as likely that a product comes from factory I. What is the probability that a given product operates properly (i.e. is not defective)? Let A denote the event of ‘operates properly’ and B denote the event ‘the product is made in factory I’. Then: P(A)

= =

P(A|B)P(B) + P(A|B c )P(B c ) 51 4 2 19 1 + = . 5 3 20 3 60

If a product is defective, what is the probability that it came from factory I? This is P(B|Ac ); by Bayes Theorem P(B|Ac )

= =

P(Ac |B)P(B) P(Ac ) (1/5)(2/3) 8 = . 9/60 9

7

1.4

Independence

The idea of independence is roughly that the probability of an event A is ‘unaffected’ by the occurrance of a (non-null) event B; that is, P(A) = P(A|B), when P(B) > 0. More formally: Definition 1.4.1. Consider probability space (Ω, F , P) and let A, B ∈ F . A and B are independent if P(A ∩ B) = P(A)P(B). More generally, a family of F −sets A1 , . . . , An (∞ > n ≥ 2) are independent if P

n \ i=1

n Y Ai = P(Ai ). i=1

Two important concepts include pairwise independence: P(Ai ∩ Aj ) = P(Ai )P(Aj ) ∀i 6= j. This does not necessarily mean that A1 , . . . , An are independent events. Another important concept is conditional independence; given an event C ∈ F , with P(C) > 0 , A, B ∈ F are conditionally independent events if P(A ∩ B|C) = P(A|C)P(B|C). This can be extended to families of sets Ai given C. Example 1.4.2. Consider a pack of playing cards, and one chooses a card completely at random (i.e. no card counting etc). One can assume that the probability of choosing a suit (e.g. spade) is independent of its rank. So, for example: P(spade king) = P(spade)P(king) =

1 13 4 = . 52 52 52

Chapter 2

Random Variables and their Distributions 2.1

Introduction

Throughout the Chapter we assume that there is a probability space (Ω, F , P), but its presence is minimized from the discussion. This is the main Chapter of the course and we focus upon random variables and their distribution (Section 2.2). In particular, a random variable is (essentially) just a map from Ω to some subset of the real line. Once we are there we introduce notions of probability through distribution functions. Sometimes the random variables take values on a countable space (possibly countably infinite) and we call such random variables discrete (Section 2.3). The probabilities of such random variables are linked to probability mass functions and we use this concept to revisit independence and conditional probability. We also consider expectation (the ‘theoretical average’) and conditional expectation for discrete random variables. These ideas are revisited when the random variables are continuous (Section 2.4). The Chapter is concluded when we discuss the convergence of random variables (Section 2.5) and, in particular, the famous central limit theorem.

2.2 2.2.1

Random Variables and Distribution Functions Random Variables

In general, we are often concerned by an experiment itself, but by an associated consequence of the outcome. For example, a gambler is often interested in his or hers profit or loss, rather that in the result that occurs. To deal with this issue, we introduce the notion of a random variable: Definition 2.2.1. A random variable is a function X : Ω → R such that for each x ∈ R, {ω ∈ Ω : X(ω) ≤ x} ∈ F . Such a function is said to be F −measurable. For the purpose of this course, the technical constraint of F −measurability can be ignored; you can content yourself to think of X(ω) as a mapping from the sample space to the real line and continue onwards. In general we omit the argument ω and just write X, with possible realized values (numbers) written in lower-case x. The distinction between a random variable X and its realized value is very important and you should maintain this convention. To move from the random variable, back to our probability measure P, we write events {ω ∈ Ω : X(ω) ≤ x} as {X ≤ x}, and hence we write P({X ≤ x}) as P(X ≤ x) to mean P({ω ∈ Ω : X(ω) ≤ x}) (recall {ω ∈ Ω : X(ω) ≤ x} ∈ F and so this makes sense). This leads us to the notion of a distribution function:

2.2.2

Distribution Functions

Definition 2.2.2. The distribution function of a random variable X is the function F : R → [0, 1] given by F (x) = P(X ≤ x), x ∈ R. Example 2.2.3. Consider flipping a fair coin twice; then Ω = {HH, T T, HT, T H}, with F = {0, 1}Ω and P(HH) = P(T T ) = P(HT ) = P(T H) = 1/4. Define a random variable X as the number of heads; so X(HH) = 2 X(HT ) = X(T H) = 1 X(T T ) = 0. 8

9

The associated distribution function is:

F (x) =

 0    1   

4 3 4

1

if if if if

x<0 0≤x<1 1≤x<2 x≥2

A distribution function has the following properties, which we state without proof. See [1] for the proof. The lemma characterizes a distribution function: a function F is a distribution function for some random variable if and only if it satisfies the three properties in the following lemma. Lemma 2.2.4. A distribution function F has the following properties: 1. limx→−∞ F (x) = 0, limx→∞ F (x) = 1, 2. If x < y then F (x) ≤ F (y), 3. F is right continuous; limδ↓0 F (x + δ) = F (x) for each x ∈ R. Example 2.2.5. Consider flipping a coin as in Example 1.2.4, which lands heads with probability p ∈ (0, 1). Let X : Ω → R be given by X(H) = 1

X(T ) = 0.

The associated distribution function is   0 1−p F (x) =  1

if x < 0 if 0 ≤ x < 1 if x ≥ 1

X is said to have a Bernoulli distribution. We end the Section with another lemma, whose proof can be found in [1]. Lemma 2.2.6. A distribution function F of random variable X, has the following properties: 1. P(X > x) = 1 − F (x) for any fixed x ∈ R, 2. P(x < X ≤ y) = F (y) − F (x) for any fixed x, y ∈ R with x < y, 3. P(X = x) = F (x) − limy↑x F (y) for any fixed x ∈ R.

2.3 2.3.1

Discrete Random Variables Probability Mass Functions

We now move onto an important class of random variable called discrete random variables. Definition 2.3.1. A random variable is said to be discrete if it takes values in some countable subset X = {x1 , x2 , . . . } of R. A discrete random variable takes values only at countable values and hence its distribution is a jump function, in that it shifts between different values when they change. An important concept is the probability mass function (PMF): Definition 2.3.2. The probability mass function of a discrete random variable X, is the function f : X → [0, 1] defined by f (x) = P(X = x). Remark 2.3.3. We generally use sans-serif notation X, Z to denote the supports of a random variable. That is the range of values for which its PMF (or PDF - as will be defined much later on) is (potentially) non-zero. Note however, the PMF may be defined on say Z or R but is taken as (potentially) non-zero only at those points in X it is always zero outside X. We call this the support of a random variable.

10

We remark that F (x) =

X

f (x) f (x) = F (x) − lim F (y) y↑x

i:xi ≤x

which provides and association between the distribution function and the PMF. We have the following result, whose proof follows easily from the above definitions and results. Lemma 2.3.4. A PMF satisfies: 1. the set of x such that f (x) 6= 0 is countable, P 2. x∈X f (x) = 1. Example 2.3.5. A coin is flipped n times and the probability one obtains a head is p ∈ (0, 1); Ω = {H, T }n . Let X denote the number of heads, which takes values in the set X = {0, 1, . . . , n} and is thus a discrete random variable. Consider x ∈ X, exactly nx points in Ω give us x heads and each point occurs with probability px (1 − p)n−x ; hence n x f (x) = p (1 − p)n−x x ∈ X. x The random variable X is said to have a Binomial distribution and this is denoted X ∼ B(n, p). Note that n! n = x (n − x)!x! with x! = x × (x − 1) × · · · × 1 and 0! = 1. Example 2.3.6. Let λ > 0 be given. A random variable X that takes values on X = {0, 1, 2, . . . , } is said have a Poisson distribution with parameter λ (denoted X ∼ P(λ)) if its PMF is: f (x) =

2.3.2

λx e−λ x!

x ∈ X.

Independence

We now extend the notion of independence of events to the domain of random variables. Recall that events A and B are independent if the joint probability is equal to the product (A does not affect B). Definition 2.3.7. Discrete random variables X and Y are independent if the events {X = x} and {Y = y} are independent for each (x, y) ∈ X × Y. To understand this idea, let X = {x1 , x2 , . . . }, Y = {y1 , y2 , . . . }, then X and Y are independent if and only if the events Ai = {X = xi }, Bj = {Y = yj } are independent for every possible pair i, j. Example 2.3.8. Consider flipping a coin once, which lands tail with probability p ∈ (0, 1). Let X be the number of heads seen and Y the number of tails. Then: P(X = Y = 1) = 0 and P(X = 1)P(Y = 1) = (1 − p)p 6= 0 so X and Y cannot be independent. A useful result (which we do not prove) that is worth noting is the following. Theorem 2.3.9. If X and Y are independent random variables and g : X → R, h : Y → R, then the random variables g(X) and h(Y ) are also independent. In full generality1 , consider a sequence of discrete random variables X1 , X2 , . . . , Xn , Xi ∈ Xi ; they are said to be independent if the events {X1 = x1 }, . . . , {Xn = xn } are independent for every possible (x1 , . . . , xn ) ∈ X1 × · · · × Xn . That is: P(X1 = x1 , . . . , Xn = xn ) =

n Y

P(Xi = xi ) ∀(x1 , . . . , xn ) ∈ X1 × · · · × Xn .

i=1 1 this

point will not make too much sense, until we see Section 2.3.4

11

2.3.3

Expectation

Throughout your statistical training, you may have encountered the notion of an average or mean value of data. In this Section we consider the idea of the average value of a random variable, which is called the expected value. Definition 2.3.10. The expected value of a discrete random variable X on X, with PMF f , denoted E[X] is defined as X E[X] := xf (x) x∈X

whenever the sum on the R.H.S. is absolutely convergent. Example 2.3.11. Recall the Poisson random variable from Example 2.3.6, X ∼ P(λ). The expected value is: E[X]

∞ X λx e−λ x x! x=0

=

= λe−λ

∞ X λx−1 (x − 1)! x=1

= λe−λ eλ = λ where we have used the Taylor series expansion for the exponential function on the third line. We now look at the idea of an expectation of a function of a random variable: Lemma 2.3.12. Given a discrete random variable X on X, with PMF f and g : X → R: X E[g(X)] = g(x)f (x) x∈X

whenever the sum on the R.H.S. is absolutely convergent. Example 2.3.13. Returning to Example 3.2.1, we have 2

E[X ]

=

∞ X

x2

x=0

=

λe−λ

=

λe−λ

=

λe−λ

= =

λx e−λ x!

∞ X x=1 ∞ X

x

λ(x−1) (x − 1)!

d x 1 (λ ) dλ (x − 1)! x=1

∞ i 1 d h X x−1 λ (λ ) dλ (x − 1)! x=1

d (λeλ ) dλ λ2 + λ λe−λ

where we have assumed that it is legitimate to swap differentiation and summation (it turns out that this is true here). An important concept is the moment generating function:. Definition 2.3.14. For a discrete random variable X the moment generating function (MGF) is X M (t) = E[eXt ] = ext f (x) t ∈ T x∈X

where T is the set of t for which

P

xt

X

e f (x) < ∞.

12

Exercise 2.3.15. Show that E[X] = M 0 (0)

E[X 2 ] = M (2) (0)

when the right-hand derivatives exist. The moment generating function is thus a simple way to obtain moments, if it is simple to differentiate M (t). Note also, that it can be proven that the MGF uniquely characterizes a distribution. Example 2.3.16. Let X ∼ P(λ) then: E[eXt ]

= = =

∞ X

ext

x=0 ∞ X

λx −λ e x!

(λet )x −λ e x! x=0 exp{λ(et − 1)}.

Then M 0 (0) = λ. An important special case of functions of random variables are: Definition 2.3.17. Given a discrete random variable X on X, with PMF f and k ∈ Z+ , the k th moment of X is E[X k ] and the k th central moment of X is E[(X − E[X])k ]. Of particular importance, is the idea of the variance: Var[X] := E[(X − E[X])2 ]. Now, we have the following calculations: X E[(X − E[X])2 ] = (x − E[X])2 f (x) x∈X

=

X

(x2 − 2xE[X] + E[X]2 )f (x)

x∈X

=

X

x2 f (x) − 2E[X]

x∈X

X

xf (x) + E[X]2

x∈X

X

f (x)

x∈X 2

= E[X 2 ] − 2E[X]2 + E[X]2 = E[X 2 ] − E[X] . Hence we have shown that Var[X] = E[X 2 ] − E[X]2 .

(2.3.1) P

Note a very important point: for any absolutelyPconvergent sum an , if an ≥ 0 for each n, then P an ≥ 0. Now as Var[X] := E[(X − E[X])2 ] = x∈X (x − E[X])2 f (x) and clearly (x − E[X])2 f (x) ≥ 0

∀x ∈ X

it follows that Variances cannot be negative2 . Example 2.3.18. Returning to Examples 3.2.1 and 2.3.13, when X ∼ P(λ), we have shown that E[X 2 ] = λ2 + λ

E[X] = λ.

Hence on using (2.3.1): Var[X] = λ. That is, for a Poisson random variable, its mean and variance are equal. 2 This

is very important and you will not be told again

13

Exercise 2.3.19. Compute the mean and variance for a Binomial distribution B(n, p) random variable. [Hint: consider the differentiation, w.r.t. x, of the equality n X n k x = (1 + x)n . k k=0

]. We now state a Theorem, whose proof we do not give. In general, it is simple to establish, once the concept of a joint distribution has been introduced; we have not done this, so we simply state the result. Theorem 2.3.20. The expectation operator has the following properties: 1. if X ≥ 0, E[X] ≥ 0 2. if a, b ∈ R then E[aX + bY ] = aE[X] + bE[Y ] 3. if X = c ∈ R always, then E[X] = c. An important result that we will use later on and whose proof is omitted is as follows. Lemma 2.3.21. If X and Y are independent then E[XY ] = E[X]E[Y ]. A notion of dependence (linear dependence) is correlation. This will be discussed in details, later on. Definition 2.3.22. X and Y are uncorrelated if E[XY ] = E[X]E[Y ]. It is important to remark that random variables that are independent are uncorrelated. However, uncorrelated variables are not necessarily independent; we will explore this idea later on. We end the section with some useful properties of the variance operator. Theorem 2.3.23. For random variables X and Y 1. For a ∈ R, Var[aX] = a2 Var[X], 2. For X, Y uncorrelated Var[X + Y ] = Var[X] + Var[Y ]. Proof. For 1. we have: Var[aX]

= E[(aX)2 ) − E[aX]2 = a2 E[X 2 ] − a2 E[X]2 = a2 Var[X]

where we have used Theorem 2.3.20 2. in the second line. Now for 2. we have: Var[(X + Y )]

= E[(X + Y − E[X + Y ])2 ) = E[X 2 + Y 2 + 2XY + E[X + Y ]2 − 2(X + Y )E[X + Y ]] = E[X 2 + Y 2 + 2XY + E[X]2 + E[Y ]2 + 2E[XY ]] − 2E[X + Y ]2 = E[X 2 ] + E[Y 2 ] + 4E[X]E[Y ] + E[X]2 + E[Y ]2 − 2(E[X]2 + E[Y ]2 + 2E[X]E[Y ]) = E[X 2 ] + E[Y 2 ] − E[X]2 − E[Y ]2 = Var[X] + Var[Y ]

where we have repeatedly used Theorem 2.3.20 2..

2.3.4

Dependence

As we saw in the previous Section, there as a need to define distributions on more than one random variable. We will do that in this Section. We start with the following definition (of which one can easily generalize to a collection of n ≥ 1 discrete random variables): Definition 2.3.24. The joint distribution function F : R2 → [0, 1] of X, Y where X and Y are discrete random variables, is given by F (x, y) = P(X ≤ x ∩ Y ≤ y). 2

Their joint mass function f : R → [0, 1] is given by f (x, y) = P(X = x ∩ Y = y).

14 y = −1

y=0

y=2

f (x)

1 18 2 18

3 18

2 18 3 18 3 18 8 18

6 18 5 18 7 18

x=1 x=2 x=3 f (y)

0 4 18 7 18

0 3 18

Table 2.1: The Joint Mass Function in Example 2.3.26. The row totals are the marginal mass function on X (and sum to 1) and respectively the column totals are the marginal mass function on Y. In general, it may be that the random variables are defined on a space Z which may not be decomposable into a cartesian product X × Y. In such scenarios we P writePthe joint support as simply Z and omit X and Y, in this scenario, we will use the notation x or y to denote sums over the supports induced by Z. This concept will be clarified during a reading of the subsequent text. Of particular importance, are the marginal PMFs of X and Y : X X f (x) = f (x, y) f (y) = f (x, y). y

x

Note that, clearly X

f (x) = 1 =

x

X

f (y).

y

Example 2.3.25. Consider Theorem 2.3.20 2. and for simplicity suppose Z = X × Y. Now, we have XX E[aX + bY ] = (ax + by)f (x, y) x∈X y∈Y

=

X

ax

x∈X

= a

X

X

f (x, y) +

y∈Y

X

by

y∈Y

xf (x) + b

x∈X

X

X

f (x, y)

x∈X

yf (y)

y∈Y

= aE[X] + bE[Y ]. Example 2.3.26. Suppose X = {1, 2, 3} and Y = {−1, 0, 2} then an example of a joint PMF can be found in Table 2.3.26. From the table, we have: E[XY ] =

XX

xyf (x, y) =

x∈X y∈Y

29 18

(just sum the 9 values in the table, multiplying each time by the product of the associated x and y. Similarly 5 7 37 6 +2 +3 = E[X] = 18 18 18 18 and 7 8 13 3 = . E[Y ] = −1 + 0 + 2 18 18 18 18 We now formalize independence in a result, which provides us a more direct way to ascertain if two random variables X and Y are independent. Lemma 2.3.27. The discrete random variables X and Y are independent if an only if f (x, y) = f (x)f (y)

∀x, y ∈ R.

More generally, X and Y are independent if and only if f (x, y) factorizes into the product g(x)h(y) with g a function only of x and h a function only of y. Example 2.3.28. Consider the joint PMF: f (x, y) =

λx e−λ λy e−λ x! y!

X = Y = {0, 1, . . . , }.

15

Now clearly f (x) =

λx e−λ x∈X x!

f (y) =

λy e−λ y ∈ Y. y!

It is also clear that via Lemma 2.3.27 the random variables X and Y are independent and identically distributed (and Poisson distributed). As in the case of a single variable, we are interested in the expectation of a functional of two random variables (strictly, in the proof of Theorem 2.3.23 we have already used the following result): P Lemma 2.3.29. E[g(X, Y )] = (x,y)∈X×Y g(x, y)f (x, y). Recall that in the previous Section, we mentioned a notion of dependence called correlation. In order to formally define this concept, we introduce first the covariance and then correlation. Definition 2.3.30. The covariance between X and Y is Cov[X, Y ] := E[(X − E[X])(Y − E[Y ])]. The correlation between X and Y is ρ(X, Y ) = p

Cov[X, Y ] Var[X]Var[Y ]

.

Exercise 2.3.31. Show that E[(X − E[X])(Y − E[Y ])] = E[XY ] − E[X]E[Y ]. Thus, independent random variables have zero correlation. Example 2.3.32. Returning to Example 2.3.26, we have: Var[X] = 233/324

Var[Y ] = 461/324

and Cov[X, Y ] = 29/18 − (37/18)(13/18) = 41/324. Thus

√ ρ(X, Y ) = 41/ 107413.

Some useful facts about correlations that we do not prove: • |ρ(X, Y )| ≤ 1. Random variables with correlation 1 or -1 are said to be perfectly positively or negatively correlated. • The correlation coefficient is 1 iff Y increases linearly with X and -1 iff Y decreases linearly as X increases. When we consider continuous random variables an example of uncorrelated random variables that are not independent, will be given.

2.3.5

Conditional Distributions and Expectations

In Section 1.3 we discussed the idea of conditional probabilities associated to events. This idea can then be extended to discrete-valued random variables: Definition 2.3.33. The conditional distribution function of Y given X, written FY |x (·|x), is defined by Fy|x (y|x) = P(Y ≤ y|X = x) for any x with P(X = x) > 0. The conditional PMF of Y given X = x is defined by f (y|x) = P(Y = y|X = x) when x is such that P(X = x) > 0.

16

We remark that, in particular, the conditional mass function of Y given X = x is: f (y|x) =

f (x, y) f (x, y) =P . f (x) y f (x, y)

Example 2.3.34. Suppose that Y ∼ P(λ) and X|Y = y ∼ B(y, p). Find the conditional probability mass function of Y |X = x. Note that the random variables lie in the space Z = {(x, y) : y ∈ {0, 1, . . . }, x ∈ {0, 1, . . . , y}}. We have f (y|x) =

f (x, y) f (x|y)f (y) = f (x) f (x)

(x, y) ∈ Z

which is a version of Bayes Theorem for discrete random variables. Note that for (x, y) ∈ Z all the PMFs above are positive. Now for x ∈ {0, 1, . . . } ∞ X y x λy e−λ f (x) = p (1 − p)y−x y! x y=x ∞ e−λ X ((1 − p)λ)y−x x! y=x (y − x)!

=

(λp)x

=

(λp)x −λp e . x!

Thus, we have for y ∈ {x, x + 1, . . . }: f (y|x) =

y x

y −λ

px (1 − p)y−x λ

e y!

(λp)x −λp x! e

which after some algebra, becomes: f (y|x) =

(λ(1 − p))y−x e−λ(1−p) (y − x)!

y ∈ {x, x + 1, . . . }.

Given the idea of a conditional distribution, we move onto the idea of a conditional expectation: Definition 2.3.35. The conditional expectation of a random variable Y , given X = x is X E[Y |X = x] = yf (y|x) y

given that the conditional PMF is well-defined. We generally write E[Y |X] or E[Y |x]. An important result associated to conditional expectations is as follows: Theorem 2.3.36. The conditional expectation satisfies: E[E[Y |X]] = E[Y ] assuming the expectations all exist. Proof. We have E[Y ]

=

X

yf (y)

y

=

X

yf (x, y)

(x,y)∈Z

=

X

yf (y|x)f (x)

(x,y)∈Z

=

XX [ yf (y|x)]f (x) x

y

= E[E[Y |X]].

17

Example 2.3.37. Let us return to Example 2.3.34. Find E[X|Y ], E[Y |X] and E[X]. From Exercise 2.3.19, you should have derived that if Z ∼ B(n, p) then E[Z] = np. Then E[X|Y = y] = yp. Thus E[X] = E[E[X|Y = y]] = pE[Y ]. As Y ∼ P(λ) E[X] = pλ. From Example 2.3.34, f (y|x) =

(λ(1 − p))y−x e−λ(1−p) (y − x)!

Thus E[Y |X = x] =

y ∈ {x, x + 1, . . . }.

∞ X (λ(1 − p))y−x e−λ(1−p) y . (y − x)! y=x

Setting u = y − x in the summation, one yields E[Y |X = x]

∞ X

=

(u + x)

u=0 ∞ X

=

u

u=0

(λ(1 − p))u e−λ(1−p) u!

(λ(1 − p))u e−λ(1−p) +x u!

= λ(1 − p) + x. Here, we have used the fact that U ∼ P(λ(1 − p)). We end the Section with a result which can be very useful in practice. Theorem 2.3.38. We have, for any g : R → R E[E[Y |X]g(X)] = E[Y g(X)] assuming the expectations all exist. Proof. We have E[Y g(X)]

=

X

yg(x)f (x, y)

(x,y)∈Z

=

X

yg(x)f (y|x)f (x)

(x,y)∈Z

=

X x

X g(x)[ yf (y|x)]f (x) y

= E[E[Y |X]g(X)].

2.4

Continuous Random Variables

In the previous Section, we considered random variables which produce real numbers within a countably finite or infinite set. However, there are of course a wide variety of applications where one sees numerical outcomes of an experiment that can lie potentially anywhere on the real-line. Thus we extend each of concepts that we saw for discrete random variables to continuous ones. In a rather informal (and incorrect) manner, one can simply think of the idea of replacing summation with integration; of course things will be more challenging than this, but one should keep this idea in the back of your mind.

18

2.4.1

Probability Density Functions

A random variable is said to be continuous if its distribution function F (x) = P(X ≤ x), can be written as Z x F (x) = f (u)du −∞

where f : R → [0, ∞), the R.H.S. is the usual Riemann integral and we will assume that the R.H.S. is differentiable. Definition 2.4.1. The function f is called the probability density function (PDF) of the continuous random variable X. We note that, under our assumptions, f (x) = F 0 (x). Example 2.4.2. One of the most commonly used PDFs is the Gaussian or normal distribution: f (x) =

1 1 √ exp{− 2 (x − µ)2 } 2σ σ 2π

x∈X=R

where µ ∈ R, σ 2 > 0. We use the notation X ∼ N (µ, σ 2 ). One of the key points associated to PDFs is as follows. The numerical value f (x) does not represent the probability that X takes the value x. The technical explanation goes far beyond the mathematical level of this course, but perhaps an intuitive reason is simply that there are uncountably infinite points in X (so assigning a probability to each point is seemingly impossible). In general, one assigns probability to sets of ‘non-zero width’. For example let A = [a, b], −∞ < a < b < ∞, then one might expect: Z P(X ∈ A) = f (x)dx. A

Indeed this holds true, but we are deliberately vague about this. We give the following result, which is not proved and should be taken as true. Lemma 2.4.3. If X has a PDF f , then R 1. X f (x)dx = 1. 2. P(X = x) = 0 for each x ∈ X. Rb 3. P(X ∈ [a, b]) = a f (x)dx, −∞ ≤ a < b ≤ ∞. Example 2.4.4. Returning to Example 2.4.2, we have Z ∞ 1 1 √ exp{− 2 (x − µ)2 }dx P(X ∈ X) = 2σ −∞ σ 2π Z ∞ 2 1 = √ e−u /2 du = 1. 2π −∞ Here, we have used the substitution u = (x − µ)/σ to go to the second line and the fact that √ R ∞ −u2 /2 e du = 2π. −∞

2.4.2

Independence

To define an idea of independence for continuous random variables, we cannot use the one for discrete random variables (recall Definition 2.3.7); the sets {X = x} and {Y = y} have zero probability and are hence trivially independent. Thus we use a new definition for independence: Definition 2.4.5. The random variables X and Y are independent if {X ≤ x}

{Y ≤ y}

are independent events for each x, y ∈ R. As for PMFs one can consider independence through PDFs; however, as we are yet to define the notion of joint PDFs, we will leave this idea for now. We note that one can show that for any real valued functions h and g (at least within some technical conditions which are assumed in this course) that if X and Y are independent, so are the random variables h(X) and g(Y ). We do not prove why.

19

2.4.3

Expectation

As for discrete random variables one can consider the idea of the average of a random variable. This is simply brought about by replacing summations with integrations: Definition 2.4.6. The expectation of a continuous random variable X with PDF f is given by Z E[X] = xf (x)dx X

whenever the integral exists. Example 2.4.7. Consider the exponential density: f (x) = λe−λx

x ∈ X = [0, ∞)

We use the notation X ∼ E(λ). Then Z ∞ Z E[X] = xλe−λx dx = [−xe−λx ]∞ + 0 0

0

∞

λ > 0.

1 1 e−λx dx = [− e−λx ]∞ 0 = λ λ

where we have used integration by parts. Example 2.4.8. Let us return to Example 2.4.2. Then Z ∞ 1 1 E[X] = x √ exp{− 2 (x − µ)2 }dx 2σ −∞ σ 2π Z ∞ 2 1 = (uσ + µ) √ e−u /2 du 2π −∞ Z ∞ 1 −u2 /2 = σ u√ e du + µ 2π −∞ 2 σ = √ [−e−u /2 ]∞ −∞ + µ 2π = µ. Here, we have used the substitution u = (x − µ)/σ to go to the second line and the fact that Z ∞ 2 1 √ e−u /2 du = 1 2π −∞ to go to the third. Example 2.4.9. Consider the gamma density: f (x) =

λα α−1 −λx x e Γ(α)

x ∈ X = [0, ∞)

λ, α > 0

R∞ where Γ(α) = 0 tα−1 e−t dt. We use the notation X ∼ G(α, λ) and note that if X ∼ G(1, λ) then X ∼ E(λ). Now Z ∞ λα E[X] = xα e−λx dx Γ(α) 0 Z ∞ λα 1 = uα e−u du Γ(α) λα+1 0 1 α = Γ(α + 1) = . λΓ(α) λ Where we have used the substitution u = λx to go the second line and Γ(α + 1) = αΓ(α) on the final line (from herein you may use that identity without proof ). We now state a useful technical result Theorem 2.4.10. If X and g(x) (g : R → R) are continuous random variables Z E[g(X)] = g(x)f (x)dx. X

20

We remark that all the extensions to expectations discussed in Section 2.3.3 can be extended to the continuous case. In particular Definitions 2.3.17 and 2.3.22 can be imported to the continuous case. In addition the results: Theorems 2.3.20 and 2.3.23 and Lemma 2.3.21 all can be extended. It is assumed that this is the case from herein. Example 2.4.11. Let us return to Example 2.4.7 Z ∞ E[X 2 ] = x2 λe−λx dx 0

= =

[−x2 e−λx ]∞ 0

2 + λ

Z

∞

xλe−λx dx

0

2 . λ2

Thus, for an exponential random variable: Var[X] =

1 . λ2

Example 2.4.12. Let us return to Example 2.4.2. Then Z ∞ 1 1 E[X 2 ] = x2 √ exp{− 2 (x − µ)2 }dx 2σ σ 2π −∞ Z ∞ 2 1 = (uσ + µ)2 √ e−u /2 du 2π −∞ Z ∞ Z ∞ 2 2 1 1 = σ2 u2 √ e−u /2 du + µ2 + 2σµ u √ e−u /2 du 2π 2π −∞ −∞ Z ∞ 2 2 σ2 = √ {[−ue−u /2 ]∞ e−u /2 du} + µ2 −∞ + 2π −∞ σ2 √ 2π + µ2 . = √ 2π √ R∞ R∞ 2 2 Here we have used −∞ u √12π e−u /2 du = 0 and −∞ e−u /2 du = 2π. Thus, for a normal random variable: Var[X] = σ 2 . Example 2.4.13. Let us return to Example 2.4.9 Now Z ∞ λα xα+1 e−λx dx E[X 2 ] = Γ(α) 0 Z ∞ 1 λα uα+1 e−u du = Γ(α) λα+2 0 1 α(α + 1) Γ(α + 2) = = 2 λ Γ(α) λ2 where we have used Γ(α + 2) = (α + 1)αΓ(α). Thus, for a gamma random variable: Var[X] =

α . λ2

To end this section we consider an important concept; the MGF for continuous random variables. Definition 2.4.14. For a continuous random variable X the moment generating function (MGF) is Z M (t) = E[eXt ] = ext f (x)dx t ∈ T X

where T is the set of t for which

xt

R X

e f (x)dx < ∞.

Exercise 2.4.15. Show that E[X] = M 0 (0) when the right-hand derivatives exist.

E[X 2 ] = M (2) (0)

21

As for discrete random variables, the moment generating function is a simple way to obtain moments, if it is simple to differentiate M (t). Note also, that it can be proven that the MGF uniquely characterizes a distribution. Example 2.4.16. Suppose X ∼ E(λ) then, supposing λ > t Z ∞ M (t) = λe−x(λ−t) dx 0

= =

i∞ λ e−x(λ−t) (λ − t) 0 λ . (λ − t)

h

−

Clearly M 0 (t) =

λ (λ − t)2

and thus E[X] = 1/λ. Example 2.4.17. Suppose X ∼ N (µ, σ 2 ) then Z ∞ 1 1 √ M (t) = exp{− 2 (x − µ)2 + xt}dx 2σ σ 2π −∞ Z ∞ 1 1 √ = exp{− 2 [(x − (µ + tσ 2 ))2 − (µ + tσ 2 )2 + µ2 ]}dx 2σ σ 2π −∞ Z ∞ 1 σ 2 t2 1 √ exp{µt + = } exp{− 2 (x − (µ + tσ 2 ))2 }dx 2 2σ σ 2π −∞ =

exp{µt +

σ 2 t2 } 2

where we have used a change-of-variables u = (x − (µ + tσ 2 ))/σ to deal with the integral.

2.4.4

Dependence

Just as for discrete random variables, one can consider the idea of joint distributions for continuous random variables. Definition 2.4.18. The joint distribution function of X and Y is the function F : R2 → [0, 1] given by F (x, y) = P(X ≤ x, Y ≤ y). Again, as for discrete random variables, one requires a PDF: Definition 2.4.19. The random variables are jointly continuous with joint PDF f : R2 → [0, ∞) if Z y Z x F (x, y) = f (u, v)dudv −∞

−∞

for each (x, y) ∈ R2 . In this course, we will assume (generally) that f (x, y) =

∂2 F (x, y). ∂x∂y

Note that, P(X = x, Y = y) = 0 and for sufficiently well defined sets A ⊆ Z ⊆ R2 Z P((X, Y ) ∈ A) = f (x, y)dxdy A

note that the R.H.S. is a double integral, but we will only write one integral sign in such contexts. As for discrete random variables, we can introduce the idea of marginal PDFs. Here, we take a little longer to consider these ideas:

22

Definition 2.4.20. The marginal distribution functions of X and Y are F (x) = lim F (x, y)

F (y) = lim F (x, y).

y→∞

As

Z

x

Z

x→∞

∞

Z f (u, y)dydu

F (x) =

Z

f (x, u)dxdu −∞

the marginal density functions of X and Y Z ∞ f (x) = f (x, y)dy

∞

F (y) =

−∞

−∞

y

Z

∞

f (y) =

−∞

−∞

f (x, y)dx. −∞

We remark that expectation is much the same for joint distributions (as for Section 2.4.3) of continuous random variables: Z Z ∞Z ∞ E[g(X, Y )] = g(x, y)f (x, y)dxdy. g(x, y)f (x, y)dxdy = −∞

Z

−∞

So as before if g(x, y) = ah(x) + bg(y) E[g(X, Y )] = aE[h(X)] + bE[g(Y )]. We now turn to independence; we state the following result with no proof. If you are unconvinced, simply take the below as a definition. Theorem 2.4.21. The random variables X and Y are independent if and only if F (x, y) = F (x)F (y) or equivalently f (x, y) = f (x)f (y). Example 2.4.22. Let Z = (R+ )2 = X × Y and f (x, y) = λ2 e−λ(x+y)

(x, y) ∈ Z, λ > 0.

Then one has f (x) = λe−λx x ∈ X

f (y) = λe−λy y ∈ Y

In addition, what is the probability that X > Y ? Z ∞Z x P(X > Y ) = f (x, y)dydx Z0 ∞ Z0 x = λe−λy dyλe−λx dx 0 0 Z ∞h ix = − e−λy λe−λx dx 0 Z0 ∞ = (1 − e−λx )λe−λx dx 0

= =

i∞ 1 − e−λx + e−2λx 2 0 1 1 1− = . 2 2

h

Again, one has the idea of covariance and correlation for continuous valued random variables: Cov[X, Y ]

= E[(X − E[X])(Y − E[Y ])] = E[XY ] − E[X]E[Y ] Z Z Z = xyf (x, y)dxdy − xf (x)dx yf (y)dy Z

ρ(X, Y )

=

Cov[X, Y ] p

Var[X]Var[Y ]

23 Example 2.4.23. Let Z = R2 and define f (x, y) =

1 2π

p

1−

ρ2

exp{−

1 (x2 + y 2 − 2xyρ)} 2(1 − ρ2 )

(x, y) ∈ Z ρ ∈ [−1, 1].

Let us check if this is a PDF; clearly it is non-negative on Z. So Z Z ∞Z ∞ 1 1 p exp{− (x2 + y 2 − 2xyρ)}dxdy f (x, y)dxdy = 2) 2 2(1 − ρ Z −∞ −∞ 2π 1 − ρ Z ∞ Z ∞ 1 1 1 2 2 2 p exp{− (y − y ρ )} exp{− (x − yρ)2 }dxdy = 2 2 2(1 − ρ ) 2(1 − ρ2 ) 2π 1 − ρ −∞ −∞ Z ∞ 1 1 = √ exp{− (y 2 − y 2 ρ2 )}dy 2(1 − ρ2 ) 2π −∞ Z ∞ 1 1 exp{− = √ (1 − ρ2 )y 2 }dy = 1. 2(1 − ρ2 ) 2π −∞ Here we have completed the square in the integral in x and used the change of variable u = (x − ρy)/(1 − ρ2 )1/2 . Thus f (x, y) defines a joint PDF. Moreover, it is called the standard bi-variate normal distribution. Now let us consider the marginal PDFs: Z ∞ 1 1 p (x2 + y 2 − 2xyρ)}dy x ∈ R exp{− f (x) = 2 2(1 − ρ2 ) −∞ 2π 1 − ρ Z ∞ 1 1 1 2 2 2 p = (x − x ρ )} exp{− (y − xρ)2 }dy exp{− 2 2 2(1 − ρ ) 2(1 − ρ2 ) 2π 1 − ρ −∞ 2 1 = √ e−x /2 . 2π That is X ∼ N (0, 1). By Similar arguments, Y ∼ N (0, 1). Now, let us consider: Z ∞Z ∞ 1 1 (x2 + y 2 − 2xyρ)}dxdy E[XY ] = exp{− xy p 2) 2 2(1 − ρ 2π 1 − ρ −∞ −∞ Z ∞ Z ∞ 1 1 1 2 2 2 p x exp{− = (y − y ρ )} (x − yρ)2 }dxdy y exp{− 2 2 2(1 − ρ ) 2(1 − ρ2 ) 2π 1 − ρ −∞ −∞ Z ∞ Z ∞ 2 1 1 2 2 2 p = (y − y ρ )} y exp{− [u(1 − ρ2 )1/2 + ρy](1 − ρ2 )1/2 e−u /2 dudy 2 2(1 − ρ ) 2π 1 − ρ2 −∞ −∞ Z ∞ √ √ 1 1 p (y 2 − y 2 ρ2 )}[(1 − ρ2 )1/2 {(1 − ρ2 )1/2 2πE[X] + ρy 2π}]dy y exp{− = 2 2(1 − ρ ) 2π 1 − ρ2 −∞ Z ∞ √ 1 1 p (y 2 − y 2 ρ2 )}(1 − ρ2 )1/2 ρy 2πdy = y exp{− 2 2 2(1 − ρ ) 2π 1 − ρ −∞ Z ∞ 2 ρ = √ y 2 e−y /2 dy = ρ. 2π −∞ Here we have used the results in Examples 2.4.8 and 2.4.12 that the first and second (raw) moment of N (µ, σ 2 ) are µ and µ2 + σ 2 . Therefore Cov[X, Y ]

=

ρ

ρ(X, Y )

=

ρ √ = ρ. 1

Note also that if ρ = 0 then: f (x, y) = so as

1 1 p exp{− (x2 + y 2 )} 2 2 2π 1 − ρ

(x, y) ∈ Z.

2 2 1 1 f (x) = √ e−x /2 x ∈ R f (y) = √ e−y /2 y ∈ R 2π 2π f (x, y) = f (x)f (y).

That is standard bi-variate normal random variables are independent if and only they are uncorrelated. Note that this does not always occur for other random variables.

24

We conclude this section with a mention of joint moment generating functions: Definition 2.4.24. For a continuous random variables X and Y the joint moment generating function is Z Xt1 +Y t2 M (t1 , t2 ) = E[e ]= ext1 +yt2 f (x, y)dxdy (t1 , t2 ) ∈ T Z R where T is the set of t1 , t2 for which Z ext1 +yt2 f (x, y)dxdy < ∞. Exercise 2.4.25. Show that ∂ , E[X] = M (t1 , t2 ) ∂t1 t1 =t2 =0

∂ E[Y ] = , M (t1 , t2 ) ∂t2 t1 =t2 =0

∂2 E[XY ] = M (t1 , t2 ) ∂t1 ∂t2 t1 =t2 =0

when the derivatives exist.

2.4.5

Conditional Distributions and Expectations

The idea of conditional distributions and expectations can be further extended to continuous random variables. However, we recall that the event {X = x} has zero probability, so how can we condition on it? It turns out that we cannot have a rigorous answer to this question (i.e. at the level of mathematics in this course), so some faith must be extended towards the following definitions. Definition 2.4.26. The conditional distributions function of Y given X = x is the function: Z y f (x, v) F (y|x) = dv −∞ f (x) for any x such that f (x) > 0. Definition 2.4.27. The conditional density function of F (y|x), written f (y|x) is given by: f (y|x) =

f (x, y) f (x)

for any x such that f (x) > 0. We remark that clearly f (x, y) . f (x, y)dy −∞

f (y|x) = R ∞

Example 2.4.28. Suppose that Y |X = x ∼ N (x, 1) and X ∼ N (0, 1), Z = R2 . Find the PDFs: f (x, y) and f (x|y). We have f (x, y) = f (y|x)f (x) so f (x, y) =

1 1 exp{− ((y − x)2 + x2 )} 2π 2

Now f (x|y) =

(x, y) ∈ Z.

f (x, y) f (y|x)f (x) = f (y) f (y)

which is Bayes theorem for continuous random-variables. Now for y ∈ R Z ∞ 1 1 f (y) = exp{− ((y − x)2 + x2 )}dx 2π 2 −∞ Z ∞ 1 1 2 exp{− (2x2 − 2xy)}dx = exp{−y /2} 2π 2 −∞ Z ∞ 1 = exp{−y 2 /2} exp{−[(x − y/2)2 − y 2 /4]}dx 2π −∞ Z ∞ 2 1 1 2 e−u /2 √ du = exp{−y /4} 2π 2 −∞ √ 1 2π = exp{−y 2 /4} √ 2π 2 1 = √ √ exp{−y 2 /4}. 2 2π

25

So Y ∼ N (0, 2). Then: f (x|y) =

1 2π

exp{− 12 ((y − x)2 + x2 )} 1 √ √ exp{−y 2 /4} 2 2π

After some calculations (exercise) one obtains: r 2 y f (x|y) = exp{−(x − )2 } 2π 2

x ∈ R.

x∈R

that is X|Y = y ∼ N (y/2, 1/2). Example 2.4.29. Let X and Y have joint PDF: f (x, y) =

1 x

(x, y) ∈ Z = {(x, y) : 0 ≤ y ≤ x ≤ 1}.

Then for x ∈ [0, 1]

x

Z

1 dy = 1. x

f (x) = 0

In addition, (x, y) ∈ Z: 1 1/x = 1 x that is if x ∈ [0, 1], then conditional upon x Y ∼ U[0,x] (the uniform distribution on [0, x]). Now, suppose we are interested in P(X 2 + Y 2 ≤ 1|X = x). Let x ≥ 0, and define f (y|x) =

A(x) := {y ∈ R : 0 ≤ y ≤ x, x2 + y 2 ≤ 1} that is, A(x) = [0, x ∧ (1 − x2 )1/2 ], where a ∧ b = min{a, b}. Thus (x ∈ [0, 1]) Z P(X 2 + Y 2 ≤ 1|X = x) = f (y|x)dy A(x) x∧(1−x2 )1/2

Z = 0

1 dy x

x ∧ (1 − x2 )1/2 . x

=

To compute P(X 2 + Y 2 ≤ 1), we note, by setting A = {(x, y) : (x, y) ∈ Z, x2 + y 2 ≤ 1}, we are interested in P(A), which is: Z P(X 2 + Y 2 ≤ 1) = f (x, y)dydx A 1

Z

Z

=

f (y|x)dyf (x)dx 0

Z

A(x) 1

= 0 2

2 1/2

Now x ≤ (1 − x )

if

x ∧ (1 − x2 )1/2 dx. x

√ x ≤ 1/ 2

hence 2

2

P(X + Y ≤ 1)

√ 1/ 2

Z =

Z dx +

0

Now we have, setting x = cos(θ) Z 1 (1 − x2 )1/2 dx √ x 1/ 2

1

√ 1/ 2

(1 − x2 )1/2 dx. x

π/4

Z =

sin(θ) tan(θ)dθ 0

= =

h

log(|sec(θ) + tan(θ)|) − sin(θ)

log(1 +

√

iπ/4 0

1 2) − √ 2

(2.4.1)

26 where we have used integration tables to obtain the integral3 . Thus, returning to (2.4.1): √ √ 1 1 P(X 2 + Y 2 ≤ 1) = √ + log(1 + 2) − √ = log(1 + 2). 2 2 Given the previous discussion, the notion of a conditional expectation for continuous random variables: Z E[g(Y )|X = x] = g(y)f (y|x)dy. As for discrete-valued random variables, we have the following result, which is again not proved. Theorem 2.4.30. Consider jointly continuous random variables X and Y with g, h : R → R with h(X), g(Y ) continuous. Then: Z Z E[h(X)g(Y )] = E[E[g(Y )|X]h(X)] = g(y)f (y|x)dy h(x)f (x)dx whenever the expectations exist. Example 2.4.31. Let us return to Example 2.4.28. Then E[XY ]

=

E[Y E[X|Y ]]

=

E[Y (Y /2)] 1 [2] = 1. 2

= Thus as Var[X] = 1 and Var[Y ] = 2

1 ρ(X, Y ) = √ . 2

Example 2.4.32. Suppose Y |X = x ∼ N (x, σ12 ) and X ∼ N (ξ, σ22 ). To find the marginal distribution of Y , one can use MGFs and conditional expectations. We have: M (t)

=

E[eY t ]

=

E[E[eY t |X]]

=

E[eXt+(σ1 t

= = =

2 2

)/2

]

e

(σ12 t2 )/2

E[eXt ]

e

(σ12 t2 )/2

eξt+(σ2 t

exp{ξt +

2 2

(σ12

)/2

+ σ22 )t2 /2}.

Here we have used Example 2.4.17 for the MGF of a normal distribution. Thus we can conclude that Y ∼ N (ξ, σ12 + σ22 ).

2.4.6

Functions of Random Variables

Consider a random variable X and g : R → R a continuous and invertible function (continuity can be weakened, but let us use it for now). Suppose Y = g(X); what is the distribution of Y ? We have: Z −1 F (y) = P(Y ≤ y) = P(g(X) ≤ y) = P(X ∈ g ((−∞, y)) = f (x)dx. g −1 (−∞,y]

Note that for A ⊆ R, then g −1 A = {x ∈ R : g(x) ∈ A}. Example 2.4.33. Let X ∼ N (0, 1) and set Z

x

Φ(x) = −∞

2 1 √ e−u /2 du 2π

to denote the standard normal CDF. Suppose Y = X 2 , then for y ≥ 0 (note Y = R+ ) √ √ P(Y ≤ y) = P(X 2 ≤ y) = P(− y ≤ X ≤ y). 3 you

will not be expected to perform such an integration in the examination

27

Now √ √ P(− y ≤ X ≤ y)

Then f (y) =

√ √ = P(X ≤ y) − P(X ≤ − y) √ √ = Φ( y) − Φ(− y) √ √ = Φ( y) − [1 − Φ( y)] √ = 2Φ( y) − 1.

d 1 1 √ F (y) = √ Φ0 ( y) = √ e−y/2 dy y 2πy

y∈Y

that is Y ∼ G(1/2, 1/2). This is also called the chi-squared distribution on one degree of freedom. We remark that Y and X are clearly not independent (if X changes so does Y and in a deterministic manner). Now E[XY ] = E[X 3 ] = 0. In addition E[X] = 0 and E[Y ] = 1. So X and Y are uncorrelated, but they are not independent. We now move onto a more general change of variable formula. Suppose X1 and X2 have a joint density on Z and we set (Y1 , Y2 ) = T (X1 , X2 ) where T : Z → T where T is differentiable and invertible and T ⊆ R2 . What is the joint density function of (Y1 , Y2 ) on T? As T is invertible, we set X1 = T1−1 (Y1 , Y2 ) X2 = T2−1 (Y1 , Y2 ) Then we define

J(y1 , y2 ) :=

∂T1−1 ∂y1 ∂T1−1 ∂y2

∂T2−1 ∂y1 ∂T2−1 ∂y2

∂T1−1 ∂T2−1 ∂T2−1 ∂T1−1 = − ∂y1 ∂y2 ∂y1 ∂y2

to be the Jacobian of transformation. Then we have the following result, which simply follows from the change of variable rule for integration: Theorem 2.4.34. If (X1 , X2 ) have joint density f (x, y) on Z, then for (Y1 , Y2 ) = T (X1 , X2 ), with T as described above, the joint density of (Y1 , Y2 ), denoted g is: g(y1 , y2 ) = f (T1−1 (y1 , y2 ), T2−1 (y1 , y2 ))|J(y1 , y2 )|

(y1 , y2 ) ∈ T.

Example 2.4.35. Suppose Z = (R+ )2 and f (x1 , x2 ) = λ2 e−λ(x1 +x2 )

(x1 , x2 ) ∈ Z.

Let (Y1 , Y2 ) = (X1 + X2 , X1 /X2 ). To find the joint density of (Y1 , Y2 ) we first note that T = Z and that (X1 , X2 ) =

Y Y Y1 1 2 , . 1 + Y2 1 + Y2

Then the Jacobian is J(y1 , y2 ) := Thus: g(y1 , y2 ) =

y2 1+y2 y1 (1+y2 )2

1 1+y2 y1 − (1+y 2 2)

y1 λ2 e−λy1 (1 + y2 )2

y1 . =− (1 + y2 )2 (y1 , y2 ) ∈ T.

One can check that indeed Y1 and Y2 are independent and, that the marginal densities are: g(y1 ) = λ2 y1 e−λy1 y1 ∈ R+

g(y2 ) =

1 y 2 ∈ R+ . (1 + y2 )2

28

Example 2.4.36. Suppose we have (X1 , X2 ) as in the case of Example 2.4.35, except we have the mapping: (Y1 , Y2 ) = (X1 , X1 + X2 ). Now, clearly Y2 ≥ Y1 , so this transformation induces the support for (Y1 , Y2 ) as: T = {(y1 , y2 ) : 0 ≤ y1 ≤ y2 , y2 ∈ R+ }. Then (X1 , X2 ) = (Y1 , Y2 − Y1 ) and clearly J(y1 , y2 ) = 1; thus g(y1 , y2 ) = λ2 e−λy2

(y1 , y2 ) ∈ T.

Example 2.4.37. Let X1 and X2 be independent N (0, 1) random variables (Z = R2 ) and let p (Y1 , Y2 ) = (X1 , ρX1 + 1 − ρ2 X2 ) ρ ∈ [−1, 1] Then T = Z and

Y2 − ρY1 (X1 , X2 ) = Y1 , p . 1 − ρ2

Then the Jacobian is

1 J(y1 , y2 ) := 0

−√ ρ

1−ρ2

√1

1−ρ2

= p 1 . 1 − ρ2

Thus g(y1 , y2 )

= =

1 exp{− (y12 + (y2 − ρy1 )2 /(1 − ρ2 )} (y1 , y2 ) ∈ T 2 2π 1 − 1 1 p exp{− (y 2 + y22 − 2y1 y2 ρ)} (y1 , y2 ) ∈ T. 2 2(1 − ρ2 ) 1 2π 1 − ρ 1

p

ρ2

On inspection of Example 2.4.23, (Y1 , Y2 ) follow a standard bivariate normal distribution. Distributions associated to the Normal Distribution In the following Subsection we investigate some distributions that are related to the normal distribution. They appear frequently in hypothesis testing, which is something that we will investigate in the following Chapter. We note that, in a similar way to the way in which joint distributions are defined (Definition 2.4.19), one can easily extend to joint distributions of n ≥ 2 variables (so that there is a joint distribution F (x1 , . . . , xn ) and density f (x1 , . . . , xn )). We begin with the idea of the chi-square distribution on n > 0 degrees of freedom: f (x) =

1 (2)n/2 Γ(n/2)

xn/2−1 e−x/2

x ∈ X = R+ .

We write X ∼ Xn2 ; you will notice that also X ∼ G(n/2, 1/2), so that E[X] = n. Note that from Table 4.2, we have that 1 1 M (t) = > t. n/2 2 (1 − 2t) We have the following result: Proposition 2.4.38. Let X1 , . . . , Xn be independent and identically distributed (i.i.d.) standard i.i.d.

normal random variables (i.e. Xi ∼ N (0, 1)). Let Z = X12 + · · · + Xn2 . Then Z ∼ Xn2 .

29

Proof. We will use moment generating functions. We have M (t)

=

E[eZt ]

=

E[et i=1 Xi ] n Y 2 E[etXi ]

Pn

=

2

i=1 2

E[eX1 t ]n .

=

Here we have used the independent property on the third line and the fact that the random variables are identically distributed on the last line. Now Z ∞ 1 1 1 X12 t E[e ] = √ exp{− (1 − 2t)x21 }dx1 >t 2 2 2π −∞ Z ∞ 1 1 exp{− u2 }du = √ 1/2 2 2π(1 − 2t) −∞ 1 = . (1 − 2t)1/2 Hence M (t) =

1 (1 − 2t)n/2

1 >t 2

and we conclude the result. The standard t−distribution on n−degrees of freedom is (n > 0): x2 −(n+1)/2 Γ([n + 1]/2) 1+ p(x) = √ x ∈ X = R. n nπΓ(n/2) We write X ∼ Tn . Then we have the following important result. Proposition 2.4.39. Let X ∼ N (0, 1) and independently Y ∼ Xn2 , n > 0. Let X T =q . Y n

Then T ∼ Tn . Proof. We have 1 2 1 1 1 y n/2−1 e− 2 y f (x, y) = √ e− 2 x n/2 2 Γ(n/2) 2π We will use Theorem 2.4.34, with the transformation defined by

T

=

Z = R × R+ .

X q

Y n

S

=

Y

and then marginalize out S. The Jacobian of transformation is: ps r 0 s J(t, s) := √t n = n. 1 2 sn Then f (t, s) = √

1 2nπ2n/2 Γ(n/2)

s sn/2−1/2 exp{− [t2 /n + 1]} 2

(t, s) ∈ R × R+ .

Then we have for t ∈ R f (t)

= = = =

Z

∞

s sn/2−1/2 exp{− [t2 /n + 1]}ds 2 2nπ2n/2 Γ(n/2) 0 Z ∞ 1 1 √ un/2−1/2 e−u du t2 n+1 2 2nπ2n/2 Γ(n/2) 0 ( 12 + 2n ) 1 1 √ Γ([n + 1]/2) t2 n+1 2nπ2n/2 Γ(n/2) ( 12 + 2n ) 2 Γ([n + 1]/2) t2 −(n+1)/2 √ 1+ n nπΓ(n/2) √

1

30

and we conclude. The last distribution that we consider is the standard F −distribution on d1 , d2 > 0 degrees of freedom: d +d d d1 /2 d 1 − 1 2 2 1 1 d1 /2−1 1 + f (x) = x x x ∈ X = R+ d2 B( d21 , d22 ) d2 where B(

Γ( d21 )Γ( d22 ) d1 d2 . , )= 2 2 2 Γ( d1 +d 2 )

We write X ∼ F(d1 ,d2 ) . We have the following result. Proposition 2.4.40. Let X ∼ Xd21 and independently Y ∼ Xd22 , d1 , d2 > 0. Let F =

X/d1 . Y /d2

Then F ∼ F(d1 ,d2 ) . Proof. We have f (x, y) =

1 1 1 1 xd1 /2−1 e− 2 x d /2 y d2 /2−1 e− 2 y 2d1 /2 Γ(d1 /2) 2 2 Γ(d2 /2)

Z = R + × R+ .

We will use Theorem 2.4.34, with the transformation defined by T S

X/d1 Y /d2 = Y

=

and then marginalize out S. The Jacobian of transformation is: d1 s 0 d s d2 1 J(t, s) := d1 t . = d2 1 d2 Then f (t, s) =

1 2d1 /2+d2 /2 Γ(d1 /2)Γ(d2 /2)

d1 s d1 st d1 /2−1 − d2d1 st d2 /2−1 −s/2 2 s e e d2 d2

T = (R+ )2 .

To shorten the subsequent notations, let g=

d d1 /2 1 1 . 2d1 /2+d2 /2 Γ(d1 /2)Γ(d2 /2) d2

Then, for t ∈ R+ f (t)

= gt = = = =

and we conclude.

Z

∞

1 d1 t exp{−s( [1 + ])}ds 2 d2 Z0 ∞ 1 d1 t −(d1 +d2 )/2 gtd1 /2−1 u(d1 +d2 )/2−1 e−u ( [1 + du ]) 2 d2 0 1 d1 t −(d1 +d2 )/2 gtd1 /2−1 ( [1 + ]) Γ((d1 + d2 )/2) 2 d2 d +d d1 +d2 d 1 − 1 2 2 gΓ((d1 + d2 )/2)2 2 td1 /2−1 1 + t d2 d +d d /2 1 d1 1 d1 /2−1 d1 − 1 2 2 t 1+ t d2 B( d21 , d22 ) d2 d1 /2−1

s

d1 +d2 2

−1

31

2.5

Convergence of Random Variables

In the following Section we provide a brief introduction to convergence of random variable and in particular two modes of convergence: • Convergence in Distribution • Convergence in Probability These properties are rather important in probability and mathematical statistics and provide a way to justify many statistical and numerical procedures. The properties are rather loosely associated to the notion of a sequence of random variables X1 , X2 , . . . , Xn (there will be infinitely many of them typically) and we will be concerned with the idea of what happens to the distribution or some functional as n → ∞. The second part of this Section will focus on perhaps the most important result in probability: the central limit theorem. This provides a characterization of a random variable: n

1 X Sn = √ Xi n i=1 with X1 , . . . , Xn independent and identically distributed with zero mean and unit variance. This particular result is very useful in hypothesis testing, which we shall see later on. Throughout, we will focus on continuous random variables, but one can extend this notion.

2.5.1

Convergence in Distribution

Consider a sequence of random variables X1 , X2 , . . . , with associated distribution functions F1 , F2 , . . . , we then have the following definition: Definition 2.5.1. We say that the sequence of distribution functions F1 , F2 , . . . converges to a distribution function F , if limn→∞ Fn (x) = F (x) at each point where F is continuous. If X has d

distribution function F , we say that Xn converges in distribution to X and write Xn → X. Example 2.5.2. Consider a sequence of random variables Xn ∈ Xn = [0, n], n ≥ 1, with x n x ∈ Xn . Fn (x) = 1 − 1 − n Then for any fixed x ∈ R+

lim Fn (x) = 1 − e−x .

n→∞

Now if X ∼ E(1): Z

x

F (x) =

e−u du = 1 − e−x .

0 d

So Xn → X, X ∼ E(1). Example 2.5.3. Consider a sequence of random variables Xn ∈ X = [0, 1], n ≥ 1, with Fn (x) = 1 −

sin(2nπx) . 2nπ

Then for any fixed x ∈ [0, 1] lim Fn (x) = x.

n→∞

Now if X ∼ U[0,1] , x ∈ [0, 1]: Z F (x) =

du = x. 0

d

So Xn → X, X ∼ U[0,1] .

x

32

2.5.2

Convergence in Probability

We now consider an alternative notion of convergence for a sequence of random variables X1 , X2 , . . . P

Definition 2.5.4. We say that Xn converges in probability to a constant c ∈ R (written Xn → c) if for every > 0 lim P(|Xn − c| > ) = 0. n→∞

We note that convergence in probability can be extended to convergence to a random variable X, but we do not do this, as it is beyond the level of this course. We have the following useful result, which we do not prove. Theorem 2.5.5. Consider a sequence of random variables X1 , X2 , . . . , with associated distribution functions F1 , F2 , . . . . If for every x ∈ Xn lim Fn (x) = c

n→∞ P

then Xn → c. Example 2.5.6. Consider a sequence of random variables Xn ∈ X = R+ , n ≥ 1, with x n . Fn (x) = 1+x Then for any fixed x ∈ X lim Fn (x) = 0.

n→∞ P

Thus Xn → 0. We finish the Section with a rather important result, which we again do not prove. It is called the weak law of large numbers: Theorem 2.5.7. Let X1 , X2 , . . . be a sequence of independent and identically distributed random variables with E[|X1 |] < ∞. Then n 1X P Xi → E[X1 ]. n i=1 Note that this result extends to functions; i.e. if g : R → R with E[|g(X1 )|] < ∞ then n

1X P g(Xi ) → E[g(X1 )]. n i=1 Example 2.5.8. Consider an integral Z I=

g(x)dx R

where we will suppose that g(x) 6= 0. Suppose we cannot calculate I. Consider any non-zero pdf f (x) on R. Then Z g(x) I= f (x)dx f R (x) and suppose that Z g(x) f (x)dx < ∞. R f (x) Then by the weak law of large numbers, if X1 , . . . , Xn are i.i.d. with pdf f then n

1 X g(Xi ) P → I. n i=1 f (Xi ) This provides a justification for a numerical method (called the Monte Carlo method) to approximate integrals.

33

2.5.3

Central Limit Theorem

We close the Section and Chapter with the central limit theorem (CLT) which we state without proof. Theorem 2.5.9. Let X1 , X2 , . . . be a sequence of independent and identically distributed random variables with E[|X1 |] < ∞ and 0 < Var[X1 ] < ∞. Then √

n

n

1 X (Xi − E[X1 ]) p n i=1 Var[X1 ]

d

→Z

where Z ∼ N (0, 1). Pn The CLT asserts that the distribution of the summation i=1 Xi suitably normalized and centered, will converge to that of a normal distribution. An often used idea, via the CLT, is normal approximation. That is, for n ‘large’ √

n

n

1 X (Xi − E[X1 ]) p n i=1 Var[X1 ]

·

∼ N (0, 1)

·

where ∼ means ‘approximately distributed as’, so we have4 , n X

·

Xi ∼ N (nE[X1 ], nVar[X1 ]).

i=1

Example Pn 2.5.10. Let X1 , . . . , Xn be i.i.d. G(a/n, b) random variables. Then the distribution of Zn = i=1 Xi is for b > t: M (t) = E[eZn t ] =

n Y

E[eXi t ] = E[ex1 ]n =

i=1

b a/n n b−t

where we have used the i.i.d. property and the MGF of a Gamma random variable (see Table 4.2). Thus Zn ∼ G(a, b) for any n ≥ 1. However, from the CLT one can reasonably approximate Zn , when n is large by a normal random variable with mean n and variance n

4 If

a a = bn b

a a = 2. nb2 b

Z ∼ N (0, 1) you can verify that if X = σZ + µ then X ∼ N (µ, σ 2 )

Chapter 3

Introduction to Statistics 3.1

Introduction

In this final Chapter, we give a very brief introduction to statistical ideas. Here the notion is that one has observed data from some sampling distribution ‘the population distribution’ and one wishes to infer what the properties of the ‘population’ are on the basis of observed samples. In particular we will be interested in estimating the parameters of sampling distributions such as the maximum likelihood method (Section 3.2) as well as testing hypotheses about parameters (3.3). We end the Chapter with an introduction to Bayesian statistics (Section 3.4) which is another alternative way to estimate parameters, which is more complex, but much richer than the MLE method.

3.2 3.2.1

Maximum Likelihood Estimation Introduction

So far in this course, we have proceeded to discuss pmfs and pdfs such as P(λ) or N (µ, σ 2 ), but we have not discussed what the parameters λ or µ might be. We discuss in this Section, a particularly important way to estimate parameters called maximum likelihood estimation (MLE). The basic idea is this. Suppose one observes data x1 , x2 , . . . , xn and we hypothesize that they follow some joint distribution Fθ (x1 , . . . , xn ), θ ∈ Θ (Θ is the parameter space e.g. Θ = R). Then the idea of maximum likelihood estimation is to find the parameter which maximizes the joint pmf/pdf of the data, that is: θˆn = argmaxθ∈Θ fθ (x1 , . . . , xn ).

3.2.2

The Method i.i.d.

Throughout this section, we assume that X1 , . . . , Xn are mutually independent. So, we have Xi ∼ Fθ , and the joint pmf/pdf is: fθ (x1 , . . . , xn ) = fθ (x1 ) × fθ (x2 ) × · · · × fθ (xn ) =

n Y

fθ (xi ).

i=1

We call fθ (x1 , . . . , xn ) the likelihood of the data. As maximizing a function is equivalent to maximizing a monotonic increasing transformation of the function, we often work with the log-likelihood : n X lθ (x1 , . . . , xn ) = log fθ (x1 , . . . , xn ) = log fθ (xi ) . i=1

If Θ is some continuous space (as it generally is for our examples) and θ = (θ1 , . . . , θd ), then we can compute the gradient vector: ∂l (x , . . . , x ) ∂lθ (x1 , . . . , xn ) θ 1 n ,..., ∇lθ (x1 , . . . , xn ) = ∂θ1 ∂θd and we would like to solve, for θ ∈ Θ (below 0 is the d−dimensional vector of zeros) ∇lθ (x1 , . . . , xn ) = 0. 34

(3.2.1)

35

The solution of this equation (assuming it exists) is a maximum if the hessian matrix is negative definite:  ∂ 2 lθ (x1 ,...,xn ) ∂ 2 lθ (x1 ,...,xn ) 2 (x1 ,...,xn )  · · · ∂ lθ∂θ ∂θ1 ∂θ2 ∂θ12 1 ∂θd   .. .. .. .. . H(θ) :=  . . . .   2 2 2 ∂ lθ (x1 ,...,xn ) ∂ lθ (x1 ,...,xn ) ∂ lθ (x1 ,...,xn ) ··· ∂θd ∂θ1 ∂θd ∂θ2 ∂θ 2 d

If the d numbers λ1 , . . . , λd which solve|λId − H(θ)| = 0, with Id the d × d identity matrix, are all negative, then θ is a local maximum of lθ (x1 , . . . , xn ). If d = 1 then this just boils down to checking whether the second derivative of the log-likelihood is negative at the solution of (3.2.1). Thus in summary, the approach we employ is as follows: 1. Compute the likelihood fθ (x1 , . . . , xn ). 2. Compute the log-likelihood lθ (x1 , . . . , xn ) and its gradient vector ∇lθ (x1 , . . . , xn ). 3. Solve ∇lθ (x1 , . . . , xn ) = 0, with respect to θ ∈ Θ, call this solution θen (we are assuming there is only one θen ). 4. If H(θen ) is negative definite, then θˆn = θen . In general, point 3. may not be possible analytically (so for example, one can use Newton’s method). However, you will not be asked to solve ∇lθ (x1 , . . . , xn ) = 0, unless there is an analytic solution.

3.2.3

Examples of Computing the MLE

Example 3.2.1. Let X1 , . . . , Xn be i.i.d. P(λ) random variables. Let us compute the MLE of λ = θ, given observations x1 , . . . , xn . First, we have fλ (x1 , . . . , xn )

=

n Y λ xi i=1

=

xi !

e−nλ λ

e−λ

Pn

i=1

xi

1 Qn

i=1

xi !

.

Second the log-likelihood is: lλ (x1 , . . . , xn ) = log(fλ (x1 , . . . , xn )) = −nλ +

n X

xi log(λ) − log(

i=1

n Y

xi !).

i=1

The gradient vector is a derivative: n

1X dlλ (x1 , . . . , xn ) = −n + xi . dλ λ i=1 Thirdly n

−n +

1X xi = 0 λ i=1

so, n

X en = 1 λ xi . n i=1 Fourthly, n

1 X d2 lλ (x1 , . . . , xn ) =− 2 xi < 0 2 dλ λ i=1 for any λ > 0 (assuming there is at least one i such that xi > 0). Thus assuming there is at least one i such that xi > 0 n X ˆn = 1 λ xi . n i=1

36 ˆ n as in Example 3.2.1, we note that Remark 3.2.2. As a theoretical justification of λ n

1X xi n i=1 will converge in probability (see Theorem 2.5.7) to E[X1 ] = λ if our assumptions hold true. That is, we recover the true parameter value; such a property is called consistency - we do not address this issue further. Example 3.2.3. Let X1 , . . . , Xn be i.i.d. E(λ) random variables. Let us compute the MLE of λ = θ, given observations x1 , . . . , xn . First, we have fλ (x1 , . . . , xn )

n Y

=

λe−λxi

i=1

= λn exp{−λ

n X

xi }.

i=1

Second the log-likelihood is: lλ (x1 , . . . , xn ) = log(fλ (x1 , . . . , xn )) = n log(λ) − λ

n X

xi .

i=1

The gradient vector is a derivative: n

n X dlλ (x1 , . . . , xn ) xi . = − dλ λ i=1 Thirdly n

n X − xi = 0 λ i=1 so en = ( 1 λ n

n X

xi )−1 .

i=1

Fourthly, d2 lλ (x1 , . . . , xn ) n = − 2 < 0. dλ2 λ Thus ˆn = ( 1 λ n

n X

xi )−1 .

i=1

Example 3.2.4. Let X1 , . . . , Xn be i.i.d. N (µ, σ 2 ) random variables. Let us compute the MLE of (µ, σ 2 ) = θ, given observations x1 , . . . , xn . First, we have fθ (x1 , . . . , xn )

=

n Y i=1

=

√

√

1 2πσ 2

1 2πσ 2

exp{−

n

1 (xi − µ)2 } 2σ 2

exp{−

n 1 X (xi − µ)2 }. 2σ 2 i=1

Second the log-likelihood is: lθ (x1 , . . . , xn ) = log(fθ (x1 , . . . , xn )) = −

n n 1 X log(2πσ 2 ) − 2 (xi − µ)2 . 2 2σ i=1

The gradient vector is n n ∂l (x , . . . , x ) ∂l (x , . . . , x ) 1 X n 1 X θ 1 n θ 1 n 2 , = (x − µ), − + (x − µ) . i i ∂µ ∂σ 2 σ 2 i=1 2σ 2 2(σ 2 )2 i=1

37

Thirdly, we must solve the equations

−

n 1 X (xi − µ) σ 2 i=1

=

0

(3.2.2)

n n 1 X + (xi − µ)2 2σ 2 2(σ 2 )2 i=1

=

0

(3.2.3)

simultaneously for µ and σ 2 . Since (3.2.2) can be solved independently of (3.2.3), we have: n

1X xi . n i=1

µ en = Thus substituting into (3.2.3), we have

n 1 X n (xi − µ en )2 = 2(σ 2 )2 i=1 2σ 2

that is:

n

σ en2 = Fourthly the hessian matrix is: " H(θ) =

− (σ12 )2

−n Pnσ2

i=1 (xi

1X (xi − µ en )2 . n i=1 # Pn − (σ12 )2 i=1 (xi − µ) Pn . n 1 2 i=1 (xi − µ) 2(σ 2 )2 − (σ 2 )3

− µ)

Note that when θen = (e µn , σ en2 ) the off-diagonal elements are exactly 0, so when solving |λI2 −H(θen )| = 0, we simply need to show that the diagonal elements are negative. Clearly for the first diagonal element n − 2 <0 σ en if (xi − µ en )2 > 0 for at least one i (we do not allow the case σ en2 , in that we assume this doesn’t occur). In addition n n 1 X n n n − (xi − µ en )2 = − 2 2 =− < 0. 2 2 2 3 2 2 2(e σn ) (e σn ) i=1 2(e σn ) (e σn ) 2(e σn2 )2

Thus θˆn =

3.3 3.3.1

n 1 X

n

i=1

n

xi ,

n

1X 1 X 2 xi − xj . n i=1 n j=1

Hypothesis Testing Introduction

The theory of hypothesis testing is concerned with the problem of determining whether or not a statistical hypothesis, that is, a statement about the probability distribution of the data, is consistent with the available sample evidence. The particular hypothesis to be tested is called the null hypothesis and is denoted by H0 . The ultimate goal is to accept or reject H0 . In addition to the null hypothesis H0 , one may also be interested in a particular set of deviations from H0 , called the alternative hypothesis and denoted by H1 . Usually, the null and the alternative hypotheses are not on an equal footing: H0 is clearly specified and of intrinsic interest, whereas H1 serves only to indicate what types of departure from H0 are of interest. A statistical test of a null hypothesis H0 is typically based on three elements: 1. a statistic T , called a test statistic 2. a partition of the possible values of T into two distinct regions: the set K of values of T that are regarded as inconsistent with H0 , called the critical or rejection region of the test, and its complement K c , called the non-rejection region 3. a decision rule that rejects the null hypothesis H0 as inconsistent with the data if the observed value of T falls in the rejection region K, and does not reject H0 if the observed value of T belongs instead to K c .

38

3.3.2

Constructing Test Statistics

In order to construct test-statistics, we start with an important result, which we do not prove. Note that all of our results are associated to normally distributed data. Theorem 3.3.1. Let X1 , . . . , Xn be i.i.d. N (µ, σ 2 ) random variables. Define n

¯n X

:=

1X Xi n i=1

s2n

:=

1 X ¯ n )2 (Xi − X n − 1 i=1

n

Then ¯ n − µ ∼ N (0, σ 2 /n). 1. X 2 2. (n − 1)s2n /σ 2 ∼ Xn−1 .

¯ n − µ and (n − 1)s2n /σ 2 are independent random variables. 3. X We note then, that by Proposition 2.4.39, it follows that ¯ n − µ)/(σ/√n) ¯ n − µ) (X (X p T (µ) := = p ∼ Tn−1 . 2 2 sn /σ s2n /n Note that as one might imagine Proposition 2.4.38 is used to prove Theorem 3.3.1. Note that one can show that the Tn−1 −distribution has a symmetric pdf, around 0. Now consider testing H0 : µ = µ0 against the two-sided alternative H1 : µ 6= µ0 (that is, we are testing whether the population mean, the ‘true’ mean of the data, is a particular value). Now we know for any µ ∈ R that T (µ) ∼ Tn−1 , thus if the null hypothesis is true T (µ0 ) should be a random variable that is consistent with a Tn−1 random variable. To construct the rejection region of the test, we must choose a confidence level, typically 95%. Then we want T (µ0 ) to lie in 95% of the probability. However, this still does not tell us what the rejection region is; this is informed by the alternative hypothesis, that H1 : µ 6= µ0 ; which indicates that a value which is inconsistent with the Tn−1 −distribution lies in each ‘tail’ of the distribution. Thus the procedure is: 1. Compute T (µ0 ). 2. Decide upon your confidence level (1 − α)%. This defines the rejection region. 3. Compute the t−values (−t, t). These are the numbers in R such that the probability (under a Tn−1 −random variable) of exceeding t is α/2 and the probability of being less than −t is α/2. 4. If −t < T (µ0 ) < t then we do not reject the null hypothesis, otherwise we reject the null hypothesis. A number of remarks are in order. First, a test can only disprove a null hypothesis. The fact that we do not reject the null on the basis of the sample evidence does not mean that the hypothesis is true. Second, it is useful to distinguish between two types of errors that can be made: • Type I error: Reject H0 when it is true. • Type II error: Do not reject H0 when it is false. In statistics, Type I error is usually regarded as the most serious. The analogy is with the judiciary system, where condemning an innocent is typically considered a much more serious problem than letting a guilty person free. A second test statistic we consider is for testing whether two samples have the same variance. Let i.i.d. 2 X1 , . . . , Xn and Y1 , . . . , Ym be independent sequences of random variables where Xi ∼ N (µX , σX ) i.i.d.

and Yi ∼ N (µY , σY2 ). Now we know from Theorem 3.3.1 that 2 2 (n − 1)s2X,n /σX ∼ Xn−1

39

and independently: 2 (m − 1)s2Y,m /σY2 ∼ Xm−1 . 2 2 Now suppose that we want to test H0 : σX = σY2 against H1 : σX 6= σY2 . Now if H0 is true, by Theorem 3.3.1 and Proposition 2.4.40. 2 2 2 F (σX ) := (n − 1)s2X,n /σX /(n − 1) (m − 1)s2Y,m /σX /(m − 1) = s2X,n /s2Y,m ∼ Fn−1,m−1

Thus, we perform the following procedure: 2 1. Compute F (σX ).

2. Decide upon your confidence level (1 − α)%. This defines the rejection region. 3. Compute the F −values (f , f ). These are the numbers in R+ such that the probability (under a Fn−1,m−1 −random variable) of exceeding f is α/2 and the probability of being less than f is α/2. 2 4. If f < F (σX ) < f then we do not reject the null hypothesis, otherwise we reject the null hypothesis.

Remark 3.3.2. All of our results concern Normal samples. However one can use the CLT (Theorem 2.5.9) to extend these tests to non-normal data. We do not follow this idea in this course.

3.4 3.4.1

Bayesian Inference Introduction

So far we have considered point estimation methods for the unknown parameter. In the following Section, we consider the idea of Bayesian inference for unknown parameters. Recall, from Examples 2.3.34 and 2.4.28, that there is a Bayes theorem for discrete and continuous random variables. Indeed, there is version of Bayes theorem that can mix continuous and discrete random variables. Let X and Y be two jointly defined random variables such that X or Y are either continuous or discrete. Then Bayes theorem is: f (y|x)f (x) f (x|y) = f (y) as long as the associated pmfs/pdfs are well defined. Throughout the section, we assume that the pmfs/pdfs are always positive on their supports.

3.4.2

Bayesian Estimation

Throughout, we will assume that we will have a random sample X1 , X2 , . . . , Xn which are conditionally independent, given a parameter θ ∈ Θ. As we will see, in Bayesian statistics, the parameter i.i.d. is a random variable. So, we assume Xi |θ ∼ F (·|θ) where F (·|θ) is assumed to be a distribution function for any θ ∈ Θ. Thus we have that the joint pmf/pdf of X1 , . . . , Xn |θ is: f (x1 , . . . , xn |θ) =

n Y

f (xi |θ).

i=1

The main key behind Bayesian statistics is the choice of a prior probability distribution for the parameter θ ∈ Θ. That is, Bayesian statisticians specify a probability distribution on the parameter θ before the data are observed. This probability distribution is supposed to reflect the information one might have before seeing the observations. To make this idea concrete, consider the following example: i.i.d.

Example 3.4.1. Suppose that we will observe data Xi |θ ∼ E(λ). Then one has to construct a probability distribution on λ ∈ R+ . A possible candidate is λ ∼ G(a, b). If one has some prior beliefs about the mean and variance of λ (say E[λ] = 10, Var[λ] = 10) then one determine what a and b are.

40

Throughout, we will write the prior pmf/pdf as π(θ). Now the way in which Bayesian inference works is to update the prior beliefs on θ via the posterior pmf/pdf. That is, ‘in the light of the data’ the distributional properties of the prior are updated. This is achieved by Bayes theorem; the posterior pmf/pdf is: π(θ|x1 , . . . , xn ) = where

f (x1 , . . . , xn |θ)π(θ) f (x1 , . . . , xn )

Z f (x1 , . . . , xn |θ)π(θ)dθ

f (x1 , . . . , xn ) = Θ

if θ is continuous and, if θ is discrete: f (x1 , . . . , xn ) =

X

f (x1 , . . . , xn |θ)π(θ).

θ∈Θ

For a Bayesian statistician, the posterior is the ‘final answer’, in that all statistical inference should be associated to the posterior. For example, if one is interested in estimating θ then one can use the posterior mean: Z θπ(θ|x1 , . . . , xn )dθ.

E[Θ|x1 , . . . , xn ] = Θ

In addition, consider θ ∈ R (or indeed any univariate component of θ), then we can compute a confidence interval, which is called an credible interval in Bayesian statistics. The highest 95%posterior-credible (HPC) interval is the shortest region [θ, θ], such that Z

θ

π(θ|x1 , . . . , xn )dθ = 0.95. θ

By shortest, we mean the [θ, θ], such that |θ − θ| is smallest. It should be clear by now, that the posterior distribution is much ‘richer’ than the MLE, in the sense that one now has a whole distribution which reflects the parameter, instead of a point estimate. What also might be apparent now, is the fact that the posterior is perhaps difficult to calculate. For example the posterior mean will require you to compute two integrations over Θ (in general) and this may not be easy to calculate in practice. In addition, it could be difficult, analytically to calculate a HPC. As a result, most Bayesian inference is done via numerical methods, based on Monte Carlo (see Example 2.5.8); we do not mention this further.

3.4.3

Examples

Despite the fact that Bayesian inference is very challenging, there are still many examples where one can do analytical calculations. Example 3.4.2. Let us consider Example 3.4.1. Here we have that Z ∞ n X ba a−1 −bλ f (x1 , . . . , xn ) = λn exp{−λ xi } λ e dλ Γ(a) 0 i=1 Z ∞ n X ba = λn+a−1 exp{−λ[ xi + b]}dλ Γ(a) 0 i=1 n+a Z ∞ ba 1 P = un+a−1 e−u du Γ(a) [ ni=1 xi + b] 0 n+a ba 1 Pn = Γ(n + a). Γ(a) [ i=1 xi + b] So, as: f (x1 , . . . , xn |λ)π(λ) = we have: π(λ|x1 , . . . , xn ) =

n X ba n+a−1 λ exp{−λ[ xi + b]} Γ(a) i=1

Pn λn+a−1 exp{−λ[ i=1 xi + b]} n+a P 1 Γ(n + a) [ n xi +b] i=1

41

i.e. λ|x1 , . . . , xn ∼ G(n + a, b +

n X

xi ).

i=1

Thus, the posterior distribution on λ is in the same family as the prior, except, with updated parameters, reflecting the data. So, for example: E[Λ|x1 , . . . , xn ] =

n+a Pn . b + i=1 xi

In comparison to the MLE in Example 3.2.3, we see that the posterior mean and MLE correspond as a, b → 0. i.i.d.

Example 3.4.3. Let Xi |λ ∼ P(λ), i ∈ {1, . . . , n}. Suppose the prior on λ is G(a, b). Then Z ∞ P n 1 ba a−1 −bλ λ i=1 xi exp{−nλ} Qn f (x1 , . . . , xn ) = λ e dλ 0 i=1 xi ! Γ(a) Z ∞ P n ba Qn λ i=1 xi +a−1 exp{−λ[n + b]}dλ = Γ(a) i=1 xi ! 0 1 Pni=1 xi +a Z ∞ Pn ba Qn u i=1 xi +a−1 e−u du = Γ(a) i=1 xi ! [n + b] 0 n 1 Pni=1 xi +a X ba Qn xi + a). Γ( = Γ(a) i=1 xi ! [n + b] i=1 So as: f (x1 , . . . , xn |λ)π(λ) =

Γ(a)

we have: π(λ|x1 , . . . , xn ) =

ba Qn

Pn

i=1 xi !

λ

i=1

xi +a−1

exp{−λ[n + b]}

Pn

xi +a−1 exp{−λ[n + b]} Pni=1 xi +a P n 1 Γ( i=1 xi + a) [n+b]

λ

i=1

i.e. λ|x1 , . . . , xn ∼ G

n X

xi + a, n + b .

i=1

Thus, the posterior distribution on λ is in the same family as the prior, except, with updated parameters, reflecting the data. So, for example: Pn xi + a . E[Λ|x1 , . . . , xn ] = i=1 n+b In comparison to the MLE in Example 3.2.1, we see that the posterior mean and MLE correspond as a, b → 0. i.i.d.

Example 3.4.4. Let Xi |µ ∼ N (µ, 1), i ∈ {1, . . . , n}. Suppose the prior on µ is N (ξ, κ). Then Z ∞ n 1 1 n 1X 1 √ f (x1 , . . . , xn ) = (xi − µ)2 } √ exp{− exp{− (µ − ξ)2 }dµ 2 2κ 2π 2πκ −∞ i=1 n 1 n 1 Z ∞ 1X 1 √ √ = exp{− (xi − µ)2 − (µ − ξ)2 }dµ. 2 i=1 2κ 2π 2πκ −∞ To compute the integral, let us first manipulate the exponent inside the integral: n

−

1X 1 (xi − µ)2 − (µ − ξ)2 2 i=1 2κ 2

Let c(ξ, κ, x1 , . . . , xn ) = − 21 [ ξκ + n

−

1X 1 (xi − µ)2 − (µ − ξ)2 2 i=1 2κ

Pn

i=1

n

=

n

1 1 ξ X ξ2 X 2 − [µ2 (n + ) − 2µ( + xi ) + + xi ] 2 κ κ i=1 κ i=1

x2i ], then n

1 1 ξ X = − [µ2 (n + ) − 2µ( + xi )] + c(ξ, κ, x1 , . . . , xn ) 2 κ κ i=1 Pn Pn ( ξ + i=1 xi ) 2 ( κξ + i=1 xi ) 2 i 1 1 h = − (n + ) µ − κ ) − ( + c(ξ, κ, x1 , . . . , xn ). 2 κ n + κ1 n + κ1

42

Let c0 (ξ, κ, x1 , . . . , xn ) = c(ξ, κ, x1 , . . . , xn ) + 21 (n + κ1 )( f (x1 , . . . , xn )

2 P ξ (κ + n i=1 xi ) , 1 n+ κ

then we have

Pn Z ∞ 1 n 1 ( ξ + i=1 xi ) 2 1 1 √ √ ) }dµ exp{c0 (ξ, κ, x1 , . . . , xn )} exp{− (n + ) µ − κ 2 κ n + κ1 2π 2πκ −∞ √ 1 n 1 2π 0 √ √ = exp{c (ξ, κ, x1 , . . . , xn )} . (n + κ1 )1/2 2π 2πκ =

So as: Pn 1 n 1 ( κξ + i=1 xi ) 2 1 1 0 √ exp{c (ξ, κ, x1 , . . . , xn )} exp{− (n+ ) µ− f (x1 , . . . , xn |µ)π(µ) = √ ) } 2 κ n + κ1 2π 2πκ we have π(µ|x1 , . . . , xn ) = i.e.

exp{− 12 (n + κ1 ) µ −

P ξ (κ + n i=1 xi ) 2 ) } 1 n+ κ

√ 2π 1 1/2 (n+ κ )

( ξ + Pn x ) 1 −1 i=1 i ) , (n + µ|x1 , . . . , xn ∼ N κ κ n + κ1

Thus, the posterior distribution on µ is in the same family as the prior, except, with updated parameters, reflecting the data. So, for example: Pn ( ξ + i=1 xi ) E[Ξ|x1 , . . . , xn ] = κ . n + κ1 In comparison to the MLE in Example 3.2.4, we see that the posterior mean and MLE correspond as κ → ∞, for any fixed ξ ∈ R. We note that in all of our examples the posterior is in the same family as the prior. This is not by chance; for every member of the exponential family, with parameter θ it is often possible to find a prior which obeys this former property. Such priors are called conjugate priors.

Chapter 4

Miscellaneous Results In the following Chapter we quote some results of use in the course. We do not cover this material in lectures and is there for your convenience and revision. The following Sections give general facts which should be taken as true. You can easily find the theory behind each of these results in any undergraduate text in maths or statistics.

4.1

Set Theory

Recall that A ∪ B means all the elements in A or all the elements of B. E.g. A = {1, 2}, B = {3, 4} then A∪B = {1, 2, 3, 4}. In addition that A∩B means all the elements in A and B. E.g. A = {1, 2, 3}, B = {3, 4} then A ∩ B = {3}. Finally recall Ac is the complement of the set A, that is everything in Ω that is not in A, e.g. Ω = {1, 2, 3} and A = {1, 2}, then Ac = {3}. Let A, B, C be sets in some state-space Ω. Then the following holds true: A∪B =B∪A A ∪ (B ∪ C) = (A ∪ B) ∪ (A ∪ C)

A ∩ B = B ∩ A ASSOCIATIVITY A ∩ (B ∩ C) = (A ∩ B) ∩ (A ∩ C)

DISTRIBUTIVITY.

Note that the second rule holds when mixing intersection and union, e.g.: A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ C). We also have De-Morgan’s laws: (A ∪ B)c = Ac ∩ B c

(A ∩ B)c = Ac ∪ B c .

A neat trick for probability: A = (A ∩ B) ∪ (A ∩ B c ).

4.2

Derivatives and Functions

Here I list some standard mathematical derivatives. Below a, b are real-valued finite constants, unless otherwise stated. d p (x ) dx d (aebx ) dx d (log(x)) dx d x (a ) dx d (sin(ax)) dx d (cos(ax)) dx d (tan(ax)) dx

=

pxp−1

=

abebx

=

1 x

=

log(a)ax

p 6= 0

x>0

= a cos(ax) = −a sin(ax) = a sec2 (ax) 43

a>0

44

Also recall the product rule; for two functions f (x) and g(x), we have d dg(x) df (x) (f (x)g(x)) = f (x) + g(x) . dx dx dx In addition the chain rule; for two functions f (x) and g(x), let h(x) = f (g(x)), then df (g(x)) dg(x) dh(x) = . dx dg(x) dx For example, suppose f (x) = log(x) and g(x) = 2 + sin(x) and set h(x) = log(2 + sin(x)), then df (g(x)) 1 = dg(x) g(x)

dg(x) = cos(x) dx

where df /dg is the derivative of log(g). Hence 1 cos(x) dh(x) = cos(x) = . dx g(x) 2 + sin(x) Note that by log(x) I mean the inverse function of the exponential function, or natural logarithm: log(ex ) = x. Recall that log(ab) = log(a) + log(b) and log(ab ) = b log(a). Note that sin2 (x) + cos2 (x) = 1.

4.3

Summation

Recall that

Pk2

x=k1

f (x), for some integer −∞ ≥ k1 ≤ k2 ≤ ∞, with f : Z → R, means f (k1 ) + f (k1 + 1) + f (k1 + 2) + · · · + f (k2 − 1) + f (k2 ).

For example k1 = 1, k2 = 3, f (x) = x, then 3 X

x = 1 + 2 + 3 = 6.

x=1

Recall that for any finite c ∈ R: k2 X

k2 X

cf (x) = c

x=k1

f (x).

x=k1

Also for two real-valued functions f (x) and g(x): k2 X

k2 k2 X X g(x). f (x) + g(x) = f (x) +

x=k1

x=k1

x=k1

Some standard summations that you may have seen in the past are listed below: 1 1−z

=

ez

=

(1 + z)n (1 + z)−n − log(1 − z) log(1 + z)

∞ X

zk

|z| < 1

GEOMETRIC

k=0 ∞ X

zk z ∈ R EXPONENTIAL k! k=0 n X n k = z n ∈ Z+ BINOMIAL k k=0 n X n+k k = z n ∈ Z+ , |z| < 1 NEGATIVE BINOMIAL k = =

k=0 ∞ X

k=1 ∞ X

zk k

|z| < 1

(−1)k+1

k=1

zk k

LOGARITHMIC |z| < 1

LOGARITHMIC

45

Note that, suppose |z| < 1, then we have n X

zk = zj

k=j

1 − z n−j+1 , 1−z

A standard trick that is used in this course is the reversal of summation and differentiation. For example, consider f : Z × Θ → R, where Θ ⊆ R, then it is valid to assume in this course that: k2 k2 X df (x, θ) dX f (x, θ) = . dθ dθ x=k1

x=k1

In words ‘the derivative of the sum is equal to the sum of the derivatives’. This trick is very useful for computing expectations w.r.t. e.g. geometric distributions. Note that this also applies to higher-order derivatives: k2 k2 X d2 f (x, θ) d2 X f (x, θ) = . 2 dθ dθ2 x=k1

x=k1

Pk4

Pk2

2 A double summation x=k1 f (x, y), with f : Z → R, −∞ ≥ k1 ≤ k2 ≤ ∞, with y=k3 f : Z → R, −∞ ≥ k3 ≤ k4 ≤ ∞, with f : Z → R means: k2 k4 X X

f (x, y)

y=k3 x=k1

=

k4 X

f (k1 , y) + · · · + f (k2 , y)

y=k3

= f (k1 , k3 ) + f (k1 , k3 + 1) + · · · + f (k1 , k4 ) + · · · + f (k2 , k3 ) + f (k2 , k3 + 1) + · · · + f (k2 , k4 ). In words: ‘keep y constant and sum over x, then for each x value sum over y’. Note again, that it will be mathematically valid to switch the order of summation in this course, i.e.: k2 k4 X X

f (x, y) =

k4 k2 X X

f (x, y).

x=k1 y=k3

y=k3 x=k1

You should be careful when switching the order of summation, when the first limit depends upon y, e.g. y k4 X X f (x, y). y=k3 x=k1

4.4

Exponential Function

For x ∈ R+ : x n n→∞ n x n lim 1 − n→∞ n lim

4.5

1+

= ex = e−x .

Taylor Series

Under assumptions that are in effect in this course, for infinitely differentiable f : R → R: f (x) =

∞ X (x − c)k k=0

4.6

k!

f (k) (c) c ∈ R.

Integration Methods

In lectures, I will give a review of ‘integration by parts’ and ‘integration by substitution’. The notes may be found online. We have the following standard properties of integrals, for c real and finite and f (x) integrable Z Z cf (x)dx = c

f (x)dx.

46

For two integrable functions f (x) and g(x) we have Z Z Z f (x) + g(x) dx = f (x)dx + g(x)dx. A table of integrals, essentially follows by reversing the derivatives that you saw previously (we omit the constant of integration which is not needed at this level): Z xp+1 xp dx = p 6= −1 p+1 Z a bx e aebx dx = b Z log(x)dx = x log(x) − x Z 1 ax dx = ax a > 0 log(a) Z 1 sin(ax)dx = − cos(ax) a Z 1 cos(ax)dx = sin(ax) a Z tan(ax)dx = log | sec(ax)|. Some ideas for integration: Z

d [f (g(x))] = g 0 (x)f 0 (g(x)) dx

g 0 (x)f 0 (g(x))dx = f (g(x)) + C.

Consider the following integration ‘trick’. Suppose you want to compute Z g(x)dx where g is a positive function. Suppose that there is a PDF f (x) such that f (x) = cg(x) where you know c. Then Z

Z

1 1 f (x)dx = . c c

g(x)dx = Recall that

Z

∞

e−u

I=

2

/2

du =

√

2π.

−∞

This can be established by the fact that Z I2 =

∞

Z

−∞

∞

e−u

2

/2−v 2 /2

dudv

−∞

and then changing to polar co-ordinates.

4.7

Distributions

A Table of discrete distributions can be found in Table 4.1 and continuous Distributions in Table 4.2. Recall that Z ∞ Γ(a) = ta−1 e−t dt a > 0. 0

Note that one can show Γ(a + 1) = aΓ(a) and for a ∈ Z+ , Γ(a) = (a − 1)!. In addition that for a, b > 0 Γ(a)Γ(b) B(a, b) = . Γ(a + b) One can show also that Z 1

ua−1 (1 − u)b−1 du.

B(a, b) = 0

Support X {0, 1} {0, 1, . . . , n} {0, 1, 2, . . . } {1, 2, . . . }

{n, n + 1, . . . }

B(1, p) B(n, p) P(λ) Ge(p)

N e(n, p)

p ∈ (0, 1), n ∈ Z+

Par. p ∈ (0, 1) p ∈ (0, 1), n ∈ Z+ λ ∈ R+ p ∈ (0, 1) 1 − qx

CDF

n/p

E[X] p np λ 1/p n(1 − p)/p2

Var[X] p(1 − p) np(1 − p) λ (1 − p)/p2 pe 1−et (1−p)

1−et (1−p) n t

pet

MGF (1 − p) + pet ((1 − p) + pet )n exp{λ(et − 1)}

Table 4.1: Table of Discrete Distributions.Note that q = 1 − p.

(1 − p)x−1 p x−1 x−n n p n−1 (1 − p)

λx e−λ x!

PMF px(1 − p)1−x n x n−x x p (1 − p)

47

Support X [a, b] R+ R+

R [0, 1]

U[a,b] E(λ) G(a, b)

N (a, b) Be(a, b) B(a, b)

x

b−1

(1 − x)

b a−1 −bx e Γ(a) x 1 1 − 2b (x−a)2 √ e 2πb −1 a−1

a

1 − e−λx

x−a b−a

1 b−a −λx

λe

CDF

PDF

a a/(a + b)

E[X] (a + b)/2 1/λ a/b ab (a+b)2 (a+b+1)

b

Var[X] (b − a)2 /12 1/λ2 a/b2 e

ebt −eat t(b−a) λ λ−t b a ( b−t ) at+bt2 /2

MGF

Table 4.2: Table of Continuous Distributions. Note that B(a, b)−1 = Γ(a + b)/[Γ(a)Γ(b)].

(a, b) ∈ R × R (a, b) ∈ (R+ )2

+

Par. −∞ < a < b < ∞ λ ∈ R+ a, b ∈ R+

48

Bibliography [1] Grimmett, G, & Stirzaker, D. (2001). Probability and Random Processes. Third Edition. Oxford: OUP. [2] Tijms, H. (2012). Understanding Probability. Third Edition. Cambridge: CUP.

49

NUS ST2334 Lecture Notes

Recommend Documents