Advanced Econometrics - Part I - Chapter 2: Finite Sample Properties Of The OLS Estimator

Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 1 University of Economics - HCMC - Vietnam Chapter 2 FINITE SAMPLE PROPERTIES OF THE OLS ESTIMATOR Y = X.β + ε with ],0[~ 2 IN σε • rank(X) = k non-stochastic. ε random → Y random. • YXXX ′′= −1)(βˆ ; βˆ is a statistics on a sample, βˆ is random because Y is random. Being random: - βˆ has a probability distribution, called the sampling distribution. - Repeatedly draw all possible random sample of size n calculate " βˆ " each time. Let explore some statistical properties of the OLS estimator βˆ & build up its sampling distribution. I. UNBIASED: βˆ = YXXX ′′ −1)( = )()( 1 εβ +′′ − XXXX = εβ XXXXXXX I ′′+′′ −− 11 )()(  = εβ XXX ′′+ −1)( E( βˆ ) = ])([ 1 εβ XXXE ′′+ − Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 2 University of Economics - HCMC - Vietnam = ])[( 1 εβ XXXE ′′+ − =  0 1 )()( εβ EXXX ′′+ − = β ⇒ ββ =) ˆ(E βˆ is an estimator of β, it is a function of the random sample (the element of Y). Note: we talk about the sample → that means we talk about Y only. Because X is a constant - fix matrix. "Repeatedly draw all possible random samples of size n → draw Y". The least squares estimator is unbiased for β (E(ε) = 0, X is non-stochastic). → ])')ˆ(ˆ)()ˆ(ˆ[()ˆ(  ββ βββββ EEEVarCov −−= εββ XXX ′′=− −1)(ˆ ) ˆ(βVarCov = ])' ˆ)(ˆ[( ββββ −−E = ])'))(()[( 11 εε XXXXXXE ′′′′ −− = ])(')[( 11 −− ′′′ XXXXXXE εε = 11 )()'()( −− ′′′ XXXEXXX εε = 121 )()( −− ′′′ XXXXXX εσ = 112 )()( −− ′′′ XXXXXX I εσ = 12 )( −′XXεσ So: ) ˆ(βVarCov = 12 )( −′XXεσ For the model: iiii eXXY ++= 3322 ~ˆ~ˆ~ ββ         = 3 2 ˆ ˆ ˆ β β β 12 )( −′XXεσ = ( )∑ ∑∑∑ ∑∑ −        − 2 32 2 3 2 2 2 232 32 2 32 ~~~~ 1 ~~~ ~~~ iiiiiii iii XXXXXXX XXX εσ Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 3 University of Economics - HCMC - Vietnam =         3 2 ˆ ˆ β β VarCov → ) ˆ(βVar = ( )∑ ∑ ∑ − 2 32 2 3 2 2 2 3 2 ~~~~ ~ iiii i XXXX Xεσ =    32 2 23 ; 2 3 2 2 2 2 32 2 2 2 ~~ )~~( 1 ~/ ii XXbetweenncorrelatiosample r ii ii i n X n X n XX X ∑∑ ∑ ∑ − εσ → ) ˆ(βVar = ∑ − )1(~ 22322 2 rX i εσ determined by: i. 2 εσ ↑ → )ˆ(βVar ↑ ii. 2 23r ↑ → )ˆ(βVar ↑ iii. Variation in Xi2 ∑ 22 ~ iX ↑ → )ˆ(βVar ↓ iv. n sample size ↑ → ) ˆ(βVar ↓ )ˆ(βVarCov = 12 )( −′XXεσ → we don't know 2 εσ → need an estimator for 2 εσ . Define: 2ˆεσ = kn ee − ' n: observations. k: number of estimators. ∑= 2' ieee = sum of squares. • Show 2ˆεσ is an unbiased estimator. e = Mε → e'e = ε'M'Mε=ε'Mε • Note: trace of a square matrix. Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 4 University of Economics - HCMC - Vietnam nn A × is the sum of its principal diagonal elements (= ∑ = n i iia 1 ). Rules: A, B nxn matrix tr(A+B) = tr(A) + tr(B) tr(A.B) = tr(B.A) tr(λA) = λtr(A) Trace is a linear operation → sum of certain elements. )'( eeE = )'( εε ME = )]'([)]'([ MtrEMtrE εεεε = = )]..[)'( 2 MItrMtrE εσεε = = )(2 Mtrεσ = )]')'(()([ 12 XXXXtrItr n −−εσ = )]')'(([ 12  kkI XXXXtrn × −−εσ = )( 2 kn −εσ And: 2 2 )()'( ε ε σ σ = − − = − kn kn kn eeE So: 22 )ˆ( εε σσ =E → 2ˆεσ is an unbiased estimator of 2 εσ . II. LINEARITY: Any estimator that is a linear function of the random sample data is called a linear estimator. Yi: random sample data.    1 1 1 .)(ˆ ×× − × =′′= nnkAk YAYXXX β where A is non-random: Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 5 University of Economics - HCMC - Vietnam               kβ β β ˆ ˆ ˆ 2 1  =             knkk n n aXa aaa aaa     21 22221 11211             nY Y Y  2 1 → 112121111 ...ˆ knYaYaYa +++=β → βˆ , OLS estimator is linear and unbiased for β. Because βˆ is a linear function of Y and Y is a linear function of ε, → if ε is normal then βˆ is normal. So the sampling distribution of the OLS estimator of β is: βˆ ~ N[β, 12 )( −′XXεσ ] III. EFFICIENCY: Suppose we have 2 unbiased estimators, 1ˆθ ; 2ˆθ for θ . Then we say 1ˆθ is more efficient than 2ˆθ if ) ˆ()ˆ( 21 θθ VarVar ≤ . If  1 1ˆ ×k θ ;  1 2ˆ ×k θ are vectors unbiased estimators of  1×k θ , then 1ˆθ is more efficient than 2ˆθ if )]ˆ()ˆ([ 21 θθ VV −=∆ is positive semi-definite. IV. GAUSS - MARKOV THEOREM: "Under the assumptions of the classical regression model, the least squares estimators of β, YXXX ′′= −1)(βˆ are the best linear unbiased estimators". (BLUE). Linear: in Y Best: Best for any alternative linear on unbiased estimators. jbVarVar jj ∀≤ )()ˆ(β . Proof: Let b is any other linear estimator of β: Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 6 University of Economics - HCMC - Vietnam    11 . ××× = nnkk YAb Unbiased: E(b) = β E(b) = E(AY) =E(AXβ + Aε) E(b) = AXβ + 0 = AXβ = β → AX =I Let A = (X'X)-1X' + C where C is any non-stochastic (k×n) matrix. 0')'(]')'[( 11 ==+=+== −− CXCXXXXXXCXXXAXI I  ]][')'[( 1 εβ ++== − XCXXXAYb εβεβ CCXXXXXXXX I +++= −− ')'(')'( 11  εεβ CXXX ++= − ')'( 1 ])')([()( ββ −−= bbEbVarCov }]'')'][(')'{[( 11 εεεε CXXXCXXXE ++= −− ]'')'('')'()'()'()'(')'[( 1111 CCXXXCCXXXXXXXXE εεεεεεεε +++= −−−− ')'('')'()'(')'( 21212112 CCXXCXCXXXXXXXXX I εεεε σσσσ +++= −−−−     )ˆ( 212 ')'( β εε σσ VarCov CCXX += − The jth diagonal element: ∑ = += n i jijj cVarbVar 1 22)ˆ()( εσβ kjVar j ,1)ˆ( =∀≥ β → )( jbVar kjVar j ,1)ˆ( =∀≥ β → jβˆ is the best linear unbiased estimator (BLUE). → jβˆ is efficient estimator (smallest variance). Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 7 University of Economics - HCMC - Vietnam V. REVIEW: STATISTICAL INFERENCE: 1. Linear function of normal random variables are also normal:    ),(~ 11 nnnn Nu ××× Σµ →    11 ××× = nnmm uPZ is normally distributed. µPuPEPuEZE === )()()( ]))'())(([()( ZEZZEZEZVarCov −−= ])')([( µµ PPuPPuE −−= ''])')([( PPPuuEP Σ=−−= Σ    µµ Then )',(~ PPPNZ Σµ 2. Chi-squared distribution: If ),0(~ 1 INZ r× then Z'Z has the Chi-squared distribution with r degree of freedom or 2 ][~' rZZ χ Z'Z r: number of these independent standard normal variables in the sum of squares: Theorem: If ),0(~ 1 INZ r× and nn A × is idempotent with rank equal to r, then: i. 2 ][~' rAZZ χ ii. )()( ArankAtrr == 3. Eigenvalue - eigenvector problem: For a square matrix nn A × , we can find n pairs of ),( 111 ×× n jj cλ such that: nn A × = ×1n jc )( 111 ×× n jj cλ j = 1,2, ... , n Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 8 University of Economics - HCMC - Vietnam normalizing: ∑ = == n j jjj ccc 1 2 )1(1' The eigenvectors are orthogonal to each other: )(0' jicc ji ≠∀= so c = [c1, c2, ..., cn] is an orthogonal matrix: 1'(' −== ccIcc ) Eigenvalue - eigenvector problem: nn A × = ×1n jc )( 111 ×× n jj cλ j = 1,2, ... , n 1' =jj cc )(0' jicc ji ≠∀= cj =               nj j j c c c  2 1 Let: nnnnn IcccccC ×× =⇒= '][ 21  → c' = c-1: orthogonal matrix: ][][][ 22112121 nnnn cccAcAcAccccAAC λλλ  === ][ 21 ncccAC = Λ=             Λ C n       λ λ λ 00 00 00 2 1 where Λ is a diagonal matrix: Λ=Λ= CCACC '' and also =Λ= )()( RankARank number of no-zero of λj's. Note: Λ=ACC ' → ')'('' 1111 CCCCACCCC Λ=Λ= −−−− Remember: 'CCA Λ= and Λ=ACC ' ; C'C = I, C' = C-1 Theorem: Let A be an idempotent matrix with rank = r and let ),0(~ 1 INZ r× then: 2 ][~' rAZZ χ and )()( AtrArank = Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 9 University of Economics - HCMC - Vietnam Proof: Λ=ACC ' , ),0(~ 1 INZ r× For A idempotent, λj = 0 or 1 Because: jjj CAC λ= → 2 jjjjj CACAAC λλ == So: 2jjC λ = jjC λ → 0)( 2 =− jjjC λλ → 0)1( =−jjjC λλ → 0=jλ or 1=jλ Write: Λ=ACC ' =                 0000 0100 0010 0001      There must be r nonzero elements of Λ , because )()()( Λ=Λ== trrankrArank since all diagonal elements are 0 or 1. (Rule: tr(A.B) = tr(B.A)) Also )()'()( AtrACCtrtr ==Λ so rAtrArank == )()(    ),' 11 ××× = nnnn ZCu ),0(~ 1 INZ n× ICCCZZECCZZCEuuE I ==== ')'(')''()'(  Contruct quadratic form: AZZZCACCCZuu '')'('' ==Λ ∑ = = n i iu 1 2 2 ][~ rχ So if ),0(~ INZ and nn A × is idempotent with rank equal to r, then 2 ][~' rAZZ χ Extension: So if ),0(~ 2 INZ σ , then 2 ][2 ~ ' r AZZ χ σ 4. Other distribution: Let Z be N(0,I) and let W be 2 ][rχ and let Z and W be independently distributed, then: Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 10 University of Economics - HCMC - Vietnam ][~ rt r W Z has the t-distribution with r degree of freedom. Let W be 2 ][rχ and let v be 2 ][sχ and W and v be independently distributed, then: r sF s v r W ~ has the F-distribution with r (numerator) and s (denominator) degree of freedom. VI. TESTING HYPOTHESIS ON INDIVIDUAL COEFFICIENT: Y = X.β + ε with ],0[~ 2 IN σε • Recall: βˆ ~ N[β, 12 )( −′XXεσ ] So jβˆ ~ N[βj, ijXX ])[( 12 −′εσ ] ]1,0[~ )'( ˆ 12 N XX jj jj − − → σ ββ but σ2, so this can't be used directly for constructing test or confidence intervals. εεεε MMMee '''' == , M is idempotent with with rank(M) = its trace = n-k. ],0[~ 2 )1( IN n σε × → ],0[~/ INσε ⇒ 2 ][22 ~ '' kn Mee −= χσ εε σ So follow theorem: kn jj jj t kn ee XX − − − − ~ )( ' )'( ˆ 2 12 σ σ ββ ⇔ kn jj jj t XX kn ee −− − − ~ )'(' ˆ 1 ˆ 2  σ ββ Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 11 University of Economics - HCMC - Vietnam ⇔ kn jj jj t XX −− − ~ )'(ˆ ˆ 12σ ββ 2 ˆ 12 ˆ)'(ˆ jjj XX β σσ =− = standard error of jβˆ . Finally: kn jj t j − − ~ ˆ ˆ 2 βˆ σ ββ This basic result enables us to test hypothesis about elements of β and to construct confidence intervals for them (note that we need the assumption of normality of ε's). EX: 3)4.1(205.0)7.0( 6.02.04.1ˆ iii xxy ++= H0: β2 = 0 H1: β2 > 0 4 05.0 02.0 )ˆ( ˆ = − = − = i jj SE t β ββ 74.1%)5( =αt d.o.f = n-k =17. 567.2%)1( =αt αtt > → reject H0. EX: H0: β1 = 1.5 H1: β2 ≠ 1.5 ( or ≥ 1.5 or ≤ 1.5) 1429.0 7.0 5.14.1 )ˆ( ˆ −= − = − = i jj SE t β ββ d.o.f = n-k =17. %5.2%5.2 Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 12 University of Economics - HCMC - Vietnam 2/αtt < ⇒ cannot reject H0 at 5%. VII. CONFIDENCE INTERVALS: Recall: kn i ii i tSE t − − = ~ )ˆ( ˆ β ββ so ααα −=−≤≤− 1]Pr[ 2/2/ ttt i α β ββ αα −=−≤ − ≤− 1] )ˆ( ˆ Pr[ 2/2/ tSE t i ii αβββββ αα −=+≤≤− 1)]ˆ(ˆ)ˆ(ˆPr[ 2/2/ iiiii SEtSEt • If we were to take a sample of size "n", construct this repeat many times then 100(1-α)% of such intervals would cover the true value of βi • If we construct the interval once, there is no guarantee that the internal will cover the true βi]. • Type of errors: size & power of tests. Type I: Reject H0 when it is true. Type II: Accept H0 when it is false. Assume: Prob(type I error) = α Prob(type II error) = β If sample size is fixed: α↓ ⇒ β↑ call α: significant level or size of the test. → Fix α and try to design the test so to minimize β. • Definition: The power of a test is 1- β. Power = 1 - Pr(accept H0/H0 false) = Pr(reject H0/H0 false) Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 13 University of Economics - HCMC - Vietnam • A test is "uniformly most powerful" if its power exceeds that of any other test (for the same choice of α) over all possible alternative hypothesis. • A test is "consistent" if its power → 1 as n →∞ for any false hypothesis. • A test is unbiased of its power never falls below α. VIII. FAMILY OF F-TEST: For general linear restrictions, unrestricted model (U-model), original model. H0: some restrictions on 1×k β . These define the restricted model (R-model): dfuESS rESSESSF U URr dfu /) /)( − = ESSR = error sum of squares from R-model: RRee′ ESSU = error sum of squares from U-model: UU ee′ r: number of restrictions in H0. dfu: degree of freedom in U-model = n-k. =2σ UESS 2σ UU ee′ 2σ εε M′ = σ ε σ ε M ′ = 2 ][~ kn−χ       − −− 2 ][2 2 )]([2 ~ ~ kn U rkn R ESS ESS χ σ χ σ → 2 ][22 ~ r UR ESSESS χ σσ − )/() /)( )/() /)( 2 2 knESS rESSESS knESS rESSESS U UR U UR − − = − − σ σ → r kn U UR F knESS rESSESS −− − ~ )/() /)( Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 14 University of Economics - HCMC - Vietnam Case 1: Join significant of all slopes: 1 12 1 1 −×       = kk β β β H0: 10 1)1( 2 −=→= ×− kr k β U-model: εβ += ×1k XY → ESSU =e'e dfu = n-k R-model: iiY εβ += 1 → Y+1βˆ → ii eYY += ∑ = −= n i iR YYESS 1 2)( → )/()1( )1/( )/(' )1/()')(( 2 2 1 2 1 knR kR knee keeYY F n i i k kn −− − = − −−− = ∑ =− − Case 2: rk rk − ×       = 2 1 1 β β β H0: 11 2 0×× = rr β U-model: εβ += XY → ESSU = UU ee′ R-model: εβ += ×− 1)( rk XY → ESSU = RRee′ ∑ = −= n i iR YYESS 1 2)( → )/() /)( knESS rESSESSF U URr kn − − =− EX: Translog of production function: εββββββ ++++++= )log(log2/)(log2/)(loglogloglog 6 2 5 2 4321 LKLKLKY 0: 6540 === βββH Cobb-Douglas restrictions. n = 27 ESSU = 0.67993 r = 3 ESSR = 0.85163 n - k = 21 Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Nam T. Hoang University of New England - Australia 15 University of Economics - HCMC - Vietnam → 768.1=− r knF . Critical value: 1.3 3 %5,21 =F → r knF − < Critical value ⇒ So do not reject H0 and conclude that are consistent with the Cobb-Douglas model. Case 3: General restrictions. 11 ××× = rkkr CR β           = 2 2 1 β β β β Restrictions: [ ] )1(1110 1 11 3 1 2 ==→ =+ ××× r R rrr β ββ  If restrictions:       =      → =    = =+ 0 1 001 110 )2( 0 1 1 32 β β ββ r Jarque - Beta statistics: H0: εi are normally distributed. H1: εi are not normally distributed. JB 22~ χ JB = SK 2 +(Kur)2 Reject H0 for large JB. Reject H0 if JB >7 (critical) or if p-value < 0.05