ࡱ > T V S c U@ 9? bjbj . q6
<+ <+ <+ 8 t+ 4 + J 2 , v- - - - g. g. g. PJ RJ RJ RJ RJ RJ RJ $ L R UN vJ / g. g. / / vJ - - J 5 5 5 / - - C Z 5 / PJ 5 ( 5 6 V > @ f? - , `ȗ Tb <+ #0 > ? $ J 0 J > x N 1 x N f? N f? l g. {. 5 . . p g. g. g. vJ vJ ( <+ 5 ^ <+ The analysis of categorical data: Fishers Exact test
Jenny V Freeman, Michael J Campbell
Introduction
In the previous tutorial we have outlined some simple methods for analysing binary data, including the comparison of two proportions using the Normal approximation to the binomial and the Chi-squared test( ADDIN REFMGR.CITE Freeman2007320The analysis of categorical dataJournal320The analysis of categorical dataFreeman,J.V.Julious,S.A.2007scopeStatisticsNot in File1821Scope161Scope1(1)). However, these methods are only approximations, although the approximations are good when the sample size is large. When the sample size is small we can evaluate all possible combinations of the data and compute what are known as exact P-values.
Fishers Exact test
When one of the expected values (note: not the observed values) in a 2x2 table is less than 5, and especially when it is less than 1, then Yates correction can be improved upon. In this case Fishers Exact test, proposed in the mid-1930s almost simultaneously by Fisher, Irwin and Yates ADDIN REFMGR.CITE Armitage2002319Statistical Methods in Medical ResearchBook, Whole319Statistical Methods in Medical ResearchArmitage,P.Berry,P.J.Matthews,J.N.S.2002StatisticsNot in File1344OxfordBlackwells2(2), can be applied. The null hypothesis for the test is that there is no association between the rows and columns of the 2x2 table, such that the probability of a subject being in a particular row is not influenced by being in a particular column. If the columns represented the study group and the rows represented the outcome, then the null hypothesis could be interpreted as the probability of having a particular outcome is not influenced by the study group, and the test evaluates whether the two study groups differ in the proportions with each outcome.
An important assumption for all of the methods outlined, including Fishers exact test, is that the binary data are independent. If the proportions are correlated, then more advanced techniques should be applied. For example in the leg ulcer example of the previous tutorial ADDIN REFMGR.CITE Freeman2007320The analysis of categorical dataJournal320The analysis of categorical dataFreeman,J.V.Julious,S.A.2007scopeStatisticsNot in File1821Scope161Scope1(1), if there were more than one leg ulcer per patient, we could not treat the outcomes as independent.
The test is based upon calculating directly the probability of obtaining the results that we have obtained (or results more extreme) if the null hypothesis is actually true, using all possible 2x2 tables that could have been observed, for the same row and column totals as the observed data. These row and column totals are also known as marginal totals. What we are trying to establish is how extreme our particular table (combination of cell frequencies) is in relation all the possible ones that could have occurred given the marginal totals.
This is best explained by a simple worked example. The data below come from an RCT comparing intra-muscular magnesium injections with placebo for the treatment of chronic fatigue syndrome ADDIN REFMGR.CITE Cox1991318Red blood cell magnesium and chronic fatigue syndromeJournal318Red blood cell magnesium and chronic fatigue syndromeCox,I.MCampbell,M.J.Dowson,D.1991riskStatisticsNot in File757760Lancet337Lancet1(3). Of the 15 patients who had the intra-muscular magnesium injections 12 felt better (80%), whereas, of the 17 on placebo, only 3 felt better (18%).
Table 1: Results of the study to examine whether intramuscular magnesium is better than placebo for the treatment of chronic fatigue syndrome
MagnesiumPlaceboTotal Felt better
Did not feel better12
33
1415
17Total151732
There are 16 different ways of rearranging the cell frequencies for the above table, whilst keeping the marginal totals the same, as illustrated below in figure 1. The result that corresponds to our observed cell frequencies is (xiii):
Figure 1: Illustration of all the different ways of rearranging cell frequencies in table 1, but with the marginal totals remaining the same
(i) 0 15
15 2(ii) 1 14
14 3(iii) 2 13
13 4(iv) 3 12
12 5(v) 4 11
11 6(vi) 5 10
10 7(vii) 6 9
9 8(viii) 7 8
8 9(ix) 8 7
7 10(x) 9 6
6 11(xi) 10 5
5 12(xii) 11 4
4 13(xiii) 12 3
3 14(xiv) 13 2
2 15(xv) 14 1
1 16(xvi) 15 0
0 17
The general form of table 1 is given in table 2 and under the null hypothesis of no association Fisher showed that the probability of obtaining the frequencies, a, b, c and d in table 2 is
EMBED Equation.3 (1)
where x! is the product of all the integers between 1 and x, e.g. 5!=1x2x3x4x5=120 (note that for the purpose of this calculation, we define 0! as 1). Thus for each of the results (i) to (xvi) the exact probability of obtaining that result can be calculated (table 3). For example, the probability of obtaining table (i) in figure 1 is EMBED Equation.3 =0.0000002
Table2: General form of table 1
Column 1Column 2TotalRow 1
Row 2a
cb
da+b
c+dTotala+cb+da+b+c+d
Table 3: Probabilities associated with each of the frequency tables above, calculated using formula 1
TableabcdP-value i
ii
iii
iv
v
vi
vii
viii
ix
x
xi
xii
xiii
xiv
xv
xvi0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
1515
14
13
12
11
10
9
8
7
6
5
4
3
2
1
015
14
13
12
11
10
9
8
7
6
5
4
3
2
1
02
3
4
5
6
7
8
9
10
11
12
13
14
15
16
170.0000002
0.0000180
0.0004417
0.0049769
0.0298613
0.1032349
0.2150728
0.2765221
0.2212177
0.1094916
0.0328475
0.0057426
0.0005469
0.0000252
0.0000005
0.0000000
From table 3 we can see that the probability of obtaining the observed frequencies for our data is that which corresponds with (xiii), which gives p=0.0005469 and the probability of obtaining our results or results more extreme (a difference that is at least as large) is the sum of the probabilities for (xiii) to (xvi) = 0.000573. This gives the one-sided P-value or obtaining our results or results more extreme, and in order to obtain the two-sided p-value there are several approaches. The first is to simply double this value, which gives p=0.0001146. A second approach is to add together all the probabilities that are the same size or smaller than the one for our particular result, in this case, all probabilities that are less than or equal to 0.0005469, which are tables (i), (ii), (iii), (xiii), (xiv), (xv) and (xvi). This gives a two-sided value of p=0.001033. Generally the difference is not great, though the first approach will always give a value greater than the second. A third approach, which is recommended by Swinscow and Campbell ADDIN REFMGR.CITE Swinscow2002209Statistics at square oneBook, Whole209Statistics at square oneSwinscow,T.D.VCampbell,M.J.2002displaying dataStatisticsNot in File10LondonBMJ Books2(4) is a compromise and is known as the mid-p method. All the values more extreme than the observed p-value are added up and these are added to one half of the observed value. This gives p=0.000759.
The criticism of the first two methods is that they are too conservative, i.e. is the null hypothesis was true, over repeated studies they would reject the null hypothesis less often than 5%. They are conditional on both sets of marginal totals being fixed, i.e. exactly 15 people being treated with magnesium and 15 feeling better. However, if the study were repeated, even with 15 and 17 in the magnesium and placebo groups respectively, we would not necessarily expect exactly 15 to feel better. The mid-p value method is less conservative, and gives approximately the correct rate of type I errors (false positives).
In either case, for our example, the P-value is less than 0.05, the nominal level for statistical significance and we can conclude that there is evidence of a statistically significant difference in the proportions feeling better between the two treatment groups. However, in common with other non-parametric tests, Fishers exact test is simply a hypothesis test. It will merely tell you whether a difference is likely, given the null hypothesis (of no difference). It gives you no information about the likely size of the difference, and so whilst we can conclude that there is a significant difference between the two treatments with respect to feeling better or not, we can draw no conclusions about the possible size of the difference.
Example data from last week
Table 1 shows the data from the previous tutorial. It is from a randomised controlled trial of community leg ulcer clinics ADDIN REFMGR.CITE Morrell1998231Cost effectiveness of community leg ulcer clinic: randomised controlled trialJournal231Cost effectiveness of community leg ulcer clinic: randomised controlled trialMorrell,C.J.Walters,S.J.Dixon,S.Collins,K.Brereton,L.M.L.Peters,J.Brooker,C.G.D.1998scopeStatisticsNot in File14871491British Medical Journal316British Medical Journal1(5), comparing the cost-effectiveness of community leg ulcer clinics with standard nursing care. The columns represent the two treatment groups, specialist leg ulcer clinic (clinic) and standard care (home), and the rows represent the outcome variable, in this case whether the leg ulcer has healed or not.
Table 1: 2 x 2 contingency table of Treatment (clinic/home) by Outcome (ulcer healed / not healed) for the Leg ulcer study
TreatmentClinicHomeTotalOutcome:
Healed
Not healed
22 (18%)
98 (82%)
17 (15%)
96 (85%)
39
194Total120 (100%)113 (100%)233
For this example the two-sided p-value from Fishers exact test is 0.599 two-sided and in this case we would not reject the null and would conclude that there is insufficient evidence to
Summary
This tutorial has described in detail Fishers exact test, for analysing simple 2x2 contingency tables when the assumptions for the chi-squared test are not met. It is tedious to do by hand, but nowadays is easily computed by most statistical packages.
ADDIN REFMGR.REFLIST Reference List
(1) Freeman JV, Julious SA. The analysis of categorical data. Scope 2007;16(1):18-21.
(2) Armitage P, Berry PJ, Matthews JNS. Statistical Methods in Medical Research. 4 ed. Oxford: Blackwells; 2002.
(3) Cox IM, Campbell MJ, Dowson D. Red blood cell magnesium and chronic fatigue syndrome. Lancet 1991;337:757-60.
(4) Swinscow TDV, Campbell MJ. Statistics at square one. 10 ed. London: BMJ Books; 2002.
(5) Morrell CJ, Walters SJ, Dixon S, Collins K, Brereton LML, Peters J, et al. Cost effectiveness of community leg ulcer clinic: randomised controlled trial. British Medical Journal 1998;316:1487-91.
When organising data such as this is it good practice to arrange the table with the grouping variable forming the columns and the outcome variable forming the rows.
Page PAGE 1
6 7 H Z \ ] j 7 8 9
" # % , 1 2 = > ) : + - 7 ? ' < @ d f Ҿںںڮڮڮ h
1 h j h Uh[ hy hg hS j hS Uh*p hM h> h j h> UhO h4 5\ hg 5h4 h4 5 A 6 7 \ ] j k
, - z { $$If a$gd gda gd q> ? 8? f i n s Y Z l m p q I \ l t u v m n o p q r ŻŔ h; h; CJ aJ h; h h; 5\ hZ h> ha 5\ j h> h> 0J 5U h> h 5\ h h 5\ h ha 5\ h 5\ h j ha Uha h h[ h
1 hq 3 ^ O O A A A A $x $If a$gd
@x $If gd kd $$If l \ H$ Z Z Z Z
t 0 6 4 4
l a P E x $If gd kd $$If l \ H$ Z Z Z Z
t 0 6 4 4
l a $x $If a$gd n o ^ Y T Y T O gd; gd gda kd $$If l \ H$ Z Z Z Z
t 0 6 4 4
l a o s y
h8BBx $If gd $x $If a$gdZ
h8BBx $If gd; $
h8BBx $If a$gdZ 7 # $
h8BBx $If a$gdZ kd $$If l 0ִ LT x <# h
t 6 4 4
l a $x $If a$gd $x $If a$gdZ
h8BBx $If gd; 7 # $
h8BBx $If a$gdZ kd $$If l [ ִ LT x <# h
t 6 4 4
l a ! !
! !
! ! ! ! ! ! ! ! !! #! $! %! '! (! )! +! ,! .! /! 1! 2! 4! 5! 6! 7! 9! :! =! >! @! B! C! D! F! G! H! J! K! L! T! V! Z! [! h! i! l! m! o! q! r! s! u! v! w! y! z! {! }! ~! ! ! ! ! ! ! ! ! ! ! ! ! h; h hZ h; hZ CJ aJ \
h8BBx $If gd; $x $If a$gdZ
h8BBx $If gd ! ! 9 % $
h8BBx $If a$gdZ kd. $$If l ִ LT x <# h
t 6 4 4
l a ! ! ! ! ! ! ! ! $x $If a$gd $x $If a$gdZ
h8BBx $If gd; ! ! ! 9 % $
h8BBx $If a$gdZ kd $$If l ִ LT x <# h
t 6 4 4
l a ! ! ! ! "! (! -! 3! 9! ?! E! K!
h8BBx $If gd
h8BBx $If gdZ $x $If a$gdZ
h8BBx $If gd; K! L! M! 9 % $
h8BBx $If a$gdZ kd $$If l ִ LT x <# h
t 6 4 4
l a M! N! O! P! Q! R! S! T! $x $If a$gd $x $If a$gdZ
h8BBx $If gd; T! U! \! 9 % $
h8BBx $If a$gdZ kdC $$If l ִ LT x <# h
t 6 4 4
l a \! b! h! n! t! z! ! ! ! ! ! !
h8BBx $If gdZ $x $If a$gdZ
h8BBx $If gd
h8BBx $If gd; ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! _" `" s" t" u" v" z" {" # # # # # # # # $ $ `$ a$ $ `& a& & & & & ' !' D' m' ' 9( @( u( 쭠왏}}}}} hMj h> h hg 5\ h h 5\ h 5\ j h h EHU-jfڗI
h CJ UVaJ mH nHsH tHj hg h EHU jǖI
h CJ UVaJ j h Uhg h hq ha hZ h; 1! ! ! ! ]" ^" 9 4 2 - - gd gda kd^
$$If l ִ LT x <# h
t 6 4 4
l a ^" {" |" # # $ $ $ #$ )$ $$If a$gdg gd
T gd )$ *$ 0$ 6$ 8$ :$ <$ >$ ^ O O A A A A $x $If a$gdg
x $If gdq kd# $$If l \ @ 8 T T
t 0 6 4 4
l a >$ B$ F$ G$ M$ Q$ U$ ]$ P E x $If gdg kd, $$If l \ @ 8 T T
t 0 6 4 4
l a $x $If a$gdg ]$ ^$ _$ `$ a$ $ $ $ $ ^ Y W Y R F F F $$If a$gd gd gdg kd $$If l \ @ 8 T T
t 0 6 4 4
l a $ $ $ $ $ $ $ $ S D D D
Bx $If gd kd $$If l ֈ 8p l 8 8 8 8 T
t 6 4 4
l a $$If a$gd $ $ $ $ $ % % % % % % % % $% &% (% *% ,% .% 0% 2% 4% 6% 8% ;% >% A% $x $If a$gd
Bx $If gd A% D% G% J% M% P% S% V% Y% \% ^% `% b% d% f% h% j% l% n% p% s% v% y% |% % % % % % $x $If a$gd % % % % % % % % % % % % % % % % % % % % % % % % % % % % % $x $If a$gd % % & & & "& ,& 6& @& J& T& ^& $x $If a$gd ^& _& `& a& - - 00 10 3 _ Z U P P P P K gd- gdMj gdg gd kd $$If l ֈ 8p l 8 8 8 8 T
t 6 4 4
l a u( ( ]) ) ) ) ?* * * , , , , - - 00 81 91 j1 k1 2 2 3 3 43 3 3 7 7 7 7 0: ;: n: : : : : : ; ; ; ; ; m> n> o> p> q> r> ? ? ? ? ? ? !? տݷ쯥շ j he-- Uhe-- j h> 0J Uhg hg 5\ j h4 Uh4 h[ h4 5\ j hg U hg 5\ hg hy h- h- h- h h j h> UhMj h> 83 3 43 53 9 9 9 9 9 9 9 ^ kd $$If l F $ Z Z
t 0 6 4 4
l a $$If a$gdg gdg
9 9 9 9 9 9 9 R G x $If gdg kd $$If l \ H$ Z Z Z Z
t 0 6 4 4
l a $$If a$gdg 9 9 9 9 9 9 9 9 9 9 9 9 $x $If a$gdg x $If gdg
Ax $If gdg 9 9 : : : : ^ S E E E $x $If a$gdg x $If gdg kd $$If l \ H$ Z Z Z Z
t 0 6 4 4
l a : : : : : : : ; ; ^ Y T Y R R M R gd- gdMj gdg kd $$If l \ H$ Z Z Z Z
t 0 6 4 4
l a ; < < `< < G= = l> m> p> q> ? ? ? ? ? ? ? "? #? %? &? gd> gd>
gdg
h ^ `gd
h ^ `gd $a$gd !? #? $? &? +? ,? 2? 3? 4? 5? 7? 8? 9? h4 h 0J mH nH u
h> 0J j h> 0J Uh> j he-- Uhe-- &? 6? 7? 8? 9?
gdg $a$gd> , 1h . A!"#$% $$If !v h5 Z 5Z 5Z 5Z #v Z :V l
t 0 6 5 Z / / / / / / $$If !v h5 Z 5Z 5Z 5Z #v Z :V l
t 0 6 5 Z / /
/ / $$If !v h5 Z 5Z 5Z 5Z #v Z :V l
t 0 6 5 Z / / / / / $$If !v h5 55h55555#v #v#vh#v#v#v#v#v:V l 0
t 6 5 55h55555/ / / / / / / / / / / $$If !v h5 55h55555#v #v#vh#v#v#v#v#v:V l [
t 6 5 55h55555/ / / / {$$If !v h5 55h55555#v #v#vh#v#v#v#v#v:V l
t 6 5 55h55555/ / / / / / / / / / / $$If !v h5 55h55555#v #v#vh#v#v#v#v#v:V l
t 6 5 55h55555/ / / / {$$If !v h5 55h55555#v #v#vh#v#v#v#v#v:V l
t 6 5 55h55555/ / / / / / / / / / / $$If !v h5 55h55555#v #v#vh#v#v#v#v#v:V l
t 6 5 55h55555/ / / / {$$If !v h5 55h55555#v #v#vh#v#v#v#v#v:V l
t 6 5 55h55555/ / / / / / / / / / / D d
b
c $ A ? ? 3 " ` ? 2 ( W90Q^B
K N `! W90Q^B
K 6 @ @ xڕK/QϹ3}:`Z">A%H$"z UIز+Xxl,l%#xnu aa;9{NE0 _
ejYRQ6yj<#Q{8P`s>)_\Ը˱`8SXN6 _ߙrGUF,%
8b lKeg`Wr٭H 2prR+VceU} DSuk)'-ۦh?.z.u/k~mQ}Ҡt7:o5A? NeJ1˜53> ԟPł h`PfB4h{P_?`Ors53\J=935Gq5*Jt})wAvq$Fդ<M"NE&J[lb ؓ03I7j D d
l#( b
c $ A ? ? 3 " ` ? 2 \lMײZ N `! \lMײZv x V xcdd``f 2
ĜL0##0KQ*
W
! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J L M N O P Q R [ U X Y Z ` a \ ] ^ _ b e f g h i j k l m n o p q r s t u v w x y z { | } ~ R o o t E n t r y F 0L TbW D a t a
K \ W o r d D o c u m e n t . O b j e c t P o o l p~ Tb0L Tb _ 1 2 3 4 6 1 8 2 6 2 F p~ Tbp~ Tb O l e
C o m p O b j f O b j I n f o
F Microsoft Equation 3.0 DS Equation Equation.3 9q =º xY
( a + b ) ! ( c + d ) ! ( a + c ) ! ( b + d ) ! ( a + b + c + d ) ! a ! b ! c ! d ! E q u a t i o n N a t i v e _ 1 2 3 4 6 8 8 6 1 4 F p~ Tbp~ Tb O l e
C o m p O b j
f
F Microsoft Equation 3.0 DS Equation Equation.3 9q v P@
1 5 ! 1 7 ! 1 5 ! 1 7 ! 3 2 ! 0 ! 1 5 ! 1 5 ! 2 ! Oh +'0 O b j I n f o
E q u a t i o n N a t i v e 1 T a b l e d O S u m m a r y I n f o r m a t i o n ( A ? d v@=P5< %!@5 @_L ĺE ,a L
*'031d++&1X +|-Ikˈ UOaY@54Xp> 3n&Ty0 ƈ* ol2 GOboKB F?\F1&TgQ3@sAc e( p321)W20C sg!h<2@Y$$If !v h5 85T5T5#v 8#vT#v:V l
t 0 6 5 85T5/ / / / / / a $$If !v h5 85T5T5#v 8#vT#v:V l
t 0 6 5 85T5/ /
/ / a $$If !v h5 85T5T5#v 8#vT#v:V l
t 0 6 5 85T5/ / / / / a $$If !v h5 585858585T#v #v8#vT:V l
t 6 5 585T/ / / / / / / / a $$If !v h5 585858585T#v #v8#vT:V l
t 6 5 585T/ / / / / / / / a $$If !v h5 Z 55Z #v Z #v#vZ :V l
t 0 6 5 Z 55Z / $$If !v h5 Z 5Z 5Z 5Z #v Z :V l
t 0 6 5 Z / / / / / / $$If !v h5 Z 5Z 5Z 5Z #v Z :V l
t 0 6 5 Z / /
/ / $$If !v h5 Z 5Z 5Z 5Z #v Z :V l
t 0 6 5 Z / / / / / 4 @
\ h t
! The analysis of categorical data icr he Authorised User uth uth Normal.dot U Authorised User 10 h Microsoft Word 10.0 @ C @ D8@ ]@ NSb Q. ՜. +,0 h D o c u m e n t S u m m a r y I n f o r m a t i o n 8 L C o m p O b j j p
Authorised Organization b V6 {
! The analysis of categorical data Title
F Microsoft Word Document
MSWordDoc Word.Document.8 9q @ @ @ N o r m a l CJ _HaJ mH sH tH : @ : H e a d i n g 1 $@& 5\ D A@ D D e f a u l t P a r a g r a p h F o n t V i@ V T a b l e N o r m a l :V 4 4
l a ( k ( N o L i s t 4 @ 4 H e a d e r
9r > @ >
F o o t n o t e T e x t CJ aJ @ &@ @ F o o t n o t e R e f e r e n c e H* 4 @ "4 F o o t e r
9r . )@ 1. > P a g e N u m b e r 97 97 6 7 \ ] j k , - z { n o s y " ( - 3 9 ? E K L M N O P Q R S T U \ b h n t z ] ^ { | # ) * 0 6 8 : < > B F G M Q U ] ^ _ ` a $ & ( * , . 0 2 4 6 8 ; > A D G J M P S V Y \ ^ ` b d f h j l n p s v y | " , 6 @ J T ^ _ ` a % % 0( 1( + + 4+ 5+ 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 4 4 `4 4 G5 5 l6 m6 p6 q6 7 7 7 7 7 "7 %7 67 :7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 x 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 x 0 x 0 x 0 x 0 x 0 x 0 x 0 x 0 x 0 x 0 x 0 x 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 x 0 x 0 x 0 x 0 x 0 x 0 x 0 x 0 x 0 x 0 x 0 x 0 x 0 0 @ 0 @ 0 0 8G @ 0 @ 0 @ 0 @ 0 @ 0 0 (H f ! u( !? 9? # + 5 A H o ! ! ! K! M! T! \! ! ^" )$ >$ $ $ $ % % % & 3 9 9 9 : ; &? 9? ! $ % & ' ( ) * , - . / 0 1 2 3 4 6 7 8 9 : ; < = > ? @ B C D E F G I 8? " 8
= ? Y l p _ s u " $ $ + / / 3 3 m6 97 QQQQ::QQQ ! 8 @ 0 (
B
S ? ( ;" :7 :7 V *urn:schemas-microsoft-com:office:smarttagsplace http://www.5iantlavalamp.com/ p q 1 2 > A B E M P Q T U \ q! r! j" r" 4 !4 f4 n4 4 4 4 4 M5 U5 q6 7 7 7 7 7 7 7 7 "7 #7 %7 &7 77 :7 + 0 | &4 G4 R4 U4 4 4 :5 B5 H5 g5 h5 5 ^6 g6 q6 6 6 7 7 7 7 7 7 7 7 "7 #7 %7 &7 77 :7 3 3 3 3 3 3 3 3 3 3 3 3 ] i o t K U ; < ` - . 1 2 3 4 5 6 7 8 : ; @ A C D F G L M O P R S U V X Y [ \ ] ^ _ ` c d g h k l m n o p
+ , 5 6 ? @ I J S T ` % % + 5+ 1 1 1 1 1 1 1 1 1 1 1 1 1
2 2 2 H4 a4 B6 p6 q6 7 7 7 7 7 7 7 7 "7 #7 %7 &7 +7 77 :7 q6 7 7 7 7 7 7 7 7 "7 #7 %7 &7 77 :7 A u t h o r i s e d U s e r k"- ~ BA: ?9mEdh h
^`o( h H . h ^`OJ QJ ^J o( h H o h p p^p`OJ QJ o( h H h @ @ ^@ `OJ QJ o( h H h ^`OJ QJ ^J o( h H o h ^`OJ QJ o( h H h ^`OJ QJ o( h H h ^`OJ QJ ^J o( h H o h P P^P`OJ QJ o( h H h ^`OJ QJ o( h H h ^`OJ QJ ^J o( h H o h p p^p`OJ QJ o( h H h @ @ ^@ `OJ QJ o( h H h ^`OJ QJ ^J o( h H o h ^`OJ QJ o( h H h ^`OJ QJ o( h H h ^`OJ QJ ^J o( h H o h P P^P`OJ QJ o( h H h ^`OJ QJ o( h H h ^`OJ QJ ^J o( h H o h p p^p`OJ QJ o( h H h @ @ ^@ `OJ QJ o( h H h ^`OJ QJ ^J o( h H o h ^`OJ QJ o( h H h ^`OJ QJ o( h H h ^`OJ QJ ^J o( h H o h P P^P`OJ QJ o( h H BA k"- 0?9mE > M d" e--
1 $= Z [ Mj S - q *p ; g y a O 4 o s ( - 9 ? K L M N O P Q R S T U \ h n z # ) * 6 : > F G M Q U ] ^ $ J p ^ _ 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 :7 R E F M G R . I n s t a n t F o r m a t dߴ
R E F M G R . L a y o u t R E F M G R . L i b r a r i e s ~s < I n s t a n t F o r m a t > < E n a b l e d > 1 < / E n a b l e d > < S c a n U n f o r m a t t e d > 1 < / S c a n U n f o r m a t t e d > < S c a n C h a n g e s > 1 < / S c a n C h a n g e s > < / I n s t a n t F o r m a t > 1< L a y o u t > < S t a r t i n g R e f n u m > V a n c o u v e r < / S t a r t i n g R e f n u m > < F o n t N a m e > T i m e s N e w R o m a n < / F o n t N a m e > < F o n t S i z e > 1 2 < / F o n t S i z e > < R e f l i s t T i t l e > R e f e r e n c e L i s t < / R e f l i s t T i t l e > < S p a c e A f t e r > 1 < / S p a c e A f t e r > < R e f l i s t O r d e r > 0 < / R e f l i s t O r d e r > < C i t a t i o n O r d e r > 0 < / C i t a t i o n O r d e r > < N u m b e r R e f e r e n c e s > 1 < / N u m b e r R e f e r e n c e s > < F i r s t L i n e I n d e n t > 0 < / F i r s t L i n e I n d e n t > < H a n g i n g I n d e n t > 0 < / H a n g i n g I n d e n t > < L i n e S p a c i n g > 0 < / L i n e S p a c i n g > < S h o w R e p r i n t > 1 < / S h o w R e p r i n t > < S h o w N o t e s > 0 < / S h o w N o t e s > < S h o w K e y w o r d s > 0 < / S h o w K e y w o r d s > < S h o r t F o r m F i e l d s > 0 < / S h o r t F o r m F i e l d s > < S h o w R e c o r d I D > 0 < / S h o w R e c o r d I D > < S h o w A b s t r a c t > 0 < / S h o w A b s t r a c t > < / L a y o u t > I < D a t a b a s e s > < L i b r a r i e s > < i t e m > a l l _ r e f e r e n c e s < / i t e m > < / L i b r a r i e s > < / D a t a b a s e s > @ 3 3 $ 3 3 97 @ U n k n o w n G z T i m e s N e w R o m a n 5 S y m b o l 3& z A r i a l ?5 z C o u r i e r N e w ; W i n g d i n g s " 1 h x ƅKL|&
Q. b Q. b ! 4 d V6 V6 ; 2q HX ? $= T h e a n a l y s i s o f c a t e g o r i c a l d a t a A u t h o r i s e d U s e r A u t h o r i s e d U s e r