WHO/V&B/01.26 ENGLISH ENGLISH ONLY DISTR.: DI STR.: GENERAL
D escr escrip iptt ion an d comp com paris ariso on o f t he me m et h o ds of cl clu st er sam sam pling pling an d lo lo t qu alit lit y assurance sampling to assess immun mm uniiz at ion cove ov erag e
Written by Stacy Hoshaw-Woodard, Ph.D Center for Biostatistics, The Ohio State University
DEPAR DEPARTM TM ENT ENT OF OF VACCINE VACCI NES S AND BIOLOGICALS World Worl d Heal Heal th Organizatio Organization n Geneva 2001
The D epartment epartment of Vaccines Vaccines and Biologicals thanks th anks th e donors w hose u nspecifie nspecified d financial financial support has made the production of t his document possible. possible.
This document was prod uced by the Vaccine accine Assessment and Mon itor ing Team Team of th e Depar tm ent of Vaccines Vaccines and Biologicals O rdering rdering code: WH O /V &B/ 01.2 01.26 6 Printed: August 2001
This document is available on t he Internet at: www.who.int/vaccines-documents/ Co pies may be requested requested from: World H ealth ealth O rganization rganization D epart ment of Vaccines Vaccines and Biologicals C H -1211 Geneva 27, 27, Switzerland Switzerland Fax: + 41 22 791 4227 • Email: Email:
[email protected] •
© World H ealth ealth O rganization rganization 2001 2001
This document is not a formal publication publication of the World H ealth ealth O rganization (WH O ), and all rights are reserved reserved by the O rganization. The do cument m ay, ay, however, be freely reviewed, reviewed, abstracted, reprodu ced and translated, in part or in wh ole, but not for sale sale nor for use in conjunction w ith commercial purpo ses. ses. The views expressed in documents by named authors are solely the responsibility of those authors.
ii
Contents
A bb rev iat ions...................................................................... ion s............................................................................................................ .............................................. ........ v 1.
Int roduction rodu ction ........... ................. ........... ........... ............ ............ ............ ........... ........... ........... ........... ........... ........... ........... ........... ............ ............ ............ ........ 1
2.
Brie Brieff des descr cripti iption on of the 30 by 7 cluster cluster sample sample used to t o assess imm un izatio n co verage ........... ................ .......... ........... ........... .......... ........... ........... ........... ............ .......... .... 2
3.
Brief Brief descr description iption o f the g eneric cluster cluster sample ...... ......... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ..... 3
4.
Brief Brief description description of lot lot quality assurance assurance sampling sampling ...... ......... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ..... .. 5
5.
Brief Brief description description of the stratified stratified sample ...... ......... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ... 7
6.
30 by 7 cluster sample versus the LQA S ...... ......... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ... 9
Bibliog raphy ...................................... ................................................................................. ............................................................................. .................................. 1 4
ii i
iv
Abbreviations
EPI
E xp an d ed P r o gr am m e o n I m m u n iz at io n
LQAS
lo t q u alit y assessm en t sam p lin g
PPS
p r o b ab ilit y p r o p o r t io n at e t o t h e siz e
SR S
st r at ified r an d o m sam p le
WH O
Wo r ld H ea ealt h O rg rgan iz at io n
v
vi
1. Intr In trod oducti uction on
The cluster sur vey used to assess assess coverage coverage of immun ization system s (30 (30 by 7 cluster) and Lo t Q uality Assessment Assessment Sampling Sampling (LQ AS) are two of th e more highly used sampling methods to assess immunization coverage. The 30 by 7 cluster survey is a mod ified ified two -stage cluster cluster sample and and t he LQ AS meth od is a typ e of strat strat ified ified sample. sample. Both o f these sampling sampling techniques can be used to ob tain overall population estimates of immunization coverage. coverage. This report pr ovides a brief description description of these two sampling methods, describes the general aspects of cluster and stratified sampling designs, compares and contrasts the two methods, and provides guidelines for immunization system staff on the setting in which each is appropriate.
WHO/V&B/01.26
1
2. Br Br ief desc descr ip t io n o f the th e 30 by 7 cluster cluster sample amp le u sed to to asse assess ss immu imm u n ization izatio n cover cover age age The 30 by 7 cluster cluster sample sample was developed developed by WH O in 1978 1978.. The goal of this sampling design was to estimate immunization coverage to within ±10 percentage points of the tru e propor tion, with 95% confidence confidence.. The 30 by 7 cluster cluster survey survey is a two -stage cluster sample. sample. Before Before the sampling begins begins,, the popu lation needs to be divided into a complete set of non-overlapping subpopulations, usually defined by geographic or political bou ndar ies. ies. These subpo pulation s are called called clusters. In the first stage, 30 of these clusters are sampled with probability proportionate to the size (PPS) (PPS) of the pop ulation in the cluster. cluster. Sampling with pro bability prop ort ionate to size allows the larger clusters to have a greater chance of being selected. The clusters are sampled with replacement, such that each cluster can be included in the sample more than on ce. In t he second second stage of sampling, seven seven subjects subjects are selected selected within each cluster. cluster. Alth ough th e sampling sampling unit is the individual subject, subject, the sampling is condu cted on t he hou sehold level. level. The subjects are chosen chosen by selecting selecting a household and every eligible subject in the household is included in the sample. With tr aditional PPS cluster cluster sampling, each each of the seven seven subjects wou ld be rand omly selected. selected. With the 30 by 7 method , however, however, only the first household is rando mly selected selected (by a variety o f different different meth ods), and all eligible eligible subjects subjects in th at ho usehold are sampled. sampled. After th e first first household is visi visited, ted, the surveyor moves to the “next” household, which is defined as the one whose front door is closest to the one just visited. visited. This process contin ues unt il all seven seven elig eligible ible subjects are found . N ot all of the first seven households visited will necessarily have an eligible subject, therefore mor e than seven seven households may have have to be visited. visited. Also less less than seve seven n hou seholds may need to be selected if there is more than one eligible subject per household. The information from each cluster is then combined to obtain an overall estimate of immunization coverage. coverage. A step-by-step guide for condu condu cting an immu immu nization coverage coverage survey survey is provided provided in th e WH O document Training for mid-level m anage anagers rs:: the EPI coverage overage survey survey W H O / EPI/ ML M / 91.1 91.10 0.
2
Description and comparison of the methods of cluster sampling and lot quality assurance sampling to assess immunization coverage
3. Brief Brief d escript escription ion o f t h e generic cluster sample
In a cluster sample, the population is divided into non-overlapping subpopulations usually based based on geographic or political bo und aries. For a simple cluster sample, sample, a random sample of subpopulations (clusters) is obtained and, within each selected cluster, each sub sub ject is is sampled. sampled. Mor e often, a tw o-st age cluster sample design design is used wh ere a random sample of clusters clusters is selec selected ted and , within each each cluster, a rando m sample of subjects. The tw o-stage design design can can be expanded expanded into a mult i-stage one, in which samples of clusters are selected within previously selected clusters. A benefit of this type of cluster sample is that a list of the units in the population is only needed for those clusters that are selected. A common modification to the cluster sample design is to select the clusters with probability prop ortionate to the size of some variabl variablee in the p opulation, such such as the population size (as in the 30 by 7 cluster sample), the number of health facilities in the region, or the numb er of immun izations given given in a week. week. This type of cluster cluster sample is said said to be self-weighting because every unit in the population has the same chance of being selected. In theory, clusters are chosen to be as heterogeneous as possible, that is, the subjects within each cluster are diverse and each cluster is somewhat representative of the pop ulation as a whole. Thu s, only a sample sample of the clusters clusters needs needs to be taken to captur e all all the variabili variability ty in th e popu lation. In p ractice, how ever, ever, clusters are are often defined based on geographic regions or political boundaries, so that conducting a cluster sample sample reduces the time and cost associated associated with th e survey. survey. In t his situation situation , the elements within the clusters may be rather homogeneous and, on average, th e clusters clusters may be very different different fro m on e ano ano th er. Because Because of th is, for a fixed fixed sample size, the variance from a cluster sample is usually larger than that from a simple random sample, and therefore the estimates are less precise. Some main advantages and disadvantages of a general cluster sample are as follows. Advantages: ·
O nly need to o btain list of units in the selec selected ted clusters.
·
Cost-effective.
Disadvantages: ·
N ot int ended for calculation calculation o f estimates estimates from from individual clusters. clusters.
·
Less precise than simple random sample.
WHO/V&B/01.26
3
D ifferences ifferences between t he 30 by by 7 cluster sample and the gen eric cluster sample The main difference between the 30 by 7 cluster sample and the generic two-stage cluster sample is that in the 30 by 7 cluster sample only the first household in each each cluster is rando mly selected. selected. An oth er issue issue with t he 30 by 7 cluster sample sample is that every eligible individual in the household is included in the sample. These two modifications can lead to some biases. If the immunization coverage is concentrated in specific areas of the cluster, this “pocketing” may lead to a biased estimate. The level of immunization coverage could be overestimated if the first household selected is located in a “pocket” of immunization, or it could be underestimated if the first household is located outside of the “po cket”. Simulation stud ies have been conduct ed to assess assess the bias that m ay be introdu ced ced into the estimate estimate by the fact fact t hat only the first first ho usehold is randomly randomly selected. selected. It w as concluded concluded t hat in mo st cases cases the bias is small small because because the num ber of clusters is is relatively relatively large. large. Results from from these stud stud ies also also reinforce the not ion th at estimates should no t be obtained for individual clusters. clusters. The pot ential for for bias could could be eliminated eliminated by random ly selecting selecting all all of the households. To do this, however, however, a list list of all hou seholds in the selected selected clusters clusters wo uld need to be obt ained, which may add a considerable amount of time and effort to the implementation of this sample. There are additional modifications that could also be introduced to decrease the chance of bias. Som e examp examples les includ e, selecting selecting every every k th household, or stratifying each of the selected clusters into regions and choosing part of the sample from each of the regions. All of these methods may also increase considerably the time needed to perform t he sampling. sampling. In the 30 by 7 cluster sample, once a household is selected, every eligible individual in th e household is selected. selected. Because Because of this, not all of the 210 subjects subjects in the sample are independent. This may introduce bias into the estimate because subjects in the same same household tend t o be h omogeneous with respect respect t o immun ization. For example, example, if women with many children are less likely to get their children immunized the immunization coverage may be underestimated. This bias could be alleviated by randomly selecting one child per household. Although it would be easy to select randomly one child per household, it would possibly increase the number of households that need to be visited to obtain the seven subjects per cluster. The sample size for the 30 by 7 cluster sample is set at 210, which provides an estimate that shou ld fall fall within 10 percentage poin ts of the tru e popu lation percentage. In most situations this is adequate, however, when immunization coverage is extremely high (e.g. only 1 person in 1000 is not immunized), estimating this pro por tion t o within 10 percentage percentage points is not very inform ative. Because Because of this, it may be necessary necessary to increase the sample size. size. This could be done by increasing both the number of clusters and the sample size per cluster, or by just increasing the sample size size in each of t he 30 clusters. Mor e extensive extensive sample sample size calculations calculations wo uld be needed to obtain the optimal combination of clusters and samples per cluster for each situation.
4
Description and comparison of the methods of cluster sampling and lot quality assurance sampling to assess immunization coverage
4. Br ief ief descript description ion o f lo lo t quality assurance sampling
Lot quality assurance assurance sampling sampling (LQ AS) AS) originated originated in t he manufacturing industry for quality control p urpo ses. ses. Manufacturers Manufacturers were interested interested in determining whether a batch, or lot, of goods met the desired desired specific specifications. ations. Rather t han checking each item in the lot to determine the number of “defects”, they decided to take a sample of the items and define the level of risks they were willing to take for not inspecting each each and every every item. Based Based on th ese risks, they w ould d ecide ecide to accept, or reject, the entir entir e lot. lot. The only o utcom e in in this typ e of sampling is is “acceptable” “acceptable” or “n ot acceptable”. Ther e is is no measure measure as to d ifferent ifferent levels levels of unacceptability. unacceptability. For example, one lot may be deemed unacceptable because of two defective items out of 10 sampled, and another lot with all 10 defective items will be assigned the same classification of unacceptable. The sample size and decision values for lot quality assurance sampling are based on the risks that th e investig investigators ators are willing willing to take. The sample size size is the nu mber of unit s that are selec selected ted from each lot. The decision decision value is is the numb er of “defective” “defective” items that need need to found before the lot is deemed deemed unacceptable. unacceptable. There are two ty pes of risk that need to be considered: (i) the risk of accepting a “bad” lot, referred to as Type I Error, and (ii) the risk of not accepting a “good” lot, referred to as Type II Err or. Based Based on the risks risks and and the hypo thesized propo rtion of defects defects in the lot, lot, the sample size and decision value can be obtained. Although it is not th e original original intention of the LQ AS method, information from lots can be combin ed to obtain the overall pro por tion of defects. defects. To do th is, the pop ulation is first divided into a complete set of non-overlapping lots. Samples are then taken from every lot, and the proportion of defective items in each lot is calculated. The overall proportion of defects in the population is estimated by taking a weighted average average of the estimated estimated pr opo rtio n of defects defects from from each of the lots. A corr esponding confidence interval can can be calculated calculated as well. well. Since the overall pro por tion of defects defects is determined determined by combining the information from each each of t he lots, the LQ AS method is an example of stratified sampling, where the lots play the role of the strata. The advantage of the LQ AS method o ver a traditional stratified sampling sampling design design is that the response for each lot is binary (acceptable or not), and therefore smaller sample sizes sizes can can be used. For a tr aditional stratified sample, sample, a confidence interval with a certain precision is usually desired for each stratum (or lot), which requires a larger sample. A m odifica odification tion to the pr eviousl eviously y described described LQ AS method method is the implementation implementation of “do uble sampling”. sampling”. The meth od is useful useful in in some situatio situatio ns to decrease decrease the sample sample size. With th e double sampling sampling metho d, a first sample sample of size n 1 is selected. selected. If more than d 1 items are found to be defective, the lot is deemed to be “unacceptable”
WHO/V&B/01.26
5
and th e sampling sampling in the lot is over. over. If the nu mber o f defects defects is less less than or equ al to d 1, another n 2 items are selected, and if more than d 2 items in the sample of n 1+ n 2 items are foun foun d to be defective, defective, the lot is deemed to be “unacceptable”. O therw ise the lot is classified as “acceptable”. In terms of immunization assessment, “acceptability” is usually determined by whether th e lot lot m eets eets some desi desired red prop ortion o f immunization coverag coverage. e. To use th e LQ AS meth od for assessing assessing immu nization coverage, coverage, some some paramet ers need to be set before the sampling sampling can begin. begin. First, the level level of immu immu nization coverage that wou ld be deemed deemed “un acceptable” acceptable” needs needs to be determined. Second, the level level of immu nization coverage that is desired desired needs to be defined. Third , the amoun t of risk risk you are willing willing to take for incor rectly judging an un acceptable acceptable lot as being acceptable. acceptable. Finally, the amount of risk you are willing to accept for deeming an acceptable lot un acceptable. acceptab le. U sually, th e risk of classifying classifying an unaccept able lot as acceptable acceptab le is set lower than the risk of classifying an acceptable lot as unacceptable because it is usually a more serious error to judge an unacceptable lot as being acceptable when it is not. The sample size size and th e decision decision value value are selec selected ted so th at the lots wit h high immunization coverage have a good chance of being classified as acceptable and tho se with po or immu nization coverage as unacceptable. The larger larger the difference difference between the level of immunization coverage defined to be unacceptable and the desired level of immunization coverage, the smaller the sample size that is needed, and the less precise your results will be. The LQ AS method method can can be used by immun ization ization system staff staff to determine whether their ind ividual ividual “lot ” is acceptable acceptable or no t using a relatively relatively small small sample. sample. Because Because of the small sample needed for assessment of an individual lot, the assessment can be made more frequently. Alth ough the LQ AS metho d is not generally associated associated with overall population estimates, the com bined samples from from all of the lot s can can be tr eated as a stratified sample and the overall population proportion of immunization can then be estimated. estimated. N ote that when it is of interest interest to o btain an overall overall population estimate, the entire sample sample needs to be selected selected in each each lot. The sampling canno canno t be stopped after the decision number of non-immunized persons is reached, because the interest is in the proportion of subjects immunized, not the proportion of acceptable acceptable lots. A guide for condu cting an LQ AS coverage coverage survey is presented in services es using using th e lot quality t echniqu echniqu e the WHO docume document nt Monitoring imm uniz ation servic (WHO/VRD/TRAM/96.01).
6
Description and comparison of the methods of cluster sampling and lot quality assurance sampling to assess immunization coverage
5. Brief Brief d escript escription ion o f t h e stra str at ifie ified d sample amp le
For a stratified stratified sample, sample, the population is divided divided into non -overlapping subpop ulations defined on th e basis basis of some some kn own characteristic that is believ believed ed to b e related related to the variable of interest . For example, the population may be stratified with respect to sex, race, geograph geograph ic region region , etc. etc. Since a rand om samp le is is taken from each and every every stratu m, a list list of every every unit in th e population is required. An overall population estimate can be obtained by a weighted average of the estimates from each of the strata. If the overall sample size is divided amon amon g the strat strat a using using prop ort ional allocation, allocation, the estimates from th e strat strat a are are self-weighting. self-weighting. O ne of the objectives objectives of this type of sampling design is to obtain estimates for each of the subpopulations, or str ata. Ther efore, the sample sample size size is usually chosen to be large enough such th at reasonably precise estimates can be obtained for each stratum. The strata are chosen to be homogenous, such that the elements within each strata are similar similar.. Therefor e, it is necessary necessary to sample subjects from each stratum to o bt ain an overall pop ulation estimate. Since th e un its within each each stratu m are similar, similar, the variability within each stratum is smaller, yielding more precise estimates. Some advantages and disadvantages of a traditional stratified random sample are as follows. Advantages: ·
·
Production of estimates and corresponding confidence intervals for each stratum. Increased precision over a SRS.
Disadvantages: ·
A list of all the units with in each stratum required.
D ifference ifference betw betw een LQA S and and tradition al strat stratified ified random sample. The standard application application of the LQ AS method method is to determine whether individual individual lots are acceptable acceptable or not . The meth od can be extended, extended, how ever, ever, to ob tain overall pop ulation estimates. This is don e by treating the lots lots as strata, and and comb ining the samples from all of the lots to create a stratified sample. sample. The main difference between the LQ AS metho d, as used to ob tain pop ulation estimates, and the tr aditional strat strat ified ified sampling design design is in in th e sample sample size. size. The sample size size using the LQ AS metho d is typically smaller than that of a traditional stratified sample because, for each lot or stratum, only a binary decision is made (“acceptable” or“not unacceptable”).
WHO/V&B/01.26
7
In the traditional stratified sample, the sample size in each stratum, or lot, is large enough to estimate the sample proportion to within some desired level of precision. Alth ough t he cost of an L Q AS sample sample is lower lower because of the d ecreased ecreased sample sample size, the inform ation gained gained from such a sample sample is is limited. limited. O ne of the benefits benefits tout ed with the LQ AS method is that the sampling sampling can can be done more frequently frequently within each lot to assess change because of the smaller sample sizes required. The downside to this, however, is that for each lot only an acceptable/not acceptable verdict is possible. Even tho ugh a lot may have und ergone significa significant nt changes, it may still be deemed unacceptable, and there is no measure of the degree of “unacceptability”. N ote that for approximately the same same amou amou nt of “precision”, “precision”, the LQ AS method actually requires a sample size that is at least as large, if not larger, than that of a stratified random sample. This is illustrat illustrat ed in the follow follow ing example. example. Example: Example: Assume Assume that the hypothesized (or desired) desired) level of immuniz ation cov cov erage erage is 80% . U sing a stratifi stratifi ed random sample, sample, to es esti mate the immunization coverage in each stratum to within 10 percentage points of the true v alue, alue, w ith 95% confidence confidence (using (using a one-sided one-sided confidence confidence interval, because because the LQ L Q AS method uses uses a one-sid one-sid ed hypothesis hy pothesis test to calculate calculate sample siz siz e), a sample sample size of 44 44 is required required for each each stratum. stratum. Alternativ ely, ass assuming the desired level of imm unizati on to be 80% and an acceptable acceptable level of immuniz ation coverage to be 70% (i.e. 10 percentage percentage point aw ay from the hypothesized value), with a probability of Type I Error set to be 5% 5% (i .e. a=0.0 a=0.05, 5, where a=1-t a=1-t he confidence confidence lev lev el) and a probabili ty of Type II Error set to be 0.10, a sample size of 156 subjects per lot is required. As the probability of Type II Error is increased (i.e. we allow a greater chance that an “acceptable” lot will be deemed “unacceptable”), the sample siz siz e decreases decreases.. For examp examp le, if the prob abilit abi lityy of Type II Error is increase increased d to 20% , a sample sample size of 109 109 per lot is required, required, and if t he probabilit y of Type II Error is increased increased t o 50% , a samp sample le siz siz e of 44 per lot i s requi requi red (which is the same sample size for the stratified sample). This example highlights the inherent differences between a stratified random sample and the LQ AS metho d. The objective of a stratified rando m sample sample is to estimate the immunization coverage with a certain precision, and the intended use of the LQ AS method is to uncover uncover t he “lots” w ith inadequate immunization immunization coverage coverage.. We may be less strict on what is deemed “acceptable” immunization coverage (i.e. (i.e. less less precise), precise), which wou ld lead to a smaller sample size size using th e LQ AS method .
8
Description and comparison of the methods of cluster sampling and lot quality assurance sampling to assess immunization coverage
6. 30 b y 7 clu clu ster sam sam p le versus rsus the th e LQ AS
Table Table 1 provides a comparison o f the 30 by 7 cluster sample and and t he LQ AS method (as used to o btain an overall popu lation estimate) with respect to k ey sampling issues issues.. The main difference between the two sampling techniques is that one is a two-stage cluster sample sample and and th e oth er a stratified random sample. The main implication implication of this is that estimates for for in dividual clusters clusters cannot be calculated calculated in t he 30 by 7 cluster sample. The only appr opr iate estimate is for the pop ulation as a whole. Altern atively, atively, wh ile decisions decisions can be made for each lot in an LQ AS sample, sample, these decisions decisions are limited limited to “acceptable” or “n ot acceptable”. In general, th e goal of the 30 by 7 cluster sample is to estimate the population proportion to within a certain level of precision, whereas in the basic LQ AS metho d the ob jective jective is to t est the hypo thesis that t he lot is unacceptable. H owever, bot h metho ds can can be used to ob tain an overall overall pop ulation estimate. The 30 by 7 cluster cluster sample sample does not r equire a rando m sample sample from all of the clusters, which makes the method easier to use and more economical, but since only a single single rando m ho usehold is selected selected in each cluster, cluster, this metho d also has a greater chance for bias (although computer simulations have shown that the bias is small). small). If the LQ AS meth od is used used to ob tain an overall overall population estimate, it requ ires a smaller smaller sample sample size than a t radition al stratified stratified sample, but it may req uire a much larger larger sample sample size size than the 30 by 7 cluster cluster sample. sample. N ote that th e LQ AS method was not originally conceived to obtain an overall population estimate, but to determine whether individual lots are acceptable or not. The 30 by 7 cluster cluster sample and and L Q AS sampling methods will be furt furt her comp ared using the following example. Example: Example: A country country is div ided i nto 200 200 non-ov non-ov erlapping erlapping districts. The main objective of the survey is to obtain the proportion of immunized children between the ages of 1–12. To use the 30 by 7 cluster sample, the population size in each of the 200 districts is obt ained. Fro m this information , 30 districts are selected selected with pro bability pro por tion ate to size. size. In each of the 30 30 clusters selected, selected, seven seven hou seholds are are selected selected yielding yielding a sample sample size of 210 210 children. children. Fro m th e sample sample information , the overall proportion of immunized children can be calculated. To use the LQ AS metho d, the unacceptable and desirable pro por tions of immu nizatio n coverage, coverage, as as well as the risks, need need to b e defined. defined. Assume that t he level of unacceptable immunization is 0.30, and the level of desired immunization coverage coverage is 0.80 0.80.. Set th e risk of accepting accepting a lot w ith imm unizat ion coverage of less less than 30% to b e 0.05 0.05,, and and t he risk of not acceptin acceptin g a lot wit h immu nization coverage greater greater th an 80% 80% at 0.10 0.10.. U sing these value values, s, the LQ AS method method indicates indicates that
WHO/V&B/01.26
9
seven children in each district be randomly selected and, if more than two children in each district are not immunized, the district is deemed to have unacceptable immun ization coverage. coverage. Each district can can be categorized categorized as “acceptable “acceptable”” or “not acceptable”. acceptable”. Althou gh the primary objective objective of an an LQ AS sample is to determine whether each lot is acceptable or not, the proportion of immun ized children in each each district can be combined to calculate calculate the overall propor tion of immunized children in the population. With this sampling sampling method, t o d etermine whether a district has acceptable immunization coverage or not, only seven children need to be sampled, however, to calculate the overall proportion of immunized children, a total of 1400 children must be sampled. This example begs the question of which is the better sampling method for this scenario. scenario. The following questions help to pro vide guidelines on wh en to use each each sampling method.
Questions
1)
Is it of inter interes estt to make infer inferen ence ce about about eac each h indiv indiv idual subpopula ubpopulation? tion?
If the answer is yes, then the LQ AS meth od w ill need to be used because making inference abou abou t th e individual individual clusters in the 30 by 7 cluster cluster sample is not appro priate and can can be very misle misleading ading.. N ote that from t he LQ AS method o nly an “acceptable “acceptable”” or “not acceptable” conclusion can be made about the individual lots, because the sample size in each lot is too small to provide accurate estimates. If the answer is no, the 30 by 7 cluster sample will be much more cost-effective than the LQ AS method.
2) 2)
Are the subpopu subpopula lations tions heter heterog ogen eneo eous us or homo homoge gene neou ouss?
If the subpopulations are heterogeneous (i.e. the subjects within the subpopulations are quite variable but there is little difference, on average, from subpopulation to subpopulation), then no information will be lost by not sampling all of the subpopulations, as with the 30 by 7 cluster sample. If the subpopulations are homogeneous (i.e. the subjects within the subpopulations are similar but there are big differences, on average, between the subpopulations), then th e LQ AS method method would b e more appropr iate beca because use ele elements ments from each each of the subp opu lations will be sampled. If the 30 by 7 cluster cluster survey is used, the estimate estimate may be biased because subpopulations with very high or very low immunization may no t be included in the sample. sample. U sing one overall estimate to describe describe the wh ole population may be misleading, and it may be more meaningful to judge each subpopulation separately.
3) 3)
How difficult difficult is it to obtain obtain a list list of all all the the units units in the the popul population ation? ?
If it is very difficult to obtain a list of population units, it may be easier to use the 30 by 7 cluster sample, because: (i) a list of units is only needed for those selected clusters, and (ii) according to the 30 by 7 cluster method, only the first household is random ly sele selected. cted.
10
Description and comparison of the methods of cluster sampling and lot quality assurance sampling to assess immunization coverage
If it is easy easy to o btain a list of the po pulation units, taking a rando m sample is relativel relatively y simple simple and and th e most unbiased unbiased method. H aving aving a list list of the population units makes the LQ AS method mu ch easi easier er to implement. implement. N ote that if it it were possible possible to random ly select select all seven seven hou seholds for the 30 by 7 cluster sample, the p ossible biases biases usually usually associated associated with this metho d would be eliminated. eliminated. H owever, the design design would then be that of a traditional cluster sample rather than a 30 by 7 sample.
4) 4)
What is the desire desired d precis precision ion of the estimate estimate? ?
The 30 by 7 cluster sample sample is set set up t o ob tain a 95% 95% confidence interval to w ithin ±10 percentage points of the popu lation parameter. The sample size calculation calculation is based on an assumed assumed prop ort ion of immun ization coverage of 0.50. 0.50. If the coverage coverage is greater greater or less less than 50% , the confidence interval is is likely likely to be somewh at narr ower than ± 10 percentage percentage point s. N everth everth eless, eless, if a greater level level of precision precision is desired, desired, the sample size size can can be increased. increased. H ow ever, ever, for a two -stage cluster cluster sample, th ere is no simple equatio equatio n to calculate calculate sample sample size. size. D ifferent ifferent comb inations of the num ber of clusters and number of units selected per cluster will yield different levels of precisi precision. on. The optimal combination combination depends upon nu merous assumptions assumptions about the population with respect to the costs of sampling additional clusters and sampling additional units per cluster, as well as the within and between cluster variability. For t he LQ AS metho d, the sample sample size size is based based on a hy pot hesis hesis test, test, not o n th e estimation of a confidence inter inter val. val. H owever, the sample size needed to obt ain an an overall estimate with a certain precision from a stratified random sample can be easily obtained using a simple equation.
5) 5)
Is the event event of intere interesst v ery ery rare rare or v ery ery common common? ?
N ote th at the sample size size for th e 30 by 7 cluster sample is is based based on a po pulation proportion of 0.50, which yields the greatest estimate of variability and therefore the mo st conservative conservative sample size. H owever, if the event of an an immun ized, or not immunized subject is very rare, a precision of ±10 percentage points may not be satisfac satisfactor tor y. In n eonatal tetanus, for example, example, the incidence is only app roximately 1 in 1000, and to obtain an estimate that would be somewhere between 0 and 10% wou ld not provide useful useful informat ion. Therefor e, for events events that are very rare or very common, a larger sample size will be needed from a cluster sample to obtain a reasonab le level level of precision precision . For example, to estimate the prevalence of neon atal tetanus to with in 50% of th e tru e value value (i.e. (i.e. between 0.0005 0.0005 and and 0.0015) 0.0015) with 95% confidence, a total sample size size of 15 352 wou ld be r equired for a simple simple rando m sample. G iven iven th at the variability variability is larger for for a cluster cluster sample than for a simple random sample, an an even larger larger sample sample wou ld be needed for the former. If we assume assume th at the variability variability from the cluster sample to be twice that of a simple simple rando m sample, the overall sample size from the simple random sample would need to be dou bled to ob tain the same level level of precision precision as from from a ccluster luster sample. sample. For t he neonatal tetanus example, an overall sample of 30 704 subjects would be needed, indicating that 1024 subjects from each of the 30 clusters would have to be sampled. The sample size size for LQ AS is based on the hyp oth esized esized (or desired) immu immu nization coverage. coverage. Therefor e the rarity of an an event event is already already taken into consideration w hen the sample size is calculated.
WHO/V&B/01.26
11
6)
What knowle knowledge dge is there there about about the the actu actual al level level of of immuniza immunization tion coverage in the populati on?
When using the LQ AS method, method, n umerous parameters parameters need need to be set set t o ob tain th e sample sample size size and decision decision value for th e lots. lots. To obt ain these values, values, assumptions about the population need to be made about the actual level of coverage that can be realistically obtained, and about the current level of coverage. The 30 by 7 cluster sample does not make any assumptions about the population, which makes it easy easy to apply to many situat situat ions. N ote, however, however, that th e sample size of 210 may not be appropriate when the interest is in an event that is very rare or very commo n (see (see Q uestion uestion 5).
7)
What What is the the bu budge dget fo for th the survey?
This may be the most important q uestion. uestion. In a perfect perfect wor ld, the sampli sampling ng method would be dictated strictly by the population characteristics and the research question of interest. In r eality, eality, how ever, ever, the sampling design design is often dictated b y cost. Th e 30 by 7 cluster sur vey has a set sample size size of 210. 210. Th is sampling sampling design is economical because only 30 clusters need to be visited, which decreases the travel time. Fur ther mor e, a list list of the all the unit s in in the pop ulation is not r equired, which makes the sampling more feasible. The LQ AS method is tout ed as being being a method that requires very very small small sample sample sizes. In r eality, eality, however, th is is no t always tru e, especially especially if it is of interest t o obt ain an an overall pop ulation estimate. In mo st scenarios, scenarios, the sample sample size size per lot is at at least 7, therefore if the population is divided into more than 30 lots, the overall sample size size will be greater than 210. 210. If the subpo pulation s are are the same as as used used for the cluster sample, there will be many more clusters than 30, and the overall sample size will be much larger than 210. 210. In add ition to t he possibly large large sample size needed to estimate estimate an overall overall pop ulation prop ort ion, the LQ AS method can also also be more costly given that samples have to be taken from every subpopulation, which increases the travel time and cost, and because of the need for a list of all the unit s in the popu lation. D oub le sampling can be used to decrease decrease the sample sample size, size, but this modification is only useful when the interest is to classify individual lots as acceptable acceptable or no t acceptab acceptab le. D oub le sampling is is not beneficial beneficial when it is intend ed to combine the information from each each of the lot s to calculate calculate an overall overall proport ion.
8)
What is the exper experien ience ce lev lev el of the field field worker workerss?
In t erms of settin settin g up the survey, th e 30 30 by 7 cluster cluster sample is very simple. It is set that 30 clusters will be selected with probability proportionate to size and with in each cluster cluster seven seven subjects will be selec selected. ted. There m ay be some difficulty difficulty in selecting the first household, given that there are a variety of methods for doing so (e.g. (e.g. going going to t he centre of th e villag village e and spinning a bott le to get a random direction, enumerating the houses in that direction and randomly selecting one).
12
Description and comparison of the methods of cluster sampling and lot quality assurance sampling to assess immunization coverage
O n th e other hand, before the sampling sampling can can begin begin for a LQ AS sample sample,, a number of parameters need to be set from which the sample size and decision value will be calculated. calculated. An d a list list of all the sample elements elements in the lots needs needs to be obt ained. H owever, owever, once th e list list is ob tained, the sampling is straightforward straightforward because because the surveyor knows exactly which units to sample. Table 1: 30 by 7 cluster sample versus the LQAS method to obtain an overall population estimate Issue
30 by 7 cluster sample
L QA S (As used to obtain an overall population estimate)
Sampling design
Two-stage cluster sample
Stratified random sample
Subpopulations
Called clusters
Called lots
Usua Usuallllyy bas based ed on geog geogra raph phic ic or poli polititica call boundaries
Usua Usuallllyy bas based ed on geog geogra raph phic ic or poli polititica call boundaries
Supposed to be heterogeneous
Supposed to be homogeneous
Sample size
N=210 (30 cl cluster ters, 7 subjects pe per cl cluster)
Dependent on the desired proportion and level of ririsks, ma may be be mu much la larger th than 21 210
List of units
No ne need for list of units
Need for list of all units in population
Basis for inference
Confidence interval for estimate
Hypothesis test
Outcome
Overa Overallll estim estimate ate of of immun immuniz izati ation on cover coverag age, e, estimat estimates es from from indivi individua duall cluste clusters rs shou should ld not not be calculated
Overa Overallll esti estima mate te of immu immuni nizat zatio ionn cover coverage age,, indivi individual dual lots are judged judged to be be accep acceptabl tablee or not
Precision
Set Set to be be withi withinn + 10 10 perce percenta ntage ge poi points nts of the the true population value
Can be set set to to diffe differen rentt level levelss
Weighting of the sample
Self-weighting
Weights need to be calculated for each lot
Cost
Decr Decrea ease sedd tra trave vell tim timee and and prep prepar arat atio ionn
Need Need to sam sample ple eac eachh lot lot,, yie yield ldin ingg hig highe herr cos costt
Reasons for potential bias
Heter Heteroge ogene neou ouss clus cluster ters, s, the the house househol holds ds are are not randomly selected, all eligible subjects in the household are sampled
Small Small samp sample less in in each each lot lot
Ease of implementation
The The samp sample le siz sizee is set set,, no need need for for lis listt of all units in the population
Need Need to deci decide de on acce accept ptab able le prop propor ortition on and and risks, need for list of units in the population
If rare event
Need to increase sample size
Built into design of study
When to use
Inte Intere rest st in in an over overal alll popu popula latition on esti estima mate te obtained at a low cost
Inte Intere rest st in in info inform rmat atio ionn from from each each lot, o t, and and a traditional stratified sample not affordable
WHO/V&B/01.26
13
Bibliography
Bennett A et al. A computer simulation of household sampling schemes for Journal of Epidem iology iology,, 1994, health surveys in developing countries. In ternational Journal 23 : 1282-1291. H arris DR, Lemeshow S. Eval Evaluation uation of th e EPI survey survey metho dology for estimating estimating Statistics Q uarterly, 1991, 44: 107-113. relative risk. World H ealth Statistics Katz J et al. Sampling Designs for Xerophthalmia prevalence surveys. International Journal of Epidemiology, 1997, 26: 1041-1048. Lanata CF et al. An evaluation of lot quality assurance sampling to monitor and improve immunization coverage. International Journal of Epidemiology, 1990, 19 : 1086-1090. Lanata CF, Black RE. Lot quality assurance sampling techniques in health World H ealth ealth surveys in developing countries: advantages and current constraints. World Statistics Q uarterly, 1991, 44: 133-139. Lemeshow S, Robinson D. Surveys to measure programme coverage and impact: a review of the methodology used by the Expanded Programme on Immunization. World H ealth Statistics Statistics Q uarterly, 1985, 38: 65-75. Lemeshow S et al. A computer simulation of the EPI survey strategy. International Journal of Epidemiology, 1985, 14: 391-399. Lemeshow S, Stroh G. Sampling techniques for evaluating health parameters in developing countries, a working paper . Prepared for: Board on Science and Technology Technology for Intern ational Development, Development, N ational Researc Research h C ouncil. 1988: 1988: N ational Academy Press, Washington Washington , D.C . Lemeshow S, Stro Stro h G . Q uality assurance assurance sampling for for evaluating health health p arameters in developing countries. Survey Methodology, 1989, 15: 71-81. Lemeshow S, Taber S. Lot quality assurance sampling: single and double sampling Statistics Q uarterly, 1991, 44: 115-132. plans. World H ealth Statistics Reichler MR et al. Cluster survey evaluation of coverage and risk factors for failure to b e immunized during the 1995 1995 N ational ational Immunization D ays in in Egypt. International Journal of Epidemiology, 1998, 27: 1083-1089.
14
Description and comparison of the methods of cluster sampling and lot quality assurance sampling to assess immunization coverage
Robertson SE et al. The lot quality technique: a global review of applications in the assessment of health services and disease surveillance. World H ealth Statistics Statistics Q uarterly, uarterly, 1997, 50: 199-209. Singh J et al. Evaluation Evaluation o f immun ization coverage coverage by lot quality assuran assuran ce sampling sampling compared with 30-cluster sampling in a primary health center in India. Bulletin of the World World H ealth ealth O rgani rganizz ation, ation, 1996, 74: 269-274. Turner AG , Magnani Magnani RJ, Shuaib Shuaib M. A n ot q uite as quick but much cleaner alternative to the Expanded Programme on Immunization (EPI) cluster survey design. International Journal of Epidemiology, 1996, 25: 198-203. Monitoring immunization services using the lot quality technique. Geneva, World H ealth ealth O rganization, rganization, 1996 1996 (unpublished document WH O / VRD / 96.0 96.01; 1; avai availabl lablee from Vaccines accines and Biologicals Biologicals,, World H ealth ealth O rganization, 1211 Geneva 27, 27, Switzerland). Training for mid-level managers: the EPI coverage survey . G eneva: eneva: World World H ealth ealth O rganization, rganization, 1991 1991 (unpub (unpub lished lished do cument WH O / EPI/ 91.1 91.10; 0; ava availa ilable ble from Vaccines accines and Biologicals, Biologicals, World World H ealth ealth O rganization, 1211 Gen eva 27, 27, Switz Switz erland).
WHO/V&B/01.26
15
