Project Objective
The One Number Census (ONC) had the goal of providing a methodology and processes to identify and adjust for the number of people and households not counted in the 2001 Census. The extent of this underenumeration was identified using a large survey covering approximately 320,000 households, the Census Coverage Survey (CCS). Statistical sample techniques were then used to produce an adjusted database from which the final census results were produced, these results also formed the new 2001 base for the Office for National Statistics (ONS) population estimates.
This report evaluates the methodology, processes and operations. It also provides an overview of the wider strategic questions of whether the ONC strategy was the correct one to adopt and what were the most and least successful aspects of the strategy. The 2001 Census results were the focus of considerable debate. From their publication in September 2002 through to September 2004, ONS undertook further work on both the ONC results and the previous population estimates resulting in adjustments to both. Publication of this updated evaluation report follows the completion of that work.
The ONC project was designed and implemented across the whole of the UK, however this report evaluates the success of the process in England and Wales only.
Background
Every effort is made to ensure everyone is counted in a census. However, no census is perfect and some people are missed. This underenumeration does not usually occur uniformly across all geographical areas or across other sub-groups of the population such as age and sex groups. Therefore, it is accepted best practice to assess the extent of any underenumeration, usually by means of a post-enumeration survey. Non-response to censuses and surveys is increasing in much of the developed world so assessing the scale and type of underenumeration is becoming increasingly important.
In the 1991 Census, 3.8% of the population was missed overall. This comprised 2.2% that was completely missed (see next paragraph) and 1.6% that was missed but imputed. In 1991, for the first time, records were imputed for households that were known from the Census operation to exist but had not returned a form. Although the overall proportion missed compared favourably with other countries, what was notable was that underenumeration was greater in certain types of areas and age-sex groups. Of most significance, however, the 1991 Census post-enumeration survey, the Census Validation Survey (CVS), did not identify the full extent and distribution of the underenumeration, its main focus was the accuracy of responses to specific questions.
These difficulties took time to resolve and four different resident population counts were available following the 1991 Census. Initially 'raw' census counts were published. Then a set of provisional estimates, was published, uprating the census count in line with the CVS. Subsequently a decision was made, in England and Wales, to base the national population estimates on demographic estimates, rather than uprated census counts, and to produce consistent counts at Local Authority (LA) level using a mathematical model. This third set of counts were then subject to a final revision following further analysis.
At geographic levels below local authority level no official estimates of underenumeration were produced, although those produced by the Economic and Social Research Council funded project 'Estimating with Confidence' have been widely used (Office of Population Censuses and Surveys (OPCS) 1993, 1994; Simpson, Cossey and Diamond, 1997). The official census tabulations therefore remained inconsistent with the population estimates.
During the extensive consultation carried out prior to the 2001 Census, users of census data were clear that they wanted one definitive set of estimates to come out of the Census. The ONC project provided a way to address these issues. It led to the strategy of adjusting the Census responses for the estimated underenumeration so that all published census statistics would add to the national estimate of the population. It built on the advances made in previous censuses: the introduction of automatic editing and imputation of missing variables in 1981, the absent household imputation in the 1991 Census (imputing records for households we knew existed from the Census operation) and in 2001 to the addition of imputed households and persons estimated by the ONC. Furthermore the mid-year population estimates were to be based on these ONC population estimates and there needed to be a simple link between them.
The aim was also to have one final set of census estimates, rather than the four different estimates in 1991. That aim was not achieved. The reasons for this and the problems it caused are discussed later.
Methodology
The ONC methodology was devised by ONS in consultation with experts from Southampton University. The ONC Steering Committee, which included a number of external experts and representatives, oversaw and guided the development of the methodology and management of the project. The key stages or processes used to achieve the aims are summarised below. Full details of the methodology can be found here. All other committee papers can also be found on the National Statistics website, detailing all the methodological research undertaken throughout the project here. The day to day progress of the ONC project was overseen by the One Number Census Project Board.
Underpinning the ONC methodology was the assumption of independence between the count of population given by the Census and CCS. For the ONC to work well, there must be no systematic relationship between the chance of a household or individual being enumerated in the Census and of being captured by the CCS. However, dependence was found to exist to a sufficient extent that adjustment was needed - see lessons learnt.
Key stages of the One Number Census methodology
England & Wales were divided into 101 areas, each with a population of about 500,000. These areas are known as Estimation Areas (EAs) and are made up of whole Local Authority Districts (LADs) or groups of smaller LADs.
Census Coverage Survey (CCS)
In each of these EAs, a sample of postcodes was drawn, with CCS interviews in each of the sampled postcodes. The aim was to identify and interview all households within each postcode. As underenumeration has historically been found to be disproportionately distributed across areas, the sample was stratified according to a Hard to Count (HtC) index and the hard-to-count areas were oversampled. The HtC index was constructed from 1991 Census variables found to be associated with underenumeration.
The CCS began three and a half weeks after Census Day. To maintain independence, CCS interviewers were not provided with the Census address lists for their areas. Maps of the CCS postcodes were supplied to interviewers and they attempted to interview every household at every address within the postcode, in effect re-enumerating the area. Comprehensive training was provided for the CCS interviewers. The CCS fieldforce was supplemented by 80 experienced ONS interviewers who normally work on the ONS social surveys; they were allocated to teams in areas expected to pose particular problems in obtaining responses.
The ONC method of estimation is based on an assumption of independence between the Census and the CCS, and considerable efforts were made to ensure that the Census and CCS were carried out separately:
the sample postcodes were kept confidential;
census managers were not able to work on the CCS;
CCS managers were not allowed to work on the Census; and
census enumerators were allowed to act as CCS interviewers, but were prevented from interviewing in the same area they had enumerated and were not informed of their postcodes until their census fieldwork had finished.
Matching the CCS and Census
The CCS records were matched with those from the Census using a combination of automated and clerical matching. The matching process had to be as accurate as possible because the number of mismatches has a direct impact on the final estimates. The five key stages of the matching process for each unique postcode were: Exact Matching; Probability Matching; Clerical Resolution; Clerical Matching and Quality Assurance (including a double matching strategy). Further details of the matching algorithms can be found in the ONC Steering Committee paper SC9814.
Estimation of populations for each EA and LAD
Populations for each EA, by age and sex, were estimated using a combination of standard statistical techniques. Estimates of the total population in postcodes covered by the CCS were based on a methodology known as Dual System Estimation (DSE). It was inevitable that some households and people would be missed by both the Census and CCS. DSE provides an estimate of the number missed by both, using the known numbers of people counted in both the Census and the CCS, those counted only by the Census and those counted only by the CCS. Having established the relationship between the Census counts and the dual system estimates in the postcodes covered by the CCS, it is then possible to estimate EA underenumeration in the areas where the CCS was not undertaken. These estimates were made for each age-sex group in each HtC group. See Steering Committee Papers SC0003A and SC0003B for more detailed methodology.
DSE requires a number of conditions to be met to ensure that error in the estimates is minimised. In particular, for an unbiased estimate independence is required between the Census and CCS. As described above, the Census and CCS were made operationally independent. Simulation work was undertaken to examine the impact if there did prove to be some 'dependence'; these simulations demonstrated that, even for quite extreme levels of dependence, the impact on the population estimates is small provided both the Census and CCS have high response rates. Details of this research can be found in Steering Committee paper SC0103. However that assumption of high response rates was not achieved in practice in some areas, leading to the need to adjust for dependence (see later).
Small area estimation techniques were then used to estimate LAD populations by age and sex. Many EAs consist of more than one LAD. With the exception of a few large LADs, LADs did not contain sufficient CCS postcodes to enable accurate direct estimates of underenumeration to be made (during the development stage it had been decided that it was not practically feasible to undertake a CCS of the massive size that would be required to provide direct estimates at LAD level). See Steering Committee Papers SC0003A and SC0003B for more detailed methodology.
Imputation of records for households and individuals
Records for households and individuals estimated to have been missed by the Census were imputed to produce a complete set of responses for each household and person, whether directly responding or imputed by the adjustment for underenumeration. The ONC imputation process had three main stages:
imputation of missed households (and the individuals within them); imputed households were geographically placed into either a physical property identified by census enumerators where no census response had been received, or into areas where similar households already existed;
imputation of missed individuals, ie. individuals missed from households where the household had been counted by the Census. Donor individuals for each type were then selected and used to impute individuals into the types of households that were likely to have missed people from their census return. This process added people to real households; and
calibration to estimates of the population. This ensured that the overall distribution of imputed individuals and households was the same as the ONC estimates of households and individuals missed by the 2001 Census.
The result was an individual level database that represented the best estimate of what would have been collected had the 2001 Census not been subject to underenumeration. Tabulations derived from this database automatically include compensation for underenumeration measured by the CCS, for all variables and all levels of geography. The detailed methodology can be found in Steering Committee Paper 'A Donor Imputation System to Create a Census Database Fully Adjusted for Underenumeration' SC9908.
EA/LAD Quality assurance
An extensive quality assurance process was undertaken. This involved a series of quality checks for each EA, aided by data, grouped by age, sex and geography, drawn from the annual mid-year population estimates produced by ONS and aggregate level administrative data. The administrative data sources and those supplying them were: patient registers (Health Authorities); state pension and child benefit (Department for Work and Pensions); children at school (Department for Education and Skills and the National Assembly for Wales); students in further and higher education (Learning Skills Council and Higher Education Statistics Agency); British armed forces personnel (Defence Analytical Services Agency) and American armed forces personnel and their dependants (United States Air Force).
The various data sources were used to calculate a range of plausible values for the number of people of each sex within five-year age groups in each geographical area. The ONC population estimates, with confidence intervals, were compared with these 'diagnostic ranges'. A range of descriptive information was also gathered to give a fuller picture of the area under consideration, for example information about the fieldwork for the Census and the CCS and feedback received from LAs commenting on past mid-year population estimates from the ONS. Demographic ratios were also calculated. All of this information was presented to Quality Assurance panel meetings consisting of specialists from ONS and Southampton University. They considered the evidence for each EA and LAD before either accepting or rejecting the estimates.
There was a predetermined contingency strategy in the event of the CCS not providing a robust estimate of undercount, such as in a situation where the fieldwork was poor. Information from similar LAs which had already passed the QA process was used to make adjustments. This process is called 'borrowing strength'. The similar LAs, or borrowing strength areas, for each LA were set out and agreed as part of the ONC consultation. The sub-national contingency strategy was required for two LAs, for Sheffield because of problems in the field with the CCS and for Shepway because of doubts about the balance of the CCS sample in that area. On the first occasion strength was borrowed from the agreed borrowing strength LAs, and on the second occasion it was possible to borrow strength from the other LAs within the EA. These strategies are covered in detail in the paper 'ONC Quality Assurance and Contingency'.
The administrative sources were not relied on to replace census estimates at any time, even when the Census data were outside the range of plausible values. The volatility of these administrative sources had been established in the ONC development phase, when investigations found none of them reliable enough to use within a 'triple system estimator'. The research showed that inaccuracies within the administrative data at individual level would lead to significant biases within the estimated populations. With hindsight, ONS should have identified the need for further investigation in some areas. For example, use of council tax has found to be helpful by the matching and LA studies (see later).
National quality assurance and dependency
Underpinning the ONC methodology is the assumption of independence between the count of population given by the Census and CCS. For the ONC to work well, there must be no systematic relationship between the chance of a household or individual being enumerated in the Census and of being captured by the CCS. The national quality assurance indicated a degree of dependence sufficient to warrant adjustment. There was a predetermined national contingency strategy to be used for such an eventuality, adjusting the Dual System Estimator with plausible target sex-ratios, or other demographic information. However, the quality assurance process had shown consistent changes in the observed sex ratios over time in the censuses of 1971, 1981, 1991 and 2001, so that it was not appropriate to base an adjustment on an assumed sex ratio. Alternative methods of adjustment were investigated and a method, using a household-level count, was developed and implemented.
Consultation
It is important that users of census data have confidence in the estimates of the population produced by the One Number Census. In arriving at the ONC methodology and processes, acceptance was sought in a number of ways, including:
the Steering Group which included key representatives from the academic and user community and interested experts;
the consultation process with census user groups;
two 'Series A' Royal Statistical Society (RSS) papers (Brown et al, 1999, Steele et al, 2002);
several RSS seminars;
the Spring 1998 Census Consultation paper '2001 A One Number Census';
the Spring 1999 Consultation paper 'A Guide to the One Number Census';
the consultation with local government Statistical Liaison Officers in Winter 2000 on the areas that should be used for the contingency strategy;
the Autumn 2001 Consultation paper 'A Quality Assurance and Contingency Strategy for the ONC';
a workshop devoted to the One Number Census project in May 1998; and
special workshops held in conjunction with Census Output Consultation Roadshow meetings.
Assessment & Lessons Learnt Strategies for estimating the population
The 2001 Census estimated the England and Wales population to be 52,041,916. When first results were published in September 2002, this was the best estimate that could be achieved by the ONC process in the time available. However, the 2001 Census results were the focus of considerable debate. There was a large discrepancy of 1.1 million between those Census estimates and the population estimates, which were based on rolling forward the 1991 Census counts by measuring annual population flows. Some LAs had large revisions to the population counts for 2001 and throughout the previous decade. This led to concerns being expressed by some LAs about their lack of confidence in the ONC results. In retrospect, ONS needed to have been better resourced to more immediately respond to those concerns.
Initially the gap between the Census results and mid-year population estimates was judged to be the result of both the demographic adjustments to the 1991 base after the Census and the known weakness in migration statistics. However, ONS was unable to explain empirically its conclusion that the difference between the Census and mid-year estimates was primarily the result of unmeasured outflows, and it gave a commitment to further work and analysis.
This further work resulted in several revisions: revisions in June 2003 to the external migration series; revisions in September 2003 of the 2001 mid-year population estimates, based on analysis of the Longitudinal Study; and further final revisions in September 2004 to the 2001-03 mid-year estimates, based on the Local Authority Studies.
The explanation of the difference between the 2001 Census based mid-2001 population estimates and the rolled forward population estimates is summarised numerically in the table below.
Study
Population Estimates Adjustments
Intercensal Discrepancy
1,140,000
1991 Adjustment
351,000
Migration Adjustment
305,000
Longitudinal Study Adjustment
(and others in September 2003)
193,000
Unexplained difference
291,000
LA Population Studies
107,000
Longitudinal Study Consequential Adjustment
(and other adjustment in September 2004)
-25,000
Remaining unexplained difference
209,000
After making the adjustments already published, the majority of the 1.1 million difference was explained. However, the table above shows that there remains about 209,000 unexplained difference. This is likely to be largely attributable to remaining difficulties in estimating migration accurately and issues associated with the usual residence definition.
The results of the LA studies show that the One Number Census (ONC) worked well in most areas, but that there were a few cases where it was not able sufficiently to adjust for exceptional circumstances. ONS concluded that better estimates of the population could be made in 15 areas; Manchester and Westminster were the authorities with the largest changes with much smaller revisions to another 13 LAs. For a full description of these studies see here.
The analysis showed that there was a need for revisions to the 2001 Census based population figures of around 107,000 for England and Wales as a whole (64,000 of which were further revisions in addition to the results provisionally announced previously). The revisions are just outside the 95% confidence interval for the population as a whole, as estimated from the ONC sample in 2002, of +/- 0.2 percent (or +/- 104,100).
There will always be issues about the ability of a census estimate to 'accurately' reflect the 'true' picture particularly for small areas with high underenumeration. More could have been done in advance to focus on the estimated confidence intervals, the assumptions underpinning them, particularly with regard to response, and the likely impact if those assumptions were not met or were pushed to their limits. It may also be that the term 'one number census' raised unrealistic expectations that everything, both the Census and the population estimates would be perfect first time.
Both the Statistics Commission and the Local Government Association have published reviews which conclude that the methodology used in 2001 was the best available and no alternative approach would have produced more reliable results overall.
It is possible that the twin objectives of the ONC, to form the base for both the Census results and the mid-year population estimates, led to one detracting from the other. At the time this was felt to be very important, to avoid the difficulties caused in 1991 with four different population counts. The Census is a micro level dataset, measuring the population at one point in time, and consistency within the dataset is important. The population estimates, however, are a macro level dataset which may need to be revised in the light of further knowledge. One option for the future would be to produce ONC-style Census results plus a first set of population estimates and then have a second phase where time can be taken to assess the results thoroughly, with the view to (if necessary) producing a final set of population estimates perhaps later.
Further discussion of options for improving population estimates for the future can be found in the study 'A demographic statistics service for the 21st century'. ONS will continue to work to extend our understanding of the conduct of the Census and of the quality of administrative data sources, in a range of types of area. The aim is to improve the methods and data sources used in producing the population estimates in order to deliver more accurate estimates at the local authority level, and also increase knowledge of their reliability. In the much longer term ONS has issued a consultation document setting out proposals for an integrated population statistics system that might be implemented when administrative sources, linked with survey sources, can provide data of sufficient reliability and coverage. Details of this are linked here.
Operational and organisational
Operationally the ONC was a major success. A new and ground-breaking methodology was successfully implemented and for the first time (both here or anywhere in the world) all census results were adjusted for measured underenumeration. Three key statistical elements (matching, estimation and imputation) were developed and operated to a high degree of professionalism.
Operational difficulties with other aspects of the Census, such as the outbreak of Foot and Mouth disease and delays in receipt of postback forms, are described in the other relevant evaluation reports.
The Census enumeration itself did not go as planned. The final postback rate of 88% was remarkably high, but it led to significant disruption in the postal flow, with forms being received quite sporadically in many areas. The poorest response rates were way below calculations. Without the CCS to estimate coverage this would have put at risk the value of the Census.
Organisationally, the project management of the whole ONC was successful and was well integrated with the rest of the Census operations. Relations with the academic team at Southampton were extremely good and created an excellent partnership which helped to develop skills within those at ONS.
Project Timetable
There was a perception amongst some users that the ONC led to a much longer timetable and to the results being delayed. The ONC process added around three months elapsed time to the original timetable, mainly to carry out national level quality assurance and as contingency for methodological problems. In the event the publication of first results was delayed by a further month. This was mainly due, not to the ONC but to considerable delays to the processing timetable. The three months that had been included within the ONC as a contingency was needed to catch up on the basic census processing timetable, leaving very little contingency for methodological problems. Even so, through careful management of the issues (outlined above) within ONS and the adaptability of ODPM, the project ultimately was able to provide the results that allowed the population estimates to be delivered in time for the Standard Spending Assessment (SSA). In implementing any major innovation, there is always a greater risk of unexpected problems and there were two such major occurrences on the ONC. One was when the imputation method caused inappropriate 'spikes' which had to be eradicated, and the other was the need not just to adjust for dependency but to develop an alternative adjustment method when the contingency strategy proved unsuitable. Both of these took significant time and resource to resolve. Although they did not cause additional delays (because of staff's commitment and willingness to work many extra hours to make the innovative methods a success), they certainly limited ONS' ability to catch up on the timetable. Two unexpected major hitches is probably less than might be expected for the level of innovation involved but it does illustrate the need to build significant contingency into the timetables for innovative processes, to a greater extent than in this case. With hindsight, it may be that the additional three months in the timetable was not long enough to allow sufficient contingency time for the unexpected.
A related lesson is that the ONC system development was carried out too late because the methodology was not finalised until a late stage. There were unexpected delays in delivery of the rehearsal data, essential for finalising the methodology and also testing and preparing the ONC systems for live data. The consequence was insufficient time to develop and test the system fully before live running began. The resources that had been estimated to be needed for live running were not enough to cope with the delays in processing which lead to a compressed timetable; once this problem was identified, staff were seconded from other areas to address this. The seconded staff had to be trained but were generally able to pick up the procedures quickly, due to the clarity of the system and the knowledge of the core team. The Matching team (matchers and expert matching quality assurers) were all agency staff. Using an agency gave the flexibility to increase or change the staffing profiles at short notice. This became important as the timetable became compressed due to changes in the EA processing order. The contract with the agency worked extremely well and most of the agency staff stayed for the full term of the project.
Quality Assurance
Operationally, the quality assurance process was successful: nothing on its scale had been done before in any other country and international experts have commented on its thoroughness and professionalism. There are issues of the ever-present time/quality balance, however; the inevitable pressure to produce census results can conflict with the need for in-depth demographic analysis.
The subsequent work by ONS has demonstrated that more analysis was required to identify the need for further adjustments in a small number of areas. The difficulty was that these issues were not resolved until 2004. For the 2011 Census it will be important to build in improvements that resolve these issues more speedily, including taking account of other sources and local knowledge as appropriate.
More information about the Census data collection difficulties would also have helped. The lack of reliable quantitative information from the Field Management Information System (FMIS) hampered some judgements (discussed more fully in the Data Collection Development evaluation report).
Although a substantial amount of information on the QA process was provided, ONS had not expected or planned for the huge demand for information for particular areas from LAs and others. With hindsight we might have predicted this better and therefore been able to respond more systematically.
Dependence
The ONC relied almost completely on the CCS as the source of information on underenumeration and subsequent work has shown this to be a shortcoming. Investigations during the development found that none of the other sources which might have been used, for example in a triple system estimator (Census, CCS and a third national population source) was reliable across the country as a whole. The CCS was designed as carefully as possible to be independent and the operational measures put in place to achieve this were successfully implemented. The CCS also achieved high response rates in the majority of areas. However dependence was found. Intuitively it does seem likely that the chance of being missed by the CCS is higher if the person in question was missed by the Census.
The measurement of such 'dependence' is extremely difficult; internationally there has been some theoretical work but nobody else, to our knowledge, has attempted the practical application of measuring 'dependence'. In hindsight, ONS should have done more in contingency planning to consider the impact on the ONC of the lack of any third national source of adequate quality.
For the future it will be important to find ways of making better use of administrative data as a third source at the data collection and estimation stages and to have a greater input of local knowledge throughout.
Conclusion
It was a huge achievement to carry out the ONC, particularly what was accomplished in the timescales. The further studies undertaken have provided lessons to be learnt but have also demonstrated that the ONC approach provides a platform on which to build for the future.
The success should be judged in the context of the Census operation that actually took place, with levels of underenumeration quite variable and sometimes extremely low, as the postal flow and the later census payroll problems created difficulties that the One Number Census could not always answer. However, in the context of a census, with large migrant flows in the last half of the decade, and considerable differences in residence status, it is clear that the Census proper on which the population statistics as a whole are founded, needs to be broadened in its basis as a platform for adjustment and analysis.
Census News
