Project objective
Conduct a Census Coverage Survey to enable underenumeration in the Census to be estimated and used to produce Census based estimates of the population of sufficient accuracy.
Background
The level of response achieved in a population census is the key measure of its quality. Response rate percentages in the high nineties have been achieved in recent censuses world-wide including the England and Wales census held in 1991 which it was estimated covered 98% of the population. However, what was particularly noticeable about the 1991 census was the differential response rate between various groups of the population and the difficulty in effectively measuring this. The One Number Census (ONC) programme was set up to research and develop a more effective methodology to measure and account for underenumeration in the 2001 Census. The key element in this process was the Census Coverage Survey (CCS) - a redesigned and significantly larger post-enumeration survey than those conducted after previous Censuses in England and Wales.
Methodology
This executive summary of the evaluation of the survey is written in isolation from other evaluation reports and therefore should not be taken out of context. The report highlights lessons learned for England and Wales.
The CCS was an interview based survey conducted independently of the Census.
The sample design was postcode based across all local authorities in England and Wales with interviews with all households in the sampled postcodes.
The CCS was designed to produce direct etimates for each of a hundred design groups, also known as Estimation Areas (average population 500,000).
The sample size was approximately 16,000 postcodes (320,000 households) and was chosen as the most optimal size to deliver the required accuracy of final estimates of the population at local authority level.
The CCS sample was stratified by a "hard to count index". This index ranged from 1 to 3 and was constructed using 1991 census information known to be associated with underenumeration (such as multi-occupied buildings, private-rented accommodation, young migrants).
The survey procedures and the questionnaire were tested extensively over the period 1997 to 2000.
In order to carry out an interview with each and every household a solid management structure was required to keep things on track and ensure the work was carried out in an organised way. The following structure was used.
34 Field Managers (FM)
|
273 Team Managers (TM) - about 8 per FM
|
3,900 Interviewers - about 14 per TM
An additional 88 experienced ONS (Social Survey Division) interviewers were integrated into the plans toprovide additional support in the most difficult areas, principally in London.
The survey started four weeks after Census day and took place from 24 May to 17 June 2001. This was a short period to minimise the number of people moving house between the two operations.
Interviewers were supplied with maps of their areas - no address lists were provided.
The survey data was collected using short (5 to 10 minute) doorstep interviews conducted by specially trained interviewers to keep the form filling burden on the public low.
A subset of Census questions were asked (those most related to underenumeration) - name, age, sex, marital status, relationship to other members of the household, ethnic group, whether the person was in education, whether the person had moved in the year before Census Day, and economic activity last week.
A team working ethos was promoted from the start for field staff so that nobody working on the survey would be in isolation.
The field methodology was designed to ensure a good response rate was achieved. A calling strategy was designed to vary the time of repeat calls, and intensive training was provided so that all staff were fully aware of (and could explain) the survey objectives.
The survey operated a flexible field force concept that meant that managers could move resources around within reason to achieve a good response rate over all areas.
A customised management and information system, Team Reporting And Communication System (TRACS), was designed to keep track of progress and share information during the fieldwork phases.
A postback self-completion survey form was left at households for which no contact had been possible over the survey period.
A national estimated response rate (number of successful interviews achieved as percentage of properties found) of 91 per cent was achieved, outstanding for a voluntary survey;
In London and the South East, known to contain the most difficult areas to enumerate, an estimated response of 84 per cent was achieved;
Of the 101 design groups covered, 58 had an estimated response over 90 per cent. Only 9 had an estimated response of less than 80 per cent, caused by isolated pockets of low response within the design group.
The postback exercise for non-response was successful, increasing the number of households covered by nearly 2 per cent;
The national refusal rate was only 5 per cent, varying between 4 and 7 per cent by area.
The overall response is certainly considered a success since the size of the CCS was chosen such that it would provide built-in contingency for lower performing areas by borrowing strength from others if need be (reference/link to ONC Quality Assurance paper on web).
The interview questionnaire worked well with no reports of systematic interviewer or interviewee error.
Maps were clear yet at times somewhat out of date.
Training programmes for field staff were well received, particularly the training video and interactive elements.
TRACS worked well providing the necessary information to enable resources to be directed to areas of lowest response. An example of this was in London. As ever with IT systems more planning and testing may have helped avoid early teething problems.
New procedures were developed to enable the survey to continue in areas affected by the foot and mouth disease outbreak. Centred around telephone interviewing where face to face interviews could not take place, these worked well particularly given they were unplanned.
Recruitment of interviewers proved more difficult than planned but sufficient interviewers were in place on time - recruitment agencies were used in some areas to provide candidates.
The fieldwork was extended in London to bolster the response rate.
The integration of the ONS interviewers helped to bolster response and provide experience in the areas of greatest need.
There was a general feeling that administrative tasks of field staff were over-bureaucratic.
Lessons learned
Although inevitable minor problems occurred these were generally overcome efficiently as a direct result of the planning that took place. No issue caused a failure in the survey programme and the work was completed successfully, providing the input needed for the One Number Census project. The key overarching lessons are:
Extensive, good and early planning is essential.
Recruitment of the field staff was a notable challenge and in conjunction with the Census itself needs careful consideration to avoid confusion and competition between the two exercises. More emphasis might be placed on local recruitment.
Good communications and current management information are essential in a survey of this size and timing.
The administrative duties involved for field staff should be as streamlined as possible.
The team working ethos of the CCS was a real strength.
Conclusion
The Census Coverage Survey was the largest one-off household survey carried out in England and Wales. It achieved its primary objective in terms of overall response rates, whilst keeping the variation in the response between areas as low as possible.
Census News
