Quality Glossary Definition: Survey
Variations: questionnaire, e-survey, telephone interview, face-to-face interview, focus group
Survey is defined as the act of examining a process or questioning a selected sample of individuals to obtain data about a service, product, or process. Data collection surveys collect information from a targeted group of people about their opinions, behavior, or knowledge. Common types of example surveys are written questionnaires, face-to-face or telephone interviews, focus groups, and electronic (e-mail or website) surveys.
Surveys are a valuable data collection and analysis tool that are commonly used with key stakeholders, especially customers and employees, to discover needs or assess satisfaction.
When to Use Surveys to Collect Data
It is helpful to use surveys when:
- Identifying customer requirements or preferences
- Assessing customer or employee satisfaction, such as identifying or prioritizing problems to address
- Evaluating proposed changes
- Assessing whether a change was successful
- Monitoring changes in customer or employee satisfaction over time
How to Administer a Survey
- Determine what you want to learn from the survey and how you will use the results.
- Determine who should be surveyed by identifying the population group. If they are too large to permit surveying everyone, decide how to obtain a sample. Decide what demographic information is needed to analyze and understand the results.
- Determine the most appropriate type of survey.
Determining Survey Type Example
- Determine whether the survey’s answers will be numerical rating, numerical ranking, yes-no, multiple choice or open-ended, or a mixture.
- Brainstorm questions and, for multiple choice, the list of possible answers. Keep in mind what you want to learn, and how you will use the results. Narrow down the list of questions to the absolute minimum that you must have in order to learn what you need to know.
- Print the questionnaire or interviewer's question list.
- Test the survey with a small group. Collect feedback.
- Which questions were confusing?
- Were any questions redundant?
- Were answer choices clear? Were they interpreted as you intended?
- Did respondents want to give feedback about topics that were not included? (Open-ended questions can be an indicator of this.)
- On average, how long did it take for a respondent to complete the survey?
- For a questionnaire, were there any typos or printing errors?
- Test the process of tabulating and analyzing the results. Is it easy? Do you have all the data you need?
- Revise the survey based on test results.
- Administer the survey.
- Tabulate and analyze the data. Decide how you will follow through. Report results and plans to everyone involved. If a sample was involved, also report and explain the margin of error and confidence level.
Survey Considerations
- Conducting a survey creates expectations for change in those asked to answer it. Do not administer a survey if action will not, or cannot, be taken as a result.
- Satisfaction surveys should be compared to objective indicators of satisfaction, such as buying patterns for customers or attendance for employees, and to objective measures of performance, such as warranty data in manufacturing or re-admission rates in hospitals. If survey results do not correlate with the other measures, work to understand whether the survey is unreliable or whether perceptions are being modified by the organization’s actions.
- Surveys of customer and employee satisfaction should be ongoing processes rather than one-time events.
- Get help from a research organization in preparing, administering, and analyzing major surveys, especially large ones or those whose results will determine significant decisions or expenditures.
Survey Example
Pearl River School District, recipient of the Malcolm Baldrige National Quality Award, surveys students, parents, and staff annually using a questionnaire developed and administered by a national research firm. This ensures statistically correct methods, allows them to watch changes over time, and enables comparison with other districts. The surveys show areas of satisfaction and dissatisfaction, but not reasons. To find causes, they must dig deeper. Recently, the annual survey showed that the high school students ranked the cafeteria service low. This was hardly surprising, since the research firm’s database showed that almost all high school students rate their cafeterias low. However, Pearl River wanted to improve this rating.
A committee of student leaders and cafeteria staff prepared a second survey to get more information. Figure 1 below shows a condensed version of their survey. Demographic information was requested at the top. A short paragraph stated the reason for the survey, some additional information respondents needed to know, and who was eligible to complete the survey. The survey was designed to be filled out quickly with check marks in appropriate boxes. A few open-ended questions were placed at the end. Data from the survey, supplemented with sales data, was analyzed and used to plan menu changes.
Figure 1: Data Collection Survey Example
Adapted from The Quality Toolbox, ASQ Quality Press.
THE SET-UP STAGE OF A SURVEY
Survey Set-up
The set-up stage is a very important part of any research, yet its value is often underestimated. Most research projects (and therefore, most surveys) which go badly wrong, do so because of inadequate attention given to setting the project up properly. For this reason, any effort to collect information, should not occur until the issues discussed in this Section are dealt with. This Section is intended to assist researchers and managers in identifying the key factors in planning the research project, analyse their requirements and select the most appropriate method of collecting the necessary information.Aims and Objectives
The first and most important step, in the collection of any information, is to establish the research objectives clearly. From this basis, it is then possible to translate these objectives into establishing specific information requirements. The objectives should define what the survey results will be used for and may include such uses as the basis for decision-making, the allocation of funds, to analyse the outcome of policies or programs (program evaluation) or to determine the direction of future operations. It is highly recommended that the number of objectives be kept to a minimum. With too many, the information gathering exercise can easily become too unwieldy and costly.The Target Population
Once our uses for the data have been determined, our next requirement is to define the target population. The target population is the group about which we wish to make inferences from the survey data. The target population should be defined in terms of :- content (eg. all persons),
- units (eg. in households),
- extent (eg. in Australia), and
- time frame (eg. 6 September 1991).
When considering the population that is to be studied, an important factor is the units that are to respond to the collection (the reporting units) and the units that are to be selected to access the reporting units (the selection units). An example of the difference between reporting units and selection units is given in Aims and Objectives, where the selection unit is a household and the reporting unit is any responsible adult resident at that household.
Methods of accessing populations are dealt with further in Frames & Population.
Conducting Preliminary Research
Establishing Data Requirements
The first consideration when clarifying information requirements is to clearly define the data items to be collected. Only with clear definitions of the terms and concepts involved can the researcher be assured that the data will be relevant to the project's aims and objectives. Standard definitions for data items should be investigated as the use of standards allows comparability between the results from the current investigation and previous data. A common way to ensure all the appropriate data items are collected is to specify, as early as possible in the planning stages of the collection, the detailed tables to be derived. Such specifications not only help clarify the data items to be collected, but the levels or subgroups for which the data is required. Assuming an appropriate collection methodology, defining the levels of interest beforehand will help to ensure an adequate sample size is chosen to achieve the desired levels of accuracy. The definition of concepts also plays a part in identifying the population. The survey objectives define the target population, but the specific data items and output requirements determine the make-up of subgroups inside the population. These subgroups make up the cells on the tables that have been determined in describing the output requirements. All data items proposed for the survey should be scrutinised closely to establish whether they are consistent with the aims and objectives of the project. It is vital that extra information is not sought because it is of interest or is related to the issues being examined. This may lead to the survey losing its focus and add considerably to the cost and time to do the project. Ideally the information requirements, whether they be factual data (eg. unemployment or inflation rates) or attitudinal data (eg. opinions on a particular topic), and the proposed methods of analysis should be put down in writing, as this can help to focus the researcher's attention on important considerations affecting the design of the collection.Financial and Time Constraints and Other Considerations
Apart from the important considerations in survey design outlined above, there are several other aspects that need to be accounted for. The financial and time constraints placed on a survey are often a deciding factor influencing survey design, as well as the size of the survey and the level of accuracy sought. It is therefore important to make both time and financial estimates for each aspect of a survey and monitor them closely. Other aspects include creating or maintaining the survey frame, designing the sample selection, choosing a data collection method, designing the questionnaire and determining the field and data processing procedures.NON-SURVEY METHODOLOGIES
As described above, the characteristics of our data items often point at the best way to collect our information. How expensive and time-consuming the data are to collect are also often important considerations when choosing whether to run a survey. The availability of similar data from other sources also plays a part, but collating data from many sources may also prove time-consuming and expensive.Existing Data Sources
The information that is required may already be available from another data source. Research of subject matter documentation, as discussed in Conducting Preliminary Research above, may provide the necessary information on some topics. Some data may also be available from government departments and agencies as a by-product of their administrative information (eg. Registrar of Births, Deaths and Marriages). Previous surveys may also have been carried out in the field of interest. Care must be taken that the recording of the data comes from reliable sources and that the definitions of the data items used closely approximate those currently of interest. This is a cheap and fast way of obtaining data so long as the existing data meet the requirements of the research.Case Study
In preparing a case study the researcher seeks to collect and analyse as much data about the chosen subject as possible from a relatively small number of cases. Case studies allow greater detail to be gathered for less cost than in most surveys, but the data is less likely to reflect the whole population.Participatory Observation
Participant observation is a method of data collection whereby the researcher seeks to become a member of the group under study, and records data from within. This method is often used in qualitative studies ie. studies that deal with variables which cannot be readily measured such as peoples perception, group dynamics, etc.Advantages and Disadvantages
The main advantages of non-survey methodologies is that they provide data, often quite detailed, in-depth data, at a relatively low cost and that they can result in substantial reductions in the time taken to produce results. However, statistical accuracy may be traded off to achieve these gains. Existing data sources may use definitions that do not match the required definitions very well, and the other non-survey methodologies use only very small numbers of subjects in the study and are open to the subjects not being representative of the target population. This does not mean that non-survey methodologies should be overlooked entirely - it only means that care should be taken when analysing their results. For example, if we chose to look at the effect of overcrowding in our hospitals, we may decide to perform a case study or control group experiment rather than a survey as these methods allow the researcher to collect information for a comparatively modest cost.SURVEY DESIGN CONSIDERATIONS
After considering all the options available to you, should you decide that a survey is the option most suited to your study, your next task will be to decide upon the survey design. The three objectives of sample surveys are to- use the data collected from a sample of population units (using the sample statistics calculated),
- to describe a population, i.e. estimate certain population parameters (descriptive statistics), and
- to test statistical hypotheses about the population.
- the survey objectives,
- the population of interest (or target population),
- the reference period for the data,
- Geographic, demographic boundaries,
- the frame and the units,
- the sample design and sample size,
- the data collection method,
- the questionnaire,
- non-sampling errors,
- non-response,
- the field procedures and data processing system, and
- the tabulations and analyses required.
SURVEY MANAGEMENT
Advertising
There are several parts of the survey where advertising is necessary. These include advertising for staff which can be done through the daily newspapers or through Centrelink and pre-survey publicity to potential respondents which can be done through an introductory letter sent to respondents. After the survey, the availability of survey results can be advertised through the media or by contacting interested parties. Publicity of the implementation of survey results can also raise the usefulness of the study. When advertising for staff, you should specify what the job is, subject of the survey, amount of interviewing work, need for prior experience if required, rate of pay, number of days interviewer is required for, whether evening/weekend work is involved and area where interviews will be conducted.Finance
The financial constraints placed on a survey is often a deciding factor influencing the survey. It is therefore important to make financial estimates for each aspect of a survey and to monitor spending (and therefore the progress of the survey) closely. Adequate resources should be allocated to survey management as it plays a central role in the conduct of a survey, particularly the most labour intensive part. Financial estimates for other aspects of a survey generally fall into three categories:- overheads,
- salaries, and
- survey and processing costs.
Recruitment
Regardless of particular collection methods used, personnel need to be recruited and trained to collect, despatch and process the survey questionnaires as well as interviewing respondents. When recruiting staff, it is necessary that they have the basic skills and attributes required to do the work, and in the case of a national survey, they must be in or near the areas where they are expected to work. The number of staff that you need to recruit depends on the size of the sample, the length and complexity of questionnaire, the timetable and the availability of funds. Staffing levels may vary during the research period depending on staff retention rates or whether there are peak periods or low periods.Training
The adequacy of training given to interviewers and processing staff has a strong influence on the quality of results obtained from the survey. Thorough training of interviewers is important because they have a wide range of tasks to perform and are the main point of contact between the respondents and the researcher. Comprehensive training of office staff should enable them to process the survey questionnaires as accurately and as quickly as possible. Training can be provided in the form of:- manuals,
- formal training courses, and
- "on the job" training.
Interviewer Training
Training for interviewers includes the purpose of the survey; the scope and coverage of the survey; a general outline of the adopted sampling approach; the format of the questionnaire; recording of the responses; correct interviewing techniques; techniques for avoiding or reducing non-response and maintaining respondent co-operation; field practice; quality control; editing; planning the workload; and administrative arrangements.Training for Processing Staff
For processing staff the training should cover the purpose of the survey; the scope and coverage of the survey; the questionnaire; the way the responses are recorded; coding; editing; key-punching instructions if the survey responses are to be punched onto a computer database; quality control; and administrative arrangements.