Jump to Navigation

Surveys as a Research Method: The Backbone of Quantitative Research is Within Your Organization's Reach


Survey Design
 

Developing good survey questions can be a difficult process, but by putting the time and effort in at this stage, you can ensure that the data you collect will be worthwhile and help you get solid answers to your research questions.  There are few things worse in the research process than doing lots of work to conduct a survey, only to find out at the end of the process that your questions did not really elicit the types of responses you expected.   

Get participatory! The process of constructing relevant questions is one of the best ways to get community members involved in the research process.

Before you begin the main portion of the survey, you might want to think about asking a few “screening questions” to make sure that the person taking the survey is the type of person you want to get information from.   These questions might be about their age, race, religion, years living in an area, an institutional affiliation, or any number of other issues. For example, the Metamorphosis Project conducted work in the LA County city of Alhambra.  We wanted to speak with adults who had lived in the community for at least two years in order to get the opinions of longer-term residents of the area.  Here was our screening procedure:
 

First, we need to ask a few questions to see if you qualify for the survey.

Q1. Are you 18 years of age or older?  (Yes or No)

  • If not, thank you very much for your time, but we are only looking to speak with people over the age of 18.

Q2. How many years have you lived in Alhambra? ___________

  • If you have lived in Alhambra for less than two years, thank you for your time, but we are only looking to speak with people who have lived in Alhambra for at least two years.


Another early consideration is whether you want open-ended or close-ended responses to your questions.  Open-ended questions leave the respondent room to answer as they please – their response could be a word, a phrase, a paragraph or more, depending on how the survey guides them.  Closed-ended responses offer a few possible responses that respondents can choose from.  Many surveys use a combination of both.

Below are some examples from the Metamorphosis Project. Often, close-ended questions ask participants to respond to a statement and place their answer on a scale, like the examples below that come from our measure of neighborhood belonging:
 

For the following statements, please note whether if you strongly agree, agree, neither agree nor disagree, disagree or strongly disagree:

You are interested in knowing what you neighbors are like.             

a) Strongly disagree b) Disagree c) Neither agree nor disagree d) Agree e) Strongly Agree

You enjoy meeting and talking with your neighbors.                         

a) Strongly disagree b) Disagree c) Neither agree nor disagree d) Agree e) Strongly Agree

Your neighbors always borrow things from you or your family.                   

a) Strongly disagree b) Disagree c) Neither agree nor disagree d) Agree e) Strongly Agree

It’s easy to become friends with your neighbors.      

a) Strongly disagree b) Disagree c) Neither agree nor disagree d) Agree e) Strongly Agree


By using this type of numerical scale, we can easily quantify the information and later analyze it using statistical techniques.  If you are interested in conducting statistical analysis, you might want to use a scale – like the neighborhood belonging measure from Metamorphosis – that have been used successfully by other researchers in the past.  This way you can be sure that the questions have been well-designed and you can also compare your data to those of other researchers.

There are several types of scales that can be used in survey questions.

Rating Scales (often called Likert Scales), like the one above, ask the respondent to rate a statement, usually on a 3-point, 5-point or 7-point scale (more than that is too much).  Likert scales can ask about things like agreement, frequency (very frequently to never), importance (very important to not at all important), quality (very good to very poor), likelihood (very likely to not likely at all), or other topics.

Ranking Scales might ask respondents to put a few options in rank order.  For instance, the Metamorphosis Project often asks respondents to rank the top two types of media they use to stay on top of community issues (like local television, radio, or the newspaper).

Here are a few open-ended examples:
 

What is the name of your neighborhood?                                                          

  • In this instance, the respondent could say anything, but their answer is likely to be no more than a few words.

What do you like most about living in your neighborhood? _________________

  • This answer could be lengthy, depending on the respondent's answer.


These answers are still up to the respondent, but there is a chance that these responses could be several sentences or even a paragraph or more, unlike in the closed-ended examples.  While this provides valuable information, it may be difficult to quantify their responses – that is, put them into numbers that can be analyzed with statistics.  This type of qualitative information provides important but different types of research data.

There are a number of other considerations that must come into account as you design the content of your survey.  Surveys should go through different revisions, when possible, and have several people work on and revise them to make sure the questions are appropriate and easy to understand. Once again, if you get participatory you can avoid a lot of the mistakes researchers often make in this area.  While we cannot go through ever single issue that should be thought about in the development of the survey, here are a few important ones to consider:

How personal will your questions be?

  • If you ask about sensitive topics, you need to make sure that the respondent is not being put in any danger depending on their answers.  Also, sensitive questions sometime do not yield accurate responses from participants.

How do I avoid socially desirable answers? 

  • Sometimes, respondents give answers to questions that they think the survey researcher wants to hear or that make them look good – that is, answers that are “socially desirable”.  For instance, how would someone respond to this question: “Do you think it is important to be compassionate to other people?”
    Very few people would be willing to admit that they think compassion is not important!  Try to avoid questions that have a clearly socially desirable answer like the one above.

How detailed or complex will your questions be? 

  • Taking a survey takes a lot of energy out of the respondent, so finding a good balance that keeps the questions simple and direct while still getting good information is important.

How long should the survey be? 

  • Again, if the survey is too long the respondent will likely lose interest after a while.  Ask only questions you “need to know” the answers to, not every single question that would be “nice to know”. 

Does question order and sequence matter?

  • Make sure that the order makes sense –a follow-up question that refers to another response, for instance, has to come after the initial question.

Again, these are some but not all of the concerns that should be in mind when developing the survey itself.  Please see our additional resources for other in-depth treatments of survey design.

Syndicate content