dana centre - Writing Questionnaires

Thinking of the right questions to ask can be a tricky business. And who do you ask them to, in what way? This document provides examples of questions to ask and questions to avoid, as well as raising issues to consider as you plan your questionnaire.

This guide tells you the fundamentals of questionnaires, however, there is not the space here to discuss these issues in detail. For further information, you may wish to consult a one of the many publications available. We recommend, Constructing questions for interviews and questionnaires- theory and practice in social research by William Foddy (1994, Cambridge University Press ISBN 0-521-46733-0)

step 1 - first define your project

As with all evaluation before you do anything else answer these questions in this order:

What do I want to find out? Who do you want to find this out from?
Why do I want to find this out?
How will this information be used?
How will I find this out - choose your methodologies (note the plural) and sample?
Who am I finding this out for and how shall I tell them the results?
How much money do I have for this project? How many staff? How much time?

step 2 - what type of survey do you want?

Okay, so you have chosen to do a survey. Here are your options for the type of survey you might want to conduct:

Self-completion questionnaires

pick-up - left lying around on tables or in rubbish bins
on-line - via email or pop-up windows on web pages

Interviews

face-to-face
telephone
postal
email

And here are some hints to help you make your choice of survey

Self-completion questions

Pros: quick, cheap

Cons: no control over sample, excludes people who can't write, mostly get negative comments

Interviews

Pros: can get representative sample, can probe for more detailed / useful answers.

Cons: time consuming, costly in terms of staff and money.

postal questionnaires

Postal questionnaires can be a very effective method of interviewing people but you must get a return rate of at least 60% for the data to be considered reliable.

Your first letter will elicit a return rate of about 10-20%. Your second (reminder letter) will increase this to about 30-40%. Hopefully your third reminder (plus repeat copy of the questionnaire for all those people with hungry dogs) should get you a return rate of 60% plus.

email questionnaires

Email questionnaires we use are entirely within the body of the email i.e. they are not an attachment. We have found that Fridays are the best days to send an email questionnaire since many people receive and answer them at work. We generally get a return rate of 50-75%.

web surveys

Web surveys - where a questionnaire pops up when someone enters a web-site or a particular part of a web-site - generally get a low response rate. On average only 2-3% of people visiting a web-site will complete such a questionnaire. This means that data from such questionnaires must be treated with caution as it is likely that segments of the audience are not being included in the survey for one reason or another. To be fair though pop-up surveys are sometimes the only way to collect data about a web-site's audience profile.

It is worth adding a 'cookie' to the questionnaire so as not to annoy regular visitors who would otherwise be sampled every time that they enter the site.

step 3 - quantitative or qualitative?

Do you want data that you can analyse statistically or do you want very detailed in-depth data? You need to decide before you choose your methodology since this will determine the type of questionnaire you design and the size of sample you will take.

Here are some questions that will help you decide:

Do you want to know how many people think or do something?
Do you want to find out what the majority view is?
Do you want to find out how views/behaviour vary according to age, gender, frequency of visiting the museum?
Do you need statistical data to convince your audience?
Do you want to understand why people do or think something?
Do you want to find out in detail what people think about something?
Do you want to find out about people's emotional response to something?
Do you want to explore the range of views that your audience have?
Are you expecting to collect a lot of data from each respondent?

If the answer is yes to questions 1-4 you need quantitative data. This means interviewing a sample of at least 100 people. But remember size of sample does not necessarily equate with quality of sample. The sample needs to be an accurate cross section of your audience and not exclude people who only visit on weekends or who don't enter the building by a particular entrance.

If the answer is yes to questions 5-9 you need qualitative data. This means interviewing people in much greater depth. The sample still needs to be carefully and rigorously selected but it will be smaller.

step 4 - choosing your questions (the big step)

Think about your visitors! What are they going to make of your questions? Will they understand what you are after? Are they willing and able to provide this information?

Visitors are not passive recipients of questions who respond in predictable ways to external stimuli. Visitors are trying to work out who you are, why you are asking these questions, how they appear to you, how they feel about themselves.

Remember these two key rules:

Be clear about what you are trying to do
Clearly explain what you are trying to do to the interviewee

However you must also be sure that your interviewees understand your questions in the way that you want them to and can give the answers you require. This depends not only on the wording of your questions but also on your initial introduction, any further explanation you provide, who you interview, where you interview them and many other factors.

When you choose your questions you need to consider many factors. The following section will give you some broad outlines of what to do and what to avoid but the best course of action is to carefully test your questionnaire before you launch your full scale survey (see step 7). What you need to do can be summarised by the acronym TAP

Topic - the topic should be clearly defined for you and for the respondent so that you are both understand what is being talked about
Applicability - The applicability of the question to each respondent should be established. Respondents should not be asked to give information that they do not have
Perspective - the perspective that the respondent should adopt when answering the questions should be specified

types of question

There are two main types of question and you need to be clear which you want to use and when

open or closed?

Open-ended questions are those where visitors have to answer in their own words for example:

"What do you think we could do to improve this exhibition?"

Open-ended questions:

provide very rich data
they do not constrain visitors to answering in your terms
elicit much more detailed answers
gain greater understanding of interviewee's opinions
allows interviewee to raise issues you did not think of

But -

difficult to record answers - may be better tape recording interviews
difficult & time-consuming to analyse data - need to categorise answers
more difficult for visitors to answer
interviewees may not give a relevant answer
interviewees may not give a full and comprehensive answer - you must:
- clearly define the scope of the question
- explain the perspective you want the interviewee to take i.e. your point of view, you and your family; your community's points of view etc.
- probe to ensure that interviewee has fully expressed their point of view

Closed questions only offer (or appear to offer) a limited set of answers for example:

Which of the following age categories do you fit into?

How much did you enjoy this exhibition?

Closed questions also include the following type of question:

Yes/no questions ("Have you visited the Science Museum before - yes / no?")
Number scales ("On a scale of 1 to 5 where 1 is not interested at all and 5 is very interested, how interested would you be in using this exhibit in an exhibition? (and don't worry I didn't make this exhibit so I wont be offended by what you say)")
Rating scales ("How would you rate the following facilities provided in the Science Of Sport exhibition? I would like you to tell me whether you think they were very good, good, average, poor or very poor.")

Some crucial points about scale questions:

Any form of scale you used must be symmetrical i.e. there should be as many positive as negative options. Don't use the sort of rating scales you see on those self-completion questionnaires in restaurants where you are offered options of; absolutely fabulous, excellent, very good, good, ok, poor.
Always include an open-ended question after a rating scale question.If you don't do this you will never know why a service is failing unless you do another time-consuming survey. For example, if interviewee rates any service 'average', 'poor' or 'very poor', ask 'What can we do to improve that?'
When analysing data from rating scales avoid the temptation of lumping people who are 'very satisfied' with those who are 'quite satisfied'. The behaviour, loyalty, needs and wants of the people who fall into these categories are very different and they should not be confused. If someone chooses 'quite satisfied' instead of 'very satisfied' that means there is something that they do not like.
Be aware that interviewees may not have any opinion about an issue. Therefore offer a 'no opinion' option. Otherwise you will get people choosing an answer at random, that does not reflect their views simply because they are trying to be helpful and complete all the questions you are asking them.
There is a tendency for interviewees to chose either the central or a positive option. Hence the distribution of answers will be skewed. Negative answers should therefore be considered as more significant. If you are getting 20% or so of your visitors rating something as poor or very poor you are looking at a serious problem.
It is often important to gauge not only people's attitude towards an issue but also how important this issue is to them. For example you might strongly agree that Bill Clinton's hair is very nice but this issue would be of relatively little importance to you when deciding what political party to vote for.

Multiple Choice

"Which one of the following statements do you agree with most?

(a) I am not especially interested in sports and rarely participate in sporting activities.

(b) I am a keen sports fan and regularly participate in sporting activities.

(d) I have a general interest in sports and occasionally participate in sporting activities.

Closed questions are .

easy for interviewees to answer
data is easy to analyse - already in pre-defined categories
easy to compare answers from different interviewees
good for large sample quantitative surveys

But .

you will only get very limited responses
restrict answers to pre-determined list
don't allow visitors to raise unexpected issues
don't allow visitors to explain or qualify their answers
you must only offer a limited number of options - people on average can only hold 5-9 items in their short-term member so don't overload them with options to choose from

Examples of rubbish questions

There are a number of things to avoid when designing questionnaire. One is using rubbish questions. Here are some examples of questions best avoided. Depressingly many of these are examples from real questionnaires that were actually used on people.

a) Loaded questions

Questions where you give, intentionally or otherwise, hints about which answer you want.

"Hades Pharmaceuticals Ltd. test new cosmetics by squirting them into the eyes of lovely fluffy rabbits while the senior executives stand around laughing demonically. Do you feel that Hades Pharmaceuticals Ltd. are a responsible and caring company? yes / no

You need to watch out for emotive language which can sway interviewees answers. For example lets try and avoid questions such as;

Are the public well informed about recent scientific developments or are they deliberately left in the dark?

However occasionally you do need to load questions just to get visitors to answer them. For example visitors are often wary about being critical. Therefore it is worth loading a question to indicate that you are expecting them to be critical.

What did you like least about this exhibition? How can we improve that?

b) Double barrelled questions

These are two (or more!) questions masquerading as one.

Should the United States cut its defence budget and spend more on social welfare?

There is no way that you could analyse data from this question. Someone may say yes meaning 'yes the US should cut its de fence budget' but not support more spending on social welfare. Or vice versa. The data collected with such a question is worthless. Double-barrelled questions can be very difficult to spot for example the question:

"For whom do you think you will vote for in the next election?"

Will you vote in the next election?

For whom will you vote at the next election?

Watch out for any question that contains the words - and, or - e.g.

Do you distrust banks and building societies?

This is of course two separate questions - one about banks and one about building societies.

c) The dreaded 'why' question

What is your favourite flavour of ice cream - why?

What would you actually answer to the second part of this question? Why questions often force people to rationalise what are irrational actions and feelings.

The reason why visitors think or do something is often what you are trying to get at but you have to be more subtle about it. You can't just rush in with the dreaded why question. Get visitors to describe in concrete terms what happened, when, under what circumstances and how they felt. You can usually infer from the answers to these questions why they did what they did.