#TipTuesday: Crafting a stellar survey

eyork

Communities hold tension for what the host organization wants it to be, and what the community members need it to be. For Community Leaders, it is in their best interest to have a close pulse on the needs of the community members and- what better way to find that out than to ask them directly.

Survey creation is both an art AND a science. With how much time people spend on their devices, they may not been keen on taking a survey (even if they have opinions to share)! So it should be taken under consideration that making and distributing a survey may only be one step in getting feedback from your member base. You may also choose to: host focus groups, speak to a select number of individuals 1:1, and/or share the survey results with your greater [external] audience. Suffice it to say, a community space should benefit both the goals of the host, and of the members: a survey can be a great tool for this.

To create a useful survey, you need to have a clear and specific objective, well-defined target audience, appropriate question types, and a clear and concise layout. The survey questions should be unbiased, easily understandable, and relevant to the objective. The order of the questions and the response options should be logical and easy to follow. Additionally, it is important to pretest the survey with a sample of the target audience to identify and correct any issues before sending it out to the larger population. Finally, the survey results should be analyzed and presented in a way that is meaningful and actionable to the survey stakeholders.

Defining your survey's objective and target audience

Before creating questions for your survey, you’ll first want to define your objective. This part doesn’t have to be complicated (even if getting the information may feel complicated). In fact- the simpler the objective is, the better: it will give you more guardrails when creating questions. If it doesn’t relate to the north star or your “why”, then consider omitting the prompt as it will likely end up being a distraction from the survey’s end goal.

Considerations for choosing your respondents:

• How big is our sample size?
• If we don’t get 100% response-rate, will the sample size be large enough?
• Do we have a target demographic? (spoiler- the answer should ALWAYS be yes, however: based on the “why” of your survey, you may choose to broaden or narrow your focus)
  • Who are we missing from the target? Are their implications to that?

Surveys IRL

Many Community Managers send out surveys to understand platform utilization from their members. Imagine there is a question about why someone logs into the community- consider using the “book end” technique: send the survey to people who login the most frequently AND to people who login infrequently to understand their why (or in the case of the latter: why NOT). The benefit is having information on where to optimize or adapt your community tactics based on each sides’ feedback.

Using surveys to understand change over time also means you may need to consider if you need to re-survey this group of people again, or a different group of people at the same time!

Selecting the appropriate question types and response options

When making an initial draft of your survey, it doesn’t have to perfect on the first go-round. In fact, it may help just to do a brain-dump of all the questions you may want to ask, even if they’re not grammatically perfect or laid out in order. Simply bulleting out your questions is a good place to start.

From there you can start to take a closer look. First thing is to take out any question that doesn’t have to do with your “why” and delete any duplicates to avoid straightlining (answering questions without putting any thought behind it).

You’ll also want to consider the types of questions you present-

• Open vs. closed-ended
• Ratings (how many stars would you give us)
• Multiple select
• Scaled (Unhelpful – to – very helpful)

Designing the survey layout and question order

Now that you have your initial questions, it’s time to fine-tune your survey. Are the questions neutral? They should be. Do you have a question where you’re really asking two things at once? Break them up into two questions. Utilize direct and simple language a large audience would easily understand. How you word questions can greatly affect how people answer, so consider providing context if necessary

Real life example- Many Community Managers run up against the platform utilization feedback “I don’t have enough time for community.” Posing the question “Do you anticipate using the community more or less on average over the next month?” is not the same question as, “Do you anticipate using the community more or less on average over the next month if we enable SSO?” Adding context to questions that would make your “why” clearer to your audience can make a big difference!

Ordering your questions should also be taken under consideration. According to the Pew Research Center, there can be an assimilation vs. contrast effect based on the order of your questions but- more importantly- your questions should unfold in logical order. If you were having a conversation with someone, would this line of questioning make sense? Do the questions start out general and develop into specifics? Is there a chronological basis to the questions. This is easily something that can be A/B *tested with trusted colleagues before sending out to the general public.

And unless there truly are required questions, make the majority of your questionnaire optional; if people run into too many questions that don’t apply to them, or they simply don’t want to answer specific questions, there’s a chance they opt out of the survey altogether. You can always add a “prefer not to respond” choice on each question and add field at the end asking if they’d be open to a follow up email to clarify any of their answers.

*Speaking of testing- always test your survey first! Send it out to someone to check for flow, grammar, typos etc. An ounce of prevention is worth a pound of cure!

Lastly, keep it short and sweet. In today’s day and age attention span is getting shorter and shorter, so the goal is to balance getting quality information as succinctly as possible.

Analyzing and presenting survey results in a meaningful way

To avoid work for the sake of work and have your survey create impact, make sure you come back to your why! Why did you create this survey in the first place? What outcome were you hoping to get clarity around? What objective were you trying to propel forward?

Make a note of anything obvious from the jump- Do an initial sweep. Are there:

• Blanks
• Outliers (like REALLY out of left field)
• Easy trends to spot

From there you can do a deeper dive- do certain segments all answer the same? Is there variance across a singular question versus the entire questionnaire? Set aside some deep thinking time to analyze what you’re seeing to avoid generalizing responses or misrepresenting the results. It’s the Sherlock method: observe the facts as they are to understand what is driving the behavior vs. making assumptions about what is happening that might make an audience answer the way that they did.

Consider creating visual elements to present your data. Especially for data sets that come from close-ended questions, rendering your information into some sort of graph will make it easier for people to see trends, or obvious answer groupings.

You should also come back to your “why did we do this survey in the first place” and present your data with your north star front and center. The answers may be what you expect- or maybe they come as a surprise! Either way, by the end of the survey, there should always be at least two things that happen:

1. Present the data internally to your relevant teams
2. Thank the participants of the survey and let them know how their responses are going to be implemented!

Many organizations overlook step number two, but for Community Leaders this is a critical step. Closing that feedback loop ensures your participating members receive the thanks they deserve and a feeling of ownership of the community that they’re helping to build right alongside your organization.

Survey Sum-up!

• Have a clearly defined objective and choose your target audience accordingly
• Think with the end in mind: what types of questions are going to solicit the most helpful responses
• Vary the question types to keep respondents engaged
• The order of your questions should make logical sense
• Consider how to (visually) represent your data to relevant stakeholders to make the greatest impact

Sources

The Visual Economy Report

Designing Community-Powered Customer Support Hub (FeverBee)

SurveyMonkey

Pew Research Center

qualtrics