One of the metrics that most communities looks at is āaverage time to first commentā or āaverage time to first answerā ā¦ but what does average mean? And which should I use to accurately measure my community's health? š„
Itās been a long time since I studied math actively (I may or may not have had a flip phone and a decidely emo haircut at the timeā¦), so I figured I would bop around the internet to brush up on the definitions:
After searching a few other sites, it sounds like the term 'average' can be used ambiguously, depending on the context of the data you are working with so I'm going to throw that term out the window for a moment and focus on 'median' and 'mean' which are much clearer to define:
TheĀ arithmetic meanĀ is the number you get by dividing the sum of a set of values by the number of values in the set.
In contrast, theĀ medianĀ is the middle number in a set of values when those values are arranged from smallest to largest.
Ok, so which matters to me? Since we normally care about this in the context of average time to first comment (or answer), let's look at a sample scenario.
A community has 5 questions that have been answered in various amounts of time.
- Question A was answered in 15 minutes
- Question B was answered in 30 minutes
- Question C was answered in 60 minutes
- Question D was answered in 120 minutes
- Question E was answered in 5 days (aka 5 days * 24 hours * 60 minutes = 7200 minutes)
The arithmetic mean would be (15 + 30 + 60 + 120 + 7200) / 5 = (7425)/5 = 1485 minutes aka about 24 hours)
The median would be 60 minutes, the number in the middle of the pack.
So which calculation has more value as a community manager?
You can find a lot of opinions on the internet (for a change, right?) by googling which is betterā¦ but the answer tends to be that mean makes sense when your data has fairly few outliers, and the median makes more sense when your data has outliers that might 'throw off the curve' and skew the data. (I found this article pretty interesting.)
Ok, so is my data skewed?
Usually, it is. Some communities might be right on top of their posts, but in busy communities, there might be discussions that go unanswered for days, months or years. If someone digs up a post from a long time ago and finally replies, it might really throw off the mean.
So which should I use?
Without going back to school for data analysis (and maybe going back to the emo haircut?), my best bet is to look at both and use my anecdotal knowledge of my community to see what makes the most sense ā the arithmetic mean which includes all outliers, or the median, which isn't distorted by outliers.
In the example above, would it be fair for me to go to my team and say, hey, what the heck, you're only answering things in 24 hours?! when 4/5 times , they are answering in under 2 hours, and maybe that post that took five days to reply to they pinged me on slack about three times (oopsss š)
This is why we use the median as the default in our analytics ā because we think it's more important to see how quickly folks are answering most of the time, rather than focusing on the arithmetic mean, which includes those occasional outliers.
To get the full picture, why not create both?
When you select, 'average time to first comment' off the hop, it'll default to the median to discount those outliers that are so common in community:
If we want to calculate the arithmetic mean, we can switch median to 'average' (which is currently the arithmetic mean in the function operator, but I've heard that this might be renamed soon for clarity):
Oof, right in the ego! And I felt so good about the first chart!
So does this mean that we are normally answering things in 6 hours or almost 4 days?
If I get a full list of discussions for that time period and their individual time to answer (which shows up in seconds and requires some excel kung fu to wrangle), and use countif to see how often we are hitting each category I can see that we only exceed that 72 hour mark about a quarter of the time, and we are under 6 hours over 50% of the time:
So for me, I think the median is a more accurate representation of how quickly we normally answer questions, while the arithmetic mean isn't as representative of our actual behaviours.
What do you think?
Which is a more valuable metric to you? What makes the most sense and provides the most information about the health of YOUR community? Tell me what you think in the comments ā¬ļøā¬ļøā¬ļøā¬ļø
