Measure Marketing Results Accurately
One of the most important (and difficult) parts of marketing is proving the beneficial effects of marketing for your company. That’s when you measure marketing results against industry metrics and your own. ANY marketer has had the conversation with a superior at any level of their career that went something like this…
Superior: You’ve been doing a great job of spinning up A/B tests for ad campaigns and email campaigns and I recognize that. However, we want to be sure that the money we’re spending on those is actually creating provable ROI.
You: Our Ad Campaigns when A/B tested have steady ROI and our email campaigns have a great CTO (Click to open rate) – Do we need to see more proof?
Superior: I just can’t shake the feeling that the A group had a better performance just by chance. Is there anything we can do?
Here’s where your dormant neurons from your old Statistics 101 class start slowly firing and you vaguely remember the words ‘t-test’.
Quickly summarized, a t-test is a statistical test that can answer the question ‘How do I know that my sample group I selected in an A/B Test did not suggest a certain A/B test result purely by chance?
As always, let’s give you the insights you need to measure marketing results and upgrade your marketing reporting without any grand-standing. You can implement a t-test for your marketing data with four easy to follow steps.
T-test in 4 Steps
Step 1: State your Hypotheses:
- What are you trying to examine? Likely: That my Sample Group A has a statistically different result from my Sample Group B – thereby reaffirming my A/B Test result. In statistical terms – your ‘null hypothesis’ is that your sample groups have the same performance. The ‘alternative hypothesis’ is that they are sufficiently different, meaning your A/B test results are valid.
- Stated explicitly, our null hypothesis is x1 = x2, and our alternative hypothesis is x1 != x2.
Step 2: Examine your results with a significance level of 0.05
- Skipping most of the Statistics 101 – generally selecting a significance level based on industry standards will give you a metric to test your sample sizes for the null and alternative hypotheses. In this case our industry standard significance level will be α=0.05.
Step 3: Calculate
- Calculate your data’s statistical significance using a combination of formulas.
- Below we’ll go through the calculation steps using some REAL Sample data taken from one of our A/B Tests we’ve run internally here at Blue Fish Group.
Step 4: Report and Profit!
- Tell your team that you are confident in the numbers reported by your analysis and to commit this to your marketing processes!
T-test Calculation Example
Let’s examine CTO Rate. Our A Version had a CTO Rate of 62.50% and our B Version had a 34.78% CTO Rate. Most marketers would stop here and just call the A Version a clear winner. However – we aren’t like most marketers and will now apply the t-test to this data to determine the likelihood that email B’s success was due to random chance.
Here’s how you begin to measure marketing results:
Identify the Data
Identify the data you need to perform the calculation:
- X1 = Average CTO rate for email A (generically, X1 & X2 are the success metrics of your samples)
- X2 = Average CTO rate for email B
- N1 = Sample Size (200)
- N2 = Sample Size (256)
- S1 = Standard Deviation of Email A (see appendix for formula)
- S2 = Standard Deviation of Email B
- Sp = Pooled Standard Deviation between the two samples
- Df1 = (N1 – 1)
- Df2 = (N2 – 1)
- Df = Df1 + Df2
- t-value = (X1-X2)/(sqrt(Sp^2/N1+Sp^2/N2))
An appendix of the equations is available at the bottom of this article. At a high level, the exercise we are conducting is to calculate what ‘t’ score results by comparing the difference in success rates, in this case CTO rate, between the two sample groups. The t-score can be referenced against a chart to determine whether or not the comparative results in success rates between two or more samples is statistically significant.
Perform Calculations
To conduct the t test, we will first perform simple calculations to derive the necessary values. Most of your values will come directly from your data. See below:
- X1 = 8/N1 = 8/200 = 4/100 = .04 = Average CTO rate for email A
- X2 = 15/256 = x/100 => x = 5.86 => .0586 = Average CTO rate for email B
- N1 = Sample Size (200)
- N2 = Sample Size (256)
- S1 = = 0.23 = Standard Deviation of Email A (See image: t-calculation)
- S2 = 0.28 = Standard Deviation of Email B (See image: t-calculation)
- Sp = .067 = Pooled Standard Deviation (See image: t-calculation)
- Df1 = (N1 – 1) = (200 – 1) = 199
- Df2 = (N2 – 1) = (256 – 1) = 255
- Df = 454
- t-value = -2.94 = t-value (See image: t-calculation)
Evaluate your Marketing Results
To conduct the t test, we will first perform simple calculations to derive the necessary values. Most of your values will come directly from your data. See below:
- X1 = 8/N1 = 8/200 = 4/100 = .04 = Average CTO rate for email A
- X2 = 15/256 = x/100 => x = 5.86 => .0586 = Average CTO rate for email B
- N1 = Sample Size (200)
- N2 = Sample Size (256)
- S1 = = 0.23 = Standard Deviation of Email A (See image: t-calculation)
- S2 = 0.28 = Standard Deviation of Email B (See image: t-calculation)
- Sp = .067 = Pooled Standard Deviation (See image: t-calculation)
- Df1 = (N1 – 1) = (200 – 1) = 199
- Df2 = (N2 – 1) = (256 – 1) = 255
- Df = 454
- t-value = -2.94 = t-value (See image: t-calculation)
Recalling that our p-value is .05 (put another way, our threshold for indicating a comparative difference in success rates is not random), any resulting p-value from our test that is less than .05 qualifies as significant.
As previously noted, our t-value referenced against the t-table indicates that the significance of the difference between the CTO rates of A and B suggests a p-value between .01 and .002. Since our p-value in this t-test is < p = .05, the difference between CTO rates for samples A and B is significant according to our t-test. Using this t-test, we have quantitatively demonstrated that the greater CTO rate of email B was highly unlikely to be a random event, and we can therefore claim with statistically justified confidence that email B is a superior approach to email A, which leads to step 4!
All Clear....or Not?
Right now you’re either thinking “oh yeah, I remember how to do that” or “what in the name of pepperoni pizza are they even talking about?” If you’re intrigued by the idea of how to measure marketing results but aren’t sure how to start, contact us to start a conversation!
Appendix
