Ad Testing and Margin of Error

Status
Not open for further replies.
BarneyFife

BarneyFife

Deputy Member
Apr 4, 2007
18
0
0
www.nakedmile.tv
I was looking at an AdWords campaign today and noticed something interesting. In one adgroup I have two identical ads. Today one has a 1.12% CTR and the other has 0%. OK, it's only 6 clicks -- so is it a statistical aberation?

In the last week the ads have a total of 40 clicks with an identical .7% CTRs. OK, looks like today was a short-term aberation. BUT, "all time," out of about 10,000 impressions, the CTRs are .96% and .64%. That's a pretty big difference.

So, how many impressions does it take to make a meaningful comparison? (Math geeks, set me straight if this calculation is off-base.) I used the Raosoft Sample Size Calculator to calculate that with a margin of error of 2%, a confidence level of 99%, and a population of 1 million, I only need a sample size of 4130.

That seems a bit small (assuming that sample size equates to impressions -- if sample size equates to clicks, that's an awfully big number).

So, putting aside any issues like "What kind of a nimrod runs two identical ads in the same adgroup," how many impressions (or clicks) does it take to do a valid split-test? From reading some posts, it looks to me like some people are trashing ads and keywords based upon sample sizes that are way too small.

(As usual, Diorex appears to have some clue about how to approach this stuff: How many clicks to test cpa offers)
 


This is a good question. I think using the number of impressions as the sample size is incorrect.

Having 4130 impressions and 2 clicks, one on each ad, does not tell you anything statistically significant.

Split tests are usually performed on equal amounts and the conclusion determined by the number that received the most clicks.

Assuming both ads received an equal amount of impressions, the proper statistical test would be the X^2 (chi-squared) test, which will determine if there is a statistically significant difference between the two groups.
 
Aqiutas you do a lot of posting here. Ever time I read a thread you already posted
 
Ibart said:
Aqiutas you do a lot of posting here. Ever time I read a thread you already posted
Click to expand...

Its because I spend 16 hours a day online and even though I'm working on my websites and campaigns I'm consistantly on wicked fire, its becoming an addiction, plus I come around to check the new posts when I need a short break from my work.
 
At first, I was also inclined to think that the "sample size" should not be the number of impressions. If you compare it to a poll, it's like asking each searcher which ad they like. In that case clicks would be the only responses, and non-clicks would be people who refuse to take the poll.

But you're not actually asking people which ad they like because they only get to see one ad. I think it's more like running two polls (one for each ad) where the question is "do you want to click on this ad?" In that case, the number of impressions for each ad would be the sample size.

At this point, margin of error becomes very important. If you have a 2% margin of error then one ad has to outperform the other by better than 2%. (That means 3% CTR vs. 1% CTR, NOT 3.02% CTR vs. 3.00 CTR). If you lower the margin of error to less than 1%, the necessary sample size goes up quite a bit.

For example, if you have competing ads with 1% CTR and 1.5% CTR and you assume 1 million potential searchers, each ad would have to get about 37,000 impressions before you could choose one with 95% confidence.

Disclaimer: This is based on the Roasoft calculator. (Also, keep in mind that I have no idea what I'm talking about.)

Also, thanks for the heads up on that tool Aequitas.
 
Status
Not open for further replies.
Top Bottom
Back