Knowing when to make a Conclusion when Testing

smaxor

smaxor

New member
Oct 17, 2006
2,515
87
0
San Diego
I work with affiliates all day that run 20 clicks to something and say it doesn't work. There is something called "statistical significance" basically how much should you test before you have a level of certainty and decide to move on?

I'm going to do a post on my blog about this, but I also found a cool calculator to figure out how certain you want to be. Typically 85-95% certain is usually what you want to aim for. We can use this calculator on impressions on an ad with the ctr, impressions on a presell with the ctr there and also impressions on an offer with the ctr there. It'll calculate out your probability to find with what certainty you can make the decision. Pretty cool stuff.

Split Test Accelerator: Faster Ad Testing Now!
 
  • Like
Reactions: mpbiz and buckeyefan


That's a pretty cool tool, is that your site? I'd like to program something slightly more in depth and for many more ads as a lot of us don't run just two versions of an ad obviously.

Great post though.
 
Jason thank you for the calculator link it has already helped . . .

2h5jsqh.jpg


Look forward to your blog post.
 
nice tip smax. I generally run mine 100 clicks.. up to 300 clicks per change on my splits to determine a winner when spiting something like ad copy or LP copy.
 
Yep, you should always make sure you have a high confidence level (statistically, faggots) before you make a decision. Sometimes what seems to be the best decision, isn't.
 
  • Like
Reactions: Rooby
smaxor said:
I work with affiliates all day that run 20 clicks to something and say it doesn't work. There is something called "statistical significance" basically how much should you test before you have a level of certainty and decide to move on?

I'm going to do a post on my blog about this, but I also found a cool calculator to figure out how certain you want to be. Typically 85-95% certain is usually what you want to aim for. We can use this calculator on impressions on an ad with the ctr, impressions on a presell with the ctr there and also impressions on an offer with the ctr there. It'll calculate out your probability to find with what certainty you can make the decision. Pretty cool stuff.

Split Test Accelerator: Faster Ad Testing Now!
Click to expand...

Smax this is pretty sick +rep
 
smaxor said:
I work with affiliates all day that run 20 clicks to something and say it doesn't work. There is something called "statistical significance" basically how much should you test before you have a level of certainty and decide to move on?

I'm going to do a post on my blog about this, but I also found a cool calculator to figure out how certain you want to be. Typically 85-95% certain is usually what you want to aim for. We can use this calculator on impressions on an ad with the ctr, impressions on a presell with the ctr there and also impressions on an offer with the ctr there. It'll calculate out your probability to find with what certainty you can make the decision. Pretty cool stuff.

Split Test Accelerator: Faster Ad Testing Now!
Click to expand...

Great tool man, thanks. +rep
Been sending about 100-200 clicks to offers and deciding whether or not to switch them, just from that small sample. This will probably help me out a ton.
 
Nice was just about to do some research about stuff like that for my AD performing tool thanks a lot :)
 
Does anyone have any good reads about the math that is involved behind this? I would be interested in reading up on exactly how the confidence number is determined. This tool looks pretty cool as well, Answers Research.
 
adigitalorange said:
Does anyone have any good reads about the math that is involved behind this? I would be interested in reading up on exactly how the confidence number is determined. This tool looks pretty cool as well, Answers Research.
Click to expand...

Yeah was wondering about that 2, just did a quick wikipedia digging and found plenty of models for stuff like this.

Not sure which way the tool works though, will pay a bit more attention to those math stuff the next days. Cuz there are ways for multiple data also which is even more interesting to me.

Running some kind of algo against all type of data we have like LP + ADs + AD Title + AD Text etc etc makes it more complex but i think would result in a better statistic and therefore makes it easyer to pick the right combination
 
juntao65 said:
:::SplitTester.com:::

For making conclusions off of ad CTR. Just thought I'd throw it in here.

Thanks for the link Smax.
Click to expand...

problem with this is that higher CTR doesn't always mean better ad. actually on a system where you're paying per click a high ctr ad can eat your lunch if it's not converting.

A lot of times for me a lower ctr ad will make as much money as a higher ctr ad with less clicks (and less cost).

I'd like to see a tool that did the math based on revenue earned vs clicks or impressions, cuz at the end of the day dollars are all that matters when determining what works best.
 
AffAholic said:
problem with this is that higher CTR doesn't always mean better ad. actually on a system where you're paying per click a high ctr ad can eat your lunch if it's not converting.

A lot of times for me a lower ctr ad will make as much money as a higher ctr ad with less clicks (and less cost).

I'd like to see a tool that did the math based on revenue earned vs clicks or impressions, cuz at the end of the day dollars are all that matters when determining what works best.
Click to expand...

Very true. The best option I've found is to just create two ad copies in prosper (one labeled A and one B), and just see which one earns more dollars given a sufficient amount of clicks/impressions.

But when operating the gurgle, higher CTR usually equates to a quality score boost which decreases your click costs across the board. Deciding on the best option for checking statistical significance is going to be dependent on how you market. A shit load of targetted keywords (ie buy acai berry online) or super high volume broad keywords (acai berry). The splittester usually gets the job done for the former.
 
You must log in or register to reply here.
Top Bottom
Back