Getting Around

Hello my name is Jeevan Padiyar. This is my personal and professional blog.  It's a place for me to think out loud and learn, talk about things that inspire me, and share my observations with the world. If you feel like my musings are misguided or just plain wrong please feel free to reach out and correct me. I would relesh the opportunity for discourse. Thanks for visiting.

Who is Jeevan?

 

Other Places you can find me on the web:

Photo Blog

Link Blog

Main | The statistics of A/B testing pt 1 »
Tuesday
Sep212010

The statistics of A/B testing pt 2.

Calculating Statistical Significance

Before I describe the hypothesis test used in Part I, I want to lay a foundation.

Let’s begin with the phrase hypothesis test, itself. A hypothesis test is a statistical procedure designed to test a claim. There are two parts to any claim being examined – the null hypothesis which is what is currently known and the alternative hypothesis which is what you are testing.

When I was learning about statistical testing, the term null hypothesis used to confuse me, but the more I began to wrap my head around it the more I realized that the null is simply the status quo. Using the example in part I, the two claims being tested are Page A and Page B. The null hypothesis for this is simply that both A and B have the same efficacy, that is they result in statistically the same number of conversions. Or put another way, there is no statistical difference between them. The alternate hypothesis on the other hand is the idea that we are concerned with. Again going back to the example in part I -we are looking at the data to determine if Page B converts better than page A, so the alternative hypothesis is Page B is > Page A at conversion (or comparably page A is < Page B – it performs worse.)

Now that we have established the null hypothesis- Page A is Equal to Page B, and the alternative hypothesis- Page A< Page B, we can begin our test.

We need to the following items to arrive at our result (*Warning a little bit of math below*):

1)      A Z score to percentile conversion table. (The chart here is for IQ tests, but it has the data we need. Ignore all columns except Z score and percentile). What is a Z score you ask? The Z score, also called the standard score, is the relative position of a single value on the bell curve of all values. Anyone who has ever been in a college class is familiar with the idea of a bell curve, or normal distribution. In statisical studies that are conducted correctly, the data also tends to follow a bell shape:

 

The Z score (or standard score) is the number of standard deviations away from the center of the bell curve (the mean), that a particular data point falls and can be correlated to a percentile. Standard scores are great because you don’t need to know the specifics of the data once you have calculated them. 2 standard deviations above the mean or the 97th percentile means the same thing to everyone.

2)      The number of people participating in each segment of the the A/B test (n1 and n2) for Page A and Page B respectively.

  1. n1 = 31500, n2= 33500

3)      The sample proportion of each sample. In our case this is conversion rate for page A (p1) and Page B (p2)

4)      The overall sample proportion  which is the total number of individuals from each sample who have converted. In the A/B test from part one this can be calculated by dividing the total number of conversions for both tests by the total number of people who saw both pages:

5)      The following formula for calculating the test statistic for the  two population proportions which is used to calculate the Z-score:

From substituting the correct values into the test statistic equation we get

In this particular case, where the alternate hypothesis is Page A < Page B the test statistic is the  Z-score. And it corresponds to a percentile of 11.51% But what does that mean?

If we subtract 100 from the percentile we get 88.5%. This is probability that the null hypothesis is false- or that Page A is not equal to Page B.  While it is high it does not quite meet the threshold of statistical significance (95% certainty), therefore conclude that the two tests are not statistically different.

So there you have it – we plowed through an uncertain situation, and using statistics came up with a definitive business decision.

PrintView Printer Friendly Version

Reader Comments (9)

Jeevan - this post is very helpful in understanding what we're really measuring. We've built an internal A/B testing framework and I now understand what it's numbers are telling us. One follow up question though.

How does one know if they've allowed for a big enough sample size?

In the example above you had samples of 31,500 and 33,500 respectively with an 88.5% confidence that the null hypothesis was false (short of the 95% needed to be relevant).

How do you know that the numbers wouldn't change with a bigger sample size (ie let each sample run up over 60,000)?

I ask because it's hard to know when to stop the test when the sample sizes are smaller.

Thanks.

January 13, 2011 | Unregistered CommenterClint Watson

viagra related overdose are growing in number. Which is why viagra is better suited as a prescription drug.

August 3, 2011 | Unregistered CommenterStefano

Interesting formula and ration about viagra dosage. You have good points on the dosage being a suspect for heart attacks.
buy viagra

August 5, 2011 | Unregistered CommenterStefano

I am apreciating it swiss made valjoux 7750 replica omega seamaster
very much.I have never read such a lovely article and I am mont blanc blancpain carrousel black flying tourbillon swiss replica watch
coming back tomorrow to continue reading.

September 26, 2011 | Unregistered Commenterswiss replica watches

I am so glad this internet thing works and your article really helped me. Might take you up on that home advice you

This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the

only one having all the leisure here! Keep up the good work.
cheap steelers jerseys on sale
ed reed jersey
Women's Patriots Jersey
Oakland Raiders Jerseys
antonio brown ersey authentic

84 pittsburgh steelers white reebok nfl jersey

Arizona Cardinals quarterback Kevin Kolb said he was Authentic Matt Schaub Jersey able topractice "a little bit" Authentic Israel Idonije Jersey on Wednesday after tests evaluating theconcussion he sustained Authentic Ryan Clark Jersey early in Authentic Johnathan Joseph Jersey Sunday's victory over SanFrancisco."The symptoms are down," he said. "It's just a matter ofmaking sure they are down Authentic Brian Urlacher Jersey long enough to Authentic Haloti Ngata Jersey where I can get out thereand then be in Authentic Joe Flacco Jersey full-speed action. That's the key right now. It's atouchy subject, and we want to make sure that we err Authentic Keiland Williams Jersey on the rightside."Unlike the situation with the Cleveland Browns and theirquarterback, Colt McCoy, the Cardinals benched Kolb after he took aknee to the head on Authentic Aaron Rodgers Jersey Arizona's Authentic Ray Lewis Jersey third offensive play against the49ers. Authentic Cedric Benson Jersey He immediately Authentic Ray Rice Jersey went to the locker room for evaluation and itwas determined he should not return to the game."I Authentic Tom Brady Jersey don't know how Authentic Mike Wallace Jersey they handled it in Cleveland," Arizona coachKen Whisenhunt said. Authentic Marques Colston Jersey "All I can speak Authentic Troy Polamalu Jersey about is the way our guysdid it and they did a great job with it. ... Authentic Jacoby Ford Jersey The number one thingis, if there is ever a doubt, you err to the side of caution Authentic Rob Gronkowski Jersey andthat's the way Authentic Tom Brady Jersey we are going Official Nick Collins Jersey to Authentic Israel Idonije Jersey proceed."Kolb said he plans to practice more as the week goes on in hopesof being able to play Sunday against the Browns.He said the football culture is better at dealing withconcussions than it used to be."I think that there are so many studies coming out now thatprove long-term Authentic Ziggy Hood Jersey effects and things like that," Kolb Authentic Chad Ochocinco Jersey said."Obviously, our health is number one, especially when it comes toyour brain. I want to be out there as much as Authentic Joe Flacco Jersey anybody, but it'sjust something you don't push."This is Kolb's second concussion in as many seasons. He wassidelined Authentic Maurkice Pouncey Jersey with one after he started last season's opener forPhiladelphia and Authentic Tony Scheffler Jersey wound Authentic Rey Maualuga Jersey up losing the starting job to hisreplacement, Michael Vick. Authentic Michael Oher Jersey That eventually led to the trade thatbrought Authentic Ndamukong Suh Jersey Kolb to Arizona.Under the NFL's revised Authentic Brian Urlacher Jersey rules, Kolb was required to see a doctornot affiliated Authentic Calvin Johnson Jersey with the Cardinals to have his conditionindependently evaluated. Authentic Jermaine Gresham Jersey He said he did so on Tuesday and that thevisit Authentic Charles Woodson Jersey "went good."Kolb said Authentic Darren McFadden Jersey some of Authentic Bernard Scott Jersey the symptoms of a concussion don't begin toshow up for him until the adrenaline of the football game Authentic Jahvid Best Jersey begins Authentic Rob Gronkowski Jersey tofade."When your adrenaline Authentic Dez Bryant Jersey is Authentic Pierre Thomas Jersey going, when Official Carson Palmer Jersey you are in the Authentic Clay Matthews Jersey game, itcovers up," he said. "It tends to cover up some of the Authentic Michael Oher Jersey symptoms.As Authentic DeMarcus Ware Jersey you Authentic Andre Smith Jersey start calming down, during Authentic A.J. Green Jersey or Authentic Aaron Hernandez Jersey after the game, then, from myexperience, a lot Authentic Andy Dalton Jersey of Authentic Antonio Brown Jersey things start to rush on Authentic Jarret Johnson Jersey you; Authentic Wes Welker Jersey the vision, thesensitivity to light and noise, Authentic Johnny Knox Jersey and all those things."Kolb was making Authentic Rashard Mendenhall Jersey just his second start after being Authentic Richard Seymour Jersey sidelined forsix games Authentic Jason Witten Jersey with a right turf toe and bruise to the side of that samefoot. The previous week, he had come Authentic Mark Ingram Jersey on strong in Authentic Torrey Smith Jersey the second halfto help Arizona beat Dallas Official Stanford Routt Jersey in overtime."It's beyond frustrating, just because we played a good secondhalf there against the Cowboys Authentic B.J. Raji Jersey and had a good Authentic Devin Hester Jersey week at practice,"he said. "We were Authentic Jay Cutler Jersey ready to Authentic Arian Foster Jersey go out Authentic Michael Huff Jersey there and Authentic Ray Lewis Jersey do what we ended updoing, which is great for our team. I wish I could have been a partof it, Authentic Nick Fairley Jersey but they did a great job again of picking right Authentic Robert Meachem Jersey up androlling along."John Skelton relieved Authentic Devin Hester Jersey Kolb last Sunday and threw three touchdownpasses as the Authentic Patrick Chung Jersey Cardinals rallied to beat the 49ers 21-19.Arizona has won three in Authentic Jerod Mayo Jersey a row and five of six. Authentic Matthew Stafford Jersey A victory Sundaywould even the Cardinals' season record at 7-7 with two games toplay.

December 14, 2011 | Unregistered CommenterMike Wallace Jersey Outlet

great article on hypothesis testing//liked it

September 7, 2012 | Unregistered Commentersalina

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>