Blog

The statistics of A/B testing pt 2.

2010-09-21T21:23:00Z

Calculating Statistical Significance

Before I describe the hypothesis test used in Part I, I want to lay a foundation.

Let’s begin with the phrase hypothesis test, itself. A hypothesis test is a statistical procedure designed to test a claim. There are two parts to any claim being examined – the null hypothesis which is what is currently known and the alternative hypothesis which is what you are testing.

When I was learning about statistical testing, the term null hypothesis used to confuse me, but the more I began to wrap my head around it the more I realized that the null is simply the status quo. Using the example in part I, the two claims being tested are Page A and Page B. The null hypothesis for this is simply that both A and B have the same efficacy, that is they result in statistically the same number of conversions. Or put another way, there is no statistical difference between them. The alternate hypothesis on the other hand is the idea that we are concerned with. Again going back to the example in part I -we are looking at the data to determine if Page B converts better than page A, so the alternative hypothesis is Page B is > Page A at conversion (or comparably page A is < Page B – it performs worse.)

Now that we have established the null hypothesis- Page A is Equal to Page B, and the alternative hypothesis- Page A< Page B, we can begin our test.

We need to the following items to arrive at our result (*Warning a little bit of math below*):

1) A Z score to percentile conversion table. (The chart here is for IQ tests, but it has the data we need. Ignore all columns except Z score and percentile). What is a Z score you ask? The Z score, also called the standard score, is the relative position of a single value on the bell curve of all values. Anyone who has ever been in a college class is familiar with the idea of a bell curve, or normal distribution. In statisical studies that are conducted correctly, the data also tends to follow a bell shape:

The Z score (or standard score) is the number of standard deviations away from the center of the bell curve (the mean), that a particular data point falls and can be correlated to a percentile. Standard scores are great because you don’t need to know the specifics of the data once you have calculated them. 2 standard deviations above the mean or the 97^th percentile means the same thing to everyone.

2) The number of people participating in each segment of the the A/B test (n1 and n2) for Page A and Page B respectively.

n1 = 31500, n2= 33500

3) The sample proportion of each sample. In our case this is conversion rate for page A (p1) and Page B (p2)

4) The overall sample proportion which is the total number of individuals from each sample who have converted. In the A/B test from part one this can be calculated by dividing the total number of conversions for both tests by the total number of people who saw both pages:

5) The following formula for calculating the test statistic for the two population proportions which is used to calculate the Z-score:

From substituting the correct values into the test statistic equation we get

In this particular case, where the alternate hypothesis is Page A < Page B the test statistic is the Z-score. And it corresponds to a percentile of 11.51% But what does that mean?

If we subtract 100 from the percentile we get 88.5%. This is probability that the null hypothesis is false- or that Page A is not equal to Page B. While it is high it does not quite meet the threshold of statistical significance (95% certainty), therefore conclude that the two tests are not statistically different.

So there you have it – we plowed through an uncertain situation, and using statistics came up with a definitive business decision.

The statistics of A/B testing pt 1

2010-09-18T18:16:22Z

Hypothesis test to find your way.

Statistical methods can help us with more than just examining trends in a given population. Used correctly we can also determine the best of two options available. This is exceptionally important in A/B testing land.

For those of you not familiar with A/B testing, here is a brief primer. A/B testing is the process of modifying site elements to increase conversions. What constitutes a conversion is defined by the site owner and can be anything from purchasing a product to visiting a deeper page within the site. In a properly set up A/B test either page A (with no change) or page B (with the change being tested) is shown randomly to a site visitor and the conversion rate is measured for that specific instance. At the end of a test the data is aggregated to determine which page, A or B converted, and site is changed to reflect the winning modification.

Below is a great diagram of A/B testing process from the Unbounce Blog

Once the data is gathered the real fun of assessing the statistical significance of A vs. B begins.

You might be asking, can’t we just look at the conversion rate and move on? The answer is sometimes. Only when there is an overwhelming winner is the decision easy. In most tests the data is less definitive. Let’s take a look at an A/B test we recently conducted at bookswim

In our experiment, we generated the following results.

Test Case	Views	Conversion Rate
Page A	31500	1.01%
Page B	33500	1.11%

Strictly based on the conversion numbers it looks like Page B converted 10% better than Page A, but when we ran the numbers through a standard hypothesis test (also known as a Z test – more on that later) we found that the two scenarios were in fact equal.

So the moral of the story is, use hypothesis testing on all your A/B data otherwise you may be setting yourself up for a surprise in your conversion optimization efforts.

In my next post I will describe how we came up with the results.

Is your data lying to you?

2010-08-29T21:17:18Z

As the book industry continues to change, we are inundated with statistics about user behavior:

49% of e-book readers are bought as gifts [Bowker]
28% of US adults are avid (5+ hours/week) readers [Verso] - 64MM avid readers
The heart of the U.S. romance novel readership is women aged 31-49 who are currently in a romantic relationship. [Romance Writers of America]

These statistical nuggets are great because in isolation they give us a glimpse into why people do what they do, and how we can adjust our business to match market needs. But how often do we blindly accept data because it comes with pretty graphs and sound bites that seem to make sense? Probably more often than we'd like to admit.

The best way to ensure that we are not led astray, is to look at what biases have been introduced into a study before using its data to make a decision. Bias is systematic favoritism in the data collection process which causes misleading results. Two types of bias are hazards in studies: selection bias and measurement bias.

Selection Bias can occur when the group that is surveyed does not accurately reflect the target of the study, or is simply too small to matter. For example, if a study claims to describe the behavior of all readers in the U.S. but only surveys 30 stay-at-home moms in Indiana, it is hardly representative of every reader in the country.
Measurement Bias occurs when the questions asked favor a specific outcome. A survey question like "Do you agree that e-books are replacing print books as the preferred medium?" will deliver very different results than one that asks readers to choose their preferred medium from among e-books, purchased p-books or books checked out from the library.

As you read a study, ask yourself the following questions to determine if the authors tried to mitigate bias. Remember: the target population is the group that you want to generalize about, and the sample is the group that you actually survey in order to make those generalizations.

1. Is the target population (sometimes called the sampling frame) well-defined? If it isn't, the study may contain people outside the target, or it may exclude people who are relevant. In researching e-book reader purchase behavior, a well-defined population could be American consumers who purchased an e-book reader either online or in a physical store over the last 2 years. But if a study only looked at online shoppers at Christmas, the results could be skewed towards gift givers, and they could not be generalized to consumers who bought e-book readers in stores.

2. Is the sample randomly selected from the target population? In a truly random sample, every member of the target population has the same chance of being included in the study. When asking this question be wary of surveys that are conducted exclusively on the web, but draw generalizations about all people. These types of studies have participants that are not randomly selected, as they only capture a slice of the traffic to a given domain, and at best can only ever speak to the habits of the users of the particular site conducting research.

3. Does the sample represent the target population? Here it is important to look at all of the characteristics of the target population to see if they are mirrored in the sample. If you are looking to figure out the book purchase habits of Americans, make sure the sample has the same diversity of ethnicity, geographic distribution and age as is reported in the latest U.S. census.

4. Is the sample large enough? The larger the sample the more accurate the results. A quick way to estimate if a sample is large enough to produce a reasonably small margin of error is to divide 1 by the square root of the sample size (Margin of Error=√Sample Size). So a 1,500 person survey would produce a margin of error of 2.58%. It is also important that the sample size in this calculation be the number of people who responded to survey, not the number of survey requests that were sent out.

5. What is the response rate for the survey? The response rate is defined as the number of people in a target population who actually responded to a given survey. If the response rate is too low, a study may only reflect people who have a strong opinion about the topic, making the results biased toward their opinions and not the larger and less vociferous target population. A "good" response rate is dependent upon the margin of error that a study is looking to achieve (or that it claims), and the size of the target population being studied. There are two factors to consider here. The first one is a no-brainer: The higher the response rate, the more accurate the study. The second is a little more subtle. The larger the target poplulation being examined, the lower the response rate required for the same level of accuracy. The linked figure helps explain the correlation graphically. (According to the chart, for a study that is looking to achieve a margin of error of +/- 5%, and is studying a population of 2000 people, the response rate needs to approach 20% to achieve the desired result.) At the end of the day, know the response rate and make sure it closely matches the stated margin of error that study purports to achieve.

6. Do the questions appear to be leading the respondents into a particular answer? If they do, run the other way! This means that the researchers' agenda is adding a measurement bias and the results aren't worth the paper they are printed on. Also be wary of any study that doesn't share its sampling method, sample characteristics and survey questions.

In the end, the goal of a survey is to accurately describe a larger population. This can only be done if great care is taken to 1) ensure that the results wouldn't change much if another sample was taken under the same conditions and to 2) reduce biases that can be introduced into the system.

Is your company investible?

2009-05-05T21:34:00Z

Fred Wilson from Union Square Ventures makes several great points in his posts on the Venture Capital math problem. (As you may remember, we were fortunate enough to have 2 members of USV speak during entrepreneur week: Albert Wenger and Andrew Parker)

But instead of rehashing how the venture capital model needs fixing, I thought it would be interesting to use the data to shed light into how an venture funds are set up and the financial metrics that drive equity investments.

First lets talk about how a venture fund works:

Venture funds make investments into illiquid and risky investments with expectation of high returns over the long term. Most funds are organized as limited partnerships in which the venture capital firm acts as the general partner (GP) and investment advisor and the investors serve as limited partners or (LPs).

The General Partner

The general partner is the person or entity that has control over management (running day to day operations) and investment decision. Usually the GP is organized as an entity such as an LLC, in order to shield the individuals who are making investment decision from personal liability associated with acting as GPs.

General partners can compensated by the fund in several ways. The most common is though a management fee which can be a percentage of the value of the fund at the end of a relevant period or straight percentage on the paid in capital--usually 2%. Moreover, the GP is entitled to a percentage of the overall profits from a fund at some point in the future, such as when a liquidity event occurs or when the fund closes. This is called carried interest and is usually 20%. "2 and 20" is how this is commonly referred, where 2% represents the management fee and 20% represents the carried interest.

Often the 20% carried interest has a limitation: It is only allocated if the funds value exceeds a certain amount. This prevents the GP from receiving payment if the fund suffered a loss. VCs who are members of the LLC which serves as the GP are therefore doubly careful about the investment decision they make. Not only do they want to wisely invest the money of their investors, but want to make investments that maximize profits of their fund, so they can participate in the upside at the funds close.

The Limited partners

Limited partners are usually institutional investors or HNW individuals. Fund investors can be employee benefit plans, insurance companies, banks, pension plans, university endowments, family offices, trusts among others. Limited partners cannot participate in the management of the fund without exposing themselves to personal liability and generally are prohibited from transferring their interests in the fund without consent of the general partner.

Lifespan and Capital calls

Venture funds usually are organized so that they have a life span between seven and ten years. Generally investors are not required to invest 100 percent of their capital in the initial closing of the fundraising round. Instead capital will be dispersed to the fund by investors on a pre-determined basis or when investments are made by the fund. The latter is known as a 'capital call' Many partnership agreements have penalties if LPs fail to meet their calls on a timely basis which can range from loss of rights to participate or exclusion from the funds profits going forward.

Economics

Because LPs typically make investments into high risk illiquid securities, they are looking for significant returns on their investments to compensate them for the increased risk. Fred Wilson does the math in his blog:

At a bare minimum

an investment needs to generate 2.5x net of fees and carry to the investors to deliver a decent return. Fees and carry bump that number to 3x gross returns.

So a 5 million dollar investment needs to generate 15MM in returns.

Looking at how much of a company a single VC owns at exit is next.

The number bandied about by most VCs is 20%. That means that each VC investor owns, on average 20% of each portfolio company. We'll use that number but to be honest I think it's lower, like 15% which makes the math even tougher.

Using the 20% number, a 5MM investment must generate 75MM (or 25X) at a minimum in order to make it worthwhile for the venture capital firm to satisfy the risk appetite of its limited partners.

As you can see VCs have to take into account the size of the potential market, and the probability of generating significant returns on their investment, not because they are greedy, but because of the metrics of success they need to meet in order run their own VIABLE businesses.

Do you agree or disagree? We'd love to hear your thoughts in the comments below.

Equity Compensation Explained

2009-04-07T21:36:00Z

Start-up technology companies often have difficulty recruiting and retaining talented employees. These companies need to attract high quality employees to build their businesses; however, they often lack the financial resources to offer their employees competitive salaries. One way to help level the compensation playing field between start-ups and established companies is with equity compensation.

Equity compensation is non-cash compensation that represents an ownership interest in the company. The two most common forms of equity compensation are stock options and restricted stock. Due to the variety of legal, accounting, and tax issues that are involved with equity compensation, proper planning is critical. Therefore, a company should seek legal and accounting advice before implementing an equity compensation plan.

A stock option is a right to purchase shares of a company’s stock at a predetermined price, which is referred to as the exercise price. The right to exercise the option and purchase shares of a company’s stock generally accrues, or "vests," over a period of time. The vesting of options over time creates an incentive for the employee to remain with the company to build its value. Option holders are not stockholders and thus are not entitled to vote their option shares or otherwise exercise any other rights of stockholders. All or a portion of the vesting of options often accelerates upon the sale of the company, unless the buyer assumes the options under its plan. Conversely, if an employee leaves the company, the vesting of stock options ceases, and the employee usually has a limited period of time to exercise the options that were vested on the employment termination date.

There are two types of stock options: incentive stock options ("ISOs") and non-qualified stock options ("NQSOs"). In the case of ISOs, and generally in the case of NQSOs, there is no tax to the option holder when the option is granted or when the option vests. The crucial distinction between ISOs and NQSOs is when the option is exercised. Generally, there is no tax to the option holder when ISOs are exercised. (However, the option holder may be subject to alternative minimum tax when ISOs are exercised.) When NQSOs are exercised, the option holder is subject to ordinary income tax on the difference between the exercise price and the fair market value of the stock on the date of exercise. The option holder is subject to capital gains tax on the sale of the stock that was purchased upon the exercise of ISOs and NQSOs. This tax is on the difference between the sales price and, in the case of ISOs the exercise price, and in the case of NQSOs the fair market value of the stock on the date of exercise.

Under the Internal Revenue Code, a stock option must satisfy several criteria to qualify as an ISO. Principal among these is that ISOs may be granted to employees only, and the exercise price of ISOs must be equal to or greater than the stock’s fair market value on the grant date. NQSOs may be granted to non-employees, such as outside directors or advisors. The exercise price of NQSOs may also be less than the stock’s fair market value on the grant date.

While stock options are appropriate for most employees, a company’s founders generally demand the voting and other rights of stockholders. However, the founders may desire to ensure that the stock owned by all of the founders is at risk and thus subject to forfeiture if a founder leaves the company. This motivates all the founders to work hard to build the company’s value, and if the stock of a departing founder is forfeited, it can be used to hire a replacement thus minimizing the dilution to the remaining founders. In addition, investors may wish to ensure that the founders are motivated to remain with the company to build its value. Restricted stock fulfills these objectives and is often used as a form of equity compensation for founders.

Unlike the grant of stock options, the grant of restricted stock is the issuance of shares of the company’s stock. A holder of restricted stock can vote the shares at stockholder meetings and has all of the other rights of a stockholder under applicable corporate law.

Restricted stock is generally subject to a repurchase right that allows the company to repurchase a portion of the founder’s stock if his or her employment is terminated by the company for cause, or if the founder voluntarily resigns within a certain period of time. This repurchase right lapses over time, freeing the stock of the restrictions in much the same way that stock options are subject to a vesting schedule.

The founder is subject to ordinary income tax as the restrictions lapse in an amount equal to difference between the purchase price of the shares and the fair market value of the stock at the time the repurchase restrictions lapse. However, the founder can file a "Section 83(b) Election" with the IRS within 30 days of the grant of the restricted stock to accelerate this tax to the grant date. If this election is made the founder is subject to ordinary income tax upon the grant of the restricted stock equal to the fair market value of the stock on the grant date. In either case, the founder is subject to capital gains tax when the stock is sold.

The two most common forms of equity compensation, stock options and restricted stock, serve similar, yet different, purposes in structuring a company’s compensation plan. Proper use of equity compensation is important in building a start-up company as it helps ensure the hiring, motivation, and retention of quality employees.

Why do companies lead their customers on?

2008-07-31T08:37:00Z

It has been a while since we last posted, and I almost was going to prolong our silent streak this morning until I saw this article in CMS Watch which spoke about how Day CQ leading their customers on re: a new version release. Now I have no idea what Day CQ is, (personally I am a fan of squarespace.com --which powers this site) but that is besides the point. I see a similar trend of over promising and under delivering across the web. This only leads to one thing, disappointing and frustrated users.

Here is our advice to companies that want to build buzz by making an "early" announcement about an undeliverable event:

DON'T
DON'T
Refer to 1 and 2.

I understand the intention. Tell people ahead of time so they can chatter about it, and then release to a primed audience. That only works when the priming is close to the event, itself. When to much time elapses, the user you hoped to excite end up talking about your inability to deliver. When the product launch actually does come, the negative energy serves to dampen or squash the buzz around your long anticipated project, regardless of how good it is. Worse yet, if the product has flaws they are amplified 10 fold by the disgruntled chatter.

Here is our suggestion.

Build your product
Test the crap out of it
A few weeks before launch make a cryptic announcement about new things comming.
Soft launch your product (without fanfare) to see if the market is really excited about it.
Let attention organically build
A week after the launch, do a hard launch with a full PR push

If you product is awesome you'll be surprise how much more effective this strategy is.

Happy 4th!!

2008-07-04T13:11:52Z

Happy 4th of July!!! Don't set too much stuff on fire.

10,500,000 Firecrackers set off at once!:

100,000 bottle rockets set off at once:

I don't care that you hired a new developer

2008-07-01T13:41:47Z

We see it all the time. Companies touting the most useless or incomprehensible information on their blogs:

We just hired a new developer today. Yay us!

Translation: We have nothing to write about.

From the Dell Channel Blog

Microsoft is scheduled to transition XP to EOL status on June 30^th, 2008. The last day to order a system pre-installed with XP from Dell will be today, June 26, 2008, in order to allow for building and assembly of the system and stay in accordance with Microsoft’s licensing policies.

Translation: We were in a hurry to post intelligible information, so we just cut and pasted a paragraph from Microsoft's technical sales documentation.

In fact a recent study published by Forester and written up in the Wall Street Journal suggests that most corporate blogs are abysmal failures, generating no interaction with the customer base they are aimed at engaging:

Forrester found that most B2B blogs are “dull, drab, and don’t stimulate discussion."
Seventy percent stuck to business or technical topics,
74% rarely get comments
56% simply regurgitated press releases or other already-public new

Our advice, if you want to engage your customers, or potential customers with a blog, give them something of value. Tell them about new developments in your industry, or how product decisions were made, or just be entertaining. Heck if you are afraid of straying to far from center, profile your customers with a case study that explains what folks should do in a similar situation.

Why? The whole purpose of a blog is establish a relationship with the community around you. You want people to walk away with the idea that:

you care about them
you are an expert in your field
you are not trying to cram product down their throat without saying hello first
you are willing to share (which goes back to caring about the people around you)
your company is made of real people, not mindless automatons, and has a personality of its own.

I mean seriously, would you ever start a conversation with someone at a bar by saying "I bought toilet paper this morning."?

If you want to check out an excellent corporate blog, take a peak at compete.com. Because they provide me with substance, I check them out daily.

New Copyright Law. Is your IP protected?

2008-06-26T12:24:52Z

A new piece of copyright legislation being debated by congress, The Shawn Bentley Orphan Act, may change the way copyright infringement awards are made by the courts.

This bill would limit the amount of damages a copyright holder could collect from an infringer if the infringer performed a diligent search for the copyright holder before using their work.

While aiming to open up unclaimed intellectual property to free use, the law creates a loophole for those who don't maintain adequate ownership records on their work.

Personally, we feel that this legislation would be effective at stemming the tide of frivolous lawsuits because

it places the burden of actively monitoring registrations on copyright owners
it keeps promotes open use of abandoned works, without the fear of legal recourse if those abandoned works gain public traction after they been re-purposed or re-introduced.

To build an Audience Create Video Often

2008-06-21T12:17:07Z

A study just published by Tubemogul indicates that online video has a very short half life. Most views (75%) occur in the first 44 days.

While there is still the potential for a longtail, 25% of the views still remain after the first month, the drop of appears to be exponential.

What does this mean for your viral video campaign (Provided that you have created valuable and compelling content)?

Create a library. Release 4 or 5 videos at the same time when you are starting your campaign. Most people will see your content in the first 30-45 days. If they enjoy what they see they will look at your other content before deciding whether or not to subscribe. When you only have one video to show, even if it is brilliant, the chances of building a regular audience are slim.
Build on your video momentum. Release a new video at least once a month, when most people are likely to find your content. This way you are capitalizing on the upswing of interest, building a larger audience with each release.
Refer back to older episodes if they are relevant to increase viewership of those earlier pieces. You won't see a huge peak in viewership of non-evergreen content, but every view increases the possibility of developing a new subscriber.
If you aren't able to create videos on a regular basis, or you miss a release date, put up a short notice informing your audience that new content is coming (and then apologize in the next release for the delay). People appreciate the honesty, and this lets them know that you aren't a flake.

If you have anything to add to the list, feel free to leave a comment.