Getting Around

Hello my name is Jeevan Padiyar. This is my personal and professional blog.  It's a place for me to think out loud and learn, talk about things that inspire me, and share my observations with the world. If you feel like my musings are misguided or just plain wrong please feel free to reach out and correct me. I would relesh the opportunity for discourse. Thanks for visiting.

Who is Jeevan?

 

Other Places you can find me on the web:

Photo Blog

Link Blog

Tuesday
Sep212010

The statistics of A/B testing pt 2.

Calculating Statistical Significance

Before I describe the hypothesis test used in Part I, I want to lay a foundation.

Let’s begin with the phrase hypothesis test, itself. A hypothesis test is a statistical procedure designed to test a claim. There are two parts to any claim being examined – the null hypothesis which is what is currently known and the alternative hypothesis which is what you are testing.

When I was learning about statistical testing, the term null hypothesis used to confuse me, but the more I began to wrap my head around it the more I realized that the null is simply the status quo. Using the example in part I, the two claims being tested are Page A and Page B. The null hypothesis for this is simply that both A and B have the same efficacy, that is they result in statistically the same number of conversions. Or put another way, there is no statistical difference between them. The alternate hypothesis on the other hand is the idea that we are concerned with. Again going back to the example in part I -we are looking at the data to determine if Page B converts better than page A, so the alternative hypothesis is Page B is > Page A at conversion (or comparably page A is < Page B – it performs worse.)

Now that we have established the null hypothesis- Page A is Equal to Page B, and the alternative hypothesis- Page A< Page B, we can begin our test.

We need to the following items to arrive at our result (*Warning a little bit of math below*):

1)      A Z score to percentile conversion table. (The chart here is for IQ tests, but it has the data we need. Ignore all columns except Z score and percentile). What is a Z score you ask? The Z score, also called the standard score, is the relative position of a single value on the bell curve of all values. Anyone who has ever been in a college class is familiar with the idea of a bell curve, or normal distribution. In statisical studies that are conducted correctly, the data also tends to follow a bell shape:

 

The Z score (or standard score) is the number of standard deviations away from the center of the bell curve (the mean), that a particular data point falls and can be correlated to a percentile. Standard scores are great because you don’t need to know the specifics of the data once you have calculated them. 2 standard deviations above the mean or the 97th percentile means the same thing to everyone.

2)      The number of people participating in each segment of the the A/B test (n1 and n2) for Page A and Page B respectively.

  1. n1 = 31500, n2= 33500

3)      The sample proportion of each sample. In our case this is conversion rate for page A (p1) and Page B (p2)

4)      The overall sample proportion  which is the total number of individuals from each sample who have converted. In the A/B test from part one this can be calculated by dividing the total number of conversions for both tests by the total number of people who saw both pages:

5)      The following formula for calculating the test statistic for the  two population proportions which is used to calculate the Z-score:

From substituting the correct values into the test statistic equation we get

In this particular case, where the alternate hypothesis is Page A < Page B the test statistic is the  Z-score. And it corresponds to a percentile of 11.51% But what does that mean?

If we subtract 100 from the percentile we get 88.5%. This is probability that the null hypothesis is false- or that Page A is not equal to Page B.  While it is high it does not quite meet the threshold of statistical significance (95% certainty), therefore conclude that the two tests are not statistically different.

So there you have it – we plowed through an uncertain situation, and using statistics came up with a definitive business decision.

Saturday
Sep182010

The statistics of A/B testing pt 1

Hypothesis test to find your way.

Statistical methods can help us with more than just examining trends in a given population. Used correctly we can also determine the best of two options available. This is exceptionally important in A/B testing land.

For those of you not familiar with A/B testing, here is a brief primer. A/B testing is the process of modifying site elements to increase conversions. What constitutes a conversion is defined by the site owner and can be anything from purchasing a product to visiting a deeper page within the site.  In a properly set up A/B test either page A (with no change) or page B (with the change being tested) is shown randomly to a site visitor and the conversion rate is measured for that specific instance. At the end of a test the data is aggregated to determine which page, A or B converted, and site is changed to reflect the winning modification.

Below is a great diagram of A/B testing process from the Unbounce Blog

Once the data is gathered the real fun of assessing the statistical significance of A vs. B begins.

You might be asking, can’t we just look at the conversion rate and move on? The answer is sometimes. Only when there is an overwhelming winner is the decision easy. In most tests the data is less definitive. Let’s take a look at an A/B test we recently conducted at bookswim 

In our experiment, we generated the following results.

Test Case

Views

Conversion Rate

Page A

31500

1.01%

Page B

33500

1.11%

 

Strictly based on the conversion numbers it looks like Page B converted 10% better than Page A, but when we ran the numbers through a standard hypothesis test (also known as a Z test – more on that later) we found that the two scenarios were in fact equal.

So the moral of the story is, use hypothesis testing on all your A/B data otherwise you may be setting yourself up for a surprise in your conversion optimization efforts.

In my next post I will describe how we came up with the results. 

Sunday
Aug292010

Is your data lying to you?

As the book industry continues to change, we are inundated with statistics about user behavior:

  • 49% of e-book readers are bought as gifts [Bowker]
  • 28% of US adults are avid (5+ hours/week) readers [Verso] - 64MM avid readers
  • The heart of the U.S. romance novel readership is women aged 31-49 who are currently in a romantic relationship. [Romance Writers of America]

These statistical nuggets are great because in isolation they give us a glimpse into why people do what they do, and how we can adjust our business to match market needs. But how often do we blindly accept data because it comes with pretty graphs and sound bites that seem to make sense? Probably more often than we'd like to admit.

The best way to ensure that we are not led astray, is to look at what biases have been introduced into a study before using its data to make a decision.   Bias is systematic favoritism in the data collection process which causes misleading results.  Two types of bias are hazards in studies: selection bias and measurement bias.

  • Selection Bias can occur when the group that is surveyed does not accurately reflect the target of the study, or is simply too small to matter. For example, if a study claims to describe the behavior of all readers in the U.S. but only surveys 30 stay-at-home moms in Indiana, it is hardly representative of every reader in the country.
  • Measurement Bias occurs when the questions asked favor a specific outcome. A survey question like "Do you agree that e-books are replacing print books as the preferred medium?" will deliver very different results than one that asks readers to choose their preferred medium from among e-books, purchased p-books or books checked out from the library.

As you read a study, ask yourself the following questions to determine if the authors tried to mitigate bias. Remember: the target population is the group that you want to generalize about, and the sample is the group that you actually survey in order to make those generalizations.

1.       Is the target population (sometimes called the sampling frame) well-defined? If it isn't, the study may contain people outside the target, or it may exclude people who are relevant. In researching e-book reader purchase behavior, a well-defined population could be American consumers who purchased an e-book reader either online or in a physical store over the last 2 years. But if a study only looked at online shoppers at Christmas, the results could be skewed towards gift givers, and they could not be generalized to consumers who bought e-book readers in stores.

2.       Is the sample randomly selected from the target population?  In a truly random sample, every member of the target population has the same chance of being included in the study. When asking this question be wary of surveys that are conducted exclusively on the web, but draw generalizations about all people. These types of studies have participants that are not randomly selected, as they only capture a slice of the traffic to a given domain, and at best can only ever speak to the habits of the users of the particular site conducting research.

3.       Does the sample represent the target population? Here it is important to look at all of the characteristics of the target population to see if they are mirrored in the sample. If you are looking to figure out the book purchase habits of Americans, make sure the sample has the same diversity of ethnicity, geographic distribution and age as is reported in the latest U.S. census. 

4.       Is the sample large enough? The larger the sample the more accurate the results. A quick way to estimate if a sample is large enough to produce a reasonably small margin of error is to divide 1 by the square root of the sample size (Margin of Error=√Sample Size). So a 1,500 person survey would produce a margin of error of 2.58%. It is also important that the sample size in this calculation be the number of people who responded to survey, not the number of survey requests that were sent out.

5.       What is the response rate for the survey? The response rate is defined as the number of people in a target population who actually responded to a given survey.  If the response rate is too low, a study may only reflect people who have a strong opinion about the topic, making the results biased toward their opinions and not the larger and less vociferous target population. A "good" response rate is dependent upon the margin of error that a study is looking to achieve (or that it claims), and the size of the target population being studied. There are two factors to consider here. The first one is a no-brainer: The higher the response rate, the more accurate the study. The second is a little more subtle. The larger the target poplulation being examined, the lower the response rate required for the same level of accuracy. The  linked figure helps explain the correlation graphically. (According to the chart, for a study that is looking to achieve a margin of error of +/- 5%, and is studying a population of 2000 people, the response rate needs to approach 20% to achieve the desired result.) At the end of the day, know the response rate and make sure it closely matches the stated margin of error that study purports to achieve.

6.       Do the questions appear to be leading the respondents into a particular answer? If they do, run the other way! This means that the researchers' agenda is adding a measurement bias and the results aren't worth the paper they are printed on. Also be wary of any study that doesn't share its sampling method, sample characteristics and survey questions.

In the end, the goal of a survey is to accurately describe a larger population. This can only be done if great care is taken to 1) ensure that the results wouldn't change much if another sample was taken under the same conditions and to 2) reduce biases that can be introduced into the system.



Tuesday
May052009

Is your company investible?

Fred Wilson from Union Square Ventures makes several great points in his posts on the Venture Capital math problem. (As you may remember, we were fortunate enough to have 2 members of USV speak during entrepreneur week: Albert Wenger and Andrew Parker)

But instead of rehashing how the venture capital model needs fixing, I thought it would be interesting to use the data to shed light into how an venture funds are set up and the financial metrics that drive equity investments.

First lets talk about how a venture fund works:

Venture funds make investments into illiquid and risky investments with expectation of high returns over the long term. Most funds are organized as limited partnerships in which the venture capital firm acts as the general partner (GP) and investment advisor and the investors serve as limited partners or (LPs).

The General Partner

The general partner is the person or entity that has control over management (running day to day operations) and investment decision. Usually the GP is organized as an entity such as an LLC, in order to shield the individuals who are making investment decision from personal liability associated with acting as GPs.

General partners can compensated by the fund in several ways. The most common is though a management fee which can be a percentage of the value of the fund at the end of a relevant period or straight percentage on the paid in capital--usually 2%. Moreover, the GP is entitled to a percentage of the overall profits from a fund at some point in the future, such as when a liquidity event occurs or when the fund closes. This is called carried interest and is usually 20%. "2 and 20" is how this is commonly referred, where 2% represents the management fee and 20% represents the carried interest.

Often the 20% carried interest has a limitation: It is only allocated if the funds value exceeds a certain amount. This prevents the GP from receiving payment if the fund suffered a loss. VCs who are members of the LLC which serves as the GP are therefore doubly careful about the investment decision they make. Not only do they want to wisely invest the money of their investors, but want to make investments that maximize profits of their fund, so they can participate in the upside at the funds close.

The Limited partners

Limited partners are usually institutional investors or HNW individuals. Fund investors can be employee benefit plans, insurance companies, banks, pension plans, university endowments, family offices, trusts among others. Limited partners cannot participate in the management of the fund without exposing themselves to personal liability and generally are prohibited from transferring their interests in the fund without consent of the general partner.

Lifespan and Capital calls

Venture funds usually are organized so that they have a life span between seven and ten years. Generally investors are not required to invest 100 percent of their capital in the initial closing of the fundraising round. Instead capital will be dispersed to the fund by investors on a pre-determined basis or when investments are made by the fund. The latter is known as a 'capital call' Many partnership agreements have penalties if LPs fail to meet their calls on a timely basis which can range from loss of rights to participate or exclusion from the funds profits going forward.

Economics

Because LPs typically make investments into high risk illiquid securities, they are looking for significant returns on their investments to compensate them for the increased risk. Fred Wilson does the math in his blog:

At a bare minimum

an investment needs to generate 2.5x net of fees and carry to the investors to deliver a decent return. Fees and carry bump that number to 3x gross returns.

So a 5 million dollar investment needs to generate 15MM in returns.

Looking at how much of a company a single VC owns at exit is next.

The number bandied about by most VCs is 20%. That means that each VC investor owns, on average 20% of each portfolio company. We'll use that number but to be honest I think it's lower, like 15% which makes the math even tougher.

Using the 20% number, a 5MM investment must generate 75MM (or 25X) at a minimum in order to make it worthwhile for the venture capital firm to satisfy the risk appetite of its limited partners.

As you can see VCs have to take into account the size of the potential market, and the probability of generating significant returns on their investment, not because they are greedy, but because of the metrics of success they need to meet in order run their own VIABLE businesses.

Do you agree or disagree? We'd love to hear your thoughts in the comments below.

 

Tuesday
Apr072009

Equity Compensation Explained

Start-up technology companies often have difficulty recruiting and retaining talented employees. These companies need to attract high quality employees to build their businesses; however, they often lack the financial resources to offer their employees competitive salaries. One way to help level the compensation playing field between start-ups and established companies is with equity compensation.

Equity compensation is non-cash compensation that represents an ownership interest in the company. The two most common forms of equity compensation are stock options and restricted stock. Due to the variety of legal, accounting, and tax issues that are involved with equity compensation, proper planning is critical. Therefore, a company should seek legal and accounting advice before implementing an equity compensation plan.

A stock option is a right to purchase shares of a company’s stock at a predetermined price, which is referred to as the exercise price. The right to exercise the option and purchase shares of a company’s stock generally accrues, or "vests," over a period of time. The vesting of options over time creates an incentive for the employee to remain with the company to build its value. Option holders are not stockholders and thus are not entitled to vote their option shares or otherwise exercise any other rights of stockholders. All or a portion of the vesting of options often accelerates upon the sale of the company, unless the buyer assumes the options under its plan. Conversely, if an employee leaves the company, the vesting of stock options ceases, and the employee usually has a limited period of time to exercise the options that were vested on the employment termination date.

There are two types of stock options: incentive stock options ("ISOs") and non-qualified stock options ("NQSOs"). In the case of ISOs, and generally in the case of NQSOs, there is no tax to the option holder when the option is granted or when the option vests. The crucial distinction between ISOs and NQSOs is when the option is exercised. Generally, there is no tax to the option holder when ISOs are exercised. (However, the option holder may be subject to alternative minimum tax when ISOs are exercised.) When NQSOs are exercised, the option holder is subject to ordinary income tax on the difference between the exercise price and the fair market value of the stock on the date of exercise. The option holder is subject to capital gains tax on the sale of the stock that was purchased upon the exercise of ISOs and NQSOs. This tax is on the difference between the sales price and, in the case of ISOs the exercise price, and in the case of NQSOs the fair market value of the stock on the date of exercise.

Under the Internal Revenue Code, a stock option must satisfy several criteria to qualify as an ISO. Principal among these is that ISOs may be granted to employees only, and the exercise price of ISOs must be equal to or greater than the stock’s fair market value on the grant date. NQSOs may be granted to non-employees, such as outside directors or advisors. The exercise price of NQSOs may also be less than the stock’s fair market value on the grant date.

While stock options are appropriate for most employees, a company’s founders generally demand the voting and other rights of stockholders. However, the founders may desire to ensure that the stock owned by all of the founders is at risk and thus subject to forfeiture if a founder leaves the company. This motivates all the founders to work hard to build the company’s value, and if the stock of a departing founder is forfeited, it can be used to hire a replacement thus minimizing the dilution to the remaining founders. In addition, investors may wish to ensure that the founders are motivated to remain with the company to build its value. Restricted stock fulfills these objectives and is often used as a form of equity compensation for founders.

Unlike the grant of stock options, the grant of restricted stock is the issuance of shares of the company’s stock. A holder of restricted stock can vote the shares at stockholder meetings and has all of the other rights of a stockholder under applicable corporate law.

Restricted stock is generally subject to a repurchase right that allows the company to repurchase a portion of the founder’s stock if his or her employment is terminated by the company for cause, or if the founder voluntarily resigns within a certain period of time. This repurchase right lapses over time, freeing the stock of the restrictions in much the same way that stock options are subject to a vesting schedule.

The founder is subject to ordinary income tax as the restrictions lapse in an amount equal to difference between the purchase price of the shares and the fair market value of the stock at the time the repurchase restrictions lapse. However, the founder can file a "Section 83(b) Election" with the IRS within 30 days of the grant of the restricted stock to accelerate this tax to the grant date. If this election is made the founder is subject to ordinary income tax upon the grant of the restricted stock equal to the fair market value of the stock on the grant date. In either case, the founder is subject to capital gains tax when the stock is sold.

The two most common forms of equity compensation, stock options and restricted stock, serve similar, yet different, purposes in structuring a company’s compensation plan. Proper use of equity compensation is important in building a start-up company as it helps ensure the hiring, motivation, and retention of quality employees.