from Pulling the Plug on JUnit Max  by Kent Beck

This note is to inform JUnit Max subscribers that I will not be actively developing Max further (extraordinary events notwithstanding). Customers wishing to cancel their subscriptions are welcome to do so, as long as they stop using Max. Anyone wanting a refund will be granted one, no questions asked. Subscribers are welcome to continue using Max for the duration of their subscription, but no further subscriptions or renewals will be accepted.

Reasons

Max addresses a genuine need: dedicated TDDers really do need better, faster feedback. Max’s approach of risk-first test execution and an embedded UI modeled on syntax error reporting successfully deliver that feedback. There is an audience who can effectively use what Max delivers.

The business problem with Max is that there just aren’t enough of us who rely on tests minute-by-minute. Even allowing for my marketing and sales mistakes, the data suggests that there are at most a few thousand Java programmers actively practicing TDD. That’s not a large enough market to sustain a business, especially as the market turned out to be price sensitive. Even if every Max subscriber successfully convinced 100 of their friends to sign up, there wouldn’t have been enough revenue.

And so, much as I love the concept of better, faster test feedback and extending the benefits of developer testing, I will henceforth be investing my time in other products and in my consulting clients. I am not closing the door to future business–if someone comes along with a 10,000 seat order or offers to buy Max in order to donate it to the Eclipse foundation, I’ll be glad to discuss the possibilities.

Lessons

I learned a lot developing, marketing, and selling Max. One thing I learned is that I love being involved in the entire business. There were days in the middle where I felt fully engaged, fully alive. I’d like more of that in my life.

Here are some more specific lessons I learned:

• Release early and often. The phrase, “If you aren’t embarrassed, you waited too long to release,” was very helpful to me. My first release was the tiniest sliver of what I imagined, but it gave me market validation (even if Erik and Saff were the only two people to buy it) and, more importantly, energy. The rapid release cycle let me experiment with features that I hoped would drive sales (none did) and fix blocking defects.
• Ship it and fix it. The product needs to provide value from the first, but it doesn’t need to provide everyone with value all the time. Early sales answer questions about the market size. Early users accept some rough edges if they get to be first and you fix the problems. Not reflexively fixing every defect was a hard transition for me, but, often, answering the next business question was tmy highest priority. I would recommend installing real-time remote error reporting for anyone bringing client software to market. It was nerve-wracking at first to see all the errors, but in the end the feedback was invaluable.
• End-to-end. I wish I had implemented a tiny sliver of the server functionality earlier. My ambition for Max was always larger than what I delivered, but implementing something of the service offering, however small, would have communicated that vision more clearly. Given the market size problem, it wouldn’t have made a difference in the end, though.
• Cut your losses. The exponential revenue growth turned down in May (negative second derivative). June was worse. It was six weeks from the first signals that the market was saturated to today.
• Lean startup. The most helpful advice I got came from Eric Ries and his Lean Startup ideas. Balancing time between the problem team and the solution team is a challenge for me, as I like making things.

Marketing JUnit Max (http://www.threeriversinstitute.org/blog/?p=299) by Kent Beck

I did market Max, just not in the same way that is appropriate for more mature products. Just as there are different engineering principles and practices for products at different stages, there are different marketing principles and practices. I’ll try to describe the differences & why I see my marketing of Max as successful.

Getting off the ground

I’m afraid this post is going to be analogy rich. One analogy is the flight of the startup, that startups go through distinct phases:

• Taxi–identify a need
• Takeoff–generate capital
• Climb–achieve scale
• Cruise–extract profit

Max got safely through the taxi phase: programmers relying on tests genuinely need better and faster feedback. It was the takeoff phase that I wasn’t able to get Max through.

In the takeoff phase you need to identify whether people will actually pay to have their (validated) need met. You need to find out how large the total market is. You need to begin selling enough to that market to generate capital to carry the product through to the next phase, either by bootstrapping or by attracting venture capital.

Prospecting

Here is the next analogy–marketing as mining. If marketing during climb and cruise is like strip-mining, extracting every last ounce of value, marketing during takeoff is more like prospecting. You drill a few holes and see what comes up (hence the opening picture). Is there revenue? How large is the total potential revenue? These are questions that can often be efficiently answered with certainty by actually marketing and selling.

If your first few holes don’t find any sales you haven’t proven there aren’t any sales, just that there aren’t any where you’ve drilled. My marketing campaign for Max was aimed at finding the size of the market. I used this blog, Twitter, and a mailing list to invite potential users into the “story of JUnit Max”. I wanted to see how far the product would spread. It isn’t until you get past the end of effective marketing that you actually know where the edges are.

Bonfires

One final analogy–product as bonfire. A product that solves the right problem for the right people at the right time is like touching a match to one twig in a really dry pile of brush. After that first touch you really don’t have much to do until it’s time to pile on more brush. Each burning piece sets off several around it.

At first Max looked like this. Friends told friends. Single sales turned into sales to whole teams. At one point a logarithmic graph of sales showed Max going cash-flow positive by August (of course, the same graph showed Max exceeding the entirely global economy in December 2010). Then the second derivative of sales went negative. Then the first derivative. No features I added or marketing activity I took had any effect.

That’s when I concluded that I had found the edges of the market. It wasn’t that I had “consumed” all the fuel, made every possible sale. Marketing and sales don’t work like that. They are necessarily stochastic. Perhaps I could have sold five times as many subscriptions by dint of hard work. By porting to other platforms like IntelliJ I could have doubled sales yet again. None of that added up to the 100x or 1000x I needed to sustain a business, though.

Marketing in Context

One of the emerging concepts in lean startups is separating the product launch from the marketing launch. Here are takes on this concept by Sean Ellis and Eric Ries. You don’t know how your product is going to be most valuable. You don’t know when your product is going to be most valuable. You don’t know to whom your product is going to be most valuable. A single launch compounds all your risks. As Cuil and Wolfram Alpha have showed most recently, even pretty good technology can’t survive a big-bang launch that doesn’t go just right. Risk management suggests staging the roll out to find answers to “how, when, and who” until the market practically demands a big launch.

Mass marketing, press releases to the mass media for example, is aimed at filling the sales pipeline. This is fine if you have a profitable product. However, Max was never profitable. Each user cost me more in support costs than I received. Max was valuable to an experienced user, but deploying it to a large audience of less dedicated testing programmers would have been ruinously expensive to me and disappointing to them.

Reflection (or rationalization)

Looking back I’m satisfied with my marketing of Max. I quickly and cheaply figured out the size of the paying market, the effects of pricing on sales, and made a small dribble of money in the process. I found that out without having the pay all the costs of engineering a completely reliable, performant Eclipse application.

In future I will use Twitter again to tell the story of development, with its ups and downs. I will use my blog to give more detailed and generally applicable lessons along the way. I will use a mailing list for product-specific discussions. All of these activities promote the product at hand, and, should it not go well, enhance any future products.

When I get to a product that is really catching fire, that’s when Andrew Binstock’s advice will come in handy. When the cost per customer has been driven down, when the benefit per customer has been driven up, when the potential market is large enough to sustain a business, that’s when pouring on kerosene makes sense. Do it too soon, though, and you just have a bunch of half-burned wood and no fire.