When the Intern Writes the Billing System

Posted April 19, 2013

In the autumn of 2005 I was finishing an internship at Fog Creek Software. My colleagues and I on Project Aardvark had successfully developed and released Copilot. After the launch, we cleaned up the few remaining bugs and added a few small, requested features. Then people started leaving. First Ben, then Yaron, and finally Michael. By the end of August, I was the only intern left.

It would stay that way for the next three months. Unlike the other interns, I didn't have to return to school immediately. I had graduated from Rose-Hulman already and was able to defer my admission to Stanford for the first quarter of classes. This gave me another three months at Fog Creek to tie up any loose ends with Copilot and start adding the next big feature. My assignment, as the last remaining intern? A subscription billing for Copilot.

(Yes, they had an intern writing the new billing system. No, I'm not making this up.)

When we first launched Copilot, there was only one option for payments: a $5 pass that was good for 24 hours of use. This was great for occasional users, since the cost was more than worth the time saved by not having to do remote support blind. But for more frequent users, it was a hassle. They'd have to enter their credit card information every time they needed to help someone. This was especially bad for companies that used it for customer support, since it forced their support people to request access to the company card on a daily basis.

Our plan was to implement a subscription billing system, similar to cell phone plans, based on minutes used per month, except with transparent, benevolent pricing. It would have several plans with a different number of included minutes and a per-minute charge for any overages. To make sure users weren't locked into something too expensive, we also said the system needed to let users change plans at any time.

My first task was to research options for storing and charging credit cards. I read dozens of marketing pages, sent emails to companies asking for info, and made phone calls to sales people. But every option had the same basic flaw: they could only do rigid, set-price subscriptions. This wouldn't work for us, since users could easily be charged a different amount each month, depending on how much they utilized the service.

Instead, we decided that we would write the whole subscription system from scratch ourselves. We already had a similar system in place for one-off purchases, so we thought we might be able to reuse some of the old code. I started writing the recurring billing code while one of our co-founders started on the credit card vault, which we called Roach Motel. (Get it? Cards check in, but they don't check out...)

We did everything possible to isolate credit cards from the rest of our infrastructure, keeping it all on a separate server stored in our secure cage in the office, instead of the data center where data-center employees might have access to it. The code was written to accept credit cards, but never to send them out again, except in encrypted form, and only to the payment processor. Access to the box required a thumb drive, held in secure storage, and a password. (Note: As of a couple years ago, this was no longer the case, as we began using a proper credit card vault instead.)

Over the next couple of months, the system came together nicely, being designed and redesigned to be flexible, so more products could be added in the future, while still being simple and robust. When I left that December, the system was up and running, billing customers each month. Overall, the project seemed like a total success.

Upon returning to Fog Creek the following summer, I found that was not the case. While I was away, FogBugz had grown from an install-only product to its own hosted solution. As with Copilot, this new SaaS-y version of FogBugz required a subscription billing system. So one of the developers on the FogBugz team added a significant amount of new code to support it.

While part of this new code was to deal with cases I had not foreseen in the original version, much more duplicated existing concepts in a different and incompatible way. (This might sound like an admonishment of the developers adding that new code, but most of the blame lies with me for not communicating, through documentation, the intent of the features they had duplicated.)

As time went on, the code base continued to grow organically. Occasionally, a change in one side of the code would cause billing errors for the other side. Bugs were introduced that caused miscalculations in customers bills. Other bugs caused duplicate payments to be triggered without properly recording them.

When Kiln came along it needed to be added to the system as well. Because of its tight integration with FogBugz, even more code warts were added on to deal with the possibility that accounts might only have Kiln, only have FogBugz, or have both.

In all, thousands of developer hours were spent developing and maintaining something that had nothing to do with the software our customers were paying us for. Time that we could have spent on writing new features and fixing bugs for customers was instead spent digging into stringy old billing code. What's worse, the system always was and, unless some great rewrite happens, will always be a mess. (Little known fact: The odds of such a rewrite happening on such a critical system are 2276709 to 1 against.) The fact is, writing billing software is not Fog Creek's core competency.

I bring this story up because it seems to closely parallel the situation that lead to Linode's recent security incident, in which they lost both their password database as well as their entire credit card database. (Fortunately, Fog Creek never suffered a similar loss of credit card data while we were still storing it ourselves.) In these sorts of security breaches, passwords are relatively easy to fix by requiring a password reset. The credit cards, however, are a much bigger problem, since they are often used many different places, requiring worried customers to update their credit card with every vendor they work with.

(Fortunately, the credit card numbers were encrypted in the database. Unfortunately, they decided to store the public and private key for those credit card numbers on the same server. According to Linode, the private key is encrypted with a "complex passphrase" which is "not guessable, sufficiently long and complex, not based on dictionary words, and not stored anywhere but in our heads".)

So how did Linode lose their entire credit card database to a hacker? Putting aside the technical details (which involve a zero-day exploit of ColdFusion), it comes down to Linode, a great VPS provider, spending time developing sensitive, complex systems that were outside of both their core competency and their main business. They, like Fog Creek, are not a payment processing company. Literally speaking, the system brings in the money, it's not what their customers pay them for. If, instead, they'd simply paid for one of the many affordable credit card vaults available, whose job it is to securely store credit card numbers, this never would have happened.

Well that sucks, what else you got?

Posted October 16, 2012

The Conversation

It was getting towards the end of the breakfast hour on the second day of Business of Software, and people started getting up to refill their coffee before the first session, leaving only Jason Cohen and me to chat while we finished our meal.

"So what do you do?" he asked me. I explained that I used to work for Fog Creek, but now I was off on my own, trying to start a company that provides photo backup for serious photographers. He asked the questions I've come to expect from people in the tech industry. "How is it different than Dropbox/BackBlaze? Where are you storing the files?" And so on.

Then he asked the key question. "How's it going so far?"

"Well, to be honest, very slowly." I proceeded to describe how we'd gotten a strong showing of interest by being featured on betali.st, only to have to drastically restructure pricing in a way that significantly cut our margins. Then, how, even with the new pricing, conversion rates were dismally low. I explained who our competition was and how we were having trouble explaining to our users why we were better. I repeated an observation that a former coworker had made, that the photography market is inverted, amateurs have plenty of money but don't care (that much) if they lose their photos, while professionals, whose businesses could be devastated be losing a single shot, don't have a lot of extra money to spend. Lastly, I pointed out that an effective marketing campaign would almost necessarily rely on convincing people that something bad (i.e. losing all of their photos) will likely happen to them sometime in the future, a particularly unappealing proposition to me.

Jason listened patiently while I explained all of this, and then summed it all up.

"Well that sucks, what else you got?"

The Realization

It had slowly been occurring to me over the past week. Snaposit was probably done. My co-founder, Jeff and I had already put it on the back burner to focus on Jeff's photo deblurring software, Blurity. Further development was becoming harder to justify.

Over the course of Business of Software, I became more open about talking with people about the problems we were running into. The hope was that the more experienced entrepreneurs I was talking to would be able to provide some reassurance. But reassurance was not forthcoming. If anything, I was hearing more reasons why it would be a tough sell.

My talk with Jason gelled it all together in that single phrase. "Well that sucks, what else you got?"

I realized it at the time, but it took me another few days before I could admit it. Keeping Snaposit going just didn't make any sense. Sure, it wasn't hemorrhaging cash, but it did have a slow leak. More importantly, it was still demanding enough attention to be detracting from other projects. To keep it running in the hopes that it might miraculously take off at some point in the future was unrealistic. It was time to shut it down.

The Causes

Poor Conversion Rates

While we were thrilled with the response from betali.st, the subsequent conversions were abysmally low. With those numbers, we would have to find traffic sources ten times larger than betali.st, every week for two years before we would even reach the level of "lifestyle business". Of course, we could hope to improve the conversion rate or the incoming traffic, but the numbers were still pretty poor. More realistically, it would take us much longer to hit the sort of revenue levels we were looking for.

Misunderstanding Our Market

In hind sight, there was a disconnect between what people said they thought was a good idea and what they would actually pay for. Almost everyone we talked to said something similar: "That's a really good idea." We heard it enough times that we really thought we were onto something. But there were two problems with this feedback: First, we were talking to more software developers, friends, and family than actual photographers. Second, we were just getting feedback on the idea itself, not on whether people would pay for it or not. Of course people think an automatic backup service is a good idea. It's like flossing, one of those things that everyone knows they should do, but the majority of people still neglect actually doing.

We also didn't fully understand what photographers wanted and needed. One of Snaposit's big features was that it backed up your entire library faster than a general backup service. It did this by compressing the photos to full resolution, high quality JPEGs, which saved anywhere from 50%-90% of the size. The downside of this, of course, is that photographers who shoot in RAW would only have a JPEG copy backed up. We figured this was a reasonable compromise. In a catastrophic event, wouldn't a photographer rather have a JPEG copy than nothing at all? It turns out the answer was no. But we didn't ask that question, because we thought the answer was so obvious.

Lastly, we wrote the first version of the Snaposit desktop application for Windows. We figured, "We're photographers who use Windows, we'll be able to find beta testers for this version and we'll get to the Mac version once all the bugs are ironed out." Nope. We couldn't find a single beta tester who used Windows.

The Loss of a Founder

For various reasons, the amount of personal runway Jeff and I had differed greatly. At a point, it became very clear that Snaposit would not be pulling in revenue before we reached the end of Jeff's runway. He decided that, in order to remain financially secure, he would shift his focus to Blurity, which was much closer to having a reliable revenue stream. I fully supported his decision, but it meant that the majority of the Snaposit work fell to me after that point. While this seemed feasible at the time, in retrospect, it did cut the chances of Snaposit's success. Unfortunately, there wasn't really any other way around it, it was just the reality of the situation. To this day, I still fully support Jeff's decision.

The Lessons

Of course, the experience was not a total loss, by any means. We learned a lot from failing. Below is a quick list of what we learned about:

  • How to apply for Y Combinator and Tech Stars.
  • How to integrate Stripe's subscription plans.
  • Python on the desktop with WxPython.
  • Mac development and deployment.
  • How to organize and form an LLC.
  • A better look at the backup industry.
  • That Jeff and I work well together.
  • How to run an effective 99designs competition.
  • Twitter's Bootstrap framework.
  • Desktop deployment and installers.
  • How to ask better questions about our market.

All told, we actually did quite well. For this experience, we spent less than $1,000. Compared to an entrepreneurship class at any college, that's quite cheap!

The Next Step

Jeff is continuing to work on Blurity. The recent press he's gotten has kicked Blurity into overdrive and he's continuing to knock it out of the park.

As for me, there are a few small side projects that could use some attention. I'm also getting back into writing. A few larger ideas are percolating, but nothing's quite finished brewing yet.

Comments for this post can be found on Hacker News.

Gatekeeper: Using FUD to get Developers to Pony Up

Posted August 29, 2012

A new user comes to your site and downloads your installer. They extract the disk image and fire up the PKG file. They're greeted with this message.


"Hmm," they think to themselves, "maybe something went wrong with the download." So they download it again, extract the disk image and run the PKG file again. Same error. The file is damaged. Now running late to pick up their kids from school, your new users leaves, likely never to come back.

This is their first introduction to Gatekeeper, a new feature of OS X 10.8, a.k.a. Mountain Lion. Intended as an anti-malware feature, Gatekeeper prevents downloaded applications from running, depending on their source. Similar to the SmartScreen in Windows, Gatekeeper presents the user with a warning when they are trying to run unsigned applications.

However, there are two important differences between Gatekeeper and SmartScreen.

First, with SmartScreen, the user has the option to continue running the application, despite not being signed. However, the default option for Gatekeeper provides no option to continue.

Gatekeeper dialog

(Note, for applications, unlike PKG files, the warning dialog is somewhat more helpful, though it still does not allow the user to actually run anything.)

This causes problems for any developers who have not paid Apple $100 for the Mac Developer Program, as the users are not left with any obvious way of continuing past the warning. This includes many developers who have chosen not to release via the App Store. Jeff and I have run into this problem already with users of both Snaposit and Blurity.

The second, and possibly more troubling , is that Apple is the only source for developer certificates. With SmartScreen, there are a wide variety of certificate authorities and brokers to choose from, each of which offers different pricing options, so the developer has a chance to comparison shop, which yields prices on the order of a third of the cost of the Mac Developer Program. But for Mac developers, Apple has created an artificial barrier instead of using existing and well established signing practices in order to make themselves a monopoly for developer certificates.

Is there any way around it?

Yes, but you're going to have to educate your users. Each and every one of them.

There are two options:

The first, and easiest, is to instruct your users to Control-Click (or right-click) the application once they've downloaded and installed it. From the context menu, the'll need to choose "Open".

Context menu

When they start the application this way, they'll be presented with a slightly different warning dialog, which allows them to continue.

Gatekeeper dialog

The next time they launch the application, they shouldn't see any warnings.

The other option is to have them disable Gatekeeper entirely. To do this, have them go to System Preferences and choose Security & Privacy.

System preferences

Once there, they'll need to click the lock in order to make changes. Once they've unlocked the preferences, they'll need to choose "Anywhere" from the list of options under "Allow applications downloaded from:"

Security preferences

This will essentially disable Gatekeeper, reverting their computer's security to a state similar to OS X Lion. Obviously, this isn't ideal, but if it's something they might run into often, it might be the best choice.

Snaposit is Live!

Posted August 02, 2012

As I wrote about a few months ago, we applied to Y Combinator's summer class this year. We got an interview with Paul Graham himself, but ultimately we were not accepted into the program. But we didn't let that stop us.

So what are we going to do?

We're going to grab our bootstraps and pull hard.

I'm proud to announce that, after several months of hard work, our product and company, Snaposit has officially launched!

We flipped the switch yesterday, moving all of our beta users onto actual subscription plans. (Heavily discounted, of course, to thank them for their help.) As with all launch days, there were a few small bugs that cropped up, but overall the day was pretty quiet, which is always a good sign.

But we're not done yet. In fact, it might be safe to say that the past few months were the easy part. Now we have the task of making people aware of Snaposit and convincing them of how useful it is.

So with the celebratory drinks out of the way, it's time to get back to work. We've done a good job of pulling our own bootstraps so far, but there's a good bit more that needs to be done before we're fully on our feet.

Go Lock Your Laptop, Right Now

Posted June 14, 2012

My father's home was burgled today. Fortunately, everyone is okay and everything that was taken can be replaced. As with most burglaries, the list of items taken is a little strange and the list of items not taken is even stranger.

One thing the burglars did get was my dad's laptop. It's a first-version MacBook Air, probably of little value at this point. But, it did have a wealth of information on it.

My first question to my dad was "Did the computer have a password?". He didn't remember, which probably means it didn't. Of course, requiring a password to log in by no means guarantees that someone can't get in to the computer. But it does pose a significant road block to most people, and that's the point of any type of security: make it difficult enough that it's no longer worth someone's time or effort to break into it.

Consider for a second what sort of data someone would have if they got a hold of your computer. If you're like most people, your browser knows most of your passwords. That means access to your email, which means access to everything else. Speaking of email, they'd probably have access to every email you've ever sent or received, especially if you use a desktop email client like Outlook or Thunderbird. And then your files. Pictures of friends and family. Financial documents.

Now consider how easily your computer could walk off. As humans, we tend to have a flawed belief that it will never happen to us, but it has to happen to someone. Laptops are designed to be portable. And desktops are pretty easy to pick up and carry off.

So consider this a reminder to at least make sure your computer have a good password. It will act as a solid castle wall, if your computer is stolen. Also, get all of your passwords into a password vault, not your browser. Personally, I like LastPass, but there are many good ones out there. (If you do choose LastPass, make sure your sessions time out after a certain amount of idle time, so the passwords are harder to use.) If you have any sensitive files on your computer, consider encrypting them with a tool such as TrueCrypt.

Lastly, make sure you secure all of your computers. Especially that one in your pocket. It's the easiest to lose, and contains as much information as any of your other computers.

Update: I forgot to add this part, but you also may want to consider installing a lojack. Personally, I use Prey, which will send you location information, screen shots, and pictures from the webcam. You can either pay them for service, or set it up yourself, which is pretty easy to do.

Update 2: A reader wrote in suggesting that I also mention using full drive encryption. If you're using Windows, BitLocker (built into some versions of Windows 7) or TrueCrypt can do this for you. On Mac, use File Vault. Full drive encryption would completely prevent an attacker from gaining any access to the data on the machine.