Be Your Own User
“Every good work of software starts by scratching a developer’s personal itch.” — Eric S. Raymond, The Cathedral and the Bazaar
The other day I got my pack of IDEO Method Cards in the mail. Each card has a quick method to help in design. While I was busy packing for a trip last night I picked them up and pulled out the first card. It was called A Day in the Life. The goal of the method is to gain a deep understanding of the context and processes in which real everyday use is happening.
This got me thinking about eating your own dog food, or taking out your own garbage, about being the first and biggest user of your own applications. I love designing and building things for myself that solve my own problems. BriteVerify’s real-time email verification api came from working on a data management system that had to verify email addresses. There was no good solution. So we built one.
However, my best experience of the benefits of being your own user has been over the past two years. In the early days of BriteVerify we had strong demand for a file or batch processing service. I have always hated dealing with the file based processing services; it always feels so dated – almost archaic. The gutter of programming work, more often than not, is simply a semi-automated manual process of pulling files from one FTP, reformatting them, dealing with a million different little formatting and character issues, processing them, and putting them back. Furthermore, the API was much more organic and would represent a lower load on the overall system. A file on the other hand could be huge and the end customer does not understand the difference between running 50 million records against a static database and running 50 million records one at a time against web based services over the internet. IO problems vs CPU problems.
Needless to say, I was opposed to doing it. But being a bootstrapped company has a way of getting you to do things. In short, we needed the money. Therefore, I built the email verification software as quickly as I could that solved the basic problems of running 1000s of email verifications in parallel. Then, I spent the next year in my own kind of everyday firedrill. As the files got bigger and bigger, I had to find new innovative ways to scale not only the file processor, but also our core web service infrastructure. Not to mention that I did indeed have to deal with an illegal character at line 1,248,321 at 4 AM on Christmas Day.
Overtime we started to learn all the tricks, to see all the problems repeat themselves. As they repeated, they became a pattern and that pattern eventually became code. If we hated doing something over and over again enough we automated it, and slowly life became more and more back to normal.
We learned how to scale, how to handle volumes most companies 10 times our size don’t handle. Over Christmas holiday one year we got an e-mail validation request to process a 70 million record file in 5 days. It was double the size of the largest single file we had ever verified. Everything went according to plan until Christmas Eve when I came home to find the email list software doubling over on itself. Our head of engineering, Brady, and I spend that night trouble shooting every possible scenario while listening to heavy metal Christmas carols until 5 AM in the morning. Needless to say my wife was not too pleased. The next day with a clear head I made a drastic change to the infrastructure and we were able to finish the file ahead of schedule. To no surprise the experience left me drained, frustrated, and hating my wonderful life. I decided I would never have another Christmas like that. Over the next few months, Brady and I took what we had learned and incorporated into the files service. Now, we comfortably and regularly handle files of 100 million+ without killing ourselves or our service. All thanks to the misery of a very metal Christmas.
When time came to begin designing a file application for the general public, known as BriteFiles, we were able to approach it from a position of deep understanding and empathy. The idea was to make an email validation system that we would love to use, something so simple that it just worked. To build something that would give people like us their Christmases back. As a result, the design and development was very detail oriented. There was not a lot of functionality questioning. There was a lot of removing functionality. When it came time to code, the tests were already there and the coding went very quickly.
What really has impressed me about Brite Files since in use for the past few months is that there have been nearly no errors. Files are such funky opaque things and so many things can go wrong. The email cleaning application has barely had a hiccup and it such a pleasure for us to use.
BriteFiles is the result of scratching our own itch. Technology is best when it is personal, when it makes your life better, not the life of some abstract user. As users, we took out our own garbage rather than outsourcing it, we intimately learned the pain of this line of work. BriteFiles was built to make our lives better first and in doing so we hope we have made the lives of our users better as well.