原文地址:Dear Startup: You have no idea how much that costs.

24 Sep 2019

The problem


Throughout my career working for tech startups I noticed a strange phenomenon. Throughout the beginning of my career I had a fair bit of what some would consider relatively “simple” web development work. Things like developing crud apps, building an API integration, etc. I noticed that even though I would give estimates for my work, my managers and their managers all the way up to the founder would have certain expectations about how long work should take regardless of my estimates, and if my estimates outpaced their expectations then there would be problems. Often times this would be in spite of the fact that many times those above me didn’t have nearly the technical expertise at all to make such an assessment. I chalked it off at the time as simply just a display of systemic mistrust within that organization and tried my best to meet their expectations in spite of them sometimes being unrealistic.


However, as the years have gone by I’ve noticed that the problem extended to almost every company I worked for. The problem became worse when I graduated from simple web development work to distributed computing and what many people would consider “big data” (though I hate the term to be honest). Many of the problems I was now solving were significantly more involved, cluster management, event driven systems, high availability, functional programing, complex distributed graph computations, topics of research and scalable data science. However, I found that the unspoken expectations of the managers above me in terms of how long certain tasks should take still on average remained roughly the same as was when I was doing simple web development work!


They wouldn’t say this at first, you would give your estimates and thoughtfully break your tasks into reasonable chunks and factor in for uncertainties and testing, but still if you sat down and pressed them they would still expect anything that would take more than a couple of weeks to be too involved, and they would assume that the problem was how you are approaching the problem, regardless of how hard the problem actually was. I found myself absolutely astonished that tech founders could be so clueless as to assume that a simple rest api integration should take the same amount of time as a real time transactional distributed ward’s clustering implementation for peta bytes of data, or a highly available complex distributed metastore. Had engineering really come that far in those few short years for this much harder work to be commoditized already? No.

他们一开始不会这么说,你深思熟虑地把你的任务分成各个部分,考虑到不确定性和测试因素,给出你的预估时间,但如果你坐下来,按他们的要求,他们仍然会期待[事情可以在几周内完成],他们关心你如何解决问题,而不管问题有多困难。我很惊讶,技术创始人竟然如此无知,以至于认为一个简单的rest api集成花费的时间与一个PB级实时事务分布式监控集群,或者一个高可用的复杂分布式元数据存储需要相同的时间。在这短短的几年里,工程技术真的走了这么远吗?因为更加艰巨的工作已经商品化了?不。

After all of these years, I finally came to one simple conclusion. With all due respect: we are completely clueless about how long things should take.


a author's pic

The Parsimonious Yachtman


Lets explore this point more by means of an extended analogy. Suppose that you wanted to start a new business as a yachting captain. Suppose that you had watched a show my wife will indulge in from time to time, “Below Deck”, and you had determined that you needed one of those gigantic multi million dollar yachts to serve the best of the best, baseball stars and movie directors will be your quarry. This is in many ways analogous to when a startup company decides that they want to serve the fortune 500, companies that have petabytes and beyond of data. However, you as a startup founder have to operate lean, and you are only willing to spend $10,000 on a boat. If you were to walk up to the owner of the multi million dollar yacht and say, I’ll give you $10,000 for that boat, you would be laughed off the dock.

让我们通过一个类比来进一步探讨这一点。假如你是个游艇船长,想开展一项新的业务,假设您看过我妻子不时沉迷“甲板下”的节目,并且您确定需要一艘价值数百万美元的巨大游艇,来服务最好的棒球明星和电影导演。这很类似于一家初创公司以拥有PB级甚至更多数据服务于《财富》 500强。但是,作为初创公司的创始人,您必须精益经营,而您只愿意在船上花费10,000美元。如果您走到价值数百万美元的游艇的老板那里说,我给您那艘船10,000美元,您将在码头上被嘲笑。

Similarly if you were to walk to a boat construction company (because what does that yacht owner know) and ask them to build a comparable boat, they wouldn’t take you seriously. So what do you do instead? Hire someone that’s an expert in boats to make it work for you. Shift the culpability of making the impossible happen to someone else. So you hire your enthusiastic new employee and and think to yourself, “He will find me exactly what I want, a tremendous yacht that can serve 20 people with luxury amenities, full kitchen, hot tub, etc.” essentially you envision in a piecemeal fashion what you saw before but couldn’t afford. Now suppose that you were to pay them simply a flat fee for their services, and then you give them the budget of $10,000. Now suppose additionally, you didn’t give them the full vision of the boat that you wanted, but simply told them “I want a boat that can transport 20 people” and left out the rest of it.

同样,如果您要去一家造船公司(because what does that yacht owner know)并要求他们建造一艘一样的船,他们也不会认真对待您。那你该怎么办呢?聘请一位船上的专家为您服务。将不可能的事情转移到了别人身。因此,您雇用了热情的新员工,然后想“他会做到我想要的,找到一艘巨大的游艇可以为20人提供豪华的设备,设施齐全的厨房,热水浴缸等”。您会开始幻想以前那些时尚而又买不起的东西,现在,您只需要向他们支付固定费用即可获得他们的服务,然后您给他们10,000美元的预算。另外假设,您没有给他们想要的船的完整全貌,只告诉他们“我想要一个能载20个人的船”,而没告诉其他的要求。

The boat expert may think to themselves, this is impossible, but I dont want to dissapoint them on my first week, so maybe I can find some type of discounted used ferry for under $50,000 and says “ok lets do this but lets up the budget to $50,000”. Its more than the founder had wanted to spend, but they are willing to make this concession. So the boat finder begins his work, however, as the different parts of the vision start to unfold, and the luxury nature of the yacht starts to reveal itself, the boat expert suddenly realizes they have just become victim to a case of misplaced culpability, and any additional needs that they surface to satisfy the needs of the “luxury yacht” vision is now a issue of their own invention! Even though the business requirements are coming from the owner, they are still surfacing the additional technical requirements needed to satisfy the vision and thus are in many ways now an obstacle to the founders original intention of acquiring a boat for under $10,000. Suddenly, the boat expert is producing problems, and not solutions, suddenly he is a problem.

船上的专家可能会认为,这是不可能的,但是我不想在第一周就让他们失望,也许我可以找到一艘价格低于50,000美元的某种廉价二手渡轮,并说:“好的,可以这样做,但是要增加预算到$ 50,000”。预算比创始人想花的更多,但他们愿意做出让步。因此,专家开始了他的工作,但是随着工作的展开,游艇的豪华性需求开始展现出来,专家突然意识到,他们成为了“misplaced culpability”的受害者,并且他们满足“豪华游艇”愿景的任何其他需求现在都是他们自己发明的问题!即使业务要求来自老板,他们仍然需要其他技术需求来满足要求, 这些需求现在已成为创始人最初打算以低于10,000美元的价格购买船的障碍。突然间,船专家正在产生问题,而不是解决方案,船专家成为了一个问题。

The parsimonious yachtman isn’t being malicious here. Its just inevietable, that no matter what the boat finding expert does, he will never be able to overcome the fact that the founder really only “values” his new acquisition with a price tag of $10,000 so anything he does contrary to this reality simply weakens his position, REGARDLESS of how right he is in his on going estimation, and additionally he’s already started the engagement by asking for a concession.


a author's pic

Back to the tech startup, suppose that you think of the development of a piece of software in terms of its TCA (total cost of acquisition), but multiplying the development hours that you spend on something by the developer salaries, and for simplicity suppose that you don’t factor in things like infrastructure costs or license costs for other software. The startup manager says they want a system built that can support X write latency/throughput, Y read/latency and throughput, but the exact nature of the product itself is a WIP due to the fact that they are exploring the market and trying to find the right clients (namely any clients) so need to keep some doors open. Additionally the manager has an idea in the back of his mind that it should take about 2 weeks to build the system by comparing it to another system that he was involved with (unbeknownst to him that system was much much simpler and didn’t have anywhere near the same requirements). The engineer comes back with this simplified description and says he can get a first version produced, but it will take a month instead of 2 weeks. The engineer was being optimistic, and it was a stretch goal already, but he’s new to the company and doesn’t want to seem like he’s a slow poke.

回到技术初创公司,假设您以TCA(总购买成本)来考虑某个软件的开发,为简单起见,您无需考虑基础设施成本或其他软件的许可成本之类的问题, 用开发时间乘以开发人员的薪水。假设,初创公司经理说,他们希望构建一个能够支持X写延迟/吞吐量、Y读/延迟和吞吐量的系统,但产品本身的确切性质是WIP,因为他们正在探索市场,并试图找到合适的客户(即任何客户),因此需要开一些门。此外,经理的想法是,对比另一个系统(他不知道这个系统要简单得多,需求也完全不相同),该系统需要大约2周的时间来构建。工程师说他可以生产出第一个版本,但是需要一个月而不是2个星期。工程师一直很乐观,但其实这已经是一个艰巨的目标,他是该公司的新手,并且不想让自己看起来很慢。

The manager after 2 weeks (his original expectations) has already started breathing down the engineers neck. But why is that? Didn’t the engineer say it would take a month? Not to mention that now that the client focus has shifted, new requirements are now pushing the estimate well beyond the month, but the estimate is not allowed to shift with shifting requirements. It is now the written gospel of expectation for the project, regardless of lip service the company may pay to “agile”.


I could take the comparison farther, but nothing can change the fact that the founder was so far off his original unspoken 2 week expectation that nothing the engineer can do will satisfy what he has in mind and his original budget of 2 weeks was all that he “really” was willing to spend, and a month was a stretch but he was willing to pay a bit more. He was doomed from the start by the manager/founders inaccurate expectation. What happens next? Most likely the engineering lead fails to live up to the expectation and is replaced on the project with another one. But that sounds kind of like an extreme scenario doesn’t it? Would you be surprised to hear that most companies I’ve worked for fell prey to some form of this pattern? Its clearly not an isolated problem.


Startup founders fool themselves into thinking that they can procure assets orders of magnitude beyond what they are willing to pay to procure those assets. In essence, when it comes to engineering assets and engineering time they fall prey to “magical thinking” (likely inspired by some depiction of a hacker they saw in a movie once). If founders/managers were to step back and think about it simply in terms of TCA and compare what they are building to what a company is charging for a similar product they might realize that they are trying to get a $10 million yacht for $10,000. Again, no one is being malicious, they are simply learning from what the history of tech enterprises that have gone before us have taught them. That skilled engineers working in a basement totally clueless to business requirements can produce tremendous value. They don’t realize that so many factors have gone into play in those success stories that went before including luck, timing, market conditions to name a few.


A Radical Proposal


The underlying problem still remains that the founder/managers (albeit uninformed, ultimately unchangeable) subconcious expectations about how long the task should take inspired by how much they value that particular component is not even in the same galaxy to what it will cost to build such a system. Some “product development methodologies” like Agile have tried to solve the problem but given us nothing more than a slightly different means to shift culpability to engineering when things aren’t happening as fast as we expect. So how do you solve that problem? The question I have started asking managers and founders is simple. “How much time do YOU want me to spend on this”. This little question has often made them pause. “What do you mean, you are supposed to provide me with an estimate and then I approve it.” Its just not that simple.


Estimates are such a loaded and dangerous endeavor for an engineer. To the point where companies mistakenly talk about estimates like its a “skill” that an engineer can learn. But the reality is that if you can make a probabalistically accurate estimate, then its likely that the task should have been automated by some other means already. In other words, its easy to estimate a task that essentially amounts to copy and pasting some well known CRUD API end point patterns, but any even remotely creative or novel work is almost guaranteed to be totally unknown. Its almost always a way to trap an engineer in a bargaining conundrum. Ie, if you make me provide an estimate first and its no where near what you have in mind (because it won’t be), then you have placed me automatically in a position of begging for additional time. Its not so surprising that this subtle form of manipulation in the sales tactic of getting us to put our cards out the table first comes from the business end of the company. So as engineers, what do we have to do?

对于工程师而言,估算是一项艰巨而危险的工作。甚至到公司错误地谈论估算,例如工程师可以学习的“技能”。但是实际情况是,如果您可以进行概率上准确的估计,则很可能该任务应该已经通过其他某种方式实现了自动化。换句话说,可以很容易地估计一项任务,该任务本质上相当于复制和粘贴一些众所周知的CRUD API端点模式,但是几乎可以保证几乎没有任何创造性或新颖性的工作。这几乎总是一种将工程师困在讨价还价难题中的方法。就是说,如果您让我先提供一个估算值,并且它与您所想的相差无几(因为它不会),那么您就自动将我置于乞求位置,需要更多时间。毫不奇怪,这种使我们首先将牌放到桌面的销售策略中的微妙操纵形式来自公司的业务部门。因此,作为工程师,我们必须做什么



But estimates are our friends, it conveys expectations back to the company. The truth is that isn’t how it actually plays out, and this needs to become our new engineering battle cry. If engineers stop giving estimates for their work and simply ask for deadlines then it changes the dynamic of the conversation. If they say that they want to spend $10,000 on acquiring the boat, then we can come back and say, well I won’t be able to give you the luxury yacht you had in mind, however, I can find a used sailboat and after we fix it up you can possible charge high end clientele for exclusive sailing excursions. It puts us back in the superior position of negotiation. Then as engineers, we have the freedom to come up with a solution that meets your expectations and simultaneously doesn’t put us in an impossible situation. Or at a minimum at least veto the idea all together as not being possible to achieve in the desired time spans. Besides 99% of the time you end up getting a deadline anyway, so why even go through the excercise of making estimates?


It will only get worse


Engineering is not getting simpler, its getting more and more complex, because we are solving harder and harder problems, which means that these things (which all haven’t been commoditized yet, in spite of the fact that many founders secretly think that all engineering problems have already been reduced to the drag and drop simplicity of building Wix websites) will take longer to solve and will have a higher TCA. As a result, the gap between the expectations of founders/managers in terms of time to implementation and the amount of time we should be spending building these systems is becoming light years apart. As a result, we continue to push out half baked solutions cobbled together with duct tape and chicken wire just barely hobbling across the finish line so we can toss it off to operational teams and burden them with our rushed decision making for the rest of the life of the company or until they burn out and leave (guess which will come first).


The problem will get much worse until engineers start hitting their employers with much needed reality checks on what it takes to solve these problems. For startup founders/managers, if I cannot appeal to your sense of pity for your employees, I hope I can at least appeal to your sense of pride: Don’t be the guy asking the $10 million yacht captain if he wants to sell it for $10,000.


Kyle Prifogle