When to Make Data-Driven Decision vs Opinion-Based Decision
A reflection on Lex Fridman's interview with Tony Fadell and my experiences at a high growth startup and a large multinational company
If you have not seen Lex Fridman’s interview with Tony Fadell, check it out here. From the nearly three hours conversation, Fadell’s insights about opinion-based decisions versus data-driven decisions resonated with me.
When you build the V1 of some revolutionary product, you can only rely on opinion-based decisions. There is no data. No history to work with. To get the team moving with the right mindset, articulating the whys of the decision makes all the difference. After shipping the V1, only then will data flow in. From V2 onwards, data-driven decisions become the name of the game.
Nothing → V1
When I joined the autonomous checkout startup Standard, the company ran on opinion-based decisions. The space did not exist a decade ago. The problems we faced were genuinely novel. There was no playbook or historical data. Our competitors were hush-hush about their internals, not wanting to leak any competitive edge.
One novel system I worked on was our AI-based anonymously tracking system. The AI follows shoppers throughout their shopping experience. But the system sometimes made a mistake; it would swap person A with person B. We had no means to inspect what happened. So I was tasked to build a debug tool for an autonomous store.
What should a debug tool for an autonomous store do? How should you debug AI data? Everything was novel, so I had to make several opinion-based calls developing this tool. I placed extra importance on the first user experience, even at the cost of additional complexity. Why? Because the tool will make or break on this first interaction.
So, I pushed for these initial requirements.
Works like any other web app with data readily available.
Has URLs that link to a specific frame for any set of cameras at any store
Require no extra software installation or special manual processes to access data
Looking back, I think these were the right calls because the tool gained company-wide adoption. When an issue arose, a QA person could quickly drop a link on Slack. Then an engineer, even on their phone, can immediately pick up the context and begin an investigation.
V1 → V2
Over my tenure at Standard, I was part of the push that took our V1 autonomous solution to V2.
The V1 product was built around having an on-premise installation. All the AI inference hardware would live on-site. There are benefits of this setup, from better latency to data security. However, from customer feedback, this was turning out to be a significant blocker. A number of stores did not have adequate storage space to house our server rack. Others we could cram it in, but we needed extra construction work to install cooling or lift electrical capacity limits.
After the first few installs, we finally have data. We had numbers for the bill of materials. We had a list of actions to perform to onboard a new location.
This data all became input into the V2 product. Engineering leadership and I pushed leadership to shift to a cloud-based architecture. When compared side-by-side, the numbers made the comparison straightforward and the decision obvious. The upfront cost for deploying V2 was more attractive than V1 for our customers. Furthermore, complaints like storage space limitations were no longer a factor. As a cherry on top, this architecture allowed engineering to explore new types of cameras.
Since moving to V2, being data-driven quickly became how decisions got made.
V3 and Beyond
Recently, I joined the job search giant, Indeed. This company is data-driven to the core having it as one of its five principles.
Even within my first month, Indeed taught me how to integrate data into everything. Every product change goes through A/B testing. While a feature can start as an opinion-based decision and be released, whether it stays on the product is determined by statistics. If it causes harm to key metrics, the feature gets dropped.
For example, I learned that an innocuous UX fix might not be benign. Indeed’s search results page had a component where a layout padding looked a little off; it felt too small. The UX team introduced a fix. Surprisingly, the variant with the extra whitespace performed worse. Some users became less likely to do a mission-critical action; thus, this fix was reverted.
Indeed takes data-driven decisions to the next level. Everything gets measured, and the Imhotep IQL tool trivializes access to and analysis of this data. Not only does the Imhotep tool track business intelligence data, but it also indexes Gitlab code commits, JIRA tickets, Confluence Wiki articles, and even interview data.
So, when performance review season starts, a manager can retrieve the relevant data points to make a promotion case. For example, have the report been giving other teammates comments on their PR this quarter? There’s a query template and a web app that summarizes that. How about an overview of the JIRA tickets closed on a team-by-team basis? That is a query away on the Imhotep IQL tool.
Conclusion
Having switched from a startup to a large company made me reflect: the two companies truly operate differently. Watching Lex’s interview with Tony Fadell, I could finally put this difference into words. At a startup like Standard, many decisions will have to be opinion-based. Some may disagree but building up the whys is how to unite a team. At a large company, decisions have to be data-driven. Decisions can start as an opinion, but your job is to quantify them to succeed as a leader.